mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Compare commits
12
Commits
| Author | SHA1 | Date | |
|---|---|---|---|
|
|
0372feaf00 | ||
|
|
71523f371b | ||
|
|
c5c44528ca | ||
|
|
164cc9e09a | ||
|
|
cbc9e9baad | ||
|
|
0da5700d26 | ||
|
|
5847d1f3fe | ||
|
|
ca7cabb752 | ||
|
|
500ad94578 | ||
|
|
9bc7de32bd | ||
|
|
656dc96e92 | ||
|
|
402f062461 |
+113
@@ -2,6 +2,119 @@
|
||||
|
||||
All notable changes to GBrain will be documented in this file.
|
||||
|
||||
## [0.32.6] - 2026-05-11
|
||||
|
||||
**Your brain learns to detect its own integrity drift.**
|
||||
**New `gbrain eval suspected-contradictions` probe + doctor + MCP wire-up.**
|
||||
|
||||
A user (Fergtic, Chronicle writeup) flagged that gbrain handles contradictions for *curated* pages via compiled-truth-plus-timeline + source-boost, but raw extracted claims don't have a supersession story. On evaluation, most of the supersession case is already handled — `takes.active` filter hides superseded takes from search; source-boost ranks curated content above bulk; recency-decay applies per-prefix half-life; compiled_truth chunks get a guaranteed slot in dedup. What's NOT measured: whether unmarked semantic contradictions actually surface in retrieval results, and whether the brain has a self-healing loop to act on them once detected.
|
||||
|
||||
v0.32.6 is a complete brain-consistency subsystem, not a one-off probe. A measurement instrument + agent-facing surface + dream-cycle integration + persistent cache + time-series tracking. The size is intentional — the goal is "trustworthy nightly cadence" not "run it once and decide."
|
||||
|
||||
### The numbers that matter
|
||||
|
||||
A full 9-commit branch behind the feature flag of "the user asked for it." 226 hermetic tests + 12 real-Postgres E2E cases. The probe ships ready for the user's brain to populate.
|
||||
|
||||
```
|
||||
new command gbrain eval suspected-contradictions [run|trend|review]
|
||||
new MCP op find_contradictions(slug?, severity?, limit?)
|
||||
new doctor check contradictions (paste-ready resolution commands)
|
||||
new dream-cycle hook synthesize phase reads prior contradictions per slug
|
||||
new schema migrations v51 (eval_contradictions_cache), v52 (eval_contradictions_runs)
|
||||
new engine methods listActiveTakesForPages, writeContradictionsRun,
|
||||
loadContradictionsTrend, getContradictionCacheEntry,
|
||||
putContradictionCacheEntry, sweepContradictionCache
|
||||
```
|
||||
|
||||
The probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), runs a date pre-filter to skip obvious quarterly-update shapes, asks an LLM judge with severity scoring, aggregates into a per-query + global report with Wilson 95% confidence interval on the headline percentage. Soft budget cap with pre-flight refuse + mid-run stop. Persistent judge cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` so prompt edits cleanly invalidate prior verdicts. `judge_errors` is first-class in the report (parse_fail, refusal, timeout, http_5xx, unknown) — silent skips were the wrong default; counting errors in the denominator keeps the headline honest.
|
||||
|
||||
### What this means for new users
|
||||
|
||||
`gbrain init` keeps OpenAI as the zero-config default. After the migration, run `gbrain eval suspected-contradictions --query "what is X" --top-k 5` to see what the probe finds against your real brain. If `gbrain doctor` flags any high-severity contradictions, each one ships with a paste-ready resolution command: `gbrain takes supersede`, `gbrain dream --phase synthesize --slug`, or `gbrain takes mark-debate`. The agent can call `find_contradictions(slug="companies/acme")` during conversations to surface findings proactively.
|
||||
|
||||
The bigger swing (chunk-level `revises` field + ranking change + synthesize-prompt coupling) is still gated on probe data — if your Wilson CI lower-bound stays <5% across a month of nightly runs, source-boost + recency-decay + curated pages are doing the job and we stop. If >15%, plan in v0.34+.
|
||||
|
||||
### To take advantage of v0.32.6
|
||||
|
||||
`gbrain upgrade` should do this automatically. If it didn't:
|
||||
|
||||
1. **Apply the migrations:**
|
||||
```bash
|
||||
gbrain apply-migrations --yes
|
||||
```
|
||||
Adds tables v51 + v52 plus their indexes. Idempotent on both PGLite and Postgres.
|
||||
|
||||
2. **Run the probe against a few real queries:**
|
||||
```bash
|
||||
gbrain eval suspected-contradictions --query "what is alice's role at acme" --top-k 5 --json
|
||||
```
|
||||
Default budget is $5 in TTY, $1 non-TTY. Judge defaults to `anthropic:claude-haiku-4-5`.
|
||||
|
||||
3. **Inspect findings in doctor:**
|
||||
```bash
|
||||
gbrain doctor
|
||||
```
|
||||
Look for the `contradictions` check. High-severity items ship with paste-ready resolution commands.
|
||||
|
||||
4. **Read the new docs**: `docs/contradictions.md` (architecture + severity rubric) and `docs/eval-bench.md` (workflow for nightly runs + trend tracking).
|
||||
|
||||
5. **No breaking changes**: existing search, ranking, synthesize, and takes behavior is unchanged. The find_contradictions MCP op is read-scope (NOT in the subagent allowlist — user-initiated only).
|
||||
|
||||
6. **Privacy posture**: probe output (slugs, chunk text, take claims) is stored in `eval_contradictions_runs.report_json` on your local brain. The build-contradictions-fixture script applies a multi-pass redactor before any output is committed to the repo; the operator must inspect every redaction.
|
||||
|
||||
### Itemized changes
|
||||
|
||||
#### Probe core (9 modules)
|
||||
|
||||
- `src/core/eval-contradictions/types.ts` — wire contract. `schema_version: 1`, `PROMPT_VERSION = '1'`, `TRUNCATION_POLICY = '1500-chars-utf8-safe'`. Stable JSON output shapes (ProbeReport, ContradictionFinding, JudgeVerdict, etc.).
|
||||
- `src/core/eval-contradictions/judge.ts` — `judgeContradiction()` is the single LLM call. Query-conditioned prompt (Codex outside-voice fix — judge sees the user's query, not just two free-form chunks). Holder context for take pairs so "Alice thinks X vs Bob thinks not-X" doesn't get flagged. UTF-8-safe truncation at `maxPairChars` (default 1500, surrogate-pair aware). C1 confidence-floor double-enforcement: orchestrator filters `contradicts: true` cases where `confidence < 0.7` even if the model ignored the prompt rule.
|
||||
- `src/core/eval-contradictions/runner.ts` — the orchestrator. Pair generation (cross-slug + intra-page), date pre-filter (3-rule), deterministic sampling, A2 budget tracker, cache integration, C2 first-class judge_errors, Wilson CI aggregation, hot_pages roll-up. `PreFlightBudgetError` is a discriminable rejection class.
|
||||
- `src/core/eval-contradictions/date-filter.ts` — 3-rule layered pre-filter (Codex fix to the naive single-rule approach). Same-paragraph-dual-date overrides the separation rule (flip-flop case sees the judge). Missing-date side always falls through to the judge.
|
||||
- `src/core/eval-contradictions/calibration.ts` — Wilson 95% confidence interval, exact-clamping at p=0 and p=1, small-sample warning when n < 30.
|
||||
- `src/core/eval-contradictions/cost-tracker.ts` — A2 soft ceiling + P3 embedding-spend tracking. Anthropic + OpenAI per-MTok pricing baked in.
|
||||
- `src/core/eval-contradictions/cache.ts` — P2 persistent cache wrapper. 5-component key (Codex fix includes prompt_version + truncation_policy). Order-independent on (a, b) via lex-sorted SHA-256 hashes. Shape-validates JSONB on read so corrupt rows treat as miss.
|
||||
- `src/core/eval-contradictions/cross-source.ts` — M6 source-tier breakdown. Reuses `DEFAULT_SOURCE_BOOSTS` prefix logic; emits {curated_vs_curated, curated_vs_bulk, bulk_vs_bulk, other} counts.
|
||||
- `src/core/eval-contradictions/severity-classify.ts` — M4 severity helpers (parse, sort, bucket, hot-page rollup).
|
||||
- `src/core/eval-contradictions/auto-supersession.ts` — M7 resolution-proposal generator. Classifies into takes_supersede / dream_synthesize / takes_mark_debate / manual_review with paste-ready CLI commands.
|
||||
- `src/core/eval-contradictions/judge-errors.ts` — typed error collector (Codex fix — silent skip was wrong; errors counted in denominator).
|
||||
- `src/core/eval-contradictions/trends.ts` — M5 time-series helpers (write/read + ASCII chart renderer).
|
||||
- `src/core/eval-contradictions/fixture-redact.ts` — privacy redactor for the gold-fixture build path (slug rewrite, name placeholders, monetary obfuscation, PII scrubber wrapper). Fail-closed via `isCleanForCommit`.
|
||||
|
||||
#### CLI + dispatch + agent surfaces
|
||||
|
||||
- `src/commands/eval-suspected-contradictions.ts` — new `gbrain eval suspected-contradictions [run|trend|review]` command. ~350 LOC. A4 empty-capture UX: `--from-capture` against empty `eval_candidates` exits 2 with hint naming `GBRAIN_CONTRIBUTOR_MODE=1`.
|
||||
- `src/commands/eval.ts` — sub-subcommand dispatch updated (~5 lines).
|
||||
- `src/commands/doctor.ts` — new `contradictions` check (M1). Severity-sorted findings with paste-ready commands; gracefully skipped pre-migration.
|
||||
- `src/core/operations.ts` — new `find_contradictions` MCP op (M3, read scope, NOT localOnly). Filter by slug substring + severity + limit.
|
||||
- `src/core/operations-descriptions.ts` — `FIND_CONTRADICTIONS_DESCRIPTION` constant.
|
||||
- `src/core/cycle/synthesize.ts` — M2 prompt injection. `loadPriorContradictionsBlock` pre-fetches the latest probe's top-5-by-severity findings once at phase start and threads them into `buildSynthesisPrompt` as an informational block. Subagent sees what to reconcile when writing to flagged slugs. Empty trend = empty block, fresh-install behavior unchanged.
|
||||
|
||||
#### Engine surface (P1 + M3 + M5 + P2)
|
||||
|
||||
- `src/core/engine.ts` + `src/core/postgres-engine.ts` + `src/core/pglite-engine.ts` — 6 new methods. P1 `listActiveTakesForPages` batches the per-page active-take fetch (single `WHERE page_id = ANY($1)` instead of N round-trips). M5 `writeContradictionsRun` + `loadContradictionsTrend` are the time-series surface. P2 `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` are the cache surface. JSONB writes use `sql.json()` on Postgres (no double-encode regression class) and `$N::jsonb` on PGLite.
|
||||
|
||||
#### Schema (2 migrations)
|
||||
|
||||
- `src/core/migrate.ts` — v51 `eval_contradictions_cache` (composite PK on 5 components; expires_at-driven TTL). v52 `eval_contradictions_runs` (Wilson CI bounds, source_tier_breakdown JSONB, full report_json blob). Both idempotent on both engines.
|
||||
- `src/core/pglite-schema.ts` + `src/schema.sql` — DDL mirror. RLS-enable lines for the two new tables.
|
||||
- `src/core/schema-embedded.ts` — regenerated.
|
||||
|
||||
#### Scripts + fixtures
|
||||
|
||||
- `scripts/build-contradictions-fixture.ts` — operator script for building the privacy-redacted gold fixture against a real brain. Interactive labeling, multi-pass redactor, pre-commit `isCleanForCommit` safety gate.
|
||||
- `test/fixtures/contradictions-mini.jsonl` — 5 redacted-style queries for CLI smoke testing.
|
||||
|
||||
#### Tests
|
||||
|
||||
- 226 hermetic unit tests across 15 files: judge (25), runner (26), trends (15), engine methods (17), cache (14), cost (12), date-filter (15), calibration (13), severity (14), judge-errors (12), cross-source (13), auto-supersession (12), fixture-redact (16), integrations (11), plus shared helpers.
|
||||
- 12 real-Postgres E2E cases in `test/e2e/eval-contradictions-postgres.test.ts` covering migrations, JSONB round-trip, cache TTL with `now()`, M5 trend TIMESTAMPTZ ordering, and the find_contradictions MCP op end-to-end. Required-on-DATABASE_URL per gbrain convention.
|
||||
|
||||
#### For contributors
|
||||
|
||||
- Three-layer cohesion in the runner: pair generation → date pre-filter → cache lookup → judge → cost track → aggregate. Hermetic via `judgeFn` + `searchFn` dependency injection — no test ever touches the real LLM gateway or hybrid search.
|
||||
- `__setChatTransportForTests` already existed at `src/core/ai/gateway.ts:421` — judge tests use direct `chatFn` injection instead (cleaner for one-shot wrappers).
|
||||
- Codex outside-voice review caught 5 fixes that are now standard: command rename (`contradictions` → `suspected-contradictions`), judge_errors as first-class output, prompt_version + truncation_policy in cache key, Wilson CI on headline, query-conditioned judge prompt. All folded in.
|
||||
- Decision-point not in TODOS.md per user preference: after a month of nightly runs, check Wilson CI lower-bound. If <5%, source-boost + recency-decay + curated pages are doing the job and the bigger swing (chunk-level `revises`) stops here. If >15%, plan for v0.34+.
|
||||
## [0.32.5] - 2026-05-11
|
||||
|
||||
**Time, place, and what you're doing — reinjected on every turn, no matter how hard the session got compacted.**
|
||||
|
||||
@@ -77,6 +77,7 @@ strict behavior when unset.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
|
||||
@@ -0,0 +1,150 @@
|
||||
# gbrain eval suspected-contradictions (v0.32.6)
|
||||
|
||||
The contradiction probe samples retrieval results, asks an LLM judge whether
|
||||
any pair contradicts on a factual claim relevant to the user's query, and
|
||||
aggregates into a calibrated report. The output is data — the operator
|
||||
decides what to act on. This doc covers the architecture, severity rubric,
|
||||
how to interpret the headline number, and when to act.
|
||||
|
||||
## Why this exists
|
||||
|
||||
gbrain handles contradictions for *curated* pages via compiled-truth-plus-
|
||||
timeline and source-boost: when `companies/acme.md` says MRR is $2M and a
|
||||
chat transcript from 2024 says MRR was $50K, the curated page outranks the
|
||||
chat. `takes.active` filtering hides explicitly-superseded takes. Recency
|
||||
decay biases ranking toward fresher content per source-tier.
|
||||
|
||||
What none of those mechanisms measure: how often do unmarked semantic
|
||||
contradictions actually surface in retrieval? Without a probe, every
|
||||
"should we build the bigger swing (chunk-level `revises` field + ranking
|
||||
change)" decision is vibes. The probe produces evidence.
|
||||
|
||||
## Architecture
|
||||
|
||||
```
|
||||
┌──────────────────────────────────────┐
|
||||
│ gbrain eval suspected-contradictions │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ For each query: hybridSearch top-K │
|
||||
│ → cross_slug_chunks + intra_page │
|
||||
│ chunk-vs-take pairs │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ Date pre-filter: skip pairs whose │
|
||||
│ dates are >30d apart (Codex fix: │
|
||||
│ same-paragraph-dual-date overrides) │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
┌──────────────────▼───────────────────┐
|
||||
│ Persistent cache lookup │
|
||||
│ (chunk_a_hash, chunk_b_hash, model, │
|
||||
│ prompt_version, truncation_policy) │
|
||||
└────────┬─────────┬────────────────────┘
|
||||
hit│ │miss
|
||||
│ ▼
|
||||
│ ┌─────────────────────────┐
|
||||
│ │ LLM judge call │
|
||||
│ │ → JudgeVerdict │
|
||||
│ │ confidence floor ≥ 0.7 │
|
||||
│ └─────────┬───────────────┘
|
||||
│ │
|
||||
▼ ▼
|
||||
┌──────────────────────────────────────┐
|
||||
│ Aggregate per-query + global stats │
|
||||
│ Wilson 95% CI on headline % │
|
||||
│ source-tier breakdown │
|
||||
│ hot pages + resolution proposals │
|
||||
└──────────────────┬───────────────────┘
|
||||
│
|
||||
▼
|
||||
ProbeReport JSON
|
||||
│
|
||||
┌──────────────────┼──────────────────────┬───────────────┐
|
||||
▼ ▼ ▼ ▼
|
||||
doctor (M1) MCP (M3) synthesize (M2) trend (M5)
|
||||
surfaces find_contradictions informational persistent
|
||||
findings op for agents block in prompt tracking
|
||||
```
|
||||
|
||||
## Severity rubric
|
||||
|
||||
The judge assigns severity per finding:
|
||||
|
||||
| Level | Rubric | Example |
|
||||
|---|---|---|
|
||||
| `low` | naming/format differences | "Alice Smith" vs "A. Smith" |
|
||||
| `medium` | factual values that may be stale | revenue figure, headcount, valuation |
|
||||
| `high` | identity / structural claims | founder/CEO/CFO role, company status |
|
||||
|
||||
Doctor sorts findings by severity DESC. The MCP op accepts a severity filter
|
||||
so agents can fetch just the high-priority items.
|
||||
|
||||
## How to interpret the headline number
|
||||
|
||||
The probe outputs `queries_with_contradiction / queries_evaluated` with a
|
||||
Wilson 95% confidence interval:
|
||||
|
||||
```
|
||||
Queries with >=1 contradiction: 12 / 50 (24%) Wilson CI 95%: 14–37%
|
||||
```
|
||||
|
||||
What this says: with 95% confidence, the true rate is between 14% and 37%.
|
||||
The 24% point estimate is the most-likely-value but bounded by sampling
|
||||
noise. **`small_sample_note` fires when n < 30** — at that scale the CI is
|
||||
too wide to act on.
|
||||
|
||||
Decision criteria for the bigger swing (chunk-level `revises` field):
|
||||
|
||||
| Wilson CI lower bound | What it says | Action |
|
||||
|---|---|---|
|
||||
| < 5% | Source-boost + recency-decay + curated pages handle the load | Stop here; this is the right scope |
|
||||
| 5–15% | Real but bounded | Operator decides whether the cost justifies the swing |
|
||||
| > 15% | Real and substantial | Plan the bigger swing in v0.34+ |
|
||||
|
||||
## When to act on findings
|
||||
|
||||
Each finding ships with a `resolution_command` field — paste-ready:
|
||||
|
||||
- `gbrain takes supersede <slug> --row N` — newer take should replace
|
||||
the older chunk text on the same page (intra_page kind).
|
||||
- `gbrain dream --phase synthesize --slug <slug>` — compiled_truth for
|
||||
the curated entity needs an update (cross_slug curated-vs-bulk).
|
||||
- `gbrain takes mark-debate <slug> --row N` — intentional disagreement
|
||||
(e.g., two opinions you want to keep both of).
|
||||
- `# manual review: <a> vs <b>` — judge wasn't sure; operator decides.
|
||||
|
||||
Run `gbrain eval suspected-contradictions review --severity high` to
|
||||
inspect findings without re-running the probe.
|
||||
|
||||
## Cost model
|
||||
|
||||
Default judge is `claude-haiku-4-5` at ~$1/Mtok in, $5/Mtok out. With
|
||||
the v0.32.6 truncation at 1500 chars per pair, ~500 input + 80 output
|
||||
tokens per judge call. Budget cap defaults to $5 in TTY / $1 non-TTY.
|
||||
|
||||
- ~$0.0006 per judge call
|
||||
- ~$0.005 per query (after date pre-filter + cache hits)
|
||||
- ~$0.50 per 100 queries
|
||||
|
||||
The persistent cache means nightly runs against the same query set
|
||||
pay near-zero on re-runs (until you bump PROMPT_VERSION).
|
||||
|
||||
## Trust posture
|
||||
|
||||
- Probe never mutates the brain. Runs only read pages/takes/chunks.
|
||||
Writes go only to `eval_contradictions_runs` and `eval_contradictions_cache`.
|
||||
- MCP `find_contradictions` is read-scope. NOT in the subagent allowlist —
|
||||
user-initiated only, not autonomous-action surface.
|
||||
- Build-fixture script is local-only. The redactor + `isCleanForCommit`
|
||||
gate makes accidental private-data commits hard, but the operator MUST
|
||||
inspect every redaction before commit.
|
||||
|
||||
## See also
|
||||
|
||||
- Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`
|
||||
- CHANGELOG: `## [0.32.6]` entry covers the whole release.
|
||||
- Cost discipline: `docs/eval-bench.md` for the recommended nightly cadence
|
||||
+ trend-tracking workflow.
|
||||
@@ -291,3 +291,40 @@ p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per the
|
||||
`test/eval-longmemeval.test.ts` perf gate). Per-question cost well under the
|
||||
500ms speed gate. 500 questions = ~13s of overhead plus your retrieval and
|
||||
LLM latency.
|
||||
|
||||
## Measuring brain consistency over time (v0.32.6)
|
||||
|
||||
`gbrain eval suspected-contradictions` is a complementary measurement
|
||||
instrument: it samples retrieval results for unmarked semantic
|
||||
contradictions (e.g., compiled_truth vs chat content, intra-page chunk
|
||||
vs active take). Where LongMemEval measures retrieval correctness on a
|
||||
fixed labeled set, the contradiction probe measures how often a real
|
||||
brain surfaces conflicting answers.
|
||||
|
||||
### Recommended nightly cadence
|
||||
|
||||
```bash
|
||||
# Once a day, against your top 50 most-frequent queries:
|
||||
gbrain eval suspected-contradictions \
|
||||
--queries-file ~/.gbrain/queries.jsonl \
|
||||
--top-k 5 \
|
||||
--budget-usd 5 \
|
||||
--output ~/.gbrain/probe-runs/$(date +%Y-%m-%d).json
|
||||
```
|
||||
|
||||
Persistent cache (`eval_contradictions_cache`) makes re-runs near-zero
|
||||
cost until you bump `PROMPT_VERSION`. Trend-track via:
|
||||
|
||||
```bash
|
||||
gbrain eval suspected-contradictions trend --days 30
|
||||
```
|
||||
|
||||
The ASCII bar chart shows total flagged per day. Headline % surfaces in
|
||||
`gbrain doctor`'s `contradictions` check with paste-ready resolution
|
||||
commands per high-severity finding.
|
||||
|
||||
### See also
|
||||
|
||||
- `docs/contradictions.md` — architecture, severity rubric, action criteria.
|
||||
- CHANGELOG `## [0.32.6]` — full release notes including the bigger-swing
|
||||
decision criteria gated on Wilson CI lower-bound.
|
||||
|
||||
@@ -177,6 +177,7 @@ strict behavior when unset.
|
||||
- `src/commands/eval-export.ts` (v0.25.0) — streams `eval_candidates` rows as NDJSON to stdout with `schema_version: 1` prefix on every line. EPIPE-safe, progress heartbeats on stderr, stable id-desc tiebreaker so `--since` windows never dupe/miss rows.
|
||||
- `src/commands/eval-prune.ts` (v0.25.0) — explicit retention cleanup. Requires `--older-than DUR`. `--dry-run` reports would-delete count.
|
||||
- `src/commands/eval-replay.ts` (v0.25.0) — contributor-facing replay tool. Reads NDJSON from `gbrain eval export`, re-runs each captured `query` / `search` op against the current brain, computes set-Jaccard@k between captured + current `retrieved_slugs`, top-1 stability rate, and latency Δ. Stable JSON shape (`schema_version: 1`) for CI gating; human mode prints a regression table. Pure Bun, zero new deps. The dev-loop half of BrainBench-Real that closes the gap between "data captured" and "data used to gate a PR." See `docs/eval-bench.md` for the workflow.
|
||||
- `src/commands/eval-suspected-contradictions.ts` + `src/core/eval-contradictions/{judge,runner,types,date-filter,cost-tracker,cache,severity-classify,cross-source,trends,calibration,judge-errors,auto-supersession,fixture-redact}.ts` (v0.32.6) — `gbrain eval suspected-contradictions [run|trend|review]`. Probe samples top-K retrieval pairs per query (cross-slug + intra-page chunk-vs-take), date pre-filters (3-rule layered — same-paragraph-dual-date overrides separation rule), LLM judge (query-conditioned per Codex; UTF-8-safe truncation; C1 confidence-floor double-enforcement; resolution_kind output drives M7 paste-ready commands), persistent cache keyed on `(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)` (Codex outside-voice fix — prompt edits cleanly invalidate prior verdicts), Wilson 95% CI calibration on the headline percentage with `small_sample_note` when n<30, judge_errors as first-class typed counters (parse_fail/refusal/timeout/http_5xx/unknown — Codex fix to bias from silent skip), M5 trend writes to `eval_contradictions_runs`, M6 source-tier breakdown reuses `DEFAULT_SOURCE_BOOSTS` prefix logic, deterministic sampling (combined_score DESC + lex tiebreaker — stable cache hit-rate across re-runs). Hermetic via `judgeFn` + `searchFn` DI in the runner; never touches the real gateway in tests. Engine surface: `BrainEngine.listActiveTakesForPages` (P1 batched), `writeContradictionsRun` + `loadContradictionsTrend` (M5), `getContradictionCacheEntry` + `putContradictionCacheEntry` + `sweepContradictionCache` (P2). Schema migrations v51 + v52. MCP op `find_contradictions` (read scope, NOT localOnly, NOT in subagent allowlist — user-initiated only). M1 doctor check surfaces high-severity findings with paste-ready resolution commands. M2 synthesize phase pre-fetches latest probe's top-5-by-severity findings and threads them into `buildSynthesisPrompt` as an informational block. 226 hermetic unit tests + 12 real-Postgres E2E. Plan: `~/.claude/plans/system-instruction-you-are-working-hashed-dewdrop.md`. Architecture doc: `docs/contradictions.md`.
|
||||
- `src/commands/eval-longmemeval.ts` + `src/eval/longmemeval/{harness,adapter,sanitize}.ts` (v0.28.1) — `gbrain eval longmemeval <dataset.jsonl>` runs the public [LongMemEval](https://huggingface.co/datasets/xiaowu0162/longmemeval) benchmark against gbrain's hybrid retrieval. Architecture: one in-memory PGLite per benchmark run created via `createBenchmarkBrain` + `withBenchmarkBrain` (NO `EphemeralBrain` class). Between questions, `TRUNCATE` over runtime-enumerated `pg_tables` so future schema migrations don't silently leak data across questions; infrastructure tables (`sources`, `config`, `gbrain_cycle_locks`, `subagent_rate_leases`) are preserved. `cli.ts` has a pre-dispatch bypass so `eval longmemeval` skips `connectEngine()` — the user's `~/.gbrain` brain is never opened. `--expansion` defaults to OFF (deterministic, no per-query Haiku call); pass `--expansion` to opt in. Default model resolves through `resolveModel()` 6-tier chain with `models.eval.longmemeval` as the new config key. Sanitization parity: `harness.ts` re-uses `INJECTION_PATTERNS` from `src/core/think/sanitize.ts` (now exported, line 22) so adding a pattern automatically covers takes AND benchmarks. Retrieved chat content is wrapped in `<chat_session id="..." date="...">` framing; the answer-gen system prompt declares the content UNTRUSTED. LLM injection seam: `runEvalLongMemEval(args, {client?: ThinkLLMClient})` lets tests stub the client so the full pipeline runs without an Anthropic API key. p50 25.9ms / p99 30.3ms warm reset+import+search on Apple Silicon (per `test/eval-longmemeval.test.ts` perf gate). Hand the JSONL output to LongMemEval's `evaluate_qa.py` to score (their published evaluator, not bundled — needs OpenAI gpt-4o per their spec).
|
||||
- `docs/eval-bench.md` (v0.25.0) — contributor guide for using captured data to benchmark retrieval changes before merging. Linked from CONTRIBUTING.md under "Running real-world eval benchmarks (touching retrieval code)".
|
||||
- `src/core/eval-capture.ts` (v0.25.0) — op-layer capture wrapper called from `src/core/operations.ts` `query` + `search` handlers. Catches MCP + CLI + subagent tool-bridge from one site. Fire-and-forget; failures route to `engine.logEvalCaptureFailure` so `gbrain doctor` sees drops cross-process. **Capture is off by default** — `isEvalCaptureEnabled` resolution: explicit `config.eval.capture` (true/false) wins, else `process.env.GBRAIN_CONTRIBUTOR_MODE === '1'`, else off. Production users get a quiet brain; contributors set `export GBRAIN_CONTRIBUTOR_MODE=1` in `.zshrc` to enable the dev loop. PII scrubber gate is independent and defaults to true regardless of CONTRIBUTOR_MODE.
|
||||
|
||||
+1
-1
@@ -1,6 +1,6 @@
|
||||
{
|
||||
"name": "gbrain",
|
||||
"version": "0.32.5",
|
||||
"version": "0.32.6",
|
||||
"description": "Postgres-native personal knowledge brain with hybrid RAG search",
|
||||
"type": "module",
|
||||
"main": "src/core/index.ts",
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
#!/usr/bin/env bun
|
||||
/**
|
||||
* scripts/build-contradictions-fixture.ts (v0.32.6, T2)
|
||||
*
|
||||
* Build a privacy-redacted gold fixture for the contradiction probe judge
|
||||
* by running the probe against the user's REAL brain and hand-labeling
|
||||
* the candidate pairs. Output: test/fixtures/contradictions-eval-gold.jsonl.
|
||||
*
|
||||
* Privacy posture (CLAUDE.md rule): the operator MUST inspect the
|
||||
* generated file before commit. The redactor (fixture-redact.ts) is
|
||||
* best-effort; the pre-commit review is the safety net. Fail-closed if
|
||||
* any pair fails the isCleanForCommit check after redaction.
|
||||
*
|
||||
* Usage:
|
||||
* bun run scripts/build-contradictions-fixture.ts \
|
||||
* [--queries-file FILE.jsonl] \
|
||||
* [--top-k N=5] \
|
||||
* [--judge MODEL=claude-haiku-4-5] \
|
||||
* [--max-pairs N=50] \
|
||||
* [--output PATH=test/fixtures/contradictions-eval-gold.jsonl] \
|
||||
* [--non-interactive]
|
||||
*
|
||||
* Interactive flow:
|
||||
* - Probe runs with --no-cache (so candidate pairs aren't pre-judged).
|
||||
* - For each candidate pair, the script prints A + B and prompts:
|
||||
* y) contradiction, n) not contradiction, s) skip
|
||||
* If y: prompt for severity (low|medium|high) and one-line axis.
|
||||
* - After labeling, redact in-memory, write JSONL with audit comments.
|
||||
* - Pre-commit safety: isCleanForCommit per line. Failures abort with
|
||||
* a sentinel string the operator must resolve manually.
|
||||
*
|
||||
* Non-interactive flow (`--non-interactive`): captures candidates with
|
||||
* NO labels, redacts, writes JSONL. Operator labels manually later.
|
||||
*/
|
||||
|
||||
import { readFileSync, writeFileSync, existsSync, mkdirSync } from 'node:fs';
|
||||
import { dirname } from 'node:path';
|
||||
import { createInterface } from 'node:readline/promises';
|
||||
import { stdin as input, stdout as output } from 'node:process';
|
||||
import { loadConfig, toEngineConfig } from '../src/core/config.ts';
|
||||
import { createEngine } from '../src/core/engine-factory.ts';
|
||||
import { connectWithRetry } from '../src/core/db.ts';
|
||||
import type { BrainEngine } from '../src/core/engine.ts';
|
||||
import { runContradictionProbe } from '../src/core/eval-contradictions/runner.ts';
|
||||
|
||||
async function connectLocalEngine(): Promise<BrainEngine> {
|
||||
const cfg = loadConfig();
|
||||
if (!cfg) throw new Error('No brain configured. Run `gbrain init` first.');
|
||||
const engineCfg = toEngineConfig(cfg);
|
||||
const engine = await createEngine(engineCfg);
|
||||
await connectWithRetry(engine, engineCfg, { noRetry: false });
|
||||
return engine;
|
||||
}
|
||||
import {
|
||||
createRedactionSession,
|
||||
isCleanForCommit,
|
||||
redactSlug,
|
||||
redactText,
|
||||
} from '../src/core/eval-contradictions/fixture-redact.ts';
|
||||
import type { ContradictionPair, Severity } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
interface ParsedFlags {
|
||||
queriesFile?: string;
|
||||
topK: number;
|
||||
judge: string;
|
||||
maxPairs: number;
|
||||
output: string;
|
||||
nonInteractive: boolean;
|
||||
help: boolean;
|
||||
}
|
||||
|
||||
function parseFlags(argv: string[]): ParsedFlags {
|
||||
const f: ParsedFlags = {
|
||||
topK: 5,
|
||||
judge: 'anthropic:claude-haiku-4-5',
|
||||
maxPairs: 50,
|
||||
output: 'test/fixtures/contradictions-eval-gold.jsonl',
|
||||
nonInteractive: false,
|
||||
help: false,
|
||||
};
|
||||
for (let i = 0; i < argv.length; i++) {
|
||||
const a = argv[i];
|
||||
const next = (): string => {
|
||||
const v = argv[++i];
|
||||
if (v === undefined) throw new Error(`flag ${a} requires a value`);
|
||||
return v;
|
||||
};
|
||||
if (a === '--help' || a === '-h') f.help = true;
|
||||
else if (a === '--queries-file') f.queriesFile = next();
|
||||
else if (a === '--top-k') f.topK = Number.parseInt(next(), 10);
|
||||
else if (a === '--judge') f.judge = next();
|
||||
else if (a === '--max-pairs') f.maxPairs = Number.parseInt(next(), 10);
|
||||
else if (a === '--output') f.output = next();
|
||||
else if (a === '--non-interactive') f.nonInteractive = true;
|
||||
else throw new Error(`unknown flag: ${a}`);
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
process.stderr.write(`Build a privacy-redacted gold fixture for the contradiction probe judge.
|
||||
|
||||
Usage:
|
||||
bun run scripts/build-contradictions-fixture.ts \\
|
||||
--queries-file FILE.jsonl # one JSON object per line, {query: "..."}
|
||||
[--top-k N=5]
|
||||
[--judge MODEL=claude-haiku-4-5]
|
||||
[--max-pairs N=50]
|
||||
[--output PATH=test/fixtures/contradictions-eval-gold.jsonl]
|
||||
[--non-interactive]
|
||||
|
||||
Output: JSONL with one labeled-and-redacted pair per line. Lines that
|
||||
fail isCleanForCommit are marked with a sentinel string the operator
|
||||
MUST resolve manually before commit. Audit log printed to stderr.
|
||||
`);
|
||||
}
|
||||
|
||||
function readQueriesFile(path: string): string[] {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const out: string[] = [];
|
||||
for (const line of raw.split(/\r?\n/)) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
if (trimmed.startsWith('{')) {
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed) as { query?: string };
|
||||
if (typeof parsed.query === 'string' && parsed.query.length > 0) {
|
||||
out.push(parsed.query);
|
||||
}
|
||||
} catch {
|
||||
// ignore
|
||||
}
|
||||
} else {
|
||||
out.push(trimmed);
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
async function promptLabel(rl: ReturnType<typeof createInterface>, pair: ContradictionPair): Promise<{
|
||||
contradicts: boolean;
|
||||
severity: Severity;
|
||||
axis: string;
|
||||
skip: boolean;
|
||||
}> {
|
||||
process.stderr.write(`\n--- Pair ---\n`);
|
||||
process.stderr.write(`A (${pair.a.slug}): ${pair.a.text.slice(0, 240)}${pair.a.text.length > 240 ? '…' : ''}\n`);
|
||||
process.stderr.write(`B (${pair.b.slug}): ${pair.b.text.slice(0, 240)}${pair.b.text.length > 240 ? '…' : ''}\n`);
|
||||
const ans = (await rl.question('Contradiction? [y/n/s skip]: ')).trim().toLowerCase();
|
||||
if (ans === 's' || ans === 'skip') {
|
||||
return { contradicts: false, severity: 'low', axis: '', skip: true };
|
||||
}
|
||||
if (ans !== 'y' && ans !== 'yes') {
|
||||
return { contradicts: false, severity: 'low', axis: '', skip: false };
|
||||
}
|
||||
let sev = (await rl.question('Severity [low/medium/high, default low]: ')).trim().toLowerCase();
|
||||
if (sev !== 'low' && sev !== 'medium' && sev !== 'high') sev = 'low';
|
||||
const axis = (await rl.question('One-line axis: ')).trim();
|
||||
return { contradicts: true, severity: sev as Severity, axis, skip: false };
|
||||
}
|
||||
|
||||
async function main(): Promise<void> {
|
||||
let flags: ParsedFlags;
|
||||
try {
|
||||
flags = parseFlags(process.argv.slice(2));
|
||||
} catch (err) {
|
||||
process.stderr.write(`Error: ${(err as Error).message}\n`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
if (flags.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
|
||||
if (!flags.queriesFile) {
|
||||
process.stderr.write(`--queries-file is required for the fixture build.\n`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
const queries = readQueriesFile(flags.queriesFile);
|
||||
if (queries.length === 0) {
|
||||
process.stderr.write(`No queries in ${flags.queriesFile}.\n`);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
process.stderr.write(`Building gold fixture against the local brain.\n`);
|
||||
process.stderr.write(`Queries: ${queries.length} Top-K: ${flags.topK} Max pairs: ${flags.maxPairs}\n`);
|
||||
process.stderr.write(`Output: ${flags.output}\n\n`);
|
||||
|
||||
const engine = await connectLocalEngine();
|
||||
try {
|
||||
// Run the probe with --no-cache so we get candidate pairs without
|
||||
// pre-judged verdicts. We don't keep verdicts; we hand-label every pair.
|
||||
// We intercept pairs via judgeFn returning contradicts:false (so nothing
|
||||
// is filtered to findings) and accumulating them for labeling instead.
|
||||
const candidatePairs: ContradictionPair[] = [];
|
||||
await runContradictionProbe({
|
||||
engine,
|
||||
queries,
|
||||
judgeModel: flags.judge,
|
||||
topK: flags.topK,
|
||||
noCache: true,
|
||||
// Wide budget so we don't hit cap during candidate collection.
|
||||
budgetUsd: 100,
|
||||
yesOverride: true,
|
||||
// Hijack the judge to collect pairs without spending tokens.
|
||||
judgeFn: async (input) => {
|
||||
candidatePairs.push({
|
||||
kind: 'cross_slug_chunks', // best-effort label; runner emits both kinds
|
||||
a: { slug: input.a.slug, chunk_id: 0, take_id: null, source_tier: 'curated', holder: input.a.holder ?? null, text: input.a.text },
|
||||
b: { slug: input.b.slug, chunk_id: 0, take_id: null, source_tier: 'curated', holder: input.b.holder ?? null, text: input.b.text },
|
||||
combined_score: 0,
|
||||
});
|
||||
return {
|
||||
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0, resolution_kind: null },
|
||||
usage: { inputTokens: 0, outputTokens: 0 },
|
||||
};
|
||||
},
|
||||
});
|
||||
|
||||
process.stderr.write(`\nCollected ${candidatePairs.length} candidate pairs.\n`);
|
||||
const capped = candidatePairs.slice(0, flags.maxPairs);
|
||||
|
||||
// Label.
|
||||
const rl = createInterface({ input, output });
|
||||
const session = createRedactionSession();
|
||||
const labeled: Array<{
|
||||
contradicts: boolean;
|
||||
severity: Severity;
|
||||
axis: string;
|
||||
query_redacted: string;
|
||||
a: { slug: string; text: string };
|
||||
b: { slug: string; text: string };
|
||||
}> = [];
|
||||
|
||||
for (let i = 0; i < capped.length; i++) {
|
||||
const pair = capped[i];
|
||||
process.stderr.write(`\n[${i + 1}/${capped.length}]`);
|
||||
let label: { contradicts: boolean; severity: Severity; axis: string; skip: boolean };
|
||||
if (flags.nonInteractive) {
|
||||
label = { contradicts: false, severity: 'low', axis: '', skip: false };
|
||||
} else {
|
||||
label = await promptLabel(rl, pair);
|
||||
if (label.skip) continue;
|
||||
}
|
||||
const redactedA = {
|
||||
slug: redactSlug(session, pair.a.slug),
|
||||
text: redactText(session, pair.a.text),
|
||||
};
|
||||
const redactedB = {
|
||||
slug: redactSlug(session, pair.b.slug),
|
||||
text: redactText(session, pair.b.text),
|
||||
};
|
||||
labeled.push({
|
||||
contradicts: label.contradicts,
|
||||
severity: label.severity,
|
||||
axis: redactText(session, label.axis),
|
||||
// Query gets redacted too, in case it referenced real names.
|
||||
query_redacted: '', // candidatePairs don't carry the query; populated by future iteration
|
||||
a: redactedA,
|
||||
b: redactedB,
|
||||
});
|
||||
}
|
||||
rl.close();
|
||||
|
||||
// Pre-commit safety: every text field must pass isCleanForCommit.
|
||||
const out: string[] = [];
|
||||
let flagged = 0;
|
||||
out.push(`# Gold fixture for contradiction probe judge (v0.32.6)`);
|
||||
out.push(`# schema_version: 1`);
|
||||
out.push(`# Generated: ${new Date().toISOString()}`);
|
||||
out.push(`# Audit (in-memory redactions applied):`);
|
||||
for (const entry of session.audit.slice(0, 100)) {
|
||||
out.push(`# ${entry}`);
|
||||
}
|
||||
out.push(`# Total redactions: ${session.audit.length}`);
|
||||
out.push(`#`);
|
||||
for (const row of labeled) {
|
||||
const cleanA = isCleanForCommit(row.a.text) && isCleanForCommit(row.a.slug);
|
||||
const cleanB = isCleanForCommit(row.b.text) && isCleanForCommit(row.b.slug);
|
||||
const sentinel = !cleanA || !cleanB ? ' [REDACT?]' : '';
|
||||
if (sentinel) flagged++;
|
||||
out.push(JSON.stringify({ ...row, ...(sentinel ? { _operator_review: 'REDACTION INCOMPLETE — fix manually before commit' } : {}) }));
|
||||
}
|
||||
|
||||
// Ensure output dir exists, then write.
|
||||
mkdirSync(dirname(flags.output), { recursive: true });
|
||||
if (existsSync(flags.output)) {
|
||||
process.stderr.write(`\nWARN: ${flags.output} already exists. Overwriting.\n`);
|
||||
}
|
||||
writeFileSync(flags.output, out.join('\n') + '\n');
|
||||
process.stderr.write(`\nWrote ${labeled.length} labeled pairs to ${flags.output}.\n`);
|
||||
if (flagged > 0) {
|
||||
process.stderr.write(`*** ${flagged} pair(s) flagged with [REDACT?] — review before commit ***\n`);
|
||||
process.exit(1);
|
||||
}
|
||||
process.stderr.write(`OK — pre-commit safety pass. Inspect the file once more before committing.\n`);
|
||||
} finally {
|
||||
await engine.disconnect();
|
||||
}
|
||||
}
|
||||
|
||||
main().catch((err) => {
|
||||
process.stderr.write(`fatal: ${(err as Error).message}\n`);
|
||||
process.exit(1);
|
||||
});
|
||||
@@ -0,0 +1,78 @@
|
||||
---
|
||||
feature_pitch:
|
||||
headline: "Brain consistency measurement instrument + self-healing loop."
|
||||
one_liner: "gbrain eval suspected-contradictions samples retrieval pairs and flags conflicts, with doctor + MCP + dream-cycle integration."
|
||||
---
|
||||
|
||||
# v0.32.6 Migration: Contradiction Probe
|
||||
|
||||
The agent ran this version migration when the user upgraded gbrain. After
|
||||
upgrade, this file is read by the auto-update agent which performs the
|
||||
following steps automatically.
|
||||
|
||||
## Mechanical (handled by `gbrain apply-migrations`)
|
||||
|
||||
Two new tables get added on first connect after upgrade:
|
||||
|
||||
- `eval_contradictions_cache` — persistent judge verdict cache, keyed on
|
||||
(chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy).
|
||||
TTL-driven (default 30 days). Bounded by `sweepContradictionCache`.
|
||||
- `eval_contradictions_runs` — one row per `gbrain eval
|
||||
suspected-contradictions` run. Source for the `trend` sub-subcommand
|
||||
and the doctor `contradictions` check.
|
||||
|
||||
Both are RLS-enabled on Postgres (matches existing eval table posture).
|
||||
On PGLite they're plain tables.
|
||||
|
||||
## Operator action (optional)
|
||||
|
||||
The probe is opt-in — it doesn't run automatically. To use it:
|
||||
|
||||
```bash
|
||||
# Run against one query:
|
||||
gbrain eval suspected-contradictions --query "what is acme's MRR" --top-k 5 --json
|
||||
|
||||
# Run against a queries file:
|
||||
gbrain eval suspected-contradictions --queries-file ~/.gbrain/queries.jsonl --top-k 5
|
||||
|
||||
# Inspect findings without re-running:
|
||||
gbrain eval suspected-contradictions review --severity high
|
||||
|
||||
# Trend over time:
|
||||
gbrain eval suspected-contradictions trend --days 30
|
||||
```
|
||||
|
||||
## Doctor integration
|
||||
|
||||
After the first probe run, `gbrain doctor` shows a `contradictions` check
|
||||
with severity breakdown + paste-ready resolution commands per HIGH-severity
|
||||
finding.
|
||||
|
||||
## MCP integration
|
||||
|
||||
The agent can call `find_contradictions(slug?, severity?, limit?)` during
|
||||
conversations to surface findings proactively. Reads from the latest probe
|
||||
run; does NOT trigger a new probe.
|
||||
|
||||
## Dream cycle integration
|
||||
|
||||
The synthesize phase reads the latest probe's top-5 highest-severity
|
||||
findings and injects them as an informational block into the synthesis
|
||||
prompt. The subagent sees what to reconcile when writing compiled_truth to
|
||||
flagged slugs. Empty trend = empty block; fresh-install behavior unchanged.
|
||||
|
||||
## Verification
|
||||
|
||||
```bash
|
||||
gbrain doctor --json | jq '.checks[] | select(.name == "contradictions")'
|
||||
```
|
||||
|
||||
Should print an entry with status "ok" (no probe runs yet) or "warn"
|
||||
(high-severity findings present). If the entry is missing, the migration
|
||||
didn't apply — run `gbrain apply-migrations --yes` and re-check.
|
||||
|
||||
## See also
|
||||
|
||||
- `docs/contradictions.md` — architecture, severity rubric, action criteria
|
||||
- `docs/eval-bench.md` — recommended nightly cadence + trend workflow
|
||||
- CHANGELOG `## [0.32.6]` — full release notes
|
||||
@@ -1632,6 +1632,86 @@ export async function runDoctor(engine: BrainEngine | null, args: string[], dbSo
|
||||
}
|
||||
}
|
||||
|
||||
// 11a-bis-3. contradictions probe summary (v0.32.6 — M1).
|
||||
//
|
||||
// Reads the most recent eval_contradictions_runs row and surfaces:
|
||||
// - headline count + severity breakdown
|
||||
// - paste-ready resolution commands per HIGH-severity finding
|
||||
// - Wilson CI band so the user knows whether the headline is trustworthy
|
||||
// Skipped (status: 'ok') when the table is empty — the probe simply hasn't
|
||||
// run yet, which is normal on a fresh install.
|
||||
progress.heartbeat('contradictions');
|
||||
try {
|
||||
const recent = await engine.loadContradictionsTrend(7);
|
||||
if (recent.length === 0) {
|
||||
checks.push({
|
||||
name: 'contradictions',
|
||||
status: 'ok',
|
||||
message: 'No probe runs in the last 7 days. Run `gbrain eval suspected-contradictions --query "..." --top-k 5` to populate.',
|
||||
});
|
||||
} else {
|
||||
const latest = recent[0];
|
||||
const report = latest.report_json as Record<string, unknown> | null;
|
||||
const perQuery = (report?.per_query as Array<{
|
||||
contradictions: Array<{
|
||||
severity: 'low' | 'medium' | 'high';
|
||||
axis: string;
|
||||
a: { slug: string };
|
||||
b: { slug: string };
|
||||
resolution_command: string;
|
||||
}>;
|
||||
}> | undefined) ?? [];
|
||||
let high = 0, medium = 0, low = 0;
|
||||
const highFindings: Array<{ a: string; b: string; axis: string; cmd: string }> = [];
|
||||
for (const q of perQuery) {
|
||||
for (const c of q.contradictions) {
|
||||
if (c.severity === 'high') {
|
||||
high++;
|
||||
highFindings.push({ a: c.a.slug, b: c.b.slug, axis: c.axis, cmd: c.resolution_command });
|
||||
} else if (c.severity === 'medium') medium++;
|
||||
else low++;
|
||||
}
|
||||
}
|
||||
const total = high + medium + low;
|
||||
if (total === 0) {
|
||||
checks.push({
|
||||
name: 'contradictions',
|
||||
status: 'ok',
|
||||
message: `Latest probe run (${latest.ran_at.slice(0, 10)}) found no suspected contradictions across ${latest.queries_evaluated} queries.`,
|
||||
});
|
||||
} else {
|
||||
const ciLow = (latest.wilson_ci_lower * 100).toFixed(0);
|
||||
const ciHigh = (latest.wilson_ci_upper * 100).toFixed(0);
|
||||
const lines = [
|
||||
`${total} suspected contradictions (high=${high} medium=${medium} low=${low}) detected by latest probe — Wilson CI 95%: ${ciLow}-${ciHigh}%.`,
|
||||
];
|
||||
for (const f of highFindings.slice(0, 3)) {
|
||||
lines.push(` HIGH: ${f.a} vs ${f.b}${f.axis ? ' — ' + f.axis : ''}`);
|
||||
lines.push(` → ${f.cmd}`);
|
||||
}
|
||||
if (highFindings.length > 3) {
|
||||
lines.push(` …and ${highFindings.length - 3} more — see \`gbrain eval suspected-contradictions review\``);
|
||||
}
|
||||
checks.push({
|
||||
name: 'contradictions',
|
||||
status: high > 0 ? 'warn' : 'ok',
|
||||
message: lines.join('\n '),
|
||||
});
|
||||
}
|
||||
}
|
||||
} catch (err) {
|
||||
const code = (err as { code?: string } | null)?.code;
|
||||
if (code === '42P01') {
|
||||
checks.push({ name: 'contradictions', status: 'ok', message: 'Skipped (eval_contradictions_runs table unavailable — apply migrations to enable)' });
|
||||
} else {
|
||||
checks.push({
|
||||
name: 'contradictions',
|
||||
status: 'warn',
|
||||
message: `Could not read contradictions trend: ${(err as Error)?.message ?? String(err)}`,
|
||||
});
|
||||
}
|
||||
}
|
||||
|
||||
// 11a-bis-2. facts_extraction_health (v0.31.2 — codex P1 #3).
|
||||
//
|
||||
// Mirrors the eval_capture check shape but reads facts:absorb rows
|
||||
|
||||
@@ -0,0 +1,365 @@
|
||||
/**
|
||||
* `gbrain eval suspected-contradictions` — v0.32.6 contradiction probe CLI.
|
||||
*
|
||||
* Three sub-subcommands:
|
||||
* - run (default): execute one probe pass; --queries-file / --query /
|
||||
* --from-capture. Cost-capped via --budget-usd; --yes overrides
|
||||
* pre-flight refusal. Writes a row to eval_contradictions_runs on
|
||||
* success, prints JSON to stdout when --json, human summary to stderr.
|
||||
* - trend: read eval_contradictions_runs and render the ASCII chart.
|
||||
* - review: surface findings from the most recent run, optionally
|
||||
* filtered by severity. Reuses the M7 resolution proposals.
|
||||
*
|
||||
* Output discipline:
|
||||
* - stderr: human-readable summary
|
||||
* - stdout: JSON (when --json is set), reserved for piping
|
||||
* - exit codes: 0 success, 1 over-budget without --yes, 2 mutually-
|
||||
* exclusive sources OR empty capture table with hint
|
||||
*/
|
||||
|
||||
import { readFileSync } from 'node:fs';
|
||||
import type { BrainEngine } from '../core/engine.ts';
|
||||
import {
|
||||
PreFlightBudgetError,
|
||||
runContradictionProbe,
|
||||
} from '../core/eval-contradictions/runner.ts';
|
||||
import { loadTrend, renderTrendChart, writeRunRow } from '../core/eval-contradictions/trends.ts';
|
||||
import {
|
||||
bucketBySeverity,
|
||||
compareSeverityDesc,
|
||||
} from '../core/eval-contradictions/severity-classify.ts';
|
||||
import type {
|
||||
ContradictionFinding,
|
||||
Severity,
|
||||
} from '../core/eval-contradictions/types.ts';
|
||||
|
||||
interface ParsedFlags {
|
||||
sub: 'run' | 'trend' | 'review';
|
||||
// run flags
|
||||
queriesFile?: string;
|
||||
query?: string;
|
||||
fromCapture?: boolean;
|
||||
topK: number;
|
||||
judge: string;
|
||||
limit?: number;
|
||||
budgetUsd: number;
|
||||
output?: string;
|
||||
maxPairChars: number;
|
||||
sampling: 'deterministic' | 'score-first';
|
||||
noCache: boolean;
|
||||
refreshCache: boolean;
|
||||
json: boolean;
|
||||
yes: boolean;
|
||||
// trend flags
|
||||
days: number;
|
||||
// review flags
|
||||
severity?: Severity;
|
||||
since?: string;
|
||||
// help
|
||||
help: boolean;
|
||||
}
|
||||
|
||||
function parseFlags(args: string[]): ParsedFlags {
|
||||
// Sub-subcommand: first positional that doesn't start with --
|
||||
let sub: 'run' | 'trend' | 'review' = 'run';
|
||||
const rest: string[] = [];
|
||||
for (let i = 0; i < args.length; i++) {
|
||||
const a = args[i];
|
||||
if (i === 0 && !a.startsWith('--')) {
|
||||
if (a === 'run' || a === 'trend' || a === 'review') {
|
||||
sub = a;
|
||||
continue;
|
||||
}
|
||||
}
|
||||
rest.push(a);
|
||||
}
|
||||
const isTty = process.stdout.isTTY === true;
|
||||
const f: ParsedFlags = {
|
||||
sub,
|
||||
topK: 5,
|
||||
judge: 'anthropic:claude-haiku-4-5',
|
||||
budgetUsd: isTty ? 5 : 1,
|
||||
maxPairChars: 1500,
|
||||
sampling: 'deterministic',
|
||||
noCache: false,
|
||||
refreshCache: false,
|
||||
json: false,
|
||||
yes: false,
|
||||
days: 30,
|
||||
help: false,
|
||||
};
|
||||
for (let i = 0; i < rest.length; i++) {
|
||||
const arg = rest[i];
|
||||
const next = (): string => {
|
||||
const v = rest[++i];
|
||||
if (v === undefined) throw new Error(`flag ${arg} requires a value`);
|
||||
return v;
|
||||
};
|
||||
if (arg === '--help' || arg === '-h') f.help = true;
|
||||
else if (arg === '--queries-file') f.queriesFile = next();
|
||||
else if (arg === '--query') f.query = next();
|
||||
else if (arg === '--from-capture') f.fromCapture = true;
|
||||
else if (arg === '--top-k') f.topK = Number.parseInt(next(), 10);
|
||||
else if (arg === '--judge') f.judge = next();
|
||||
else if (arg === '--limit') f.limit = Number.parseInt(next(), 10);
|
||||
else if (arg === '--budget-usd') f.budgetUsd = Number.parseFloat(next());
|
||||
else if (arg === '--output') f.output = next();
|
||||
else if (arg === '--max-pair-chars') f.maxPairChars = Number.parseInt(next(), 10);
|
||||
else if (arg === '--sampling') {
|
||||
const v = next();
|
||||
if (v !== 'deterministic' && v !== 'score-first') {
|
||||
throw new Error('--sampling must be deterministic|score-first');
|
||||
}
|
||||
f.sampling = v;
|
||||
}
|
||||
else if (arg === '--no-cache') f.noCache = true;
|
||||
else if (arg === '--refresh-cache') f.refreshCache = true;
|
||||
else if (arg === '--json') f.json = true;
|
||||
else if (arg === '--yes' || arg === '-y') f.yes = true;
|
||||
else if (arg === '--days') f.days = Number.parseInt(next(), 10);
|
||||
else if (arg === '--severity') {
|
||||
const v = next();
|
||||
if (v !== 'low' && v !== 'medium' && v !== 'high') {
|
||||
throw new Error('--severity must be low|medium|high');
|
||||
}
|
||||
f.severity = v;
|
||||
}
|
||||
else if (arg === '--since') f.since = next();
|
||||
else {
|
||||
throw new Error(`unknown flag: ${arg}`);
|
||||
}
|
||||
}
|
||||
return f;
|
||||
}
|
||||
|
||||
function printHelp(): void {
|
||||
console.error(`Usage:
|
||||
gbrain eval suspected-contradictions [run]
|
||||
[--queries-file FILE.jsonl | --query "..." | --from-capture]
|
||||
[--top-k N=5] [--judge MODEL=claude-haiku-4-5]
|
||||
[--limit N] [--budget-usd N] [--output FILE]
|
||||
[--max-pair-chars N=1500] [--sampling deterministic|score-first]
|
||||
[--no-cache] [--refresh-cache] [--json] [--yes]
|
||||
|
||||
gbrain eval suspected-contradictions trend [--days N=30] [--json]
|
||||
|
||||
gbrain eval suspected-contradictions review
|
||||
[--severity low|medium|high] [--since YYYY-MM-DD]
|
||||
|
||||
The probe samples top-K retrieval pairs and asks an LLM judge whether
|
||||
any pair contradicts on a factual claim relevant to the query. Outputs
|
||||
JSON (stable schema_version: 1) and a human summary.
|
||||
`);
|
||||
}
|
||||
|
||||
/** Read --queries-file as JSONL or plain-text-one-query-per-line. */
|
||||
function readQueriesFile(path: string): string[] {
|
||||
const raw = readFileSync(path, 'utf8');
|
||||
const queries: string[] = [];
|
||||
for (const line of raw.split(/\r?\n/)) {
|
||||
const trimmed = line.trim();
|
||||
if (!trimmed) continue;
|
||||
if (trimmed.startsWith('{')) {
|
||||
try {
|
||||
const parsed = JSON.parse(trimmed) as { query?: string };
|
||||
if (typeof parsed.query === 'string' && parsed.query.length > 0) {
|
||||
queries.push(parsed.query);
|
||||
}
|
||||
} catch {
|
||||
// ignore malformed line
|
||||
}
|
||||
} else {
|
||||
queries.push(trimmed);
|
||||
}
|
||||
}
|
||||
return queries;
|
||||
}
|
||||
|
||||
/** Detect non-empty eval_candidates; exit 2 with hint when empty (A4). */
|
||||
async function loadFromCapture(engine: BrainEngine, limit?: number): Promise<string[]> {
|
||||
const rows = await engine.executeRaw<{ query: string }>(
|
||||
`SELECT query FROM eval_candidates WHERE query IS NOT NULL ORDER BY id DESC LIMIT $1`,
|
||||
[limit ?? 100],
|
||||
);
|
||||
if (!rows || rows.length === 0) {
|
||||
console.error(
|
||||
`--from-capture: no rows in eval_candidates. Captures are off by default in v0.25.0+.\n` +
|
||||
`Enable with:\n` +
|
||||
` export GBRAIN_CONTRIBUTOR_MODE=1\n` +
|
||||
`or set 'eval.capture: true' in your gbrain config. Re-run queries to populate, then try again.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
return rows.map((r) => r.query);
|
||||
}
|
||||
|
||||
function exclusiveOneOf(...flags: Array<unknown>): boolean {
|
||||
let count = 0;
|
||||
for (const f of flags) if (f) count++;
|
||||
return count === 1;
|
||||
}
|
||||
|
||||
async function runRun(engine: BrainEngine, f: ParsedFlags): Promise<void> {
|
||||
if (!exclusiveOneOf(f.queriesFile, f.query, f.fromCapture)) {
|
||||
console.error(
|
||||
`Must pass exactly one of: --queries-file FILE, --query "...", --from-capture.`,
|
||||
);
|
||||
process.exit(2);
|
||||
}
|
||||
|
||||
let queries: string[] = [];
|
||||
if (f.queriesFile) queries = readQueriesFile(f.queriesFile);
|
||||
else if (f.query) queries = [f.query];
|
||||
else if (f.fromCapture) queries = await loadFromCapture(engine, f.limit);
|
||||
|
||||
if (typeof f.limit === 'number' && f.limit > 0) {
|
||||
queries = queries.slice(0, f.limit);
|
||||
}
|
||||
|
||||
if (queries.length === 0) {
|
||||
console.error('No queries to evaluate.');
|
||||
process.exit(1);
|
||||
}
|
||||
|
||||
console.error(
|
||||
`Contradiction probe: ${queries.length} queries, top-${f.topK}, judge=${f.judge}, budget=$${f.budgetUsd.toFixed(2)}.`,
|
||||
);
|
||||
|
||||
// Refresh-cache: sweep before run so the cache misses on this pass.
|
||||
if (f.refreshCache) {
|
||||
const swept = await engine.sweepContradictionCache();
|
||||
console.error(`Swept ${swept} expired cache rows before run.`);
|
||||
}
|
||||
|
||||
try {
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries,
|
||||
judgeModel: f.judge,
|
||||
topK: f.topK,
|
||||
sampling: f.sampling,
|
||||
budgetUsd: f.budgetUsd,
|
||||
yesOverride: f.yes,
|
||||
maxPairChars: f.maxPairChars,
|
||||
noCache: f.noCache,
|
||||
});
|
||||
|
||||
// Persist to runs table (M5).
|
||||
await writeRunRow(engine, out.report, out.report.duration_ms);
|
||||
|
||||
// Human summary.
|
||||
const r = out.report;
|
||||
const pct = (n: number) => (n * 100).toFixed(0);
|
||||
const lines: string[] = [];
|
||||
lines.push(``);
|
||||
lines.push(`Results: ${r.queries_evaluated} queries, top-${r.top_k} each, judge=${r.judge_model}`);
|
||||
lines.push(` Queries with >=1 contradiction: ${r.queries_with_contradiction} / ${r.queries_evaluated} (${pct(r.queries_with_contradiction / Math.max(1, r.queries_evaluated))}%)`);
|
||||
lines.push(` Wilson CI 95%: ${pct(r.calibration.wilson_ci_95.lower)}–${pct(r.calibration.wilson_ci_95.upper)}%`);
|
||||
if (r.calibration.small_sample_note) {
|
||||
lines.push(` Note: ${r.calibration.small_sample_note}`);
|
||||
}
|
||||
lines.push(` Total contradictions flagged: ${r.total_contradictions_flagged}`);
|
||||
lines.push(` Judge errors: ${r.judge_errors.total} (parse_fail=${r.judge_errors.parse_fail} timeout=${r.judge_errors.timeout} http_5xx=${r.judge_errors.http_5xx} refusal=${r.judge_errors.refusal})`);
|
||||
lines.push(` Cache: ${r.cache.hits} hits / ${r.cache.misses} misses (${pct(r.cache.hit_rate)}% hit-rate)`);
|
||||
lines.push(` Source-tier breakdown:`);
|
||||
lines.push(` curated_vs_curated: ${r.source_tier_breakdown.curated_vs_curated}`);
|
||||
lines.push(` curated_vs_bulk: ${r.source_tier_breakdown.curated_vs_bulk}`);
|
||||
lines.push(` bulk_vs_bulk: ${r.source_tier_breakdown.bulk_vs_bulk}`);
|
||||
lines.push(` other: ${r.source_tier_breakdown.other}`);
|
||||
lines.push(` Cost: $${r.cost_usd.total.toFixed(4)} (judge $${r.cost_usd.judge.toFixed(4)} + embedding $${r.cost_usd.embedding.toFixed(6)})`);
|
||||
lines.push(` Duration: ${r.duration_ms}ms`);
|
||||
if (r.hot_pages.length > 0) {
|
||||
lines.push(` Hot pages:`);
|
||||
for (const p of r.hot_pages.slice(0, 5)) {
|
||||
lines.push(` ${p.slug} (${p.appearances}, max ${p.max_severity})`);
|
||||
}
|
||||
}
|
||||
if (out.capHitMidRun) {
|
||||
lines.push(` *** budget cap hit mid-run; report is partial ***`);
|
||||
}
|
||||
console.error(lines.join('\n'));
|
||||
|
||||
if (f.output) {
|
||||
const { writeFileSync } = await import('node:fs');
|
||||
writeFileSync(f.output, JSON.stringify(r, null, 2));
|
||||
console.error(`Details: ${f.output}`);
|
||||
}
|
||||
|
||||
if (f.json) {
|
||||
console.log(JSON.stringify(r, null, 2));
|
||||
}
|
||||
|
||||
if (out.capHitMidRun && !f.yes) {
|
||||
// Cap was hit; we already wrote a partial. Exit non-zero to signal.
|
||||
process.exit(1);
|
||||
}
|
||||
} catch (err) {
|
||||
if (err instanceof PreFlightBudgetError) {
|
||||
console.error(`Pre-flight refused: ${err.message}`);
|
||||
process.exit(1);
|
||||
}
|
||||
throw err;
|
||||
}
|
||||
}
|
||||
|
||||
async function runTrend(engine: BrainEngine, f: ParsedFlags): Promise<void> {
|
||||
const rows = await loadTrend(engine, f.days);
|
||||
if (f.json) {
|
||||
console.log(JSON.stringify({ schema_version: 1, days: f.days, rows }, null, 2));
|
||||
return;
|
||||
}
|
||||
console.error(renderTrendChart(rows));
|
||||
}
|
||||
|
||||
async function runReview(engine: BrainEngine, f: ParsedFlags): Promise<void> {
|
||||
const rows = await loadTrend(engine, 90);
|
||||
if (rows.length === 0) {
|
||||
console.error('No probe runs in the last 90 days. Run the probe first.');
|
||||
process.exit(1);
|
||||
}
|
||||
const latest = rows[0];
|
||||
const report = latest.report_json;
|
||||
if (!report || !report.per_query) {
|
||||
console.error('Latest run has no findings to review.');
|
||||
return;
|
||||
}
|
||||
const allFindings: ContradictionFinding[] = report.per_query.flatMap((q) => q.contradictions);
|
||||
const filtered = f.severity ? allFindings.filter((c) => c.severity === f.severity) : allFindings;
|
||||
if (filtered.length === 0) {
|
||||
console.error(`No findings${f.severity ? ` at severity=${f.severity}` : ''}.`);
|
||||
return;
|
||||
}
|
||||
filtered.sort((a, b) => compareSeverityDesc(a.severity, b.severity));
|
||||
const buckets = bucketBySeverity(filtered);
|
||||
for (const sev of ['high', 'medium', 'low'] as const) {
|
||||
const items = buckets[sev];
|
||||
if (items.length === 0) continue;
|
||||
console.error(`\n${sev.toUpperCase()} severity (${items.length}):`);
|
||||
for (const item of items) {
|
||||
console.error(` - ${item.a.slug} vs ${item.b.slug}`);
|
||||
if (item.axis) console.error(` axis: ${item.axis}`);
|
||||
console.error(` → ${item.resolution_command}`);
|
||||
}
|
||||
}
|
||||
}
|
||||
|
||||
export async function runEvalSuspectedContradictions(
|
||||
engine: BrainEngine,
|
||||
args: string[],
|
||||
): Promise<void> {
|
||||
let flags: ParsedFlags;
|
||||
try {
|
||||
flags = parseFlags(args);
|
||||
} catch (err) {
|
||||
console.error(`Error: ${(err as Error).message}`);
|
||||
printHelp();
|
||||
process.exit(2);
|
||||
}
|
||||
if (flags.help) {
|
||||
printHelp();
|
||||
return;
|
||||
}
|
||||
if (flags.sub === 'run') return runRun(engine, flags);
|
||||
if (flags.sub === 'trend') return runTrend(engine, flags);
|
||||
if (flags.sub === 'review') return runReview(engine, flags);
|
||||
}
|
||||
@@ -45,6 +45,13 @@ export async function runEvalCommand(engine: BrainEngine, args: string[]): Promi
|
||||
const { runEvalCrossModal } = await import('./eval-cross-modal.ts');
|
||||
process.exit(await runEvalCrossModal(args.slice(1)));
|
||||
}
|
||||
if (sub === 'suspected-contradictions') {
|
||||
// v0.32.6 — contradiction probe. Engine connected (calls hybridSearch +
|
||||
// the eval_contradictions_cache + _runs tables). Matches the `replay`
|
||||
// dispatch pattern.
|
||||
const { runEvalSuspectedContradictions } = await import('./eval-suspected-contradictions.ts');
|
||||
return runEvalSuspectedContradictions(engine, args.slice(1));
|
||||
}
|
||||
|
||||
const opts = parseArgs(args);
|
||||
|
||||
|
||||
@@ -268,6 +268,12 @@ export async function runPhaseSynthesize(
|
||||
);
|
||||
}
|
||||
|
||||
// v0.32.6 M2: pre-fetch prior contradictions from the most recent probe
|
||||
// run (if any). Surfaced as an informational block to the synthesize
|
||||
// subagent so it knows which slugs it should reconcile if it writes to
|
||||
// them. Best-effort — a probe that's never run is a normal early state.
|
||||
const priorContradictionsBlock = await loadPriorContradictionsBlock(engine);
|
||||
|
||||
// Discover.
|
||||
const transcripts = opts.inputFile
|
||||
? loadAdHocTranscript(opts.inputFile, config.minChars, config.excludePatterns, opts.bypassDreamGuard)
|
||||
@@ -388,7 +394,7 @@ export async function runPhaseSynthesize(
|
||||
const isChunked = chunks.length > 1;
|
||||
for (let i = 0; i < chunks.length; i++) {
|
||||
const childData: SubagentHandlerData = {
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length),
|
||||
prompt: buildSynthesisPrompt(t, chunks[i], i, chunks.length, priorContradictionsBlock),
|
||||
model: config.model,
|
||||
max_turns: 30,
|
||||
allowed_slug_prefixes: allowedSlugPrefixes,
|
||||
@@ -704,11 +710,58 @@ Two reasons max, one phrase each.`;
|
||||
* collection time). Sonnet still gets the chunked seed via the prompt's
|
||||
* `USE THIS in slugs` rule for the happy path.
|
||||
*/
|
||||
/**
|
||||
* v0.32.6 M2 — Load prior probe findings into an informational block.
|
||||
* Returns '' if no probe runs exist or the engine doesn't know how (pre-v33
|
||||
* brain that hasn't applied migrations). Best-effort and silent on failure.
|
||||
*/
|
||||
async function loadPriorContradictionsBlock(engine: BrainEngine): Promise<string> {
|
||||
try {
|
||||
const rows = await engine.loadContradictionsTrend(30);
|
||||
if (!rows || rows.length === 0) return '';
|
||||
const latest = rows[0];
|
||||
const report = latest.report_json as Record<string, unknown> | null;
|
||||
const perQuery = (report?.per_query as Array<{
|
||||
contradictions: Array<{
|
||||
severity: 'low' | 'medium' | 'high';
|
||||
axis: string;
|
||||
a: { slug: string };
|
||||
b: { slug: string };
|
||||
}>;
|
||||
}> | undefined) ?? [];
|
||||
const findings: Array<{ severity: string; axis: string; a: string; b: string }> = [];
|
||||
for (const q of perQuery) {
|
||||
for (const c of q.contradictions) {
|
||||
findings.push({ severity: c.severity, axis: c.axis, a: c.a.slug, b: c.b.slug });
|
||||
}
|
||||
}
|
||||
if (findings.length === 0) return '';
|
||||
// Sort by severity DESC (high first); take top 5 to keep prompt bounded.
|
||||
const rank: Record<string, number> = { high: 3, medium: 2, low: 1 };
|
||||
findings.sort((x, y) => (rank[y.severity] ?? 0) - (rank[x.severity] ?? 0));
|
||||
const top = findings.slice(0, 5);
|
||||
const lines = top.map((f) => ` - [${f.severity}] ${f.a} vs ${f.b}${f.axis ? ' — ' + f.axis : ''}`);
|
||||
return [
|
||||
'',
|
||||
'PRIOR DETECTED CONTRADICTIONS (latest probe run, severity DESC, top 5):',
|
||||
...lines,
|
||||
'',
|
||||
'If your synthesis writes to any of these slugs, reconcile the contradiction',
|
||||
'in the compiled_truth instead of recreating it. Either update to the newer/',
|
||||
'correct value, mark the older claim as historical, or note the conflict',
|
||||
'explicitly. Ignore findings irrelevant to what this transcript covers.',
|
||||
].join('\n');
|
||||
} catch {
|
||||
return '';
|
||||
}
|
||||
}
|
||||
|
||||
function buildSynthesisPrompt(
|
||||
t: DiscoveredTranscript,
|
||||
chunkText: string,
|
||||
chunkIdx: number,
|
||||
chunkTotal: number,
|
||||
priorContradictionsBlock = '',
|
||||
): string {
|
||||
const dateHint = t.inferredDate ?? today();
|
||||
const baseSlugSegment = sanitizeForSlug(t.basename) || `session-${dateHint}`;
|
||||
@@ -727,7 +780,7 @@ function buildSynthesisPrompt(
|
||||
CONTEXT
|
||||
- Today's date: ${dateHint}
|
||||
- Transcript hash suffix (USE THIS in slugs): ${hashSuffix}
|
||||
- Source file basename: ${baseSlugSegment}${chunkBanner}
|
||||
- Source file basename: ${baseSlugSegment}${chunkBanner}${priorContradictionsBlock}
|
||||
|
||||
OUTPUT POLICY (ALL of these are required)
|
||||
1. Quote the user verbatim. Do not paraphrase memorable phrasings.
|
||||
|
||||
@@ -846,6 +846,116 @@ export interface BrainEngine {
|
||||
getDreamVerdict(filePath: string, contentHash: string): Promise<DreamVerdict | null>;
|
||||
putDreamVerdict(filePath: string, contentHash: string, verdict: DreamVerdictInput): Promise<void>;
|
||||
|
||||
// ============================================================
|
||||
// v0.32.6 Contradiction probe — batched takes fetch + cache + trends
|
||||
// ============================================================
|
||||
|
||||
/**
|
||||
* Batch fetch: for each page_id in the input array, return the page's
|
||||
* currently-active takes. Single query under the hood (`WHERE page_id =
|
||||
* ANY($1) AND active = true`); replaces the O(K) loop of listTakes calls
|
||||
* the contradiction probe would otherwise pay per probe-query.
|
||||
*
|
||||
* Returns a Map keyed on page_id; pages with no active takes get an empty
|
||||
* array (NOT undefined) so callers can avoid existence checks.
|
||||
*
|
||||
* Honors `takesHoldersAllowList` for MCP scope enforcement (mirrors
|
||||
* listTakes contract). Pass undefined from trusted local callers.
|
||||
*/
|
||||
listActiveTakesForPages(
|
||||
pageIds: number[],
|
||||
opts?: { takesHoldersAllowList?: string[] },
|
||||
): Promise<Map<number, Take[]>>;
|
||||
|
||||
/**
|
||||
* Persist a single contradiction-probe run row. Caller supplies a full
|
||||
* `ContradictionsRunRow`-shaped object; the engine inserts as-is.
|
||||
*
|
||||
* Idempotent on `run_id`: re-inserting an existing run_id is a no-op
|
||||
* (caller passes ISO-timestamp-shaped run_ids that won't collide
|
||||
* unintentionally). Returns true iff a row was inserted.
|
||||
*/
|
||||
writeContradictionsRun(row: {
|
||||
run_id: string;
|
||||
judge_model: string;
|
||||
prompt_version: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}): Promise<boolean>;
|
||||
|
||||
/**
|
||||
* Load contradiction-probe run history within the last N days, ordered
|
||||
* newest first. Used by `gbrain eval suspected-contradictions trend` and
|
||||
* by the doctor `contradictions` check. `report_json` and
|
||||
* `source_tier_breakdown` are parsed JSONB columns.
|
||||
*/
|
||||
loadContradictionsTrend(days: number): Promise<Array<{
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
judge_model: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}>>;
|
||||
|
||||
/**
|
||||
* Cache lookup for the contradiction probe's persistent judge cache (P2).
|
||||
* Returns the verdict JSON if a row exists with matching key AND non-expired
|
||||
* `expires_at`. NULL means cache miss (judge call needed).
|
||||
*
|
||||
* Key shape mirrors the table primary key: (chunk_a_hash, chunk_b_hash,
|
||||
* model_id, prompt_version, truncation_policy). Codex's outside-voice
|
||||
* critique fixed the key to include prompt_version + truncation_policy so
|
||||
* prompt edits cleanly invalidate prior verdicts.
|
||||
*/
|
||||
getContradictionCacheEntry(key: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
}): Promise<Record<string, unknown> | null>;
|
||||
|
||||
/**
|
||||
* Upsert a contradiction-probe judge verdict into the persistent cache.
|
||||
* `ttl_seconds` controls expires_at (default 30 days from now). Caller
|
||||
* supplies pre-hashed chunk text + the verdict to cache.
|
||||
*
|
||||
* ON CONFLICT DO UPDATE so re-runs refresh expires_at; this is the simplest
|
||||
* shape for "I judged the same pair again with the same config, slide the
|
||||
* TTL forward."
|
||||
*/
|
||||
putContradictionCacheEntry(opts: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
verdict: Record<string, unknown>;
|
||||
ttl_seconds?: number;
|
||||
}): Promise<void>;
|
||||
|
||||
/**
|
||||
* Sweep expired cache entries. Returns count deleted. Periodic call from
|
||||
* cache.ts — keeps the table bounded without requiring a cron.
|
||||
*/
|
||||
sweepContradictionCache(): Promise<number>;
|
||||
|
||||
// ============================================================
|
||||
// v0.31 Hot memory — facts table operations
|
||||
// ============================================================
|
||||
|
||||
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* eval-contradictions/auto-supersession — M7 resolution proposal generator.
|
||||
*
|
||||
* For each contradiction finding, classify into a resolution kind and emit
|
||||
* a paste-ready CLI command. The probe NEVER auto-applies; the user runs
|
||||
* the command themselves. The proposal is descriptive, not directive.
|
||||
*
|
||||
* Classification logic (deterministic, no LLM):
|
||||
*
|
||||
* intra_page_chunk_take pair → takes_supersede if the take is newer
|
||||
* (`since_date` or take row vs chunk),
|
||||
* else manual_review.
|
||||
* cross_slug_chunks pair → dream_synthesize if both sides cite the
|
||||
* same canonical-page slug-prefix
|
||||
* (companies/, people/, etc.) and one is
|
||||
* bulk-tier,
|
||||
* → takes_mark_debate if the judge's
|
||||
* resolution_kind hinted that direction
|
||||
* (e.g., two opinion-shaped pairs), else
|
||||
* manual_review.
|
||||
*
|
||||
* The orchestrator may override these with the judge's `resolution_kind`
|
||||
* field when present — the judge has signal we don't.
|
||||
*/
|
||||
|
||||
import type {
|
||||
ContradictionFinding,
|
||||
ContradictionPair,
|
||||
JudgeVerdict,
|
||||
ResolutionKind,
|
||||
} from './types.ts';
|
||||
|
||||
export interface ResolutionProposal {
|
||||
resolution_kind: ResolutionKind;
|
||||
resolution_command: string;
|
||||
}
|
||||
|
||||
const CURATED_ENTITY_PREFIXES = ['companies/', 'people/', 'deals/', 'projects/'];
|
||||
|
||||
function isCuratedEntitySlug(slug: string): boolean {
|
||||
return CURATED_ENTITY_PREFIXES.some((p) => slug.toLowerCase().startsWith(p));
|
||||
}
|
||||
|
||||
/**
|
||||
* Choose a resolution kind for the pair. The judge's hint (when present)
|
||||
* wins for cross_slug pairs because it has semantic context this rule-based
|
||||
* pass doesn't. For intra_page pairs we trust the structural heuristic since
|
||||
* the judge can't see take_id metadata directly.
|
||||
*/
|
||||
export function classifyResolution(
|
||||
pair: ContradictionPair,
|
||||
judgeHint: ResolutionKind | null,
|
||||
): ResolutionKind {
|
||||
if (pair.kind === 'intra_page_chunk_take') {
|
||||
// One side is a take (b, by convention in the runner). If the take is
|
||||
// active and the chunk text is older, supersede makes sense. We default
|
||||
// to takes_supersede; if context is ambiguous the user can pick manual.
|
||||
if (pair.b.take_id !== null) return 'takes_supersede';
|
||||
if (pair.a.take_id !== null) return 'takes_supersede';
|
||||
return 'manual_review';
|
||||
}
|
||||
// cross_slug: judge hint wins if it's specific.
|
||||
if (judgeHint === 'dream_synthesize' || judgeHint === 'takes_mark_debate') {
|
||||
return judgeHint;
|
||||
}
|
||||
if (judgeHint === 'takes_supersede' || judgeHint === 'manual_review') {
|
||||
return judgeHint;
|
||||
}
|
||||
// Structural fallback: if either side is a curated entity page, propose
|
||||
// a synthesize run on the curated slug to reconcile.
|
||||
if (isCuratedEntitySlug(pair.a.slug) || isCuratedEntitySlug(pair.b.slug)) {
|
||||
return 'dream_synthesize';
|
||||
}
|
||||
return 'manual_review';
|
||||
}
|
||||
|
||||
/**
|
||||
* Render the paste-ready CLI command for the chosen resolution. Operator
|
||||
* runs this verbatim; the command may itself prompt for confirmation.
|
||||
*/
|
||||
export function renderResolutionCommand(
|
||||
pair: ContradictionPair,
|
||||
kind: ResolutionKind,
|
||||
): string {
|
||||
switch (kind) {
|
||||
case 'takes_supersede': {
|
||||
// Prefer the slug of the take side (intra_page) or the curated side.
|
||||
const takeSide = pair.b.take_id !== null ? pair.b : (pair.a.take_id !== null ? pair.a : pair.a);
|
||||
const takeId = takeSide.take_id ?? '<row>';
|
||||
return `gbrain takes supersede ${takeSide.slug} --row ${takeId}`;
|
||||
}
|
||||
case 'dream_synthesize': {
|
||||
const curatedSide = isCuratedEntitySlug(pair.a.slug)
|
||||
? pair.a
|
||||
: (isCuratedEntitySlug(pair.b.slug) ? pair.b : pair.a);
|
||||
return `gbrain dream --phase synthesize --slug ${curatedSide.slug}`;
|
||||
}
|
||||
case 'takes_mark_debate': {
|
||||
const takeSide = pair.b.take_id !== null ? pair.b : (pair.a.take_id !== null ? pair.a : pair.a);
|
||||
const takeId = takeSide.take_id ?? '<row>';
|
||||
return `gbrain takes mark-debate ${takeSide.slug} --row ${takeId}`;
|
||||
}
|
||||
case 'manual_review':
|
||||
default:
|
||||
return `# manual review: ${pair.a.slug} vs ${pair.b.slug}`;
|
||||
}
|
||||
}
|
||||
|
||||
/** Convenience: classify + render in one step. */
|
||||
export function proposeResolution(
|
||||
pair: ContradictionPair,
|
||||
judgeHint: ResolutionKind | null,
|
||||
): ResolutionProposal {
|
||||
const kind = classifyResolution(pair, judgeHint);
|
||||
return {
|
||||
resolution_kind: kind,
|
||||
resolution_command: renderResolutionCommand(pair, kind),
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Promote a ContradictionPair + JudgeVerdict to a ContradictionFinding by
|
||||
* filling in severity/axis/confidence + resolution proposal. Used by the
|
||||
* runner aggregation pass.
|
||||
*/
|
||||
export function pairToFinding(
|
||||
pair: ContradictionPair,
|
||||
verdict: JudgeVerdict,
|
||||
): ContradictionFinding {
|
||||
const prop = proposeResolution(pair, verdict.resolution_kind);
|
||||
return {
|
||||
...pair,
|
||||
severity: verdict.severity,
|
||||
axis: verdict.axis,
|
||||
confidence: verdict.confidence,
|
||||
resolution_kind: prop.resolution_kind,
|
||||
resolution_command: prop.resolution_command,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,132 @@
|
||||
/**
|
||||
* eval-contradictions/cache — P2 persistent judge cache wrapper.
|
||||
*
|
||||
* Thin orchestration over the engine's getContradictionCacheEntry +
|
||||
* putContradictionCacheEntry + sweepContradictionCache methods. Owns:
|
||||
* - Stable content hashing (sha256, lower-case hex).
|
||||
* - The cache key shape that includes prompt_version + truncation_policy
|
||||
* (Codex outside-voice fix).
|
||||
* - Order-independence: (a, b) and (b, a) hash to the same key by
|
||||
* sorting the two hashes lexicographically.
|
||||
* - In-process counters for the run's cache hit-rate report.
|
||||
*
|
||||
* The judge's response shape (JudgeVerdict) round-trips through JSONB.
|
||||
* Reads parse the JSONB column back into a typed verdict; writes accept the
|
||||
* verdict object directly (sql.json on Postgres, $N::jsonb on PGLite).
|
||||
*/
|
||||
|
||||
import { createHash } from 'node:crypto';
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { PROMPT_VERSION, TRUNCATION_POLICY } from './types.ts';
|
||||
import type { CacheStats, JudgeVerdict } from './types.ts';
|
||||
|
||||
/** Stable sha256 hex of a string. UTF-8 input. */
|
||||
export function hashContent(text: string): string {
|
||||
return createHash('sha256').update(text, 'utf8').digest('hex');
|
||||
}
|
||||
|
||||
/**
|
||||
* Order-independent cache key: a and b sorted lex so (a, b) and (b, a)
|
||||
* collide. This matters because the orchestrator may emit pairs in either
|
||||
* direction depending on retrieval order; the verdict is symmetric.
|
||||
*/
|
||||
export function buildCacheKey(opts: {
|
||||
textA: string;
|
||||
textB: string;
|
||||
modelId: string;
|
||||
}): {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
} {
|
||||
const hA = hashContent(opts.textA);
|
||||
const hB = hashContent(opts.textB);
|
||||
const [first, second] = hA <= hB ? [hA, hB] : [hB, hA];
|
||||
return {
|
||||
chunk_a_hash: first,
|
||||
chunk_b_hash: second,
|
||||
model_id: opts.modelId,
|
||||
prompt_version: PROMPT_VERSION,
|
||||
truncation_policy: TRUNCATION_POLICY,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Type guard: validates a JSONB blob actually parses to a JudgeVerdict.
|
||||
* Defensive — if the cache row was written under an older prompt_version
|
||||
* but somehow survived a version bump, we want to detect the shape
|
||||
* mismatch and treat it as a miss rather than crash downstream.
|
||||
*/
|
||||
function isJudgeVerdict(raw: unknown): raw is JudgeVerdict {
|
||||
if (!raw || typeof raw !== 'object') return false;
|
||||
const v = raw as Record<string, unknown>;
|
||||
return (
|
||||
typeof v.contradicts === 'boolean' &&
|
||||
typeof v.severity === 'string' &&
|
||||
typeof v.confidence === 'number' &&
|
||||
typeof v.axis === 'string'
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* In-process cache wrapper. One instance per probe run; tracks hits/misses
|
||||
* for the report. Uses the BrainEngine's persistent backing.
|
||||
*/
|
||||
export class JudgeCache {
|
||||
private hits = 0;
|
||||
private misses = 0;
|
||||
private engine: BrainEngine;
|
||||
private modelId: string;
|
||||
private ttlSeconds: number;
|
||||
private disabled: boolean;
|
||||
|
||||
constructor(opts: {
|
||||
engine: BrainEngine;
|
||||
modelId: string;
|
||||
/** Default 30 days. Zero disables persistence (in-memory only via miss-write skip). */
|
||||
ttlSeconds?: number;
|
||||
/** If true, never read or write — every call is a miss. */
|
||||
disabled?: boolean;
|
||||
}) {
|
||||
this.engine = opts.engine;
|
||||
this.modelId = opts.modelId;
|
||||
this.ttlSeconds = opts.ttlSeconds ?? 30 * 86400;
|
||||
this.disabled = !!opts.disabled;
|
||||
}
|
||||
|
||||
async lookup(textA: string, textB: string): Promise<JudgeVerdict | null> {
|
||||
if (this.disabled) {
|
||||
this.misses++;
|
||||
return null;
|
||||
}
|
||||
const key = buildCacheKey({ textA, textB, modelId: this.modelId });
|
||||
const raw = await this.engine.getContradictionCacheEntry(key);
|
||||
if (raw && isJudgeVerdict(raw)) {
|
||||
this.hits++;
|
||||
return raw;
|
||||
}
|
||||
this.misses++;
|
||||
return null;
|
||||
}
|
||||
|
||||
async store(textA: string, textB: string, verdict: JudgeVerdict): Promise<void> {
|
||||
if (this.disabled) return;
|
||||
const key = buildCacheKey({ textA, textB, modelId: this.modelId });
|
||||
await this.engine.putContradictionCacheEntry({
|
||||
...key,
|
||||
verdict: verdict as unknown as Record<string, unknown>,
|
||||
ttl_seconds: this.ttlSeconds,
|
||||
});
|
||||
}
|
||||
|
||||
stats(): CacheStats {
|
||||
const total = this.hits + this.misses;
|
||||
return {
|
||||
hits: this.hits,
|
||||
misses: this.misses,
|
||||
hit_rate: total === 0 ? 0 : this.hits / total,
|
||||
};
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,69 @@
|
||||
/**
|
||||
* eval-contradictions/calibration — Wilson confidence interval on the headline %.
|
||||
*
|
||||
* The probe outputs a fraction like "12/50 queries had a suspected contradiction
|
||||
* (24%)." Without a CI, that single number is overclaimed: 24% on n=50 could be
|
||||
* anywhere from ~14% to ~37% at 95% confidence. Saying "24% with 95% CI 14-37"
|
||||
* is the difference between a defensible measurement and a vibes-based number.
|
||||
*
|
||||
* Why Wilson over normal-approximation: stable at small n and at extreme p
|
||||
* (close to 0 or 1). The normal approximation breaks where we care most.
|
||||
*
|
||||
* n < 30 returns a small_sample_note string so the consumer can disclaim the
|
||||
* bound rather than treat it as actionable.
|
||||
*/
|
||||
|
||||
import type { Calibration, WilsonCI } from './types.ts';
|
||||
|
||||
/** 95% confidence z-score. */
|
||||
const Z_95 = 1.959963984540054;
|
||||
|
||||
/**
|
||||
* Wilson score interval for a binomial proportion at 95% confidence.
|
||||
*
|
||||
* Returns the point estimate (k/n) and lower/upper bounds. Edge cases:
|
||||
* - n === 0: returns all zeros. Caller decides UX.
|
||||
* - k > n: clamps k to n.
|
||||
* - k < 0: clamps to 0.
|
||||
*/
|
||||
export function wilsonCI(numerator: number, denominator: number): WilsonCI {
|
||||
if (denominator <= 0) {
|
||||
return { point: 0, lower: 0, upper: 0 };
|
||||
}
|
||||
const k = Math.max(0, Math.min(numerator, denominator));
|
||||
const n = denominator;
|
||||
const p = k / n;
|
||||
const z = Z_95;
|
||||
const z2 = z * z;
|
||||
const center = (p + z2 / (2 * n)) / (1 + z2 / n);
|
||||
const margin = (z * Math.sqrt((p * (1 - p)) / n + z2 / (4 * n * n))) / (1 + z2 / n);
|
||||
// Pin exact boundaries: when k === 0 the lower bound must be exactly 0;
|
||||
// when k === n the upper bound must be exactly 1. Otherwise floating-point
|
||||
// residuals (6e-18, 0.9999...) leak through and confuse callers.
|
||||
const lowerRaw = Math.max(0, center - margin);
|
||||
const upperRaw = Math.min(1, center + margin);
|
||||
return {
|
||||
point: p,
|
||||
lower: k === 0 ? 0 : lowerRaw,
|
||||
upper: k === n ? 1 : upperRaw,
|
||||
};
|
||||
}
|
||||
|
||||
/** Build the calibration block for the ProbeReport. n < 30 triggers small-sample note. */
|
||||
export function buildCalibration(opts: {
|
||||
queriesTotal: number;
|
||||
queriesWithContradiction: number;
|
||||
}): Calibration {
|
||||
const clean = Math.max(0, opts.queriesTotal - opts.queriesWithContradiction);
|
||||
const ci = wilsonCI(opts.queriesWithContradiction, opts.queriesTotal);
|
||||
const cal: Calibration = {
|
||||
queries_total: opts.queriesTotal,
|
||||
queries_judged_clean: clean,
|
||||
queries_with_contradiction: opts.queriesWithContradiction,
|
||||
wilson_ci_95: ci,
|
||||
};
|
||||
if (opts.queriesTotal < 30) {
|
||||
cal.small_sample_note = `n=${opts.queriesTotal} is below 30; the 95% CI is too wide to act on. Run more queries before drawing conclusions.`;
|
||||
}
|
||||
return cal;
|
||||
}
|
||||
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* eval-contradictions/cost-tracker — A2 + P3 cumulative cost accounting.
|
||||
*
|
||||
* Per the v0.32.6 plan: --budget-usd is a soft ceiling enforced two ways:
|
||||
* 1. Pre-flight estimate. Refuses to start (exit 1) without --yes if the
|
||||
* conservative upper bound exceeds the cap.
|
||||
* 2. Mid-run cumulative tracker. After every judge call, if the running
|
||||
* total exceeds the cap, the orchestrator stops with a partial report.
|
||||
*
|
||||
* Codex correctly flagged that "hard ceiling" is overclaimed since token
|
||||
* estimates are approximate until the provider returns actual usage. The
|
||||
* tracker uses actual post-call accounting from the gateway response; the
|
||||
* pre-flight estimate is a function of declared per-call budgets and pair
|
||||
* counts. Both are documented in the output via `cost_usd.estimate_note`.
|
||||
*
|
||||
* Codex finding P3: include embedding cost so the budget cap is honest. The
|
||||
* probe pays a tiny per-query embedding fee on --query and --queries-file
|
||||
* paths (eval_candidates rows from --from-capture are pre-embedded). Tiny
|
||||
* in absolute dollars but the contract matters.
|
||||
*/
|
||||
|
||||
import type { CostBreakdown } from './types.ts';
|
||||
|
||||
/**
|
||||
* Per-million-token prices (USD). Update when models bump. These are
|
||||
* approximate — provider accounting after the call is authoritative.
|
||||
*/
|
||||
const ANTHROPIC_PRICING: Record<string, { input: number; output: number }> = {
|
||||
// Haiku 4.5: ~$1/Mtok in, $5/Mtok out (current as of 2026-05).
|
||||
'claude-haiku-4-5': { input: 1.0, output: 5.0 },
|
||||
'anthropic:claude-haiku-4-5': { input: 1.0, output: 5.0 },
|
||||
// Sonnet 4.6: ~$3/Mtok in, $15/Mtok out.
|
||||
'claude-sonnet-4-6': { input: 3.0, output: 15.0 },
|
||||
'anthropic:claude-sonnet-4-6': { input: 3.0, output: 15.0 },
|
||||
// Opus 4.7: ~$5/Mtok in, $25/Mtok out.
|
||||
'claude-opus-4-7': { input: 5.0, output: 25.0 },
|
||||
'anthropic:claude-opus-4-7': { input: 5.0, output: 25.0 },
|
||||
};
|
||||
|
||||
/** OpenAI text-embedding-3-large: ~$0.13/Mtok (current as of 2026-05). */
|
||||
const OPENAI_EMBEDDING_PRICE_PER_MTOK = 0.13;
|
||||
|
||||
/** Default per-call token budget for the judge. ~500 in, ~80 out. Tunable. */
|
||||
const DEFAULT_PER_CALL_INPUT_TOKENS = 500;
|
||||
const DEFAULT_PER_CALL_OUTPUT_TOKENS = 80;
|
||||
|
||||
const ESTIMATE_NOTE =
|
||||
'approximate; provider accounting is post-call. --budget-usd is a soft ceiling — mid-run stop on cumulative > cap.';
|
||||
|
||||
function pricingFor(modelId: string): { input: number; output: number } {
|
||||
return ANTHROPIC_PRICING[modelId] ?? ANTHROPIC_PRICING['claude-haiku-4-5'];
|
||||
}
|
||||
|
||||
/**
|
||||
* Conservative upper-bound estimate. Used pre-flight to decide whether to
|
||||
* refuse without --yes. NEVER use this number as "actual cost" — that's
|
||||
* the cumulative tracker's job.
|
||||
*/
|
||||
export function estimateUpperBoundCost(opts: {
|
||||
pairCount: number;
|
||||
queryCount: number;
|
||||
judgeModel: string;
|
||||
perCallInputTokens?: number;
|
||||
perCallOutputTokens?: number;
|
||||
}): number {
|
||||
const judgePricing = pricingFor(opts.judgeModel);
|
||||
const inTok = opts.perCallInputTokens ?? DEFAULT_PER_CALL_INPUT_TOKENS;
|
||||
const outTok = opts.perCallOutputTokens ?? DEFAULT_PER_CALL_OUTPUT_TOKENS;
|
||||
const judgeCost =
|
||||
opts.pairCount * ((inTok / 1_000_000) * judgePricing.input + (outTok / 1_000_000) * judgePricing.output);
|
||||
// Conservative embedding cost: assume ~50 tokens per query.
|
||||
const embedCost = opts.queryCount * (50 / 1_000_000) * OPENAI_EMBEDDING_PRICE_PER_MTOK;
|
||||
return judgeCost + embedCost;
|
||||
}
|
||||
|
||||
/** Mutable accumulator. Use for mid-run tracking + final breakdown. */
|
||||
export class CostTracker {
|
||||
private judgeUsd = 0;
|
||||
private embeddingUsd = 0;
|
||||
private cap: number;
|
||||
|
||||
constructor(opts: { capUsd: number }) {
|
||||
this.cap = Math.max(0, opts.capUsd);
|
||||
}
|
||||
|
||||
recordJudgeCall(modelId: string, usage: { inputTokens: number; outputTokens: number }): void {
|
||||
const p = pricingFor(modelId);
|
||||
this.judgeUsd +=
|
||||
(usage.inputTokens / 1_000_000) * p.input + (usage.outputTokens / 1_000_000) * p.output;
|
||||
}
|
||||
|
||||
recordEmbeddingCall(tokens: number): void {
|
||||
this.embeddingUsd += (tokens / 1_000_000) * OPENAI_EMBEDDING_PRICE_PER_MTOK;
|
||||
}
|
||||
|
||||
judge(): number { return this.judgeUsd; }
|
||||
embedding(): number { return this.embeddingUsd; }
|
||||
total(): number { return this.judgeUsd + this.embeddingUsd; }
|
||||
capUsd(): number { return this.cap; }
|
||||
|
||||
/** Returns true iff cumulative spend exceeds the configured cap. */
|
||||
exceededCap(): boolean {
|
||||
return this.total() > this.cap;
|
||||
}
|
||||
|
||||
/** Final breakdown for the ProbeReport. */
|
||||
finalize(): CostBreakdown {
|
||||
return {
|
||||
judge: round6(this.judgeUsd),
|
||||
embedding: round6(this.embeddingUsd),
|
||||
total: round6(this.total()),
|
||||
estimate_note: ESTIMATE_NOTE,
|
||||
};
|
||||
}
|
||||
}
|
||||
|
||||
function round6(n: number): number {
|
||||
return Math.round(n * 1_000_000) / 1_000_000;
|
||||
}
|
||||
@@ -0,0 +1,65 @@
|
||||
/**
|
||||
* eval-contradictions/cross-source — M6 source-tier breakdown.
|
||||
*
|
||||
* Maps each pair-member's slug to a tier ('curated' | 'bulk' | 'other'),
|
||||
* then counts pairs by tier combination so the probe report can answer
|
||||
* "where do the contradictions live?": between two curated pages (worst —
|
||||
* the canonical narrative is internally inconsistent), between curated and
|
||||
* bulk (the cleanup target), or between two bulk pages (lowest concern).
|
||||
*
|
||||
* Tier classification reuses the existing source-boost map: prefixes with
|
||||
* boost > 1.0 are 'curated', boost < 1.0 are 'bulk', and 1.0 (or unknown)
|
||||
* is 'other'. Longest-prefix-match wins, matching how source-boost.ts
|
||||
* itself classifies during ranking.
|
||||
*/
|
||||
|
||||
import { DEFAULT_SOURCE_BOOSTS } from '../search/source-boost.ts';
|
||||
import type { ContradictionPair, SourceTier, SourceTierBreakdown } from './types.ts';
|
||||
|
||||
/**
|
||||
* Classify a slug into a tier. Longest-prefix-match. Unknown/baseline slugs
|
||||
* map to 'other' (not 'bulk') so the probe doesn't quietly mis-label new
|
||||
* directories.
|
||||
*/
|
||||
export function classifySlugTier(slug: string): SourceTier {
|
||||
if (!slug) return 'other';
|
||||
const lower = slug.toLowerCase();
|
||||
// Match longest prefix first.
|
||||
const prefixes = Object.keys(DEFAULT_SOURCE_BOOSTS).sort((a, b) => b.length - a.length);
|
||||
for (const prefix of prefixes) {
|
||||
if (lower.startsWith(prefix)) {
|
||||
const boost = DEFAULT_SOURCE_BOOSTS[prefix];
|
||||
if (boost > 1.05) return 'curated';
|
||||
if (boost < 0.95) return 'bulk';
|
||||
return 'other';
|
||||
}
|
||||
}
|
||||
return 'other';
|
||||
}
|
||||
|
||||
function bucketKey(a: SourceTier, b: SourceTier): keyof SourceTierBreakdown {
|
||||
// Order-independent: 'curated' beats 'bulk' beats 'other' for the cross-tier label.
|
||||
const has = (t: SourceTier) => a === t || b === t;
|
||||
if (a === 'curated' && b === 'curated') return 'curated_vs_curated';
|
||||
if (a === 'bulk' && b === 'bulk') return 'bulk_vs_bulk';
|
||||
if (has('curated') && has('bulk')) return 'curated_vs_bulk';
|
||||
return 'other';
|
||||
}
|
||||
|
||||
/** Build the breakdown across a set of pairs. */
|
||||
export function buildSourceTierBreakdown(
|
||||
pairs: readonly ContradictionPair[],
|
||||
): SourceTierBreakdown {
|
||||
const out: SourceTierBreakdown = {
|
||||
curated_vs_curated: 0,
|
||||
curated_vs_bulk: 0,
|
||||
bulk_vs_bulk: 0,
|
||||
other: 0,
|
||||
};
|
||||
for (const pair of pairs) {
|
||||
const tierA = classifySlugTier(pair.a.slug);
|
||||
const tierB = classifySlugTier(pair.b.slug);
|
||||
out[bucketKey(tierA, tierB)]++;
|
||||
}
|
||||
return out;
|
||||
}
|
||||
@@ -0,0 +1,157 @@
|
||||
/**
|
||||
* eval-contradictions/date-filter — A1 three-rule date pre-filter.
|
||||
*
|
||||
* Goal: skip the obvious quarterly-update case (Acme MRR $50K in 2024 vs
|
||||
* Acme MRR $2M in 2026) before paying for an LLM judge call. Without this,
|
||||
* timeline-shaped content dominates judge calls and inflates cost on a
|
||||
* brain with lots of /daily/, /meetings/, or quarterly snapshots.
|
||||
*
|
||||
* Codex flagged the naive rule ("both have dates AND dates differ → skip")
|
||||
* as too blunt: "Alice is CFO in Jan / Alice is not CFO in Mar" is a real
|
||||
* contradiction-or-update that the pre-filter must NOT silently kill. So
|
||||
* the rules layer:
|
||||
*
|
||||
* 1. BOTH chunks contain explicit YYYY-like dates AND the dates differ
|
||||
* by more than DATE_SEPARATION_DAYS → SKIP (the obvious case).
|
||||
* 2. EITHER chunk lacks an explicit date → DO NOT skip; let judge decide.
|
||||
* 3. SAME paragraph in either chunk contains two distinct dates → DO NOT
|
||||
* skip; this is the flip-flop case ("in Jan I said X, in Mar I said not-X"
|
||||
* written in a single paragraph). Judge sees it.
|
||||
*
|
||||
* The detector is intentionally conservative. False negatives (NOT skipping
|
||||
* when we could have) cost LLM tokens. False positives (skipping a real
|
||||
* contradiction) cost the whole point of the probe. Errs toward FN.
|
||||
*/
|
||||
|
||||
const DATE_SEPARATION_DAYS = 30;
|
||||
|
||||
/**
|
||||
* Permissive date matcher. Recognized shapes (priority order):
|
||||
* YYYY-MM-DD → groups 1,2,3
|
||||
* YYYY/MM/DD → groups 4,5,6
|
||||
* Mon DD YYYY (e.g. Jan 15 2024) → groups 7 (month), 8 (day), 9 (year)
|
||||
* Mon YYYY (e.g. Jan 2024) → groups 10 (month), 11 (year)
|
||||
* Q1-4 YYYY → group 12
|
||||
* bare YYYY → group 13
|
||||
*
|
||||
* The Mon-DD-YYYY alternative must come BEFORE Mon-YYYY so we capture the day
|
||||
* when it's present. Bare YYYY comes last to avoid stealing the year from a
|
||||
* fuller pattern.
|
||||
*/
|
||||
const DATE_REGEX =
|
||||
/\b(?:(\d{4})-(\d{2})-(\d{2})|(\d{4})\/(\d{2})\/(\d{2})|(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\w*\.?\s+(\d{1,2}),?\s+(\d{4})|(Jan|Feb|Mar|Apr|May|Jun|Jul|Aug|Sep|Oct|Nov|Dec)\w*\.?\s+(\d{4})|Q[1-4]\s+(\d{4})|(\d{4}))\b/g;
|
||||
|
||||
const MONTH_INDEX: Record<string, number> = {
|
||||
Jan: 0, Feb: 1, Mar: 2, Apr: 3, May: 4, Jun: 5,
|
||||
Jul: 6, Aug: 7, Sep: 8, Oct: 9, Nov: 10, Dec: 11,
|
||||
};
|
||||
|
||||
export interface DateFilterDecision {
|
||||
skip: boolean;
|
||||
reason:
|
||||
| 'both_explicit_separated'
|
||||
| 'one_or_both_missing_dates'
|
||||
| 'same_paragraph_dual_date'
|
||||
| 'overlapping_or_close';
|
||||
}
|
||||
|
||||
export interface DateFilterInput {
|
||||
textA: string;
|
||||
textB: string;
|
||||
}
|
||||
|
||||
/** Extract date tokens from text. Returns parsed Date objects (UTC midnight). */
|
||||
export function extractDates(text: string): Date[] {
|
||||
if (!text) return [];
|
||||
const dates: Date[] = [];
|
||||
// Reset lastIndex because the regex has the /g flag and is reused across calls.
|
||||
DATE_REGEX.lastIndex = 0;
|
||||
let m: RegExpExecArray | null;
|
||||
while ((m = DATE_REGEX.exec(text)) !== null) {
|
||||
const parsed = parseDateMatch(m);
|
||||
if (parsed) dates.push(parsed);
|
||||
}
|
||||
return dates;
|
||||
}
|
||||
|
||||
function parseDateMatch(m: RegExpExecArray): Date | null {
|
||||
let year: number;
|
||||
let month = 0;
|
||||
let day = 1;
|
||||
if (m[1] && m[2] && m[3]) {
|
||||
// YYYY-MM-DD
|
||||
year = +m[1]; month = +m[2] - 1; day = +m[3];
|
||||
} else if (m[4] && m[5] && m[6]) {
|
||||
// YYYY/MM/DD
|
||||
year = +m[4]; month = +m[5] - 1; day = +m[6];
|
||||
} else if (m[7] && m[8] && m[9]) {
|
||||
// Mon DD YYYY
|
||||
year = +m[9]; month = MONTH_INDEX[m[7]] ?? 0; day = +m[8];
|
||||
} else if (m[10] && m[11]) {
|
||||
// Mon YYYY (no day) — assume first of month
|
||||
year = +m[11]; month = MONTH_INDEX[m[10]] ?? 0; day = 1;
|
||||
} else if (m[12]) {
|
||||
// Q1-4 YYYY
|
||||
year = +m[12];
|
||||
} else if (m[13]) {
|
||||
// bare YYYY
|
||||
year = +m[13];
|
||||
} else {
|
||||
return null;
|
||||
}
|
||||
if (year < 1900 || year > 2100) return null;
|
||||
return new Date(Date.UTC(year, month, day));
|
||||
}
|
||||
|
||||
/**
|
||||
* Does any paragraph in the text contain two distinct dates? "Distinct" means
|
||||
* dates whose UTC-day differs. A paragraph is split on blank lines.
|
||||
*/
|
||||
export function hasSameParagraphDualDate(text: string): boolean {
|
||||
if (!text) return false;
|
||||
const paragraphs = text.split(/\n\s*\n/);
|
||||
for (const p of paragraphs) {
|
||||
const dates = extractDates(p);
|
||||
if (dates.length < 2) continue;
|
||||
const days = new Set(dates.map((d) => Math.floor(d.getTime() / 86400000)));
|
||||
if (days.size >= 2) return true;
|
||||
}
|
||||
return false;
|
||||
}
|
||||
|
||||
/**
|
||||
* Return the absolute difference in days between the latest date in A
|
||||
* and the latest date in B. Returns Infinity if either side has no dates.
|
||||
*/
|
||||
function maxDateSeparationDays(a: Date[], b: Date[]): number {
|
||||
if (a.length === 0 || b.length === 0) return Infinity;
|
||||
const maxA = Math.max(...a.map((d) => d.getTime()));
|
||||
const maxB = Math.max(...b.map((d) => d.getTime()));
|
||||
return Math.abs(maxA - maxB) / 86400000;
|
||||
}
|
||||
|
||||
/**
|
||||
* The decision function called by the runner. Returns `{ skip, reason }`.
|
||||
*
|
||||
* Order matters: same-paragraph dual-date wins over the separation rule
|
||||
* because flip-flops are exactly what we DO want the judge to see, even
|
||||
* when other dates in the chunks are far apart.
|
||||
*/
|
||||
export function shouldSkipForDateMismatch(input: DateFilterInput): DateFilterDecision {
|
||||
if (
|
||||
hasSameParagraphDualDate(input.textA) ||
|
||||
hasSameParagraphDualDate(input.textB)
|
||||
) {
|
||||
return { skip: false, reason: 'same_paragraph_dual_date' };
|
||||
}
|
||||
const datesA = extractDates(input.textA);
|
||||
const datesB = extractDates(input.textB);
|
||||
if (datesA.length === 0 || datesB.length === 0) {
|
||||
return { skip: false, reason: 'one_or_both_missing_dates' };
|
||||
}
|
||||
const sep = maxDateSeparationDays(datesA, datesB);
|
||||
if (sep > DATE_SEPARATION_DAYS) {
|
||||
return { skip: true, reason: 'both_explicit_separated' };
|
||||
}
|
||||
return { skip: false, reason: 'overlapping_or_close' };
|
||||
}
|
||||
@@ -0,0 +1,169 @@
|
||||
/**
|
||||
* eval-contradictions/fixture-redact — T2 privacy redaction for gold fixture build.
|
||||
*
|
||||
* The fixture build script runs against the user's real brain to label
|
||||
* candidate contradiction pairs. Before commit, names + identifiers MUST be
|
||||
* scrubbed per CLAUDE.md privacy rule: "Never reference real people, companies,
|
||||
* funds, or private agent names in any public-facing artifact."
|
||||
*
|
||||
* Pass model (deterministic per session via a salt):
|
||||
* 1. PII via the v0.25.0 scrubber (emails, phones, SSN, JWT, credit cards).
|
||||
* 2. Slug rewrites: people/<name> → people/alice-example, companies/<name>
|
||||
* → companies/acme-example, deals/<name> → deals/acme-seed-example, etc.
|
||||
* Stable mapping within a session so the same name maps consistently.
|
||||
* 3. Quoted-name detection: capitalized firstname-lastname patterns.
|
||||
* 4. Numeric obfuscation: revenue / funding figures multiplied by a salt
|
||||
* scalar (preserves order-of-magnitude shape).
|
||||
*
|
||||
* Fail-closed: if the redactor can't determine a clean rewrite for a token
|
||||
* it flagged as potentially private, it emits a sentinel string `[REDACT?]`
|
||||
* that the operator must resolve before commit. The build script's
|
||||
* pre-commit review surfaces every redaction made.
|
||||
*/
|
||||
|
||||
import { scrubPii } from '../eval-capture-scrub.ts';
|
||||
|
||||
const SLUG_PREFIX_REWRITES: Record<string, string> = {
|
||||
'people/': 'people/',
|
||||
'companies/': 'companies/',
|
||||
'deals/': 'deals/',
|
||||
'projects/': 'projects/',
|
||||
'meetings/': 'meetings/',
|
||||
};
|
||||
|
||||
const PLACEHOLDER_POOL: Record<string, string[]> = {
|
||||
'people/': ['alice', 'bob', 'charlie', 'diana', 'eve', 'frank', 'grace', 'hank'],
|
||||
'companies/': ['acme', 'widget-co', 'globex', 'initech', 'piedpiper', 'hooli', 'pinnacle'],
|
||||
'deals/': ['acme-seed', 'widget-series-a', 'globex-series-b', 'initech-seed'],
|
||||
'projects/': ['project-alpha', 'project-beta', 'project-gamma'],
|
||||
'meetings/': [], // dates are kept; meeting-id segment redacted to numeric.
|
||||
};
|
||||
|
||||
/** First+last name detector. Two-word capitalized run, length 2..40. */
|
||||
const QUOTED_NAME_REGEX = /\b([A-Z][a-z]{1,19})\s+([A-Z][a-z]{1,19})\b/g;
|
||||
|
||||
/** Revenue / funding numeric tokens (e.g., $50K, $2M MRR, $1.2B). */
|
||||
const MONETARY_REGEX = /\$\s*(\d+(?:\.\d+)?)\s*([KMB])\b/gi;
|
||||
|
||||
export interface RedactionSession {
|
||||
/** Per-session deterministic mapping: raw slug-suffix → placeholder. */
|
||||
slugMap: Map<string, string>;
|
||||
/** Quoted-name mapping (lower-case full name → "Firstname Lastname-example"). */
|
||||
nameMap: Map<string, string>;
|
||||
/** Pool offset per prefix so we cycle through placeholders. */
|
||||
poolOffset: Record<string, number>;
|
||||
/** Salt for the numeric obfuscation; deterministic per session. */
|
||||
numericSalt: number;
|
||||
/** Audit trail of every redaction performed (for pre-commit review). */
|
||||
audit: string[];
|
||||
}
|
||||
|
||||
export function createRedactionSession(): RedactionSession {
|
||||
return {
|
||||
slugMap: new Map(),
|
||||
nameMap: new Map(),
|
||||
poolOffset: {
|
||||
'people/': 0,
|
||||
'companies/': 0,
|
||||
'deals/': 0,
|
||||
'projects/': 0,
|
||||
'meetings/': 0,
|
||||
},
|
||||
numericSalt: 1.7, // multiply revenues by 1.7 to obscure (deterministic).
|
||||
audit: [],
|
||||
};
|
||||
}
|
||||
|
||||
/** Allocate a placeholder for an unmapped slug. */
|
||||
function allocatePlaceholder(session: RedactionSession, prefix: string, raw: string): string {
|
||||
if (session.slugMap.has(raw)) return session.slugMap.get(raw)!;
|
||||
const pool = PLACEHOLDER_POOL[prefix] ?? [];
|
||||
if (pool.length === 0) {
|
||||
const next = `${prefix}redacted-${session.slugMap.size + 1}`;
|
||||
session.slugMap.set(raw, next);
|
||||
return next;
|
||||
}
|
||||
const idx = session.poolOffset[prefix] % pool.length;
|
||||
session.poolOffset[prefix] = (session.poolOffset[prefix] ?? 0) + 1;
|
||||
const placeholder = `${prefix}${pool[idx]}-example`;
|
||||
session.slugMap.set(raw, placeholder);
|
||||
return placeholder;
|
||||
}
|
||||
|
||||
/** Rewrite a single slug, mapping its tail to a placeholder. Idempotent per session. */
|
||||
export function redactSlug(session: RedactionSession, slug: string): string {
|
||||
for (const prefix of Object.keys(SLUG_PREFIX_REWRITES)) {
|
||||
if (slug.startsWith(prefix)) {
|
||||
const placeholder = allocatePlaceholder(session, prefix, slug);
|
||||
if (placeholder !== slug) {
|
||||
session.audit.push(`slug: ${slug} → ${placeholder}`);
|
||||
}
|
||||
return placeholder;
|
||||
}
|
||||
}
|
||||
return slug;
|
||||
}
|
||||
|
||||
/** Allocate a quoted-name placeholder (per-session deterministic). */
|
||||
function allocateNamePlaceholder(session: RedactionSession, lowerName: string): string {
|
||||
if (session.nameMap.has(lowerName)) return session.nameMap.get(lowerName)!;
|
||||
const peopleCount = session.poolOffset['people/'] ?? 0;
|
||||
const pool = PLACEHOLDER_POOL['people/'];
|
||||
const first = pool[peopleCount % pool.length];
|
||||
// Bump the people offset so name + slug pools stay in sync visually.
|
||||
session.poolOffset['people/'] = peopleCount + 1;
|
||||
const placeholder = `${first.charAt(0).toUpperCase()}${first.slice(1)} Example`;
|
||||
session.nameMap.set(lowerName, placeholder);
|
||||
return placeholder;
|
||||
}
|
||||
|
||||
/** Replace quoted Firstname Lastname runs in a string. */
|
||||
export function redactNames(session: RedactionSession, text: string): string {
|
||||
if (!text) return text;
|
||||
return text.replace(QUOTED_NAME_REGEX, (match, first: string, last: string) => {
|
||||
const key = `${first} ${last}`.toLowerCase();
|
||||
const placeholder = allocateNamePlaceholder(session, key);
|
||||
if (placeholder !== match) {
|
||||
session.audit.push(`name: "${match}" → "${placeholder}"`);
|
||||
}
|
||||
return placeholder;
|
||||
});
|
||||
}
|
||||
|
||||
/** Replace $50K / $2M / $1.2B style tokens by multiplying by the session salt. */
|
||||
export function redactMonetary(session: RedactionSession, text: string): string {
|
||||
if (!text) return text;
|
||||
return text.replace(MONETARY_REGEX, (match, value: string, suffix: string) => {
|
||||
const v = parseFloat(value);
|
||||
if (!Number.isFinite(v)) return match;
|
||||
const obfuscated = (v * session.numericSalt).toFixed(1).replace(/\.0$/, '');
|
||||
const out = `$${obfuscated}${suffix.toUpperCase()}`;
|
||||
session.audit.push(`monetary: ${match} → ${out}`);
|
||||
return out;
|
||||
});
|
||||
}
|
||||
|
||||
/** Full redaction pass for an arbitrary text payload. PII first, then names, then monetary. */
|
||||
export function redactText(session: RedactionSession, text: string): string {
|
||||
let out = scrubPii(text);
|
||||
out = redactNames(session, out);
|
||||
out = redactMonetary(session, out);
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Pre-commit safety: returns true iff redacted text contains no obviously
|
||||
* sensitive tokens. Conservative — only flags shape-matches the session
|
||||
* itself didn't authorize. Returning false should block commit until the
|
||||
* operator resolves the flagged token.
|
||||
*/
|
||||
export function isCleanForCommit(text: string): boolean {
|
||||
// Match: capitalized two-word names that haven't been rewritten (no
|
||||
// " Example" suffix), JWT-shaped tokens, raw email patterns. These should
|
||||
// all have been caught upstream.
|
||||
const looksLikeRawName = /\b[A-Z][a-z]{2,19}\s+[A-Z][a-z]{2,19}\b/.test(text);
|
||||
if (looksLikeRawName && !text.includes(' Example')) return false;
|
||||
const looksLikeEmail = /[\w.+-]+@[\w-]+\.[\w.-]+/.test(text);
|
||||
if (looksLikeEmail) return false;
|
||||
return true;
|
||||
}
|
||||
@@ -0,0 +1,73 @@
|
||||
/**
|
||||
* eval-contradictions/judge-errors — first-class judge error collection.
|
||||
*
|
||||
* Codex caught a real bug: per-pair skip on judge throws (the C2 decision)
|
||||
* biases the headline number downward IF errors cluster around messy
|
||||
* contradiction-like pairs. The fix is to count errors in the denominator
|
||||
* and surface them with a typed reason in the output, not bury them in stderr.
|
||||
*
|
||||
* The `note` field on the counts block is for the human reader of the JSON.
|
||||
* It says explicitly that errors are counted, not hidden.
|
||||
*/
|
||||
|
||||
import type { JudgeErrorKind, JudgeErrorRow, JudgeErrorsCounts } from './types.ts';
|
||||
|
||||
const ERROR_NOTE =
|
||||
'errors counted toward denominator; do not silently disappear from the report';
|
||||
|
||||
/** Classify a thrown error into one of the typed kinds. Conservative; defaults to 'unknown'. */
|
||||
export function classifyError(err: unknown): JudgeErrorKind {
|
||||
if (!err || typeof err !== 'object') return 'unknown';
|
||||
const msg = (err as Error).message?.toLowerCase?.() ?? '';
|
||||
if (msg.includes('parse') || msg.includes('json') || msg.includes('repair')) {
|
||||
return 'parse_fail';
|
||||
}
|
||||
if (msg.includes('refus') || msg.includes("can't help") || msg.includes('cannot help')) {
|
||||
return 'refusal';
|
||||
}
|
||||
if (msg.includes('timeout') || msg.includes('timed out') || msg.includes('aborted')) {
|
||||
return 'timeout';
|
||||
}
|
||||
if (
|
||||
msg.includes('500') ||
|
||||
msg.includes('502') ||
|
||||
msg.includes('503') ||
|
||||
msg.includes('504') ||
|
||||
msg.includes('overload')
|
||||
) {
|
||||
return 'http_5xx';
|
||||
}
|
||||
return 'unknown';
|
||||
}
|
||||
|
||||
/** Mutable collector. Calling code pushes rows; finalize() returns the counts block. */
|
||||
export class JudgeErrorCollector {
|
||||
private rows: JudgeErrorRow[] = [];
|
||||
|
||||
record(pairId: string, err: unknown): void {
|
||||
const kind = classifyError(err);
|
||||
const reason = err instanceof Error ? err.message : String(err);
|
||||
this.rows.push({ kind, pair_id: pairId, reason });
|
||||
}
|
||||
|
||||
rowsOut(): readonly JudgeErrorRow[] {
|
||||
return this.rows;
|
||||
}
|
||||
|
||||
finalize(): JudgeErrorsCounts {
|
||||
const counts: JudgeErrorsCounts = {
|
||||
parse_fail: 0,
|
||||
refusal: 0,
|
||||
timeout: 0,
|
||||
http_5xx: 0,
|
||||
unknown: 0,
|
||||
total: 0,
|
||||
note: ERROR_NOTE,
|
||||
};
|
||||
for (const row of this.rows) {
|
||||
counts[row.kind]++;
|
||||
counts.total++;
|
||||
}
|
||||
return counts;
|
||||
}
|
||||
}
|
||||
@@ -0,0 +1,292 @@
|
||||
/**
|
||||
* eval-contradictions/judge — the LLM contradiction judge wrapper.
|
||||
*
|
||||
* One-call, one-pair: send Statement A + Statement B + the user's query to
|
||||
* the chat gateway and parse the verdict JSON. The prompt is the canonical
|
||||
* text bumped via PROMPT_VERSION when edits land.
|
||||
*
|
||||
* Codex fixes incorporated:
|
||||
* - Query-conditioned: the judge sees the user's query so it can decide
|
||||
* "contradiction relevant to what was asked" instead of free-form pair
|
||||
* disagreement (Codex outside-voice finding).
|
||||
* - Confidence floor double-enforcement (C1): if the model says
|
||||
* contradicts: true with confidence < 0.7, the orchestrator downgrades
|
||||
* to false. Belt-and-suspenders against models that ignore the prompt.
|
||||
* - judge_errors as first-class: throws are typed and counted in the
|
||||
* denominator — see judge-errors.ts for the collector shape.
|
||||
*
|
||||
* Provider-neutral via the gateway. Hermetically testable via
|
||||
* gateway.__setChatTransportForTests.
|
||||
*/
|
||||
|
||||
import { chat, type ChatResult } from '../ai/gateway.ts';
|
||||
import { parseSeverity } from './severity-classify.ts';
|
||||
import type { JudgeVerdict, ResolutionKind } from './types.ts';
|
||||
|
||||
const FENCE_RE = /```(?:json)?\s*\n?([\s\S]*?)```/i;
|
||||
|
||||
/**
|
||||
* Generic 3-strategy LLM JSON parser. Throws when no strategy works rather
|
||||
* than fabricating an empty object — caller maps to judge_errors.parse_fail.
|
||||
*
|
||||
* (We don't reuse parseModelJSON from cross-modal-eval because that one is
|
||||
* shape-specific to {scores, overall, improvements} and rejects our verdict
|
||||
* payload. Same 4-strategy spirit, narrower contract.)
|
||||
*/
|
||||
export function parseJudgeJSON(text: string): unknown {
|
||||
if (!text) throw new Error('parseJudgeJSON: empty response');
|
||||
// Strategy 1: direct parse (strict JSON).
|
||||
try {
|
||||
return JSON.parse(text);
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
// Strategy 2: strip ```json fences.
|
||||
const fenceMatch = text.match(FENCE_RE);
|
||||
if (fenceMatch && fenceMatch[1]) {
|
||||
try {
|
||||
return JSON.parse(fenceMatch[1].trim());
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
}
|
||||
// Strategy 3: common-repairs pass — trailing commas, single→double quotes.
|
||||
const cleaned = text
|
||||
.replace(FENCE_RE, (_, inner) => inner)
|
||||
.replace(/,(\s*[}\]])/g, '$1')
|
||||
.replace(/(['"])?([\w-]+)\1?\s*:/g, '"$2":')
|
||||
.trim();
|
||||
// Extract the first {...} block if there's surrounding prose.
|
||||
const braceMatch = cleaned.match(/\{[\s\S]*\}/);
|
||||
if (braceMatch) {
|
||||
try {
|
||||
return JSON.parse(braceMatch[0]);
|
||||
} catch {
|
||||
// fall through
|
||||
}
|
||||
}
|
||||
throw new Error('parseJudgeJSON: all strategies failed');
|
||||
}
|
||||
|
||||
/** Default per-pair text budget (UTF-8-safe truncation). C4 default. */
|
||||
export const DEFAULT_MAX_PAIR_CHARS = 1500;
|
||||
|
||||
/**
|
||||
* UTF-8-safe truncation: cap at maxChars but never split a multi-byte
|
||||
* character. Returns the text unchanged if already under the limit.
|
||||
*
|
||||
* Pattern reused from src/core/minions/handlers/subagent-audit.ts which
|
||||
* faces the same multi-byte concern.
|
||||
*/
|
||||
export function truncateUtf8(text: string, maxChars: number): string {
|
||||
if (!text) return '';
|
||||
if (text.length <= maxChars) return text;
|
||||
// Walk back from maxChars to land at a complete code-point boundary.
|
||||
// UTF-16 surrogate pairs occupy two code units; if maxChars lands inside
|
||||
// one, drop both halves so we don't keep half an emoji.
|
||||
let end = maxChars;
|
||||
if (end > 0 && end < text.length) {
|
||||
const unitAtEnd = text.charCodeAt(end);
|
||||
const unitBefore = text.charCodeAt(end - 1);
|
||||
const isHighSurrogate = (c: number) => c >= 0xd800 && c <= 0xdbff;
|
||||
const isLowSurrogate = (c: number) => c >= 0xdc00 && c <= 0xdfff;
|
||||
// Case 1: about to split between high(end-1) and low(end) — drop both.
|
||||
if (isHighSurrogate(unitBefore) && isLowSurrogate(unitAtEnd)) {
|
||||
end -= 1;
|
||||
} else if (isHighSurrogate(unitBefore)) {
|
||||
// Stray high surrogate at end — drop it.
|
||||
end -= 1;
|
||||
} else if (isLowSurrogate(unitBefore)) {
|
||||
// We're inside an emoji and end-1 is the low surrogate; back up to
|
||||
// BEFORE the high surrogate (drop both halves).
|
||||
end -= 2;
|
||||
}
|
||||
}
|
||||
return text.slice(0, Math.max(0, end));
|
||||
}
|
||||
|
||||
export interface JudgeInput {
|
||||
/** The user's query for the search that retrieved both members. */
|
||||
query: string;
|
||||
/** Statement A: slug + text + optional source-tier + holder (if take). */
|
||||
a: { slug: string; text: string; source_tier?: string; holder?: string | null };
|
||||
b: { slug: string; text: string; source_tier?: string; holder?: string | null };
|
||||
/** Provider:model id; routed through gateway.chat. */
|
||||
model: string;
|
||||
/** UTF-8-safe truncation limit per pair member. C4 flag. */
|
||||
maxPairChars?: number;
|
||||
/** Test hook: pass a stubbed chat for hermetic tests. Production passes undefined → real gateway. */
|
||||
chatFn?: typeof chat;
|
||||
/** Abort signal for cancellation. */
|
||||
abortSignal?: AbortSignal;
|
||||
}
|
||||
|
||||
export interface JudgeOutput {
|
||||
verdict: JudgeVerdict;
|
||||
/** Token usage from the gateway. Forwarded to the cost tracker. */
|
||||
usage: { inputTokens: number; outputTokens: number };
|
||||
}
|
||||
|
||||
/**
|
||||
* Validated resolution_kind values. Anything outside this set defaults to
|
||||
* 'manual_review' (the safe, no-action option).
|
||||
*/
|
||||
function parseResolutionKind(value: unknown): ResolutionKind | null {
|
||||
if (
|
||||
value === 'takes_supersede' ||
|
||||
value === 'dream_synthesize' ||
|
||||
value === 'takes_mark_debate' ||
|
||||
value === 'manual_review'
|
||||
) {
|
||||
return value;
|
||||
}
|
||||
return null;
|
||||
}
|
||||
|
||||
/**
|
||||
* Validate the raw parsed JSON against the JudgeVerdict shape. Throws on
|
||||
* fundamentally-broken shape (missing contradicts/confidence) so the caller
|
||||
* counts it under judge_errors.parse_fail rather than fabricating a verdict.
|
||||
*
|
||||
* C1 enforcement: contradicts:true with confidence < 0.7 is downgraded to
|
||||
* false (belt-and-suspenders against models ignoring the prompt rule).
|
||||
*/
|
||||
export function normalizeVerdict(raw: unknown): JudgeVerdict {
|
||||
if (!raw || typeof raw !== 'object') {
|
||||
throw new Error('judge JSON missing or not an object');
|
||||
}
|
||||
const v = raw as Record<string, unknown>;
|
||||
const rawContradicts = v.contradicts;
|
||||
if (typeof rawContradicts !== 'boolean') {
|
||||
throw new Error('judge JSON missing required field: contradicts');
|
||||
}
|
||||
const rawConfidence = v.confidence;
|
||||
if (typeof rawConfidence !== 'number' || !Number.isFinite(rawConfidence)) {
|
||||
throw new Error('judge JSON missing or invalid confidence');
|
||||
}
|
||||
const clampedConfidence = Math.min(1, Math.max(0, rawConfidence));
|
||||
const severity = parseSeverity(v.severity);
|
||||
const axisRaw = typeof v.axis === 'string' ? v.axis : '';
|
||||
const resolutionKind = parseResolutionKind(v.resolution_kind);
|
||||
|
||||
// C1 double-enforce: contradicts:true requires confidence >= 0.7.
|
||||
let contradicts = rawContradicts;
|
||||
if (contradicts && clampedConfidence < 0.7) {
|
||||
contradicts = false;
|
||||
}
|
||||
|
||||
return {
|
||||
contradicts,
|
||||
severity,
|
||||
axis: contradicts ? axisRaw : '',
|
||||
confidence: clampedConfidence,
|
||||
resolution_kind: contradicts ? (resolutionKind ?? 'manual_review') : null,
|
||||
};
|
||||
}
|
||||
|
||||
/**
|
||||
* Build the judge prompt. Query-conditioned (Codex fix) — the model sees
|
||||
* what the user actually asked so it can decide whether the disagreement is
|
||||
* relevant to the query.
|
||||
*
|
||||
* Holder is shown when present (take pairs): "Garry holds X" vs "Garry
|
||||
* holds not-X" is a flip; "Alice holds X" vs "Bob holds not-X" is not.
|
||||
*/
|
||||
export function buildJudgePrompt(opts: {
|
||||
query: string;
|
||||
a: { slug: string; text: string; source_tier?: string; holder?: string | null };
|
||||
b: { slug: string; text: string; source_tier?: string; holder?: string | null };
|
||||
maxPairChars: number;
|
||||
}): string {
|
||||
const a = truncateUtf8(opts.a.text, opts.maxPairChars);
|
||||
const b = truncateUtf8(opts.b.text, opts.maxPairChars);
|
||||
const aMeta = [opts.a.slug, opts.a.source_tier && `source-tier ${opts.a.source_tier}`, opts.a.holder && `holder ${opts.a.holder}`].filter(Boolean).join(', ');
|
||||
const bMeta = [opts.b.slug, opts.b.source_tier && `source-tier ${opts.b.source_tier}`, opts.b.holder && `holder ${opts.b.holder}`].filter(Boolean).join(', ');
|
||||
return [
|
||||
'You are a contradiction judge for a personal knowledge brain. The user',
|
||||
'ran a search and got two results back. Decide whether the two statements',
|
||||
"contradict each other in a way that would mislead someone trying to",
|
||||
"answer the user's query.",
|
||||
'',
|
||||
`User's query: ${opts.query}`,
|
||||
'',
|
||||
`Statement A (${aMeta}):`,
|
||||
a,
|
||||
'',
|
||||
`Statement B (${bMeta}):`,
|
||||
b,
|
||||
'',
|
||||
'Rules:',
|
||||
'- Different timeframes for the same dynamic property are NOT contradictions',
|
||||
' (e.g., MRR was $50K in 2024 vs $2M in 2026 — both true at their time).',
|
||||
'- Different timeframes for a static identity claim MAY BE a contradiction',
|
||||
' (e.g., "Alice is CFO of Acme" vs "Alice left Acme" if dates suggest one',
|
||||
' supersedes the other).',
|
||||
'- Subjective opinions held at different times by the SAME holder may be',
|
||||
' a contradiction (a flip). Opinions held by DIFFERENT holders are not.',
|
||||
'- Different aspects of the same entity are NOT contradictions.',
|
||||
"- Incidental disagreements unrelated to the user's query do NOT count.",
|
||||
' Judge only on claims relevant to what the user asked.',
|
||||
'',
|
||||
'Reply with JSON ONLY:',
|
||||
'{',
|
||||
' "contradicts": true | false,',
|
||||
' "severity": "low" | "medium" | "high",',
|
||||
' "axis": "<one-line: what they disagree about, or empty>",',
|
||||
' "confidence": 0.0..1.0,',
|
||||
' "resolution_kind": "takes_supersede" | "dream_synthesize" | "takes_mark_debate" | "manual_review" | null',
|
||||
'}',
|
||||
'',
|
||||
'Severity rubric:',
|
||||
'- low: naming/format differences (Alice Smith vs A. Smith).',
|
||||
'- medium: factual values that may be stale (revenue, headcount).',
|
||||
'- high: identity / structural claims (founder/CEO/CFO role, status).',
|
||||
'',
|
||||
'Reply contradicts:true only when confidence >= 0.7.',
|
||||
].join('\n');
|
||||
}
|
||||
|
||||
/** Detect refusal-shaped responses. Caller maps to judge_errors.refusal. */
|
||||
function isRefusalResponse(result: ChatResult): boolean {
|
||||
if (result.stopReason === 'refusal') return true;
|
||||
const txt = result.text?.toLowerCase?.() ?? '';
|
||||
return (
|
||||
txt.includes("i can't help") ||
|
||||
txt.includes('i cannot help') ||
|
||||
txt.includes('refuse to answer')
|
||||
);
|
||||
}
|
||||
|
||||
/**
|
||||
* Main entry. Calls the gateway, parses JSON, normalizes the verdict with
|
||||
* C1 confidence enforcement. Throws on parse / refusal / transport errors;
|
||||
* caller wraps in try/catch and records via JudgeErrorCollector.
|
||||
*/
|
||||
export async function judgeContradiction(input: JudgeInput): Promise<JudgeOutput> {
|
||||
const maxPairChars = input.maxPairChars ?? DEFAULT_MAX_PAIR_CHARS;
|
||||
const prompt = buildJudgePrompt({
|
||||
query: input.query,
|
||||
a: input.a,
|
||||
b: input.b,
|
||||
maxPairChars,
|
||||
});
|
||||
const callFn = input.chatFn ?? chat;
|
||||
const result = await callFn({
|
||||
model: input.model,
|
||||
messages: [{ role: 'user', content: prompt }],
|
||||
maxTokens: 200,
|
||||
abortSignal: input.abortSignal,
|
||||
});
|
||||
if (isRefusalResponse(result)) {
|
||||
throw new Error('judge refused to answer');
|
||||
}
|
||||
const raw = parseJudgeJSON(result.text);
|
||||
const verdict = normalizeVerdict(raw);
|
||||
return {
|
||||
verdict,
|
||||
usage: {
|
||||
inputTokens: result.usage.input_tokens ?? 0,
|
||||
outputTokens: result.usage.output_tokens ?? 0,
|
||||
},
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,367 @@
|
||||
/**
|
||||
* eval-contradictions/runner — the orchestrator.
|
||||
*
|
||||
* One run of `gbrain eval suspected-contradictions`:
|
||||
* 1. Load queries from one of three sources (file, single, capture).
|
||||
* --from-capture detects an empty eval_candidates table and exits 2.
|
||||
* 2. For each query, run hybridSearch (engine-side, embedding cost
|
||||
* tracked separately).
|
||||
* 3. Generate pairs:
|
||||
* - cross_slug_chunks across the top-K results
|
||||
* - intra_page_chunk_take for each unique page's active takes
|
||||
* (P1 batched via engine.listActiveTakesForPages)
|
||||
* 4. Apply the A1 date pre-filter; pairs that pass go to the cache (P2)
|
||||
* or judge.
|
||||
* 5. Sort by combined retrieval score; deterministic vs score-first
|
||||
* sampling controls the order pairs are judged (A3).
|
||||
* 6. Track cost (A2 soft ceiling): pre-flight estimate + mid-run
|
||||
* cumulative stop.
|
||||
* 7. judge_errors counted as first-class (Codex fix); failed pairs go
|
||||
* into the typed counters, not stderr.
|
||||
* 8. Aggregate per-query + global stats + Wilson CI + source-tier
|
||||
* breakdown + hot pages.
|
||||
*
|
||||
* Returns a ProbeReport plus a side-channel `judgeErrors` array for the
|
||||
* doctor integration. Pure orchestration — no filesystem, no CLI parsing.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import { hybridSearch } from '../search/hybrid.ts';
|
||||
import type { SearchResult } from '../types.ts';
|
||||
import { buildCalibration } from './calibration.ts';
|
||||
import { JudgeCache } from './cache.ts';
|
||||
import { CostTracker, estimateUpperBoundCost } from './cost-tracker.ts';
|
||||
import { buildSourceTierBreakdown, classifySlugTier } from './cross-source.ts';
|
||||
import { shouldSkipForDateMismatch } from './date-filter.ts';
|
||||
import { judgeContradiction, type JudgeInput, type JudgeOutput } from './judge.ts';
|
||||
import { JudgeErrorCollector } from './judge-errors.ts';
|
||||
import { buildHotPages } from './severity-classify.ts';
|
||||
import { pairToFinding } from './auto-supersession.ts';
|
||||
import {
|
||||
PROMPT_VERSION,
|
||||
SCHEMA_VERSION,
|
||||
TRUNCATION_POLICY,
|
||||
type ContradictionFinding,
|
||||
type ContradictionPair,
|
||||
type PairMember,
|
||||
type PerQueryResult,
|
||||
type ProbeReport,
|
||||
} from './types.ts';
|
||||
|
||||
const DEFAULT_TOP_K = 5;
|
||||
const DEFAULT_JUDGE_MODEL = 'anthropic:claude-haiku-4-5';
|
||||
const DEFAULT_MAX_PAIR_CHARS = 1500;
|
||||
|
||||
/** Caller-supplied judge function signature; defaults to judgeContradiction. */
|
||||
export type JudgeFn = (input: JudgeInput) => Promise<JudgeOutput>;
|
||||
|
||||
export interface RunnerOpts {
|
||||
engine: BrainEngine;
|
||||
queries: string[];
|
||||
judgeModel?: string;
|
||||
topK?: number;
|
||||
/** Pair-sampling policy (A3). 'deterministic' uses combined_score DESC. */
|
||||
sampling?: 'deterministic' | 'score-first';
|
||||
/** USD cap for the run. Soft ceiling enforced pre-flight + mid-run. */
|
||||
budgetUsd?: number;
|
||||
/** True iff user passed --yes; allows over-budget pre-flight to proceed. */
|
||||
yesOverride?: boolean;
|
||||
/** UTF-8-safe per-pair truncation (C4). */
|
||||
maxPairChars?: number;
|
||||
/** Disable the persistent cache (P2). Useful for benchmark runs. */
|
||||
noCache?: boolean;
|
||||
/** Test hooks: override the judge and the search functions. */
|
||||
judgeFn?: JudgeFn;
|
||||
searchFn?: (engine: BrainEngine, query: string, opts: { limit: number }) => Promise<SearchResult[]>;
|
||||
abortSignal?: AbortSignal;
|
||||
}
|
||||
|
||||
export interface RunnerResult {
|
||||
report: ProbeReport;
|
||||
/** Detailed error rows (Codex fix — first-class, not stderr). */
|
||||
judgeErrorRows: ReadonlyArray<{ kind: string; pair_id: string; reason: string }>;
|
||||
/** True iff the run stopped early because cumulative cost > budget. */
|
||||
capHitMidRun: boolean;
|
||||
/** True iff pre-flight refused (only set when --yes was not passed). */
|
||||
preFlightRefused: boolean;
|
||||
}
|
||||
|
||||
/** Custom error class for pre-flight budget refusal. */
|
||||
export class PreFlightBudgetError extends Error {
|
||||
constructor(public readonly estimatedUsd: number, public readonly capUsd: number) {
|
||||
super(`Estimated cost $${estimatedUsd.toFixed(4)} exceeds --budget-usd cap $${capUsd.toFixed(2)}; pass --yes to override.`);
|
||||
this.name = 'PreFlightBudgetError';
|
||||
}
|
||||
}
|
||||
|
||||
/** Build a pair key for judge_errors row identification. Stable per run. */
|
||||
function pairId(pair: ContradictionPair): string {
|
||||
const a = pair.a.chunk_id ?? `take-${pair.a.take_id}`;
|
||||
const b = pair.b.chunk_id ?? `take-${pair.b.take_id}`;
|
||||
return `${pair.kind}:${pair.a.slug}#${a}:${pair.b.slug}#${b}`;
|
||||
}
|
||||
|
||||
/** Convert a SearchResult into a PairMember (chunk shape). */
|
||||
function searchResultToMember(r: SearchResult): PairMember {
|
||||
return {
|
||||
slug: r.slug,
|
||||
chunk_id: r.chunk_id,
|
||||
take_id: null,
|
||||
source_tier: classifySlugTier(r.slug),
|
||||
holder: null,
|
||||
text: r.chunk_text,
|
||||
};
|
||||
}
|
||||
|
||||
/** Convert a Take into a PairMember. */
|
||||
function takeToMember(take: { id: number; page_slug: string; claim: string; holder: string }, source_tier: ReturnType<typeof classifySlugTier>): PairMember {
|
||||
return {
|
||||
slug: take.page_slug,
|
||||
chunk_id: null,
|
||||
take_id: take.id,
|
||||
source_tier,
|
||||
holder: take.holder,
|
||||
text: take.claim,
|
||||
};
|
||||
}
|
||||
|
||||
/** Build cross-slug pairs from the top-K (every distinct-slug pair once). */
|
||||
function generateCrossSlugPairs(results: SearchResult[]): ContradictionPair[] {
|
||||
const out: ContradictionPair[] = [];
|
||||
for (let i = 0; i < results.length; i++) {
|
||||
for (let j = i + 1; j < results.length; j++) {
|
||||
if (results[i].slug === results[j].slug) continue; // skip same-slug
|
||||
const a = searchResultToMember(results[i]);
|
||||
const b = searchResultToMember(results[j]);
|
||||
out.push({
|
||||
kind: 'cross_slug_chunks',
|
||||
a, b,
|
||||
combined_score: (results[i].score ?? 0) + (results[j].score ?? 0),
|
||||
});
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** Build intra-page pairs: for each result page, pair its chunks with the page's takes. */
|
||||
async function generateIntraPagePairs(
|
||||
engine: BrainEngine,
|
||||
results: SearchResult[],
|
||||
): Promise<ContradictionPair[]> {
|
||||
if (results.length === 0) return [];
|
||||
// Unique page_ids only.
|
||||
const pageIds = Array.from(new Set(results.map((r) => r.page_id)));
|
||||
const takesByPage = await engine.listActiveTakesForPages(pageIds);
|
||||
const out: ContradictionPair[] = [];
|
||||
for (const r of results) {
|
||||
const takes = takesByPage.get(r.page_id) ?? [];
|
||||
if (takes.length === 0) continue;
|
||||
const chunkMember = searchResultToMember(r);
|
||||
for (const t of takes) {
|
||||
const takeMember = takeToMember(t, chunkMember.source_tier);
|
||||
out.push({
|
||||
kind: 'intra_page_chunk_take',
|
||||
a: chunkMember,
|
||||
b: takeMember,
|
||||
// Take has no retrieval score; weight 1.0 so intra-page pairs surface
|
||||
// alongside cross-slug ones in deterministic ordering.
|
||||
combined_score: (r.score ?? 0) + 1.0,
|
||||
});
|
||||
}
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Sort pairs by the chosen sampling policy.
|
||||
*
|
||||
* - deterministic: by combined_score DESC, then (slug-a, slug-b) lex.
|
||||
* Stable for measurement — re-runs surface the same pairs in the same
|
||||
* order, so cache hit-rate doesn't depend on RNG.
|
||||
* - score-first: same as deterministic for v1 (A3 reduction; if we add
|
||||
* triage-mode-specific behavior later it diverges here).
|
||||
*/
|
||||
function sortPairs(
|
||||
pairs: ContradictionPair[],
|
||||
sampling: 'deterministic' | 'score-first',
|
||||
): ContradictionPair[] {
|
||||
void sampling; // unused param flag for forward compatibility
|
||||
return [...pairs].sort((x, y) => {
|
||||
if (y.combined_score !== x.combined_score) return y.combined_score - x.combined_score;
|
||||
if (x.a.slug !== y.a.slug) return x.a.slug < y.a.slug ? -1 : 1;
|
||||
if (x.b.slug !== y.b.slug) return x.b.slug < y.b.slug ? -1 : 1;
|
||||
return 0;
|
||||
});
|
||||
}
|
||||
|
||||
/**
|
||||
* Orchestrate one run. The runner is engine-aware (needs hybridSearch and
|
||||
* the persistent cache + run-row writers); callers pass an array of query
|
||||
* strings — CLI flag parsing lives in the command file, not here.
|
||||
*/
|
||||
export async function runContradictionProbe(opts: RunnerOpts): Promise<RunnerResult> {
|
||||
const startedAt = Date.now();
|
||||
const judgeModel = opts.judgeModel ?? DEFAULT_JUDGE_MODEL;
|
||||
const topK = Math.max(1, opts.topK ?? DEFAULT_TOP_K);
|
||||
const sampling = opts.sampling ?? 'deterministic';
|
||||
const budgetUsd = opts.budgetUsd ?? 5.0;
|
||||
const maxPairChars = opts.maxPairChars ?? DEFAULT_MAX_PAIR_CHARS;
|
||||
const judgeFn = opts.judgeFn ?? judgeContradiction;
|
||||
const searchFn =
|
||||
opts.searchFn ??
|
||||
((engine, query, o) => hybridSearch(engine, query, { limit: o.limit }));
|
||||
|
||||
const errs = new JudgeErrorCollector();
|
||||
const tracker = new CostTracker({ capUsd: budgetUsd });
|
||||
const cache = new JudgeCache({ engine: opts.engine, modelId: judgeModel, disabled: !!opts.noCache });
|
||||
|
||||
// Pre-flight: pair count = queries × min(topK*(topK-1)/2 + topK, 50).
|
||||
// Conservative upper bound; the actual count depends on takes per page.
|
||||
const conservativePairsPerQuery = (topK * (topK - 1)) / 2 + topK * 2;
|
||||
const estimated = estimateUpperBoundCost({
|
||||
pairCount: opts.queries.length * conservativePairsPerQuery,
|
||||
queryCount: opts.queries.length,
|
||||
judgeModel,
|
||||
});
|
||||
if (estimated > budgetUsd && !opts.yesOverride) {
|
||||
throw new PreFlightBudgetError(estimated, budgetUsd);
|
||||
}
|
||||
|
||||
const perQuery: PerQueryResult[] = [];
|
||||
const allFindings: ContradictionFinding[] = [];
|
||||
const allPairs: ContradictionPair[] = [];
|
||||
let capHitMidRun = false;
|
||||
let queriesWithContradiction = 0;
|
||||
|
||||
for (const query of opts.queries) {
|
||||
if (opts.abortSignal?.aborted) break;
|
||||
if (capHitMidRun) {
|
||||
// Emit empty per-query so denominators stay honest.
|
||||
perQuery.push({
|
||||
query,
|
||||
result_count: 0,
|
||||
contradictions: [],
|
||||
pairs_skipped_by_date: 0,
|
||||
pairs_cache_hit: 0,
|
||||
pairs_judged: 0,
|
||||
});
|
||||
continue;
|
||||
}
|
||||
|
||||
// Search.
|
||||
const results = await searchFn(opts.engine, query, { limit: topK });
|
||||
|
||||
// Pairs.
|
||||
const cross = generateCrossSlugPairs(results);
|
||||
const intra = await generateIntraPagePairs(opts.engine, results);
|
||||
const allPairsForQuery = [...cross, ...intra];
|
||||
allPairs.push(...allPairsForQuery);
|
||||
|
||||
// Date pre-filter.
|
||||
const survivedDate: ContradictionPair[] = [];
|
||||
let skippedByDate = 0;
|
||||
for (const p of allPairsForQuery) {
|
||||
const decision = shouldSkipForDateMismatch({ textA: p.a.text, textB: p.b.text });
|
||||
if (decision.skip) {
|
||||
skippedByDate++;
|
||||
continue;
|
||||
}
|
||||
survivedDate.push(p);
|
||||
}
|
||||
|
||||
// Sort.
|
||||
const sorted = sortPairs(survivedDate, sampling);
|
||||
|
||||
// Judge each pair.
|
||||
const findings: ContradictionFinding[] = [];
|
||||
let cacheHits = 0;
|
||||
let judged = 0;
|
||||
for (const pair of sorted) {
|
||||
if (opts.abortSignal?.aborted) break;
|
||||
if (tracker.exceededCap()) {
|
||||
capHitMidRun = true;
|
||||
break;
|
||||
}
|
||||
// Cache lookup.
|
||||
const cached = await cache.lookup(pair.a.text, pair.b.text);
|
||||
if (cached) {
|
||||
cacheHits++;
|
||||
if (cached.contradicts) {
|
||||
findings.push(pairToFinding(pair, cached));
|
||||
}
|
||||
continue;
|
||||
}
|
||||
// Judge call.
|
||||
try {
|
||||
const out = await judgeFn({
|
||||
query,
|
||||
a: { slug: pair.a.slug, text: pair.a.text, source_tier: pair.a.source_tier, holder: pair.a.holder },
|
||||
b: { slug: pair.b.slug, text: pair.b.text, source_tier: pair.b.source_tier, holder: pair.b.holder },
|
||||
model: judgeModel,
|
||||
maxPairChars,
|
||||
abortSignal: opts.abortSignal,
|
||||
});
|
||||
tracker.recordJudgeCall(judgeModel, out.usage);
|
||||
await cache.store(pair.a.text, pair.b.text, out.verdict);
|
||||
judged++;
|
||||
if (out.verdict.contradicts) {
|
||||
findings.push(pairToFinding(pair, out.verdict));
|
||||
}
|
||||
} catch (err) {
|
||||
errs.record(pairId(pair), err);
|
||||
}
|
||||
}
|
||||
|
||||
if (findings.length > 0) queriesWithContradiction++;
|
||||
perQuery.push({
|
||||
query,
|
||||
result_count: results.length,
|
||||
contradictions: findings,
|
||||
pairs_skipped_by_date: skippedByDate,
|
||||
pairs_cache_hit: cacheHits,
|
||||
pairs_judged: judged,
|
||||
});
|
||||
allFindings.push(...findings);
|
||||
}
|
||||
|
||||
// Aggregate.
|
||||
const cacheStats = cache.stats();
|
||||
const judgeErrors = errs.finalize();
|
||||
const cost = tracker.finalize();
|
||||
const calibration = buildCalibration({
|
||||
queriesTotal: opts.queries.length,
|
||||
queriesWithContradiction,
|
||||
});
|
||||
const breakdown = buildSourceTierBreakdown(allPairs);
|
||||
const hotPages = buildHotPages(allFindings);
|
||||
const runId = new Date(startedAt).toISOString().replace(/[:.]/g, '-').replace(/-(?=\d{3}Z$)/, '.');
|
||||
const durationMs = Date.now() - startedAt;
|
||||
|
||||
const report: ProbeReport = {
|
||||
schema_version: SCHEMA_VERSION,
|
||||
run_id: runId,
|
||||
judge_model: judgeModel,
|
||||
prompt_version: PROMPT_VERSION,
|
||||
truncation_policy: TRUNCATION_POLICY,
|
||||
top_k: topK,
|
||||
sampling,
|
||||
queries_evaluated: opts.queries.length,
|
||||
queries_with_contradiction: queriesWithContradiction,
|
||||
total_contradictions_flagged: allFindings.length,
|
||||
calibration,
|
||||
judge_errors: judgeErrors,
|
||||
cost_usd: cost,
|
||||
cache: cacheStats,
|
||||
duration_ms: durationMs,
|
||||
source_tier_breakdown: breakdown,
|
||||
per_query: perQuery,
|
||||
hot_pages: hotPages,
|
||||
};
|
||||
|
||||
return {
|
||||
report,
|
||||
judgeErrorRows: errs.rowsOut(),
|
||||
capHitMidRun,
|
||||
preFlightRefused: false,
|
||||
};
|
||||
}
|
||||
@@ -0,0 +1,75 @@
|
||||
/**
|
||||
* eval-contradictions/severity-classify — M4 severity helpers.
|
||||
*
|
||||
* The judge prompt asks the LLM to assign severity (low | medium | high).
|
||||
* This module:
|
||||
* - Validates the judge's claimed severity against an enum (defaults to 'low'
|
||||
* on garbage input rather than throwing).
|
||||
* - Buckets findings by severity for doctor-style output sort.
|
||||
* - Computes per-page max_severity for the hot_pages roll-up.
|
||||
*
|
||||
* The rubric lives in the judge prompt itself (low = naming/format,
|
||||
* medium = value/state, high = identity/structural). This module is pure
|
||||
* post-processing; it does NOT re-classify or override the LLM's call.
|
||||
*/
|
||||
|
||||
import type { ContradictionFinding, HotPage, Severity } from './types.ts';
|
||||
|
||||
const SEVERITY_RANK: Record<Severity, number> = { low: 1, medium: 2, high: 3 };
|
||||
|
||||
/** Validate a severity string; default to 'low' on unknown input. */
|
||||
export function parseSeverity(value: unknown): Severity {
|
||||
if (value === 'low' || value === 'medium' || value === 'high') return value;
|
||||
return 'low';
|
||||
}
|
||||
|
||||
/** Compare for descending sort: high > medium > low. */
|
||||
export function compareSeverityDesc(a: Severity, b: Severity): number {
|
||||
return SEVERITY_RANK[b] - SEVERITY_RANK[a];
|
||||
}
|
||||
|
||||
/** Return findings grouped by severity. Order within each group preserved from input. */
|
||||
export function bucketBySeverity(
|
||||
findings: readonly ContradictionFinding[],
|
||||
): Record<Severity, ContradictionFinding[]> {
|
||||
const out: Record<Severity, ContradictionFinding[]> = { low: [], medium: [], high: [] };
|
||||
for (const f of findings) {
|
||||
out[f.severity].push(f);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* Roll up appearances across all findings into per-page totals + max severity.
|
||||
* Sorted by appearances DESC, then max_severity DESC for stable ties.
|
||||
*/
|
||||
export function buildHotPages(
|
||||
findings: readonly ContradictionFinding[],
|
||||
limit = 10,
|
||||
): HotPage[] {
|
||||
const acc = new Map<string, { count: number; maxSev: Severity }>();
|
||||
const touch = (slug: string, sev: Severity) => {
|
||||
const prior = acc.get(slug);
|
||||
if (!prior) {
|
||||
acc.set(slug, { count: 1, maxSev: sev });
|
||||
return;
|
||||
}
|
||||
prior.count++;
|
||||
if (SEVERITY_RANK[sev] > SEVERITY_RANK[prior.maxSev]) {
|
||||
prior.maxSev = sev;
|
||||
}
|
||||
};
|
||||
for (const f of findings) {
|
||||
touch(f.a.slug, f.severity);
|
||||
if (f.b.slug !== f.a.slug) touch(f.b.slug, f.severity);
|
||||
}
|
||||
const rows: HotPage[] = Array.from(acc.entries()).map(([slug, v]) => ({
|
||||
slug,
|
||||
appearances: v.count,
|
||||
max_severity: v.maxSev,
|
||||
}));
|
||||
rows.sort(
|
||||
(x, y) => y.appearances - x.appearances || SEVERITY_RANK[y.max_severity] - SEVERITY_RANK[x.max_severity],
|
||||
);
|
||||
return rows.slice(0, limit);
|
||||
}
|
||||
@@ -0,0 +1,99 @@
|
||||
/**
|
||||
* eval-contradictions/trends — M5 time-series helpers.
|
||||
*
|
||||
* Thin orchestration over engine.writeContradictionsRun + loadContradictionsTrend.
|
||||
* Render helpers produce the plain-text chart for `gbrain eval
|
||||
* suspected-contradictions trend`. Pure data structures — no LLM or filesystem.
|
||||
*/
|
||||
|
||||
import type { BrainEngine } from '../engine.ts';
|
||||
import type { ProbeReport, SourceTierBreakdown } from './types.ts';
|
||||
export type { ProbeReport, SourceTierBreakdown } from './types.ts';
|
||||
|
||||
export interface TrendRow {
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
judge_model: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: SourceTierBreakdown;
|
||||
/** Full ProbeReport blob; consumed by `review` sub-subcommand. */
|
||||
report_json: ProbeReport;
|
||||
}
|
||||
|
||||
/** Write one row per run. Returns true iff inserted (idempotent on run_id). */
|
||||
export async function writeRunRow(
|
||||
engine: BrainEngine,
|
||||
report: ProbeReport,
|
||||
durationMs: number,
|
||||
): Promise<boolean> {
|
||||
return engine.writeContradictionsRun({
|
||||
run_id: report.run_id,
|
||||
judge_model: report.judge_model,
|
||||
prompt_version: report.prompt_version,
|
||||
queries_evaluated: report.queries_evaluated,
|
||||
queries_with_contradiction: report.queries_with_contradiction,
|
||||
total_contradictions_flagged: report.total_contradictions_flagged,
|
||||
wilson_ci_lower: report.calibration.wilson_ci_95.lower,
|
||||
wilson_ci_upper: report.calibration.wilson_ci_95.upper,
|
||||
judge_errors_total: report.judge_errors.total,
|
||||
cost_usd_total: report.cost_usd.total,
|
||||
duration_ms: durationMs,
|
||||
source_tier_breakdown: report.source_tier_breakdown as unknown as Record<string, unknown>,
|
||||
report_json: report as unknown as Record<string, unknown>,
|
||||
});
|
||||
}
|
||||
|
||||
/** Load the last N days of runs, newest first. */
|
||||
export async function loadTrend(engine: BrainEngine, days: number): Promise<TrendRow[]> {
|
||||
const rows = await engine.loadContradictionsTrend(days);
|
||||
return rows.map((r) => ({
|
||||
run_id: r.run_id,
|
||||
ran_at: r.ran_at,
|
||||
judge_model: r.judge_model,
|
||||
queries_evaluated: r.queries_evaluated,
|
||||
queries_with_contradiction: r.queries_with_contradiction,
|
||||
total_contradictions_flagged: r.total_contradictions_flagged,
|
||||
wilson_ci_lower: r.wilson_ci_lower,
|
||||
wilson_ci_upper: r.wilson_ci_upper,
|
||||
judge_errors_total: r.judge_errors_total,
|
||||
cost_usd_total: r.cost_usd_total,
|
||||
duration_ms: r.duration_ms,
|
||||
source_tier_breakdown: (r.source_tier_breakdown ?? { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 }) as unknown as SourceTierBreakdown,
|
||||
report_json: r.report_json as unknown as ProbeReport,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* Plain-text chart. ~10 columns wide; each row shows the run, headline pct,
|
||||
* Wilson CI bounds, and a simple ASCII bar of `total_flagged` against a
|
||||
* 0..max scale.
|
||||
*/
|
||||
export function renderTrendChart(rows: readonly TrendRow[]): string {
|
||||
if (rows.length === 0) {
|
||||
return 'No contradiction-probe runs in this window. Run `gbrain eval suspected-contradictions` to populate.';
|
||||
}
|
||||
const max = Math.max(1, ...rows.map((r) => r.total_contradictions_flagged));
|
||||
const barWidth = 30;
|
||||
const lines: string[] = [];
|
||||
lines.push('Date Model Q WithCx Flag CI95 Bar');
|
||||
lines.push('----------- ------------------ - ------ ---- ------------- ' + '-'.repeat(barWidth));
|
||||
for (const r of rows) {
|
||||
const date = r.ran_at.slice(0, 10);
|
||||
const model = r.judge_model.split(':').pop()!.slice(0, 18).padEnd(18);
|
||||
const q = String(r.queries_evaluated).padStart(2);
|
||||
const withCx = String(r.queries_with_contradiction).padStart(6);
|
||||
const flag = String(r.total_contradictions_flagged).padStart(4);
|
||||
const ci = `${(r.wilson_ci_lower * 100).toFixed(0).padStart(2)}-${(r.wilson_ci_upper * 100).toFixed(0).padStart(2)}%`.padEnd(13);
|
||||
const fill = Math.round((r.total_contradictions_flagged / max) * barWidth);
|
||||
const bar = '#'.repeat(fill) + '.'.repeat(barWidth - fill);
|
||||
lines.push(`${date} ${model} ${q} ${withCx} ${flag} ${ci} ${bar}`);
|
||||
}
|
||||
return lines.join('\n');
|
||||
}
|
||||
@@ -0,0 +1,199 @@
|
||||
/**
|
||||
* eval-contradictions/types — stable shapes for the contradiction probe.
|
||||
*
|
||||
* `schema_version: 1` is the wire contract for `gbrain eval suspected-contradictions --json`.
|
||||
* `PROMPT_VERSION` is the cache-key discriminator: bumping this invalidates every
|
||||
* cached judge verdict from prior runs, which is the point — when the prompt edits,
|
||||
* old verdicts are no longer trustworthy.
|
||||
*
|
||||
* Adding fields: append-only, default-tolerant. Renaming fields or changing
|
||||
* field types is a schema_version bump.
|
||||
*/
|
||||
|
||||
export const SCHEMA_VERSION = 1 as const;
|
||||
|
||||
/** Bump when the judge prompt in judge.ts changes meaningfully. */
|
||||
export const PROMPT_VERSION = '1' as const;
|
||||
|
||||
/** Truncation policy string baked into the cache key. */
|
||||
export const TRUNCATION_POLICY = '1500-chars-utf8-safe' as const;
|
||||
|
||||
export type ContradictionKind = 'cross_slug_chunks' | 'intra_page_chunk_take';
|
||||
|
||||
export type Severity = 'low' | 'medium' | 'high';
|
||||
|
||||
export type ResolutionKind =
|
||||
| 'takes_supersede'
|
||||
| 'dream_synthesize'
|
||||
| 'takes_mark_debate'
|
||||
| 'manual_review';
|
||||
|
||||
export type SourceTier = 'curated' | 'bulk' | 'other';
|
||||
|
||||
/**
|
||||
* Judge's verdict for a single pair. Either the judge ran cleanly and we have
|
||||
* scoring, or it failed and we have a typed error to surface in the report.
|
||||
*/
|
||||
export interface JudgeVerdict {
|
||||
contradicts: boolean;
|
||||
severity: Severity;
|
||||
/** One-line description of what they disagree about, or empty when no contradiction. */
|
||||
axis: string;
|
||||
confidence: number;
|
||||
resolution_kind: ResolutionKind | null;
|
||||
}
|
||||
|
||||
/** Error classes counted toward the run's denominator (NOT silent skips). */
|
||||
export type JudgeErrorKind = 'parse_fail' | 'refusal' | 'timeout' | 'http_5xx' | 'unknown';
|
||||
|
||||
export interface JudgeErrorRow {
|
||||
kind: JudgeErrorKind;
|
||||
pair_id: string;
|
||||
reason: string;
|
||||
}
|
||||
|
||||
export interface JudgeErrorsCounts {
|
||||
parse_fail: number;
|
||||
refusal: number;
|
||||
timeout: number;
|
||||
http_5xx: number;
|
||||
unknown: number;
|
||||
total: number;
|
||||
/** Surfaced verbatim in output so users know errors are counted, not silent. */
|
||||
note: string;
|
||||
}
|
||||
|
||||
/** One end of a pair (chunk or take). Shape unified across kinds. */
|
||||
export interface PairMember {
|
||||
slug: string;
|
||||
/** Present for cross_slug_chunks; null when this end is a take. */
|
||||
chunk_id: number | null;
|
||||
/** Present for intra_page_chunk_take when this end is a take. */
|
||||
take_id: number | null;
|
||||
source_tier: SourceTier;
|
||||
/** Takes-only: who holds the take (`garry`, `alice`, ...). */
|
||||
holder: string | null;
|
||||
text: string;
|
||||
}
|
||||
|
||||
export interface ContradictionPair {
|
||||
kind: ContradictionKind;
|
||||
a: PairMember;
|
||||
b: PairMember;
|
||||
/** Sum of both members' retrieval scores. Used for deterministic ordering. */
|
||||
combined_score: number;
|
||||
}
|
||||
|
||||
export interface ContradictionFinding extends ContradictionPair {
|
||||
severity: Severity;
|
||||
axis: string;
|
||||
confidence: number;
|
||||
resolution_kind: ResolutionKind;
|
||||
resolution_command: string;
|
||||
}
|
||||
|
||||
export interface PerQueryResult {
|
||||
query: string;
|
||||
result_count: number;
|
||||
contradictions: ContradictionFinding[];
|
||||
/** Pairs the date pre-filter rejected before any judge call. Diagnostic only. */
|
||||
pairs_skipped_by_date: number;
|
||||
/** Pairs the cache satisfied without a judge call. */
|
||||
pairs_cache_hit: number;
|
||||
/** Pairs the judge actually scored. */
|
||||
pairs_judged: number;
|
||||
}
|
||||
|
||||
export interface SourceTierBreakdown {
|
||||
curated_vs_curated: number;
|
||||
curated_vs_bulk: number;
|
||||
bulk_vs_bulk: number;
|
||||
/** Anything that didn't fit the curated/bulk binary. */
|
||||
other: number;
|
||||
}
|
||||
|
||||
export interface WilsonCI {
|
||||
point: number;
|
||||
lower: number;
|
||||
upper: number;
|
||||
}
|
||||
|
||||
export interface Calibration {
|
||||
queries_total: number;
|
||||
queries_judged_clean: number;
|
||||
queries_with_contradiction: number;
|
||||
wilson_ci_95: WilsonCI;
|
||||
/** Emitted when n < 30 so the user knows the bounds are too wide to act on. */
|
||||
small_sample_note?: string;
|
||||
}
|
||||
|
||||
export interface CostBreakdown {
|
||||
judge: number;
|
||||
embedding: number;
|
||||
total: number;
|
||||
estimate_note: string;
|
||||
}
|
||||
|
||||
export interface CacheStats {
|
||||
hits: number;
|
||||
misses: number;
|
||||
hit_rate: number;
|
||||
}
|
||||
|
||||
export interface HotPage {
|
||||
slug: string;
|
||||
appearances: number;
|
||||
max_severity: Severity;
|
||||
}
|
||||
|
||||
export interface ProbeReport {
|
||||
schema_version: typeof SCHEMA_VERSION;
|
||||
run_id: string;
|
||||
judge_model: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
top_k: number;
|
||||
sampling: 'deterministic' | 'score-first';
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
calibration: Calibration;
|
||||
judge_errors: JudgeErrorsCounts;
|
||||
cost_usd: CostBreakdown;
|
||||
cache: CacheStats;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: SourceTierBreakdown;
|
||||
per_query: PerQueryResult[];
|
||||
hot_pages: HotPage[];
|
||||
}
|
||||
|
||||
/** Shape persisted to `eval_contradictions_runs` table. Mirrors the columns. */
|
||||
export interface ContradictionsRunRow {
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
schema_version: number;
|
||||
judge_model: string;
|
||||
prompt_version: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: SourceTierBreakdown;
|
||||
report_json: ProbeReport;
|
||||
}
|
||||
|
||||
/** Shape persisted to `eval_contradictions_cache` table. */
|
||||
export interface ContradictionsCacheRow {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
verdict: JudgeVerdict;
|
||||
created_at: string;
|
||||
expires_at: string;
|
||||
}
|
||||
@@ -2596,6 +2596,82 @@ export const MIGRATIONS: Migration[] = [
|
||||
WHERE row_num IS NOT NULL;
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 52,
|
||||
name: 'eval_contradictions_cache',
|
||||
// v0.32.6 — P2 persistent judge cache for the contradiction probe.
|
||||
//
|
||||
// Composite primary key includes prompt_version + truncation_policy
|
||||
// (Codex outside-voice fix). Without these, a prompt edit would silently
|
||||
// serve stale verdicts to consumers. The cache key is the FULL
|
||||
// configuration that produced the verdict; bumping any component
|
||||
// invalidates prior entries cleanly.
|
||||
//
|
||||
// TTL via expires_at — readers can WHERE expires_at > now() to ignore
|
||||
// stale rows; an explicit DELETE WHERE expires_at <= now() sweep runs
|
||||
// periodically (lives in cache.ts orchestration, not here).
|
||||
//
|
||||
// verdict JSONB carries the full JudgeVerdict shape (contradicts,
|
||||
// severity, axis, confidence, resolution_kind) so a cache hit is a
|
||||
// complete answer without needing a second column.
|
||||
//
|
||||
// Idempotent across PGLite and Postgres; engine-agnostic DDL.
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_cache (
|
||||
chunk_a_hash TEXT NOT NULL,
|
||||
chunk_b_hash TEXT NOT NULL,
|
||||
model_id TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
truncation_policy TEXT NOT NULL,
|
||||
verdict JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
PRIMARY KEY (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_cache_expires_idx
|
||||
ON eval_contradictions_cache (expires_at);
|
||||
`,
|
||||
},
|
||||
{
|
||||
version: 53,
|
||||
name: 'eval_contradictions_runs',
|
||||
// v0.32.6 — M5 time-series tracking for the contradiction probe.
|
||||
//
|
||||
// One row per `gbrain eval suspected-contradictions` run. The headline
|
||||
// numbers (queries_evaluated, with_contradiction, total_flagged) plus
|
||||
// Wilson 95% CI bounds enable `gbrain eval suspected-contradictions
|
||||
// trend [--days N]` to plot brain consistency over time.
|
||||
//
|
||||
// report_json carries the full ProbeReport for replay/inspection.
|
||||
// source_tier_breakdown is also surfaced as a top-level JSONB column
|
||||
// so trend queries can group by tier without parsing the full report.
|
||||
//
|
||||
// No FK to other tables: this is an append-only metrics log, not a
|
||||
// relational record. Trend reads filter on ran_at.
|
||||
//
|
||||
// Idempotent across PGLite and Postgres.
|
||||
sql: `
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
ran_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
schema_version INTEGER NOT NULL DEFAULT 1,
|
||||
judge_model TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
queries_evaluated INTEGER NOT NULL,
|
||||
queries_with_contradiction INTEGER NOT NULL,
|
||||
total_contradictions_flagged INTEGER NOT NULL,
|
||||
wilson_ci_lower REAL NOT NULL,
|
||||
wilson_ci_upper REAL NOT NULL,
|
||||
judge_errors_total INTEGER NOT NULL,
|
||||
cost_usd_total REAL NOT NULL,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
source_tier_breakdown JSONB NOT NULL,
|
||||
report_json JSONB NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_runs_ran_at_idx
|
||||
ON eval_contradictions_runs (ran_at DESC);
|
||||
`,
|
||||
},
|
||||
];
|
||||
|
||||
export const LATEST_VERSION = MIGRATIONS.length > 0
|
||||
|
||||
@@ -63,3 +63,17 @@ export const SEARCH_DESCRIPTION =
|
||||
"Keyword search using full-text search. For personal/emotional questions, " +
|
||||
"prefer get_recent_salience or find_anomalies — they surface activity bursts " +
|
||||
"without needing a search term.";
|
||||
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
// v0.32.6 — contradiction probe MCP surface (M3)
|
||||
// ──────────────────────────────────────────────────────────────────────────────
|
||||
|
||||
export const FIND_CONTRADICTIONS_DESCRIPTION =
|
||||
"v0.32.6 — return suspected-contradiction findings from the most recent " +
|
||||
"`gbrain eval suspected-contradictions` probe run, optionally filtered by slug " +
|
||||
"and/or severity. Use this when the user asks 'what's inconsistent in my " +
|
||||
"brain', 'show me contradictions about Acme', 'high-severity issues only', or " +
|
||||
"wants to act on the probe's findings without re-running it. Returns " +
|
||||
"{contradictions: [{a, b, severity, axis, confidence, resolution_command}]}. " +
|
||||
"Reads the cached run row — does NOT trigger a new probe; users run " +
|
||||
"`gbrain eval suspected-contradictions` for that.";
|
||||
|
||||
@@ -28,6 +28,7 @@ import {
|
||||
LIST_PAGES_DESCRIPTION,
|
||||
QUERY_DESCRIPTION,
|
||||
SEARCH_DESCRIPTION,
|
||||
FIND_CONTRADICTIONS_DESCRIPTION,
|
||||
} from './operations-descriptions.ts';
|
||||
|
||||
// --- Types ---
|
||||
@@ -2204,6 +2205,72 @@ const find_anomalies: Operation = {
|
||||
cliHints: { name: 'anomalies' },
|
||||
};
|
||||
|
||||
const find_contradictions: Operation = {
|
||||
name: 'find_contradictions',
|
||||
description: FIND_CONTRADICTIONS_DESCRIPTION,
|
||||
scope: 'read',
|
||||
// Reads eval_contradictions_runs.report_json for the latest run, then
|
||||
// filters in-memory by slug and severity. No new probe is triggered;
|
||||
// the agent surfaces what's already on disk.
|
||||
params: {
|
||||
slug: {
|
||||
type: 'string',
|
||||
description: 'Optional slug filter; matches either side of a pair (substring match on slug).',
|
||||
},
|
||||
severity: {
|
||||
type: 'string',
|
||||
enum: ['low', 'medium', 'high'],
|
||||
description: 'Optional severity filter.',
|
||||
},
|
||||
limit: {
|
||||
type: 'number',
|
||||
description: 'Max findings to return. Default 20.',
|
||||
},
|
||||
},
|
||||
handler: async (ctx, p) => {
|
||||
const limit = typeof p.limit === 'number' && p.limit > 0 ? Math.min(p.limit, 100) : 20;
|
||||
const slugFilter = typeof p.slug === 'string' ? p.slug.toLowerCase() : null;
|
||||
const sevFilter = (p.severity === 'low' || p.severity === 'medium' || p.severity === 'high')
|
||||
? p.severity
|
||||
: null;
|
||||
const rows = await ctx.engine.loadContradictionsTrend(30);
|
||||
if (rows.length === 0) {
|
||||
return { contradictions: [], note: 'No probe runs in the last 30 days; run `gbrain eval suspected-contradictions` first.' };
|
||||
}
|
||||
const latest = rows[0];
|
||||
const report = latest.report_json as Record<string, unknown> | null;
|
||||
const perQuery = (report?.per_query as Array<{
|
||||
contradictions: Array<{
|
||||
kind: string;
|
||||
severity: 'low' | 'medium' | 'high';
|
||||
axis: string;
|
||||
confidence: number;
|
||||
a: { slug: string; chunk_id: number | null; take_id: number | null };
|
||||
b: { slug: string; chunk_id: number | null; take_id: number | null };
|
||||
resolution_kind: string;
|
||||
resolution_command: string;
|
||||
}>;
|
||||
}> | undefined) ?? [];
|
||||
const findings = perQuery.flatMap((q) => q.contradictions);
|
||||
const filtered = findings.filter((f) => {
|
||||
if (sevFilter && f.severity !== sevFilter) return false;
|
||||
if (slugFilter) {
|
||||
const sA = f.a.slug.toLowerCase();
|
||||
const sB = f.b.slug.toLowerCase();
|
||||
if (!sA.includes(slugFilter) && !sB.includes(slugFilter)) return false;
|
||||
}
|
||||
return true;
|
||||
});
|
||||
return {
|
||||
run_id: latest.run_id,
|
||||
ran_at: latest.ran_at,
|
||||
contradictions: filtered.slice(0, limit),
|
||||
total_in_run: findings.length,
|
||||
};
|
||||
},
|
||||
cliHints: { name: 'find-contradictions' },
|
||||
};
|
||||
|
||||
const get_recent_transcripts: Operation = {
|
||||
name: 'get_recent_transcripts',
|
||||
description: GET_RECENT_TRANSCRIPTS_DESCRIPTION,
|
||||
@@ -2699,6 +2766,8 @@ export const operations: Operation[] = [
|
||||
get_recent_salience, find_anomalies, get_recent_transcripts,
|
||||
// v0.31: hot memory (facts table)
|
||||
extract_facts, recall, forget_fact,
|
||||
// v0.32.6: contradiction probe MCP surface (M3)
|
||||
find_contradictions,
|
||||
];
|
||||
|
||||
export const operationsByName = Object.fromEntries(
|
||||
|
||||
@@ -2205,6 +2205,180 @@ export class PGLiteEngine implements BrainEngine {
|
||||
return result.rows.length;
|
||||
}
|
||||
|
||||
/** v0.32.6 P1 — batched per-page active-takes fetch for the contradiction probe. */
|
||||
async listActiveTakesForPages(
|
||||
pageIds: number[],
|
||||
opts: { takesHoldersAllowList?: string[] } = {},
|
||||
): Promise<Map<number, Take[]>> {
|
||||
const out = new Map<number, Take[]>();
|
||||
for (const pid of pageIds) out.set(pid, []);
|
||||
if (pageIds.length === 0) return out;
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT t.*, p.slug AS page_slug
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.page_id = ANY($1::int[])
|
||||
AND t.active = true
|
||||
AND ($2::text[] IS NULL OR t.holder = ANY($2::text[]))
|
||||
ORDER BY t.page_id, t.row_num`,
|
||||
[pageIds, opts.takesHoldersAllowList ?? null]
|
||||
);
|
||||
for (const r of rows) {
|
||||
const take = takeRowToTake(r as Record<string, unknown>);
|
||||
const bucket = out.get(take.page_id);
|
||||
if (bucket) bucket.push(take);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/** v0.32.6 M5 — persist a probe run row. Idempotent on run_id. */
|
||||
async writeContradictionsRun(row: {
|
||||
run_id: string;
|
||||
judge_model: string;
|
||||
prompt_version: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}): Promise<boolean> {
|
||||
const result = await this.db.query(
|
||||
`INSERT INTO eval_contradictions_runs (
|
||||
run_id, judge_model, prompt_version,
|
||||
queries_evaluated, queries_with_contradiction, total_contradictions_flagged,
|
||||
wilson_ci_lower, wilson_ci_upper, judge_errors_total,
|
||||
cost_usd_total, duration_ms,
|
||||
source_tier_breakdown, report_json
|
||||
) VALUES (
|
||||
$1, $2, $3,
|
||||
$4, $5, $6,
|
||||
$7, $8, $9,
|
||||
$10, $11,
|
||||
$12::jsonb, $13::jsonb
|
||||
)
|
||||
ON CONFLICT (run_id) DO NOTHING`,
|
||||
[
|
||||
row.run_id, row.judge_model, row.prompt_version,
|
||||
row.queries_evaluated, row.queries_with_contradiction, row.total_contradictions_flagged,
|
||||
row.wilson_ci_lower, row.wilson_ci_upper, row.judge_errors_total,
|
||||
row.cost_usd_total, row.duration_ms,
|
||||
row.source_tier_breakdown, row.report_json,
|
||||
]
|
||||
);
|
||||
return (result.affectedRows ?? 0) > 0;
|
||||
}
|
||||
|
||||
/** v0.32.6 M5 — read probe runs from the last N days. */
|
||||
async loadContradictionsTrend(days: number): Promise<Array<{
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
judge_model: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}>> {
|
||||
const cutoff = new Date(Date.now() - Math.max(0, days) * 86400000);
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT run_id, ran_at, judge_model,
|
||||
queries_evaluated, queries_with_contradiction, total_contradictions_flagged,
|
||||
wilson_ci_lower, wilson_ci_upper, judge_errors_total,
|
||||
cost_usd_total, duration_ms,
|
||||
source_tier_breakdown, report_json
|
||||
FROM eval_contradictions_runs
|
||||
WHERE ran_at >= $1
|
||||
ORDER BY ran_at DESC`,
|
||||
[cutoff]
|
||||
);
|
||||
return (rows as Record<string, unknown>[]).map((r) => ({
|
||||
run_id: r.run_id as string,
|
||||
ran_at: r.ran_at instanceof Date ? (r.ran_at as Date).toISOString() : String(r.ran_at),
|
||||
judge_model: r.judge_model as string,
|
||||
queries_evaluated: Number(r.queries_evaluated),
|
||||
queries_with_contradiction: Number(r.queries_with_contradiction),
|
||||
total_contradictions_flagged: Number(r.total_contradictions_flagged),
|
||||
wilson_ci_lower: Number(r.wilson_ci_lower),
|
||||
wilson_ci_upper: Number(r.wilson_ci_upper),
|
||||
judge_errors_total: Number(r.judge_errors_total),
|
||||
cost_usd_total: Number(r.cost_usd_total),
|
||||
duration_ms: Number(r.duration_ms),
|
||||
source_tier_breakdown: r.source_tier_breakdown as Record<string, unknown>,
|
||||
report_json: r.report_json as Record<string, unknown>,
|
||||
}));
|
||||
}
|
||||
|
||||
/** v0.32.6 P2 — cache lookup; returns verdict JSON or null. */
|
||||
async getContradictionCacheEntry(key: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
}): Promise<Record<string, unknown> | null> {
|
||||
const { rows } = await this.db.query(
|
||||
`SELECT verdict FROM eval_contradictions_cache
|
||||
WHERE chunk_a_hash = $1
|
||||
AND chunk_b_hash = $2
|
||||
AND model_id = $3
|
||||
AND prompt_version = $4
|
||||
AND truncation_policy = $5
|
||||
AND expires_at > now()
|
||||
LIMIT 1`,
|
||||
[key.chunk_a_hash, key.chunk_b_hash, key.model_id, key.prompt_version, key.truncation_policy]
|
||||
);
|
||||
if (rows.length === 0) return null;
|
||||
return (rows[0] as Record<string, unknown>).verdict as Record<string, unknown>;
|
||||
}
|
||||
|
||||
/** v0.32.6 P2 — cache upsert with TTL refresh on conflict. */
|
||||
async putContradictionCacheEntry(opts: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
verdict: Record<string, unknown>;
|
||||
ttl_seconds?: number;
|
||||
}): Promise<void> {
|
||||
const ttl = Math.max(60, opts.ttl_seconds ?? 30 * 86400);
|
||||
const expiresAt = new Date(Date.now() + ttl * 1000);
|
||||
await this.db.query(
|
||||
`INSERT INTO eval_contradictions_cache (
|
||||
chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy,
|
||||
verdict, expires_at
|
||||
) VALUES ($1, $2, $3, $4, $5, $6::jsonb, $7)
|
||||
ON CONFLICT (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
DO UPDATE SET
|
||||
verdict = EXCLUDED.verdict,
|
||||
expires_at = EXCLUDED.expires_at,
|
||||
created_at = now()`,
|
||||
[
|
||||
opts.chunk_a_hash, opts.chunk_b_hash, opts.model_id,
|
||||
opts.prompt_version, opts.truncation_policy,
|
||||
opts.verdict, expiresAt,
|
||||
]
|
||||
);
|
||||
}
|
||||
|
||||
/** v0.32.6 P2 — periodic sweep of expired cache rows. */
|
||||
async sweepContradictionCache(): Promise<number> {
|
||||
const result = await this.db.query(
|
||||
`DELETE FROM eval_contradictions_cache WHERE expires_at <= now()`
|
||||
);
|
||||
return result.affectedRows ?? 0;
|
||||
}
|
||||
|
||||
async listTakes(opts: TakesListOpts = {}): Promise<Take[]> {
|
||||
const limit = clampSearchLimit(opts.limit, 100, 500);
|
||||
const offset = Math.max(0, Math.floor(opts.offset ?? 0));
|
||||
|
||||
@@ -501,6 +501,51 @@ CREATE TABLE IF NOT EXISTS eval_takes_quality_runs (
|
||||
CREATE INDEX IF NOT EXISTS eval_takes_quality_runs_trend_idx
|
||||
ON eval_takes_quality_runs (rubric_version, created_at DESC);
|
||||
|
||||
-- ============================================================
|
||||
-- eval_contradictions_cache (v0.32.6): persistent judge verdicts for the
|
||||
-- contradiction probe. Composite key includes prompt_version + truncation_
|
||||
-- policy so prompt edits cleanly invalidate prior verdicts (Codex fix).
|
||||
-- TTL via expires_at; sweep runs periodically from cache.ts.
|
||||
-- ============================================================
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_cache (
|
||||
chunk_a_hash TEXT NOT NULL,
|
||||
chunk_b_hash TEXT NOT NULL,
|
||||
model_id TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
truncation_policy TEXT NOT NULL,
|
||||
verdict JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
PRIMARY KEY (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_cache_expires_idx
|
||||
ON eval_contradictions_cache (expires_at);
|
||||
|
||||
-- ============================================================
|
||||
-- eval_contradictions_runs (v0.32.6): time-series tracking for the probe.
|
||||
-- One row per 'gbrain eval suspected-contradictions' run; source for the
|
||||
-- 'trend' sub-subcommand and the doctor 'contradictions' check.
|
||||
-- ============================================================
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
ran_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
schema_version INTEGER NOT NULL DEFAULT 1,
|
||||
judge_model TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
queries_evaluated INTEGER NOT NULL,
|
||||
queries_with_contradiction INTEGER NOT NULL,
|
||||
total_contradictions_flagged INTEGER NOT NULL,
|
||||
wilson_ci_lower REAL NOT NULL,
|
||||
wilson_ci_upper REAL NOT NULL,
|
||||
judge_errors_total INTEGER NOT NULL,
|
||||
cost_usd_total REAL NOT NULL,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
source_tier_breakdown JSONB NOT NULL,
|
||||
report_json JSONB NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_runs_ran_at_idx
|
||||
ON eval_contradictions_runs (ran_at DESC);
|
||||
|
||||
-- ============================================================
|
||||
-- access_tokens: legacy bearer tokens for remote MCP access
|
||||
-- ============================================================
|
||||
|
||||
@@ -2353,6 +2353,196 @@ export class PostgresEngine implements BrainEngine {
|
||||
return result.length;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.6 — batched per-page active-takes fetch (P1). One round-trip
|
||||
* regardless of how many pages the caller passes. Honors holder allow-list
|
||||
* for MCP scope enforcement. Pages with no active takes get an empty array.
|
||||
*/
|
||||
async listActiveTakesForPages(
|
||||
pageIds: number[],
|
||||
opts: { takesHoldersAllowList?: string[] } = {},
|
||||
): Promise<Map<number, Take[]>> {
|
||||
const out = new Map<number, Take[]>();
|
||||
for (const pid of pageIds) out.set(pid, []);
|
||||
if (pageIds.length === 0) return out;
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT t.*, p.slug AS page_slug
|
||||
FROM takes t
|
||||
JOIN pages p ON p.id = t.page_id
|
||||
WHERE t.page_id = ANY(${pageIds}::int[])
|
||||
AND t.active = true
|
||||
AND (
|
||||
${opts.takesHoldersAllowList ?? null}::text[] IS NULL
|
||||
OR t.holder = ANY(${opts.takesHoldersAllowList ?? null}::text[])
|
||||
)
|
||||
ORDER BY t.page_id, t.row_num
|
||||
`;
|
||||
for (const r of rows) {
|
||||
const take = takeRowToTake(r as Record<string, unknown>);
|
||||
const bucket = out.get(take.page_id);
|
||||
if (bucket) bucket.push(take);
|
||||
}
|
||||
return out;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.6 — persist a contradiction-probe run row (M5). Idempotent on
|
||||
* run_id via ON CONFLICT DO NOTHING. Returns true iff a row was inserted.
|
||||
*/
|
||||
async writeContradictionsRun(row: {
|
||||
run_id: string;
|
||||
judge_model: string;
|
||||
prompt_version: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}): Promise<boolean> {
|
||||
const sql = this.sql;
|
||||
const result = await sql`
|
||||
INSERT INTO eval_contradictions_runs (
|
||||
run_id, judge_model, prompt_version,
|
||||
queries_evaluated, queries_with_contradiction, total_contradictions_flagged,
|
||||
wilson_ci_lower, wilson_ci_upper, judge_errors_total,
|
||||
cost_usd_total, duration_ms,
|
||||
source_tier_breakdown, report_json
|
||||
) VALUES (
|
||||
${row.run_id}, ${row.judge_model}, ${row.prompt_version},
|
||||
${row.queries_evaluated}, ${row.queries_with_contradiction}, ${row.total_contradictions_flagged},
|
||||
${row.wilson_ci_lower}, ${row.wilson_ci_upper}, ${row.judge_errors_total},
|
||||
${row.cost_usd_total}, ${row.duration_ms},
|
||||
${sql.json(row.source_tier_breakdown as Parameters<typeof sql.json>[0])},
|
||||
${sql.json(row.report_json as Parameters<typeof sql.json>[0])}
|
||||
)
|
||||
ON CONFLICT (run_id) DO NOTHING
|
||||
`;
|
||||
return result.count > 0;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.6 — load probe runs from the last N days, newest first (M5).
|
||||
* Used by `trend` sub-subcommand and the doctor `contradictions` check.
|
||||
*/
|
||||
async loadContradictionsTrend(days: number): Promise<Array<{
|
||||
run_id: string;
|
||||
ran_at: string;
|
||||
judge_model: string;
|
||||
queries_evaluated: number;
|
||||
queries_with_contradiction: number;
|
||||
total_contradictions_flagged: number;
|
||||
wilson_ci_lower: number;
|
||||
wilson_ci_upper: number;
|
||||
judge_errors_total: number;
|
||||
cost_usd_total: number;
|
||||
duration_ms: number;
|
||||
source_tier_breakdown: Record<string, unknown>;
|
||||
report_json: Record<string, unknown>;
|
||||
}>> {
|
||||
const sql = this.sql;
|
||||
const cutoff = new Date(Date.now() - Math.max(0, days) * 86400000);
|
||||
const rows = await sql`
|
||||
SELECT run_id, ran_at, judge_model,
|
||||
queries_evaluated, queries_with_contradiction, total_contradictions_flagged,
|
||||
wilson_ci_lower, wilson_ci_upper, judge_errors_total,
|
||||
cost_usd_total, duration_ms,
|
||||
source_tier_breakdown, report_json
|
||||
FROM eval_contradictions_runs
|
||||
WHERE ran_at >= ${cutoff}
|
||||
ORDER BY ran_at DESC
|
||||
`;
|
||||
return rows.map((r) => ({
|
||||
run_id: r.run_id as string,
|
||||
ran_at: (r.ran_at instanceof Date ? r.ran_at.toISOString() : String(r.ran_at)),
|
||||
judge_model: r.judge_model as string,
|
||||
queries_evaluated: Number(r.queries_evaluated),
|
||||
queries_with_contradiction: Number(r.queries_with_contradiction),
|
||||
total_contradictions_flagged: Number(r.total_contradictions_flagged),
|
||||
wilson_ci_lower: Number(r.wilson_ci_lower),
|
||||
wilson_ci_upper: Number(r.wilson_ci_upper),
|
||||
judge_errors_total: Number(r.judge_errors_total),
|
||||
cost_usd_total: Number(r.cost_usd_total),
|
||||
duration_ms: Number(r.duration_ms),
|
||||
source_tier_breakdown: r.source_tier_breakdown as Record<string, unknown>,
|
||||
report_json: r.report_json as Record<string, unknown>,
|
||||
}));
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.6 — judge cache lookup (P2). Returns verdict JSON for a non-
|
||||
* expired row matching the full 5-component key, else NULL.
|
||||
*/
|
||||
async getContradictionCacheEntry(key: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
}): Promise<Record<string, unknown> | null> {
|
||||
const sql = this.sql;
|
||||
const rows = await sql`
|
||||
SELECT verdict
|
||||
FROM eval_contradictions_cache
|
||||
WHERE chunk_a_hash = ${key.chunk_a_hash}
|
||||
AND chunk_b_hash = ${key.chunk_b_hash}
|
||||
AND model_id = ${key.model_id}
|
||||
AND prompt_version = ${key.prompt_version}
|
||||
AND truncation_policy = ${key.truncation_policy}
|
||||
AND expires_at > now()
|
||||
LIMIT 1
|
||||
`;
|
||||
if (rows.length === 0) return null;
|
||||
return rows[0].verdict as Record<string, unknown>;
|
||||
}
|
||||
|
||||
/**
|
||||
* v0.32.6 — judge cache upsert. ON CONFLICT DO UPDATE refreshes verdict +
|
||||
* slides expires_at forward; same-key re-runs are safe.
|
||||
*/
|
||||
async putContradictionCacheEntry(opts: {
|
||||
chunk_a_hash: string;
|
||||
chunk_b_hash: string;
|
||||
model_id: string;
|
||||
prompt_version: string;
|
||||
truncation_policy: string;
|
||||
verdict: Record<string, unknown>;
|
||||
ttl_seconds?: number;
|
||||
}): Promise<void> {
|
||||
const sql = this.sql;
|
||||
const ttl = Math.max(60, opts.ttl_seconds ?? 30 * 86400);
|
||||
const expiresAt = new Date(Date.now() + ttl * 1000);
|
||||
await sql`
|
||||
INSERT INTO eval_contradictions_cache (
|
||||
chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy,
|
||||
verdict, expires_at
|
||||
) VALUES (
|
||||
${opts.chunk_a_hash}, ${opts.chunk_b_hash}, ${opts.model_id},
|
||||
${opts.prompt_version}, ${opts.truncation_policy},
|
||||
${sql.json(opts.verdict as Parameters<typeof sql.json>[0])}, ${expiresAt}
|
||||
)
|
||||
ON CONFLICT (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
DO UPDATE SET
|
||||
verdict = EXCLUDED.verdict,
|
||||
expires_at = EXCLUDED.expires_at,
|
||||
created_at = now()
|
||||
`;
|
||||
}
|
||||
|
||||
/** v0.32.6 — periodic sweep of expired cache rows. */
|
||||
async sweepContradictionCache(): Promise<number> {
|
||||
const sql = this.sql;
|
||||
const result = await sql`
|
||||
DELETE FROM eval_contradictions_cache WHERE expires_at <= now()
|
||||
`;
|
||||
return result.count ?? 0;
|
||||
}
|
||||
|
||||
async listTakes(opts: TakesListOpts = {}): Promise<Take[]> {
|
||||
const sql = this.sql;
|
||||
const limit = clampSearchLimit(opts.limit, 100, 500);
|
||||
|
||||
@@ -818,6 +818,47 @@ CREATE TABLE IF NOT EXISTS eval_takes_quality_runs (
|
||||
CREATE INDEX IF NOT EXISTS eval_takes_quality_runs_trend_idx
|
||||
ON eval_takes_quality_runs (rubric_version, created_at DESC);
|
||||
|
||||
-- eval_contradictions_cache (v0.32.6): persistent judge verdicts for the
|
||||
-- contradiction probe. Composite primary key includes prompt_version +
|
||||
-- truncation_policy so any prompt edit cleanly invalidates prior verdicts
|
||||
-- (Codex outside-voice fix). TTL via expires_at; cache.ts sweeps periodically.
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_cache (
|
||||
chunk_a_hash TEXT NOT NULL,
|
||||
chunk_b_hash TEXT NOT NULL,
|
||||
model_id TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
truncation_policy TEXT NOT NULL,
|
||||
verdict JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
PRIMARY KEY (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_cache_expires_idx
|
||||
ON eval_contradictions_cache (expires_at);
|
||||
|
||||
-- eval_contradictions_runs (v0.32.6): time-series tracking for the probe.
|
||||
-- One row per run; source for the \`trend\` sub-subcommand and the doctor
|
||||
-- \`contradictions\` check. report_json carries the full ProbeReport for replay.
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
ran_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
schema_version INTEGER NOT NULL DEFAULT 1,
|
||||
judge_model TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
queries_evaluated INTEGER NOT NULL,
|
||||
queries_with_contradiction INTEGER NOT NULL,
|
||||
total_contradictions_flagged INTEGER NOT NULL,
|
||||
wilson_ci_lower REAL NOT NULL,
|
||||
wilson_ci_upper REAL NOT NULL,
|
||||
judge_errors_total INTEGER NOT NULL,
|
||||
cost_usd_total REAL NOT NULL,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
source_tier_breakdown JSONB NOT NULL,
|
||||
report_json JSONB NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_runs_ran_at_idx
|
||||
ON eval_contradictions_runs (ran_at DESC);
|
||||
|
||||
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS \$\$
|
||||
BEGIN
|
||||
@@ -871,6 +912,9 @@ BEGIN
|
||||
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_takes_quality_runs ENABLE ROW LEVEL SECURITY;
|
||||
-- v0.32.6 contradiction probe tables
|
||||
ALTER TABLE eval_contradictions_cache ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_contradictions_runs ENABLE ROW LEVEL SECURITY;
|
||||
-- v0.26 OAuth 2.1 tables
|
||||
ALTER TABLE oauth_clients ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE oauth_tokens ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
@@ -814,6 +814,47 @@ CREATE TABLE IF NOT EXISTS eval_takes_quality_runs (
|
||||
CREATE INDEX IF NOT EXISTS eval_takes_quality_runs_trend_idx
|
||||
ON eval_takes_quality_runs (rubric_version, created_at DESC);
|
||||
|
||||
-- eval_contradictions_cache (v0.32.6): persistent judge verdicts for the
|
||||
-- contradiction probe. Composite primary key includes prompt_version +
|
||||
-- truncation_policy so any prompt edit cleanly invalidates prior verdicts
|
||||
-- (Codex outside-voice fix). TTL via expires_at; cache.ts sweeps periodically.
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_cache (
|
||||
chunk_a_hash TEXT NOT NULL,
|
||||
chunk_b_hash TEXT NOT NULL,
|
||||
model_id TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
truncation_policy TEXT NOT NULL,
|
||||
verdict JSONB NOT NULL,
|
||||
created_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
expires_at TIMESTAMPTZ NOT NULL,
|
||||
PRIMARY KEY (chunk_a_hash, chunk_b_hash, model_id, prompt_version, truncation_policy)
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_cache_expires_idx
|
||||
ON eval_contradictions_cache (expires_at);
|
||||
|
||||
-- eval_contradictions_runs (v0.32.6): time-series tracking for the probe.
|
||||
-- One row per run; source for the `trend` sub-subcommand and the doctor
|
||||
-- `contradictions` check. report_json carries the full ProbeReport for replay.
|
||||
CREATE TABLE IF NOT EXISTS eval_contradictions_runs (
|
||||
run_id TEXT PRIMARY KEY,
|
||||
ran_at TIMESTAMPTZ NOT NULL DEFAULT now(),
|
||||
schema_version INTEGER NOT NULL DEFAULT 1,
|
||||
judge_model TEXT NOT NULL,
|
||||
prompt_version TEXT NOT NULL,
|
||||
queries_evaluated INTEGER NOT NULL,
|
||||
queries_with_contradiction INTEGER NOT NULL,
|
||||
total_contradictions_flagged INTEGER NOT NULL,
|
||||
wilson_ci_lower REAL NOT NULL,
|
||||
wilson_ci_upper REAL NOT NULL,
|
||||
judge_errors_total INTEGER NOT NULL,
|
||||
cost_usd_total REAL NOT NULL,
|
||||
duration_ms INTEGER NOT NULL,
|
||||
source_tier_breakdown JSONB NOT NULL,
|
||||
report_json JSONB NOT NULL
|
||||
);
|
||||
CREATE INDEX IF NOT EXISTS eval_contradictions_runs_ran_at_idx
|
||||
ON eval_contradictions_runs (ran_at DESC);
|
||||
|
||||
-- NOTIFY trigger for real-time job events (Postgres only, not PGLite)
|
||||
CREATE OR REPLACE FUNCTION notify_minion_job_change() RETURNS trigger AS $$
|
||||
BEGIN
|
||||
@@ -867,6 +908,9 @@ BEGIN
|
||||
ALTER TABLE eval_candidates ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_capture_failures ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_takes_quality_runs ENABLE ROW LEVEL SECURITY;
|
||||
-- v0.32.6 contradiction probe tables
|
||||
ALTER TABLE eval_contradictions_cache ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE eval_contradictions_runs ENABLE ROW LEVEL SECURITY;
|
||||
-- v0.26 OAuth 2.1 tables
|
||||
ALTER TABLE oauth_clients ENABLE ROW LEVEL SECURITY;
|
||||
ALTER TABLE oauth_tokens ENABLE ROW LEVEL SECURITY;
|
||||
|
||||
@@ -0,0 +1,308 @@
|
||||
/**
|
||||
* E2E — Postgres-specific contradiction-probe behavior (v0.32.6, T1).
|
||||
*
|
||||
* PGLite covers the contract; this file exercises Postgres-only surfaces
|
||||
* that PGLite can't:
|
||||
* 1. The actual JSONB round-trip through postgres.js — sql.json() vs
|
||||
* double-encode regression class.
|
||||
* 2. Migrations v51 + v52 apply cleanly on a real PG instance and the
|
||||
* tables come out with the expected column shapes.
|
||||
* 3. The P1 batched listActiveTakesForPages uses ANY($1::int[]) which
|
||||
* has subtly different semantics on real PG.
|
||||
* 4. The full M5 trend write+read with PostgreSQL's TIMESTAMPTZ
|
||||
* semantics and ORDER BY ran_at DESC stability.
|
||||
* 5. The P2 cache TTL semantics with real `now()` and ON CONFLICT
|
||||
* DO UPDATE.
|
||||
* 6. The find_contradictions MCP op end-to-end via the dispatch path.
|
||||
*
|
||||
* Runs only when DATABASE_URL is set. Skips gracefully otherwise.
|
||||
*/
|
||||
|
||||
import { afterAll, beforeAll, beforeEach, describe, expect, test } from 'bun:test';
|
||||
import { PostgresEngine } from '../../src/core/postgres-engine.ts';
|
||||
import { writeRunRow, loadTrend } from '../../src/core/eval-contradictions/trends.ts';
|
||||
import { JudgeCache, buildCacheKey } from '../../src/core/eval-contradictions/cache.ts';
|
||||
import type { ProbeReport } from '../../src/core/eval-contradictions/types.ts';
|
||||
import { operationsByName, type OperationContext } from '../../src/core/operations.ts';
|
||||
|
||||
const DATABASE_URL = process.env.DATABASE_URL;
|
||||
|
||||
let engine: PostgresEngine | null = null;
|
||||
|
||||
beforeAll(async () => {
|
||||
if (!DATABASE_URL) {
|
||||
console.log('[e2e/eval-contradictions] DATABASE_URL not set — skipping.');
|
||||
return;
|
||||
}
|
||||
engine = new PostgresEngine();
|
||||
await engine.connect({ database_url: DATABASE_URL });
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
if (engine) await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
if (!engine) return;
|
||||
await engine.executeRaw('DELETE FROM eval_contradictions_runs');
|
||||
await engine.executeRaw('DELETE FROM eval_contradictions_cache');
|
||||
});
|
||||
|
||||
function mkReport(opts: Partial<ProbeReport> = {}): ProbeReport {
|
||||
return {
|
||||
schema_version: 1,
|
||||
run_id: opts.run_id ?? 'pg-test',
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
prompt_version: '1',
|
||||
truncation_policy: '1500-chars-utf8-safe',
|
||||
top_k: 5,
|
||||
sampling: 'deterministic',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 12,
|
||||
total_contradictions_flagged: 18,
|
||||
calibration: {
|
||||
queries_total: 50,
|
||||
queries_judged_clean: 38,
|
||||
queries_with_contradiction: 12,
|
||||
wilson_ci_95: { point: 0.24, lower: 0.14, upper: 0.37 },
|
||||
},
|
||||
judge_errors: { parse_fail: 1, refusal: 0, timeout: 0, http_5xx: 2, unknown: 0, total: 3, note: 'n' },
|
||||
cost_usd: { judge: 1.18, embedding: 0.005, total: 1.185, estimate_note: 'approx' },
|
||||
cache: { hits: 87, misses: 213, hit_rate: 0.29 },
|
||||
duration_ms: 45000,
|
||||
source_tier_breakdown: { curated_vs_curated: 2, curated_vs_bulk: 11, bulk_vs_bulk: 5, other: 0 },
|
||||
per_query: [],
|
||||
hot_pages: [],
|
||||
...opts,
|
||||
};
|
||||
}
|
||||
|
||||
describe('E2E: eval_contradictions migrations applied cleanly', () => {
|
||||
test('eval_contradictions_cache and eval_contradictions_runs tables exist', async () => {
|
||||
if (!engine) return;
|
||||
const rows = await engine.executeRaw<{ table_name: string }>(
|
||||
`SELECT table_name FROM information_schema.tables
|
||||
WHERE table_schema = 'public'
|
||||
AND table_name IN ('eval_contradictions_cache', 'eval_contradictions_runs')
|
||||
ORDER BY table_name`,
|
||||
);
|
||||
expect(rows.length).toBe(2);
|
||||
expect(rows[0].table_name).toBe('eval_contradictions_cache');
|
||||
expect(rows[1].table_name).toBe('eval_contradictions_runs');
|
||||
});
|
||||
|
||||
test('eval_contradictions_runs has Wilson CI columns', async () => {
|
||||
if (!engine) return;
|
||||
const cols = await engine.executeRaw<{ column_name: string; data_type: string }>(
|
||||
`SELECT column_name, data_type FROM information_schema.columns
|
||||
WHERE table_name = 'eval_contradictions_runs'
|
||||
AND column_name IN ('wilson_ci_lower', 'wilson_ci_upper')
|
||||
ORDER BY column_name`,
|
||||
);
|
||||
expect(cols.length).toBe(2);
|
||||
expect(cols[0].data_type).toBe('real');
|
||||
expect(cols[1].data_type).toBe('real');
|
||||
});
|
||||
|
||||
test('eval_contradictions_cache composite PK includes prompt_version + truncation_policy', async () => {
|
||||
if (!engine) return;
|
||||
const cols = await engine.executeRaw<{ column_name: string }>(
|
||||
`SELECT a.attname AS column_name
|
||||
FROM pg_index i
|
||||
JOIN pg_class c ON c.oid = i.indrelid
|
||||
JOIN pg_attribute a ON a.attrelid = c.oid AND a.attnum = ANY(i.indkey)
|
||||
WHERE i.indisprimary AND c.relname = 'eval_contradictions_cache'
|
||||
ORDER BY a.attname`,
|
||||
);
|
||||
const names = cols.map((c) => c.column_name);
|
||||
expect(names).toContain('prompt_version');
|
||||
expect(names).toContain('truncation_policy');
|
||||
expect(names).toContain('chunk_a_hash');
|
||||
expect(names).toContain('chunk_b_hash');
|
||||
expect(names).toContain('model_id');
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E: JSONB round-trip on Postgres (regression class)', () => {
|
||||
test('writeContradictionsRun → loadTrend preserves nested objects, not strings', async () => {
|
||||
if (!engine) return;
|
||||
await writeRunRow(engine, mkReport({
|
||||
run_id: 'jsonb-1',
|
||||
source_tier_breakdown: { curated_vs_curated: 7, curated_vs_bulk: 8, bulk_vs_bulk: 9, other: 0 },
|
||||
}), 100);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows.length).toBe(1);
|
||||
// The classic v0.12 double-encode bug stores '{"curated_vs_curated":7,...}' as a string.
|
||||
// We must see a parsed object, not a string.
|
||||
expect(typeof rows[0].source_tier_breakdown).toBe('object');
|
||||
expect(rows[0].source_tier_breakdown.curated_vs_curated).toBe(7);
|
||||
expect(rows[0].source_tier_breakdown.curated_vs_bulk).toBe(8);
|
||||
expect(typeof rows[0].report_json).toBe('object');
|
||||
expect(rows[0].report_json.schema_version).toBe(1);
|
||||
});
|
||||
|
||||
test('postgres jsonb_typeof confirms object shape (defense in depth)', async () => {
|
||||
if (!engine) return;
|
||||
await writeRunRow(engine, mkReport({ run_id: 'jsonb-2' }), 100);
|
||||
const rows = await engine.executeRaw<{ kind: string }>(
|
||||
`SELECT jsonb_typeof(source_tier_breakdown) AS kind
|
||||
FROM eval_contradictions_runs
|
||||
WHERE run_id = 'jsonb-2'`,
|
||||
);
|
||||
expect(rows[0].kind).toBe('object');
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E: P2 persistent cache with real now()', () => {
|
||||
test('lookup returns null for missing key, upsert + lookup round-trips', async () => {
|
||||
if (!engine) return;
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test' });
|
||||
expect(await cache.lookup('text-a', 'text-b')).toBeNull();
|
||||
await cache.store('text-a', 'text-b', {
|
||||
contradicts: true, severity: 'high', axis: 'pg-test', confidence: 0.9, resolution_kind: 'dream_synthesize',
|
||||
});
|
||||
const hit = await cache.lookup('text-a', 'text-b');
|
||||
expect(hit).not.toBeNull();
|
||||
expect(hit?.contradicts).toBe(true);
|
||||
expect(hit?.severity).toBe('high');
|
||||
});
|
||||
|
||||
test('expired rows hidden from lookup; sweepContradictionCache deletes', async () => {
|
||||
if (!engine) return;
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-pg-test', ttlSeconds: 60 });
|
||||
await cache.store('expire-me-a', 'expire-me-b', {
|
||||
contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null,
|
||||
});
|
||||
// Backdate expires_at by 1 second.
|
||||
const key = buildCacheKey({ textA: 'expire-me-a', textB: 'expire-me-b', modelId: 'haiku-pg-test' });
|
||||
await engine.executeRaw(
|
||||
`UPDATE eval_contradictions_cache
|
||||
SET expires_at = now() - interval '1 second'
|
||||
WHERE chunk_a_hash = $1 AND chunk_b_hash = $2`,
|
||||
[key.chunk_a_hash, key.chunk_b_hash],
|
||||
);
|
||||
expect(await cache.lookup('expire-me-a', 'expire-me-b')).toBeNull();
|
||||
const swept = await engine.sweepContradictionCache();
|
||||
expect(swept).toBeGreaterThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('different prompt_version is a different cache key (Codex fix)', async () => {
|
||||
if (!engine) return;
|
||||
const cache1 = new JudgeCache({ engine, modelId: 'haiku-pg-test' });
|
||||
await cache1.store('shared-a', 'shared-b', {
|
||||
contradicts: true, severity: 'medium', axis: '', confidence: 0.85, resolution_kind: 'manual_review',
|
||||
});
|
||||
// Direct engine call with a different prompt_version should miss.
|
||||
const wrong = await engine.getContradictionCacheEntry({
|
||||
chunk_a_hash: buildCacheKey({ textA: 'shared-a', textB: 'shared-b', modelId: 'haiku-pg-test' }).chunk_a_hash,
|
||||
chunk_b_hash: buildCacheKey({ textA: 'shared-a', textB: 'shared-b', modelId: 'haiku-pg-test' }).chunk_b_hash,
|
||||
model_id: 'haiku-pg-test',
|
||||
prompt_version: 'OTHER-VERSION',
|
||||
truncation_policy: '1500-chars-utf8-safe',
|
||||
});
|
||||
expect(wrong).toBeNull();
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E: M5 trend semantics on Postgres', () => {
|
||||
test('trend ordered newest first with TIMESTAMPTZ', async () => {
|
||||
if (!engine) return;
|
||||
await writeRunRow(engine, mkReport({ run_id: 'older' }), 100);
|
||||
// Add a small delay so the second row gets a strictly-later now().
|
||||
await new Promise((r) => setTimeout(r, 50));
|
||||
await writeRunRow(engine, mkReport({ run_id: 'newer' }), 100);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows[0].run_id).toBe('newer');
|
||||
expect(rows[1].run_id).toBe('older');
|
||||
});
|
||||
|
||||
test('days window filters via ran_at >= cutoff', async () => {
|
||||
if (!engine) return;
|
||||
await writeRunRow(engine, mkReport({ run_id: 'recent' }), 100);
|
||||
// Backdate one row to 10 days ago.
|
||||
await engine.executeRaw(
|
||||
`UPDATE eval_contradictions_runs SET ran_at = now() - interval '10 days' WHERE run_id = $1`,
|
||||
['recent'],
|
||||
);
|
||||
const oneDayRows = await loadTrend(engine, 1);
|
||||
expect(oneDayRows.length).toBe(0);
|
||||
const fifteenDayRows = await loadTrend(engine, 15);
|
||||
expect(fifteenDayRows.length).toBe(1);
|
||||
});
|
||||
});
|
||||
|
||||
describe('E2E: find_contradictions MCP op on Postgres', () => {
|
||||
test('returns "no probe runs" note on empty table', async () => {
|
||||
if (!engine) return;
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: {} as OperationContext['config'],
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} } as unknown as OperationContext['logger'],
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
};
|
||||
const result = await op.handler(ctx, {}) as { contradictions: unknown[]; note?: string };
|
||||
expect(result.contradictions).toEqual([]);
|
||||
expect(result.note).toContain('No probe runs');
|
||||
});
|
||||
|
||||
test('returns latest run findings with slug+severity filters', async () => {
|
||||
if (!engine) return;
|
||||
await writeRunRow(engine, mkReport({
|
||||
run_id: 'pg-mcp',
|
||||
per_query: [{
|
||||
query: 'q',
|
||||
result_count: 5,
|
||||
pairs_skipped_by_date: 0,
|
||||
pairs_cache_hit: 0,
|
||||
pairs_judged: 3,
|
||||
contradictions: [
|
||||
{
|
||||
kind: 'cross_slug_chunks',
|
||||
a: { slug: 'companies/acme-example', chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
|
||||
b: { slug: 'openclaw/chat/x', chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' },
|
||||
combined_score: 1.5,
|
||||
severity: 'high',
|
||||
axis: 'MRR figure',
|
||||
confidence: 0.9,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
resolution_command: 'gbrain dream --phase synthesize --slug companies/acme-example',
|
||||
},
|
||||
{
|
||||
kind: 'cross_slug_chunks',
|
||||
a: { slug: 'people/alice-example', chunk_id: 3, take_id: null, source_tier: 'curated', holder: null, text: 'c' },
|
||||
b: { slug: 'people/alice-smith-example', chunk_id: 4, take_id: null, source_tier: 'curated', holder: null, text: 'd' },
|
||||
combined_score: 1.2,
|
||||
severity: 'low',
|
||||
axis: 'name format',
|
||||
confidence: 0.75,
|
||||
resolution_kind: 'manual_review',
|
||||
resolution_command: 'gbrain takes mark-debate people/alice-example --row 1',
|
||||
},
|
||||
],
|
||||
}],
|
||||
}), 100);
|
||||
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const ctx: OperationContext = {
|
||||
engine,
|
||||
config: {} as OperationContext['config'],
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} } as unknown as OperationContext['logger'],
|
||||
dryRun: false,
|
||||
remote: true,
|
||||
};
|
||||
|
||||
const all = await op.handler(ctx, {}) as { contradictions: unknown[]; total_in_run: number };
|
||||
expect(all.contradictions.length).toBe(2);
|
||||
expect(all.total_in_run).toBe(2);
|
||||
|
||||
const highOnly = await op.handler(ctx, { severity: 'high' }) as { contradictions: Array<{ severity: string }> };
|
||||
expect(highOnly.contradictions.length).toBe(1);
|
||||
expect(highOnly.contradictions[0].severity).toBe('high');
|
||||
|
||||
const slugFiltered = await op.handler(ctx, { slug: 'acme' }) as { contradictions: unknown[] };
|
||||
expect(slugFiltered.contradictions.length).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,139 @@
|
||||
/**
|
||||
* M7 auto-supersession proposal generator tests.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
classifyResolution,
|
||||
pairToFinding,
|
||||
proposeResolution,
|
||||
renderResolutionCommand,
|
||||
} from '../src/core/eval-contradictions/auto-supersession.ts';
|
||||
import type {
|
||||
ContradictionPair,
|
||||
JudgeVerdict,
|
||||
} from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
function mkCrossSlugPair(slugA: string, slugB: string): ContradictionPair {
|
||||
return {
|
||||
kind: 'cross_slug_chunks',
|
||||
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
|
||||
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'bulk', holder: null, text: 'b' },
|
||||
combined_score: 1,
|
||||
};
|
||||
}
|
||||
|
||||
function mkIntraPagePair(pageSlug: string, takeId: number): ContradictionPair {
|
||||
return {
|
||||
kind: 'intra_page_chunk_take',
|
||||
a: { slug: pageSlug, chunk_id: 5, take_id: null, source_tier: 'curated', holder: null, text: 'chunk text' },
|
||||
b: { slug: pageSlug, chunk_id: null, take_id: takeId, source_tier: 'curated', holder: 'garry', text: 'take claim' },
|
||||
combined_score: 1,
|
||||
};
|
||||
}
|
||||
|
||||
describe('classifyResolution', () => {
|
||||
test('intra_page pair → takes_supersede when take_id present', () => {
|
||||
const pair = mkIntraPagePair('people/alice', 42);
|
||||
expect(classifyResolution(pair, null)).toBe('takes_supersede');
|
||||
});
|
||||
|
||||
test('cross_slug + judge hint dream_synthesize → honored', () => {
|
||||
const pair = mkCrossSlugPair('companies/acme', 'openclaw/chat/x');
|
||||
expect(classifyResolution(pair, 'dream_synthesize')).toBe('dream_synthesize');
|
||||
});
|
||||
|
||||
test('cross_slug + judge hint takes_mark_debate → honored', () => {
|
||||
const pair = mkCrossSlugPair('originals/talk', 'writing/essay');
|
||||
expect(classifyResolution(pair, 'takes_mark_debate')).toBe('takes_mark_debate');
|
||||
});
|
||||
|
||||
test('cross_slug + no judge hint + curated entity → dream_synthesize fallback', () => {
|
||||
const pair = mkCrossSlugPair('companies/acme', 'openclaw/chat/x');
|
||||
expect(classifyResolution(pair, null)).toBe('dream_synthesize');
|
||||
const pair2 = mkCrossSlugPair('people/alice', 'daily/2026-05-01');
|
||||
expect(classifyResolution(pair2, null)).toBe('dream_synthesize');
|
||||
});
|
||||
|
||||
test('cross_slug + neither side is curated entity → manual_review', () => {
|
||||
const pair = mkCrossSlugPair('daily/x', 'openclaw/chat/y');
|
||||
expect(classifyResolution(pair, null)).toBe('manual_review');
|
||||
});
|
||||
|
||||
test('cross_slug + judge hint manual_review honored', () => {
|
||||
const pair = mkCrossSlugPair('companies/acme', 'openclaw/chat/x');
|
||||
expect(classifyResolution(pair, 'manual_review')).toBe('manual_review');
|
||||
});
|
||||
});
|
||||
|
||||
describe('renderResolutionCommand', () => {
|
||||
test('takes_supersede emits gbrain takes supersede with row id', () => {
|
||||
const pair = mkIntraPagePair('people/alice', 7);
|
||||
const cmd = renderResolutionCommand(pair, 'takes_supersede');
|
||||
expect(cmd).toBe('gbrain takes supersede people/alice --row 7');
|
||||
});
|
||||
|
||||
test('dream_synthesize targets the curated entity side', () => {
|
||||
const pair = mkCrossSlugPair('openclaw/chat/x', 'companies/acme');
|
||||
const cmd = renderResolutionCommand(pair, 'dream_synthesize');
|
||||
expect(cmd).toBe('gbrain dream --phase synthesize --slug companies/acme');
|
||||
});
|
||||
|
||||
test('takes_mark_debate emits mark-debate with row id', () => {
|
||||
const pair = mkIntraPagePair('people/alice', 12);
|
||||
const cmd = renderResolutionCommand(pair, 'takes_mark_debate');
|
||||
expect(cmd).toBe('gbrain takes mark-debate people/alice --row 12');
|
||||
});
|
||||
|
||||
test('manual_review emits a no-op comment naming both slugs', () => {
|
||||
const pair = mkCrossSlugPair('daily/x', 'openclaw/chat/y');
|
||||
const cmd = renderResolutionCommand(pair, 'manual_review');
|
||||
expect(cmd).toContain('manual review');
|
||||
expect(cmd).toContain('daily/x');
|
||||
expect(cmd).toContain('openclaw/chat/y');
|
||||
});
|
||||
|
||||
test('takes_supersede with missing take_id falls back to row placeholder', () => {
|
||||
const pair = mkCrossSlugPair('companies/acme', 'people/alice');
|
||||
const cmd = renderResolutionCommand(pair, 'takes_supersede');
|
||||
expect(cmd).toContain('<row>');
|
||||
});
|
||||
});
|
||||
|
||||
describe('proposeResolution (classify + render combined)', () => {
|
||||
test('intra_page → takes_supersede with paste-ready command', () => {
|
||||
const pair = mkIntraPagePair('people/alice', 42);
|
||||
const p = proposeResolution(pair, null);
|
||||
expect(p.resolution_kind).toBe('takes_supersede');
|
||||
expect(p.resolution_command).toBe('gbrain takes supersede people/alice --row 42');
|
||||
});
|
||||
|
||||
test('cross_slug curated → dream_synthesize on curated slug', () => {
|
||||
const pair = mkCrossSlugPair('openclaw/chat/foo', 'companies/acme');
|
||||
const p = proposeResolution(pair, null);
|
||||
expect(p.resolution_kind).toBe('dream_synthesize');
|
||||
expect(p.resolution_command).toBe('gbrain dream --phase synthesize --slug companies/acme');
|
||||
});
|
||||
});
|
||||
|
||||
describe('pairToFinding', () => {
|
||||
test('merges pair + verdict into a finding', () => {
|
||||
const pair = mkIntraPagePair('people/alice', 7);
|
||||
const verdict: JudgeVerdict = {
|
||||
contradicts: true,
|
||||
severity: 'high',
|
||||
axis: 'CFO role status',
|
||||
confidence: 0.92,
|
||||
resolution_kind: 'takes_supersede',
|
||||
};
|
||||
const finding = pairToFinding(pair, verdict);
|
||||
expect(finding.severity).toBe('high');
|
||||
expect(finding.axis).toBe('CFO role status');
|
||||
expect(finding.confidence).toBe(0.92);
|
||||
expect(finding.resolution_kind).toBe('takes_supersede');
|
||||
expect(finding.resolution_command).toContain('gbrain takes supersede');
|
||||
expect(finding.kind).toBe(pair.kind);
|
||||
expect(finding.a).toEqual(pair.a);
|
||||
expect(finding.b).toEqual(pair.b);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,145 @@
|
||||
/**
|
||||
* Cache wrapper tests — P2 with prompt-version + truncation in the key
|
||||
* (Codex fix). Hits the real PGLite engine end-to-end.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
buildCacheKey,
|
||||
hashContent,
|
||||
JudgeCache,
|
||||
} from '../src/core/eval-contradictions/cache.ts';
|
||||
import { PROMPT_VERSION, TRUNCATION_POLICY } from '../src/core/eval-contradictions/types.ts';
|
||||
import type { JudgeVerdict } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
const verdictHit: JudgeVerdict = {
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'MRR vs ARR',
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
};
|
||||
|
||||
describe('hashContent', () => {
|
||||
test('produces stable 64-char hex sha256', () => {
|
||||
const h = hashContent('hello');
|
||||
expect(h.length).toBe(64);
|
||||
expect(h).toMatch(/^[0-9a-f]+$/);
|
||||
});
|
||||
|
||||
test('different inputs hash differently', () => {
|
||||
expect(hashContent('a')).not.toBe(hashContent('b'));
|
||||
});
|
||||
|
||||
test('same input stable across calls', () => {
|
||||
expect(hashContent('test')).toBe(hashContent('test'));
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildCacheKey', () => {
|
||||
test('sorts hashes lex so (a, b) === (b, a)', () => {
|
||||
const k1 = buildCacheKey({ textA: 'first', textB: 'second', modelId: 'haiku' });
|
||||
const k2 = buildCacheKey({ textA: 'second', textB: 'first', modelId: 'haiku' });
|
||||
expect(k1).toEqual(k2);
|
||||
});
|
||||
|
||||
test('includes the prompt_version + truncation_policy constants', () => {
|
||||
const k = buildCacheKey({ textA: 'x', textB: 'y', modelId: 'haiku' });
|
||||
expect(k.prompt_version).toBe(PROMPT_VERSION);
|
||||
expect(k.truncation_policy).toBe(TRUNCATION_POLICY);
|
||||
});
|
||||
|
||||
test('model_id pass-through', () => {
|
||||
const k = buildCacheKey({ textA: 'x', textB: 'y', modelId: 'sonnet' });
|
||||
expect(k.model_id).toBe('sonnet');
|
||||
});
|
||||
});
|
||||
|
||||
describe('JudgeCache wrapper', () => {
|
||||
test('miss returns null and increments misses', async () => {
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
const hit = await cache.lookup('text-a', 'text-b');
|
||||
expect(hit).toBeNull();
|
||||
expect(cache.stats().misses).toBe(1);
|
||||
expect(cache.stats().hits).toBe(0);
|
||||
});
|
||||
|
||||
test('store then lookup returns the verdict', async () => {
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
await cache.store('text-a', 'text-b', verdictHit);
|
||||
const hit = await cache.lookup('text-a', 'text-b');
|
||||
expect(hit).not.toBeNull();
|
||||
expect(hit?.contradicts).toBe(true);
|
||||
expect(hit?.severity).toBe('medium');
|
||||
expect(cache.stats().hits).toBe(1);
|
||||
});
|
||||
|
||||
test('order-independence: (a,b) and (b,a) both hit', async () => {
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
await cache.store('first', 'second', verdictHit);
|
||||
const hit = await cache.lookup('second', 'first');
|
||||
expect(hit).not.toBeNull();
|
||||
expect(hit?.contradicts).toBe(true);
|
||||
});
|
||||
|
||||
test('different model_id is a separate key', async () => {
|
||||
const cache1 = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
const cache2 = new JudgeCache({ engine, modelId: 'sonnet-test' });
|
||||
await cache1.store('a', 'b', verdictHit);
|
||||
const hit = await cache2.lookup('a', 'b');
|
||||
expect(hit).toBeNull();
|
||||
});
|
||||
|
||||
test('disabled cache always misses, never stores', async () => {
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test', disabled: true });
|
||||
await cache.store('a', 'b', verdictHit);
|
||||
const hit = await cache.lookup('a', 'b');
|
||||
expect(hit).toBeNull();
|
||||
// And a fresh non-disabled cache should also see nothing (store was a no-op).
|
||||
const fresh = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
expect(await fresh.lookup('a', 'b')).toBeNull();
|
||||
});
|
||||
|
||||
test('hit_rate computed correctly', async () => {
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
await cache.store('x', 'y', verdictHit);
|
||||
await cache.lookup('x', 'y'); // hit
|
||||
await cache.lookup('m', 'n'); // miss
|
||||
await cache.lookup('p', 'q'); // miss
|
||||
const s = cache.stats();
|
||||
expect(s.hits).toBe(1);
|
||||
expect(s.misses).toBe(2);
|
||||
expect(s.hit_rate).toBeCloseTo(1 / 3, 5);
|
||||
});
|
||||
|
||||
test('lookup defends against corrupt cache rows (shape validation)', async () => {
|
||||
// Inject a row directly that doesn't match JudgeVerdict shape.
|
||||
const key = buildCacheKey({ textA: 'corrupt-a', textB: 'corrupt-b', modelId: 'haiku-test' });
|
||||
await engine.putContradictionCacheEntry({
|
||||
...key,
|
||||
verdict: { unrelated_field: 'whatever' },
|
||||
});
|
||||
const cache = new JudgeCache({ engine, modelId: 'haiku-test' });
|
||||
const hit = await cache.lookup('corrupt-a', 'corrupt-b');
|
||||
expect(hit).toBeNull();
|
||||
expect(cache.stats().misses).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,95 @@
|
||||
/**
|
||||
* Calibration tests — Wilson CI and small-sample annotation.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import { buildCalibration, wilsonCI } from '../src/core/eval-contradictions/calibration.ts';
|
||||
|
||||
describe('wilsonCI', () => {
|
||||
test('zero denominator returns all zeros', () => {
|
||||
expect(wilsonCI(0, 0)).toEqual({ point: 0, lower: 0, upper: 0 });
|
||||
});
|
||||
|
||||
test('half-and-half on n=100 brackets 0.5', () => {
|
||||
const ci = wilsonCI(50, 100);
|
||||
expect(ci.point).toBe(0.5);
|
||||
expect(ci.lower).toBeGreaterThan(0.39);
|
||||
expect(ci.lower).toBeLessThan(0.41);
|
||||
expect(ci.upper).toBeGreaterThan(0.59);
|
||||
expect(ci.upper).toBeLessThan(0.61);
|
||||
});
|
||||
|
||||
test('zero successes on n=20 gives a useful upper bound', () => {
|
||||
const ci = wilsonCI(0, 20);
|
||||
expect(ci.point).toBe(0);
|
||||
expect(ci.lower).toBe(0);
|
||||
// 95% upper bound for 0/20 is ~16%.
|
||||
expect(ci.upper).toBeGreaterThan(0.1);
|
||||
expect(ci.upper).toBeLessThan(0.2);
|
||||
});
|
||||
|
||||
test('all successes on n=10 gives a non-zero lower bound', () => {
|
||||
const ci = wilsonCI(10, 10);
|
||||
expect(ci.point).toBe(1);
|
||||
expect(ci.upper).toBe(1);
|
||||
expect(ci.lower).toBeGreaterThan(0.6);
|
||||
});
|
||||
|
||||
test('clamps numerator above denominator', () => {
|
||||
const ci = wilsonCI(15, 10);
|
||||
expect(ci.point).toBe(1);
|
||||
});
|
||||
|
||||
test('clamps negative numerator to zero', () => {
|
||||
const ci = wilsonCI(-3, 10);
|
||||
expect(ci.point).toBe(0);
|
||||
});
|
||||
|
||||
test('typical 12/50 surfaces a roughly 14-37 band', () => {
|
||||
const ci = wilsonCI(12, 50);
|
||||
expect(ci.point).toBeCloseTo(0.24, 2);
|
||||
expect(ci.lower).toBeGreaterThan(0.13);
|
||||
expect(ci.lower).toBeLessThan(0.16);
|
||||
expect(ci.upper).toBeGreaterThan(0.35);
|
||||
expect(ci.upper).toBeLessThan(0.39);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildCalibration', () => {
|
||||
test('emits small_sample_note when queriesTotal < 30', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 10, queriesWithContradiction: 2 });
|
||||
expect(cal.small_sample_note).toBeTruthy();
|
||||
expect(cal.small_sample_note).toContain('n=10');
|
||||
});
|
||||
|
||||
test('omits small_sample_note for n >= 30', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 50, queriesWithContradiction: 12 });
|
||||
expect(cal.small_sample_note).toBeUndefined();
|
||||
});
|
||||
|
||||
test('queries_judged_clean is total minus contradictions', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 50, queriesWithContradiction: 12 });
|
||||
expect(cal.queries_judged_clean).toBe(38);
|
||||
});
|
||||
|
||||
test('handles all-contradiction case', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 5, queriesWithContradiction: 5 });
|
||||
expect(cal.queries_judged_clean).toBe(0);
|
||||
expect(cal.wilson_ci_95.point).toBe(1);
|
||||
});
|
||||
|
||||
test('handles zero-contradiction case', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 50, queriesWithContradiction: 0 });
|
||||
expect(cal.wilson_ci_95.point).toBe(0);
|
||||
expect(cal.wilson_ci_95.lower).toBe(0);
|
||||
expect(cal.wilson_ci_95.upper).toBeGreaterThan(0);
|
||||
expect(cal.wilson_ci_95.upper).toBeLessThan(0.1);
|
||||
});
|
||||
|
||||
test('handles zero queries gracefully', () => {
|
||||
const cal = buildCalibration({ queriesTotal: 0, queriesWithContradiction: 0 });
|
||||
expect(cal.queries_judged_clean).toBe(0);
|
||||
expect(cal.wilson_ci_95).toEqual({ point: 0, lower: 0, upper: 0 });
|
||||
expect(cal.small_sample_note).toBeTruthy();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,134 @@
|
||||
/**
|
||||
* Cost tracker tests — pre-flight estimator + mid-run cumulative + cap behavior.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
CostTracker,
|
||||
estimateUpperBoundCost,
|
||||
} from '../src/core/eval-contradictions/cost-tracker.ts';
|
||||
|
||||
describe('estimateUpperBoundCost', () => {
|
||||
test('zero pairs and zero queries → ~zero cost', () => {
|
||||
const c = estimateUpperBoundCost({
|
||||
pairCount: 0,
|
||||
queryCount: 0,
|
||||
judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
expect(c).toBe(0);
|
||||
});
|
||||
|
||||
test('haiku is cheaper than sonnet for the same pair count', () => {
|
||||
const haiku = estimateUpperBoundCost({
|
||||
pairCount: 100,
|
||||
queryCount: 10,
|
||||
judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
const sonnet = estimateUpperBoundCost({
|
||||
pairCount: 100,
|
||||
queryCount: 10,
|
||||
judgeModel: 'claude-sonnet-4-6',
|
||||
});
|
||||
expect(sonnet).toBeGreaterThan(haiku);
|
||||
});
|
||||
|
||||
test('scales linearly in pair count', () => {
|
||||
const c50 = estimateUpperBoundCost({
|
||||
pairCount: 50, queryCount: 0, judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
const c100 = estimateUpperBoundCost({
|
||||
pairCount: 100, queryCount: 0, judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
expect(c100).toBeCloseTo(c50 * 2, 8);
|
||||
});
|
||||
|
||||
test('embedding cost included even with zero pairs', () => {
|
||||
const c = estimateUpperBoundCost({
|
||||
pairCount: 0,
|
||||
queryCount: 1000,
|
||||
judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
expect(c).toBeGreaterThan(0);
|
||||
expect(c).toBeLessThan(0.01); // 1000 queries × 50 tok × $0.13/Mtok = ~$0.0065
|
||||
});
|
||||
|
||||
test('unknown model falls back to haiku pricing', () => {
|
||||
const c1 = estimateUpperBoundCost({
|
||||
pairCount: 100, queryCount: 0, judgeModel: 'made-up-model',
|
||||
});
|
||||
const c2 = estimateUpperBoundCost({
|
||||
pairCount: 100, queryCount: 0, judgeModel: 'claude-haiku-4-5',
|
||||
});
|
||||
expect(c1).toBeCloseTo(c2, 8);
|
||||
});
|
||||
});
|
||||
|
||||
describe('CostTracker', () => {
|
||||
test('fresh tracker has zero totals', () => {
|
||||
const t = new CostTracker({ capUsd: 5 });
|
||||
expect(t.judge()).toBe(0);
|
||||
expect(t.embedding()).toBe(0);
|
||||
expect(t.total()).toBe(0);
|
||||
expect(t.exceededCap()).toBe(false);
|
||||
});
|
||||
|
||||
test('records judge calls cumulatively', () => {
|
||||
const t = new CostTracker({ capUsd: 5 });
|
||||
t.recordJudgeCall('claude-haiku-4-5', { inputTokens: 500, outputTokens: 80 });
|
||||
const after1 = t.judge();
|
||||
t.recordJudgeCall('claude-haiku-4-5', { inputTokens: 500, outputTokens: 80 });
|
||||
expect(t.judge()).toBeCloseTo(after1 * 2, 8);
|
||||
});
|
||||
|
||||
test('records embedding cost separately', () => {
|
||||
const t = new CostTracker({ capUsd: 5 });
|
||||
t.recordEmbeddingCall(1000);
|
||||
expect(t.judge()).toBe(0);
|
||||
expect(t.embedding()).toBeGreaterThan(0);
|
||||
});
|
||||
|
||||
test('exceededCap fires when cumulative crosses budget', () => {
|
||||
const t = new CostTracker({ capUsd: 0.001 });
|
||||
// 100 haiku calls easily exceeds $0.001
|
||||
for (let i = 0; i < 100; i++) {
|
||||
t.recordJudgeCall('claude-haiku-4-5', { inputTokens: 500, outputTokens: 80 });
|
||||
}
|
||||
expect(t.exceededCap()).toBe(true);
|
||||
});
|
||||
|
||||
test('finalize includes estimate_note explaining soft-ceiling semantics', () => {
|
||||
const t = new CostTracker({ capUsd: 5 });
|
||||
const out = t.finalize();
|
||||
expect(out.estimate_note).toContain('approximate');
|
||||
expect(out.estimate_note).toContain('soft ceiling');
|
||||
});
|
||||
|
||||
test('finalize sums judge + embedding into total', () => {
|
||||
const t = new CostTracker({ capUsd: 5 });
|
||||
t.recordJudgeCall('claude-haiku-4-5', { inputTokens: 1000, outputTokens: 100 });
|
||||
t.recordEmbeddingCall(500);
|
||||
const out = t.finalize();
|
||||
expect(out.total).toBeCloseTo(out.judge + out.embedding, 6);
|
||||
});
|
||||
|
||||
test('zero cap exceededCap fires on any spend', () => {
|
||||
const t = new CostTracker({ capUsd: 0 });
|
||||
expect(t.exceededCap()).toBe(false);
|
||||
t.recordEmbeddingCall(10);
|
||||
expect(t.exceededCap()).toBe(true);
|
||||
});
|
||||
|
||||
test('negative cap clamped to zero', () => {
|
||||
const t = new CostTracker({ capUsd: -5 });
|
||||
expect(t.capUsd()).toBe(0);
|
||||
});
|
||||
|
||||
test('values are rounded to 6 decimals in the breakdown', () => {
|
||||
const t = new CostTracker({ capUsd: 100 });
|
||||
t.recordJudgeCall('claude-haiku-4-5', { inputTokens: 1, outputTokens: 1 });
|
||||
const out = t.finalize();
|
||||
// 6e-6 rounded ok; no scientific-notation tail in JSON output expected.
|
||||
expect(Number.isFinite(out.judge)).toBe(true);
|
||||
expect(Number.isFinite(out.total)).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,119 @@
|
||||
/**
|
||||
* Cross-source tier breakdown tests (M6).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
buildSourceTierBreakdown,
|
||||
classifySlugTier,
|
||||
} from '../src/core/eval-contradictions/cross-source.ts';
|
||||
import type { ContradictionPair } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
function mkPair(slugA: string, slugB: string): ContradictionPair {
|
||||
return {
|
||||
kind: 'cross_slug_chunks',
|
||||
a: { slug: slugA, chunk_id: 1, take_id: null, source_tier: 'curated', holder: null, text: 'a' },
|
||||
b: { slug: slugB, chunk_id: 2, take_id: null, source_tier: 'curated', holder: null, text: 'b' },
|
||||
combined_score: 1,
|
||||
};
|
||||
}
|
||||
|
||||
describe('classifySlugTier', () => {
|
||||
test('curated prefixes (boost > 1.0)', () => {
|
||||
expect(classifySlugTier('originals/talks/foo')).toBe('curated');
|
||||
expect(classifySlugTier('concepts/widget')).toBe('curated');
|
||||
expect(classifySlugTier('writing/essay-1')).toBe('curated');
|
||||
expect(classifySlugTier('people/alice')).toBe('curated');
|
||||
expect(classifySlugTier('companies/acme')).toBe('curated');
|
||||
});
|
||||
|
||||
test('bulk prefixes (boost < 1.0)', () => {
|
||||
expect(classifySlugTier('daily/2026-05-10')).toBe('bulk');
|
||||
expect(classifySlugTier('media/x/post-123')).toBe('bulk');
|
||||
expect(classifySlugTier('openclaw/chat/session-1')).toBe('bulk');
|
||||
});
|
||||
|
||||
test('baseline prefixes map to other (boost = 1.0)', () => {
|
||||
expect(classifySlugTier('yc/something')).toBe('other');
|
||||
expect(classifySlugTier('civic/whatever')).toBe('other');
|
||||
});
|
||||
|
||||
test('unknown prefix maps to other', () => {
|
||||
expect(classifySlugTier('made-up-prefix/page')).toBe('other');
|
||||
expect(classifySlugTier('random/thing')).toBe('other');
|
||||
});
|
||||
|
||||
test('empty slug maps to other', () => {
|
||||
expect(classifySlugTier('')).toBe('other');
|
||||
});
|
||||
|
||||
test('longest-prefix-match wins', () => {
|
||||
// media/articles/ is curated (1.1), media/x/ is bulk (0.7).
|
||||
expect(classifySlugTier('media/articles/foo')).toBe('curated');
|
||||
expect(classifySlugTier('media/x/bar')).toBe('bulk');
|
||||
});
|
||||
|
||||
test('case-insensitive', () => {
|
||||
expect(classifySlugTier('Originals/Talks/Foo')).toBe('curated');
|
||||
expect(classifySlugTier('OPENCLAW/CHAT/whatever')).toBe('bulk');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildSourceTierBreakdown', () => {
|
||||
test('empty input yields all zeros', () => {
|
||||
const out = buildSourceTierBreakdown([]);
|
||||
expect(out).toEqual({
|
||||
curated_vs_curated: 0,
|
||||
curated_vs_bulk: 0,
|
||||
bulk_vs_bulk: 0,
|
||||
other: 0,
|
||||
});
|
||||
});
|
||||
|
||||
test('curated_vs_curated counts both-curated pairs', () => {
|
||||
const out = buildSourceTierBreakdown([
|
||||
mkPair('originals/a', 'concepts/b'),
|
||||
mkPair('people/x', 'companies/y'),
|
||||
]);
|
||||
expect(out.curated_vs_curated).toBe(2);
|
||||
expect(out.curated_vs_bulk).toBe(0);
|
||||
});
|
||||
|
||||
test('curated_vs_bulk counts mixed pairs (order-independent)', () => {
|
||||
const out = buildSourceTierBreakdown([
|
||||
mkPair('originals/a', 'daily/2026-05-10'),
|
||||
mkPair('openclaw/chat/session-1', 'people/alice'),
|
||||
]);
|
||||
expect(out.curated_vs_bulk).toBe(2);
|
||||
expect(out.curated_vs_curated).toBe(0);
|
||||
expect(out.bulk_vs_bulk).toBe(0);
|
||||
});
|
||||
|
||||
test('bulk_vs_bulk counts both-bulk pairs', () => {
|
||||
const out = buildSourceTierBreakdown([
|
||||
mkPair('daily/2026-05-10', 'openclaw/chat/session-1'),
|
||||
]);
|
||||
expect(out.bulk_vs_bulk).toBe(1);
|
||||
});
|
||||
|
||||
test('other catches unrecognized prefixes', () => {
|
||||
const out = buildSourceTierBreakdown([
|
||||
mkPair('random/foo', 'yc/bar'),
|
||||
mkPair('made-up/x', 'civic/y'),
|
||||
]);
|
||||
expect(out.other).toBe(2);
|
||||
});
|
||||
|
||||
test('mixed input correctly partitions', () => {
|
||||
const out = buildSourceTierBreakdown([
|
||||
mkPair('originals/a', 'concepts/b'), // curated_vs_curated
|
||||
mkPair('people/x', 'openclaw/chat/y'), // curated_vs_bulk
|
||||
mkPair('daily/x', 'media/x/y'), // bulk_vs_bulk
|
||||
mkPair('yc/x', 'civic/y'), // other (both baseline)
|
||||
]);
|
||||
expect(out.curated_vs_curated).toBe(1);
|
||||
expect(out.curated_vs_bulk).toBe(1);
|
||||
expect(out.bulk_vs_bulk).toBe(1);
|
||||
expect(out.other).toBe(1);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,142 @@
|
||||
/**
|
||||
* Date pre-filter tests — A1 three-rule pre-filter.
|
||||
*
|
||||
* Codex's critique: the naive "both have dates AND dates differ → skip" rule
|
||||
* would miss real contradictions. These tests pin the layered rules:
|
||||
* - Same-paragraph dual dates → DO NOT skip (flip-flop case).
|
||||
* - One side missing dates → DO NOT skip.
|
||||
* - Both sides explicit AND separated by >30 days → SKIP (the obvious case).
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
extractDates,
|
||||
hasSameParagraphDualDate,
|
||||
shouldSkipForDateMismatch,
|
||||
} from '../src/core/eval-contradictions/date-filter.ts';
|
||||
|
||||
describe('extractDates', () => {
|
||||
test('YYYY-MM-DD', () => {
|
||||
const dates = extractDates('On 2024-08-12 we shipped');
|
||||
expect(dates.length).toBe(1);
|
||||
expect(dates[0].getUTCFullYear()).toBe(2024);
|
||||
expect(dates[0].getUTCMonth()).toBe(7);
|
||||
expect(dates[0].getUTCDate()).toBe(12);
|
||||
});
|
||||
|
||||
test('YYYY/MM/DD', () => {
|
||||
const dates = extractDates('happened 2024/08/12');
|
||||
expect(dates.length).toBe(1);
|
||||
expect(dates[0].getUTCFullYear()).toBe(2024);
|
||||
});
|
||||
|
||||
test('quarter strings', () => {
|
||||
const dates = extractDates('shipped in Q2 2024');
|
||||
expect(dates.length).toBe(1);
|
||||
expect(dates[0].getUTCFullYear()).toBe(2024);
|
||||
});
|
||||
|
||||
test('bare year', () => {
|
||||
const dates = extractDates('back in 2024 we');
|
||||
expect(dates.length).toBe(1);
|
||||
expect(dates[0].getUTCFullYear()).toBe(2024);
|
||||
});
|
||||
|
||||
test('multiple dates', () => {
|
||||
const dates = extractDates('from 2024-01-15 to 2026-03-22');
|
||||
expect(dates.length).toBe(2);
|
||||
});
|
||||
|
||||
test('no dates returns empty array', () => {
|
||||
expect(extractDates('plain text no dates here')).toEqual([]);
|
||||
expect(extractDates('')).toEqual([]);
|
||||
});
|
||||
|
||||
test('rejects implausible years', () => {
|
||||
expect(extractDates('written in 1066')).toEqual([]);
|
||||
expect(extractDates('back in 3050')).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('hasSameParagraphDualDate', () => {
|
||||
test('two dates in one paragraph → true', () => {
|
||||
const text = 'In Jan 2024 I thought X.\nIn Mar 2024 I changed my mind to not-X.';
|
||||
expect(hasSameParagraphDualDate(text)).toBe(true);
|
||||
});
|
||||
|
||||
test('two dates in different paragraphs → false', () => {
|
||||
const text = 'In 2024 we did X.\n\nIn 2026 we did Y.';
|
||||
expect(hasSameParagraphDualDate(text)).toBe(false);
|
||||
});
|
||||
|
||||
test('only one date in text → false', () => {
|
||||
expect(hasSameParagraphDualDate('on 2024-01-15 we shipped')).toBe(false);
|
||||
});
|
||||
|
||||
test('two SAME dates in one paragraph → false (not distinct)', () => {
|
||||
expect(hasSameParagraphDualDate('on 2024-01-15 and also 2024-01-15')).toBe(false);
|
||||
});
|
||||
|
||||
test('empty text → false', () => {
|
||||
expect(hasSameParagraphDualDate('')).toBe(false);
|
||||
});
|
||||
});
|
||||
|
||||
describe('shouldSkipForDateMismatch', () => {
|
||||
test('both explicit + >30 days apart → SKIP (the obvious quarterly case)', () => {
|
||||
const d = shouldSkipForDateMismatch({
|
||||
textA: 'Acme MRR was $50K (2024-08-01)',
|
||||
textB: 'Acme MRR was $2M (2026-03-15)',
|
||||
});
|
||||
expect(d.skip).toBe(true);
|
||||
expect(d.reason).toBe('both_explicit_separated');
|
||||
});
|
||||
|
||||
test('both explicit + within 30 days → do NOT skip', () => {
|
||||
const d = shouldSkipForDateMismatch({
|
||||
textA: 'on 2024-08-01 Alice was CFO',
|
||||
textB: 'on 2024-08-20 Alice was not CFO',
|
||||
});
|
||||
expect(d.skip).toBe(false);
|
||||
expect(d.reason).toBe('overlapping_or_close');
|
||||
});
|
||||
|
||||
test('one side missing date → do NOT skip', () => {
|
||||
const d = shouldSkipForDateMismatch({
|
||||
textA: 'Alice is the CFO',
|
||||
textB: 'Alice left the company on 2026-03-01',
|
||||
});
|
||||
expect(d.skip).toBe(false);
|
||||
expect(d.reason).toBe('one_or_both_missing_dates');
|
||||
});
|
||||
|
||||
test('both sides missing dates → do NOT skip', () => {
|
||||
const d = shouldSkipForDateMismatch({
|
||||
textA: 'Acme is profitable',
|
||||
textB: 'Acme is unprofitable',
|
||||
});
|
||||
expect(d.skip).toBe(false);
|
||||
expect(d.reason).toBe('one_or_both_missing_dates');
|
||||
});
|
||||
|
||||
test('same-paragraph dual-date overrides separation rule', () => {
|
||||
// Even though the OTHER chunk has a date 100 days later, the same-paragraph
|
||||
// flip in A means we must NOT skip.
|
||||
const d = shouldSkipForDateMismatch({
|
||||
textA: 'In Jan 2024 I said X. In Mar 2024 I reversed to not-X.',
|
||||
textB: 'In 2026 I still hold not-X.',
|
||||
});
|
||||
expect(d.skip).toBe(false);
|
||||
expect(d.reason).toBe('same_paragraph_dual_date');
|
||||
});
|
||||
|
||||
test('regression: regex lastIndex reset between calls', () => {
|
||||
// The /g regex shares lastIndex across calls; if not reset, the second
|
||||
// call returns wrong results. This test pins the reset.
|
||||
const text = 'see 2024-01-15 and 2026-03-22 and 2025-06-10';
|
||||
const a = extractDates(text);
|
||||
const b = extractDates(text);
|
||||
expect(a.length).toBe(b.length);
|
||||
expect(a.length).toBe(3);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,252 @@
|
||||
/**
|
||||
* Engine method tests for v0.32.6 contradiction probe surfaces.
|
||||
*
|
||||
* Five new methods land on `BrainEngine`:
|
||||
* - listActiveTakesForPages (P1, batched per-page active-take fetch)
|
||||
* - writeContradictionsRun + loadContradictionsTrend (M5, time-series)
|
||||
* - getContradictionCacheEntry + putContradictionCacheEntry + sweepContradictionCache (P2)
|
||||
*
|
||||
* Hermetic against PGLite via the canonical block. The Postgres impls mirror
|
||||
* the same SQL; their parity is exercised in the E2E suite.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
async function seedPage(slug: string, title: string, body = ''): Promise<number> {
|
||||
const compiled = body || `body for ${slug}`;
|
||||
await engine.putPage(slug, {
|
||||
title,
|
||||
type: 'concept',
|
||||
frontmatter: {},
|
||||
compiled_truth: compiled,
|
||||
timeline: '',
|
||||
});
|
||||
const page = await engine.getPage(slug);
|
||||
return page!.id;
|
||||
}
|
||||
|
||||
describe('listActiveTakesForPages (P1)', () => {
|
||||
test('returns empty map entries for pages with no takes', async () => {
|
||||
const id1 = await seedPage('test/p1', 'P1');
|
||||
const id2 = await seedPage('test/p2', 'P2');
|
||||
const out = await engine.listActiveTakesForPages([id1, id2]);
|
||||
expect(out.get(id1)).toEqual([]);
|
||||
expect(out.get(id2)).toEqual([]);
|
||||
});
|
||||
|
||||
test('returns active takes grouped by page_id', async () => {
|
||||
const id1 = await seedPage('test/p1', 'P1');
|
||||
const id2 = await seedPage('test/p2', 'P2');
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: id1, row_num: 1, claim: 'p1 claim 1', kind: 'fact', holder: 'garry', weight: 1, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
{ page_id: id1, row_num: 2, claim: 'p1 claim 2', kind: 'fact', holder: 'garry', weight: 1, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
{ page_id: id2, row_num: 1, claim: 'p2 claim 1', kind: 'take', holder: 'garry', weight: 0.5, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
]);
|
||||
const out = await engine.listActiveTakesForPages([id1, id2]);
|
||||
expect(out.get(id1)?.length).toBe(2);
|
||||
expect(out.get(id2)?.length).toBe(1);
|
||||
expect(out.get(id1)?.[0].claim).toBe('p1 claim 1');
|
||||
});
|
||||
|
||||
test('excludes inactive (superseded) takes', async () => {
|
||||
const id1 = await seedPage('test/p1', 'P1');
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: id1, row_num: 1, claim: 'old', kind: 'fact', holder: 'garry', weight: 1, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
]);
|
||||
await engine.supersedeTake(id1, 1, {
|
||||
claim: 'new', kind: 'fact', holder: 'garry', weight: 1, active: true,
|
||||
});
|
||||
const out = await engine.listActiveTakesForPages([id1]);
|
||||
expect(out.get(id1)?.length).toBe(1);
|
||||
expect(out.get(id1)?.[0].claim).toBe('new');
|
||||
});
|
||||
|
||||
test('honors takesHoldersAllowList', async () => {
|
||||
const id1 = await seedPage('test/p1', 'P1');
|
||||
await engine.addTakesBatch([
|
||||
{ page_id: id1, row_num: 1, claim: 'garry take', kind: 'take', holder: 'garry', weight: 1, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
{ page_id: id1, row_num: 2, claim: 'alice take', kind: 'take', holder: 'alice', weight: 1, since_date: undefined, source: undefined, active: true, superseded_by: null },
|
||||
]);
|
||||
const out = await engine.listActiveTakesForPages([id1], { takesHoldersAllowList: ['garry'] });
|
||||
expect(out.get(id1)?.length).toBe(1);
|
||||
expect(out.get(id1)?.[0].holder).toBe('garry');
|
||||
});
|
||||
|
||||
test('empty pageIds returns empty map (no SQL roundtrip)', async () => {
|
||||
const out = await engine.listActiveTakesForPages([]);
|
||||
expect(out.size).toBe(0);
|
||||
});
|
||||
});
|
||||
|
||||
describe('writeContradictionsRun + loadContradictionsTrend (M5)', () => {
|
||||
const baseRow = {
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
prompt_version: '1',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 12,
|
||||
total_contradictions_flagged: 18,
|
||||
wilson_ci_lower: 0.14,
|
||||
wilson_ci_upper: 0.37,
|
||||
judge_errors_total: 3,
|
||||
cost_usd_total: 1.18,
|
||||
duration_ms: 45000,
|
||||
source_tier_breakdown: { curated_vs_curated: 2, curated_vs_bulk: 11, bulk_vs_bulk: 5, other: 0 },
|
||||
report_json: { schema_version: 1, run_id: 'test', cached: false },
|
||||
};
|
||||
|
||||
test('writes a row and reads it back from the trend', async () => {
|
||||
const inserted = await engine.writeContradictionsRun({ ...baseRow, run_id: 'r1' });
|
||||
expect(inserted).toBe(true);
|
||||
const trend = await engine.loadContradictionsTrend(30);
|
||||
expect(trend.length).toBe(1);
|
||||
expect(trend[0].run_id).toBe('r1');
|
||||
expect(trend[0].queries_with_contradiction).toBe(12);
|
||||
expect(trend[0].wilson_ci_lower).toBeCloseTo(0.14, 5);
|
||||
expect(trend[0].source_tier_breakdown).toEqual(baseRow.source_tier_breakdown);
|
||||
expect(trend[0].report_json.schema_version).toBe(1);
|
||||
});
|
||||
|
||||
test('idempotent on duplicate run_id', async () => {
|
||||
await engine.writeContradictionsRun({ ...baseRow, run_id: 'dup' });
|
||||
const second = await engine.writeContradictionsRun({ ...baseRow, run_id: 'dup' });
|
||||
expect(second).toBe(false);
|
||||
const trend = await engine.loadContradictionsTrend(30);
|
||||
expect(trend.length).toBe(1);
|
||||
});
|
||||
|
||||
test('trend returns newest first', async () => {
|
||||
await engine.writeContradictionsRun({ ...baseRow, run_id: 'old' });
|
||||
// PGLite ran_at uses now() at insert time; sequential inserts get monotonic timestamps.
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
await engine.writeContradictionsRun({ ...baseRow, run_id: 'new' });
|
||||
const trend = await engine.loadContradictionsTrend(30);
|
||||
expect(trend.length).toBe(2);
|
||||
expect(trend[0].run_id).toBe('new');
|
||||
expect(trend[1].run_id).toBe('old');
|
||||
});
|
||||
|
||||
test('days window filters older entries (zero-day window returns nothing)', async () => {
|
||||
await engine.writeContradictionsRun({ ...baseRow, run_id: 'r1' });
|
||||
const trend = await engine.loadContradictionsTrend(0);
|
||||
// cutoff = now - 0 days = now. Row inserted with ran_at = now() will be on the boundary;
|
||||
// accept either 0 or 1 results to avoid flakes on the millisecond boundary.
|
||||
expect(trend.length).toBeLessThanOrEqual(1);
|
||||
});
|
||||
|
||||
test('JSONB columns round-trip as objects, not strings (postgres-jsonb regression class)', async () => {
|
||||
await engine.writeContradictionsRun({
|
||||
...baseRow,
|
||||
run_id: 'jsonb-test',
|
||||
source_tier_breakdown: { curated_vs_curated: 99, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
|
||||
report_json: { nested: { value: 42, list: [1, 2, 3] } },
|
||||
});
|
||||
const trend = await engine.loadContradictionsTrend(1);
|
||||
expect(typeof trend[0].source_tier_breakdown).toBe('object');
|
||||
expect(typeof trend[0].report_json).toBe('object');
|
||||
expect((trend[0].source_tier_breakdown as Record<string, unknown>).curated_vs_curated).toBe(99);
|
||||
expect((trend[0].report_json.nested as Record<string, unknown>).value).toBe(42);
|
||||
});
|
||||
});
|
||||
|
||||
describe('contradiction cache (P2)', () => {
|
||||
const baseKey = {
|
||||
chunk_a_hash: 'sha-aaa',
|
||||
chunk_b_hash: 'sha-bbb',
|
||||
model_id: 'anthropic:claude-haiku-4-5',
|
||||
prompt_version: '1',
|
||||
truncation_policy: '1500-chars-utf8-safe',
|
||||
};
|
||||
const verdict = {
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'MRR vs ARR',
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
};
|
||||
|
||||
test('miss returns null on fresh cache', async () => {
|
||||
const hit = await engine.getContradictionCacheEntry(baseKey);
|
||||
expect(hit).toBeNull();
|
||||
});
|
||||
|
||||
test('put then get returns the verdict object (JSONB round-trip)', async () => {
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
|
||||
const hit = await engine.getContradictionCacheEntry(baseKey);
|
||||
expect(hit).not.toBeNull();
|
||||
expect((hit as Record<string, unknown>).contradicts).toBe(true);
|
||||
expect((hit as Record<string, unknown>).severity).toBe('medium');
|
||||
});
|
||||
|
||||
test('different prompt_version is a different cache key (Codex fix)', async () => {
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
|
||||
const wrong = await engine.getContradictionCacheEntry({ ...baseKey, prompt_version: '2' });
|
||||
expect(wrong).toBeNull();
|
||||
});
|
||||
|
||||
test('different truncation_policy is a different cache key', async () => {
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
|
||||
const wrong = await engine.getContradictionCacheEntry({ ...baseKey, truncation_policy: '500-chars' });
|
||||
expect(wrong).toBeNull();
|
||||
});
|
||||
|
||||
test('upsert refreshes verdict on conflict', async () => {
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict });
|
||||
await engine.putContradictionCacheEntry({
|
||||
...baseKey,
|
||||
verdict: { ...verdict, contradicts: false, severity: 'low' },
|
||||
});
|
||||
const hit = await engine.getContradictionCacheEntry(baseKey);
|
||||
expect((hit as Record<string, unknown>).contradicts).toBe(false);
|
||||
expect((hit as Record<string, unknown>).severity).toBe('low');
|
||||
});
|
||||
|
||||
test('expired entries are not returned by get', async () => {
|
||||
// Insert with TTL=60 (minimum allowed), then manually backdate expires_at via raw SQL.
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict, ttl_seconds: 60 });
|
||||
await engine.executeRaw(
|
||||
`UPDATE eval_contradictions_cache
|
||||
SET expires_at = $1
|
||||
WHERE chunk_a_hash = $2 AND chunk_b_hash = $3`,
|
||||
[new Date(Date.now() - 1000), baseKey.chunk_a_hash, baseKey.chunk_b_hash]
|
||||
);
|
||||
const hit = await engine.getContradictionCacheEntry(baseKey);
|
||||
expect(hit).toBeNull();
|
||||
});
|
||||
|
||||
test('sweep deletes expired entries', async () => {
|
||||
await engine.putContradictionCacheEntry({ ...baseKey, verdict, ttl_seconds: 60 });
|
||||
await engine.putContradictionCacheEntry({
|
||||
...baseKey,
|
||||
chunk_a_hash: 'sha-fresh',
|
||||
verdict,
|
||||
});
|
||||
await engine.executeRaw(
|
||||
`UPDATE eval_contradictions_cache
|
||||
SET expires_at = $1
|
||||
WHERE chunk_a_hash = $2`,
|
||||
[new Date(Date.now() - 1000), baseKey.chunk_a_hash]
|
||||
);
|
||||
const swept = await engine.sweepContradictionCache();
|
||||
expect(swept).toBe(1);
|
||||
const remaining = await engine.getContradictionCacheEntry({ ...baseKey, chunk_a_hash: 'sha-fresh' });
|
||||
expect(remaining).not.toBeNull();
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,137 @@
|
||||
/**
|
||||
* Fixture redactor tests — T2 privacy redaction passes.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
createRedactionSession,
|
||||
isCleanForCommit,
|
||||
redactMonetary,
|
||||
redactNames,
|
||||
redactSlug,
|
||||
redactText,
|
||||
} from '../src/core/eval-contradictions/fixture-redact.ts';
|
||||
|
||||
describe('redactSlug', () => {
|
||||
test('people/<name> → people/alice-example (deterministic per session)', () => {
|
||||
const s = createRedactionSession();
|
||||
const out = redactSlug(s, 'people/garry-tan');
|
||||
expect(out).toMatch(/^people\/.+-example$/);
|
||||
});
|
||||
|
||||
test('same raw slug maps consistently within a session', () => {
|
||||
const s = createRedactionSession();
|
||||
const a = redactSlug(s, 'people/garry-tan');
|
||||
const b = redactSlug(s, 'people/garry-tan');
|
||||
expect(a).toBe(b);
|
||||
});
|
||||
|
||||
test('different raw slugs map to different placeholders', () => {
|
||||
const s = createRedactionSession();
|
||||
const a = redactSlug(s, 'people/garry-tan');
|
||||
const b = redactSlug(s, 'people/paul-graham');
|
||||
expect(a).not.toBe(b);
|
||||
});
|
||||
|
||||
test('companies/, deals/, projects/ all rewrite', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactSlug(s, 'companies/y-combinator')).toMatch(/^companies\/.+-example$/);
|
||||
expect(redactSlug(s, 'deals/y-seed')).toMatch(/^deals\/.+-example$/);
|
||||
expect(redactSlug(s, 'projects/secret')).toMatch(/^projects\/.+-example$/);
|
||||
});
|
||||
|
||||
test('unrecognized prefix passes through unchanged', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactSlug(s, 'random/page')).toBe('random/page');
|
||||
expect(redactSlug(s, 'concepts/foo')).toBe('concepts/foo');
|
||||
});
|
||||
|
||||
test('audit trail records every redaction', () => {
|
||||
const s = createRedactionSession();
|
||||
redactSlug(s, 'people/garry');
|
||||
redactSlug(s, 'companies/yc');
|
||||
expect(s.audit.length).toBe(2);
|
||||
expect(s.audit[0]).toContain('garry');
|
||||
expect(s.audit[1]).toContain('yc');
|
||||
});
|
||||
});
|
||||
|
||||
describe('redactNames', () => {
|
||||
test('Firstname Lastname → Alice Example', () => {
|
||||
const s = createRedactionSession();
|
||||
const out = redactNames(s, 'I met Mackenzie Burnett yesterday');
|
||||
expect(out).toMatch(/I met .+ Example yesterday/);
|
||||
});
|
||||
|
||||
test('same name maps consistently', () => {
|
||||
const s = createRedactionSession();
|
||||
const out1 = redactNames(s, 'Garry Tan');
|
||||
const out2 = redactNames(s, 'Garry Tan');
|
||||
expect(out1).toBe(out2);
|
||||
});
|
||||
|
||||
test('different names map to different placeholders', () => {
|
||||
const s = createRedactionSession();
|
||||
const out1 = redactNames(s, 'Alice Smith');
|
||||
const out2 = redactNames(s, 'Bob Jones');
|
||||
expect(out1).not.toBe(out2);
|
||||
});
|
||||
|
||||
test('does not match lowercase strings (not name-shaped)', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactNames(s, 'hello world')).toBe('hello world');
|
||||
});
|
||||
|
||||
test('does not match single names', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactNames(s, 'Alice said hi')).toBe('Alice said hi');
|
||||
});
|
||||
});
|
||||
|
||||
describe('redactMonetary', () => {
|
||||
test('$50K → multiplied by salt', () => {
|
||||
const s = createRedactionSession();
|
||||
const out = redactMonetary(s, 'MRR is $50K');
|
||||
expect(out).not.toBe('MRR is $50K');
|
||||
expect(out).toMatch(/\$\d+(\.\d+)?K/);
|
||||
});
|
||||
|
||||
test('$2M, $1.5B all rewritten', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactMonetary(s, 'raised $2M')).not.toBe('raised $2M');
|
||||
expect(redactMonetary(s, 'valued at $1.5B')).not.toBe('valued at $1.5B');
|
||||
});
|
||||
|
||||
test('non-monetary numbers pass through', () => {
|
||||
const s = createRedactionSession();
|
||||
expect(redactMonetary(s, 'we have 50 customers')).toBe('we have 50 customers');
|
||||
});
|
||||
});
|
||||
|
||||
describe('redactText (full pass)', () => {
|
||||
test('chains PII + name + monetary', () => {
|
||||
const s = createRedactionSession();
|
||||
const out = redactText(s, 'Met Mackenzie Burnett, MRR $50K, email me at foo@bar.com');
|
||||
expect(out).not.toContain('Mackenzie Burnett');
|
||||
expect(out).not.toContain('foo@bar.com');
|
||||
expect(out).not.toContain('$50K');
|
||||
});
|
||||
});
|
||||
|
||||
describe('isCleanForCommit', () => {
|
||||
test('clean text passes', () => {
|
||||
expect(isCleanForCommit('Alice Example met Bob Example at companies/acme-example')).toBe(true);
|
||||
});
|
||||
|
||||
test('raw name shape blocks commit', () => {
|
||||
expect(isCleanForCommit('Met Mackenzie Burnett')).toBe(false);
|
||||
});
|
||||
|
||||
test('raw email blocks commit', () => {
|
||||
expect(isCleanForCommit('contact me at foo@bar.com')).toBe(false);
|
||||
});
|
||||
|
||||
test('empty text is clean', () => {
|
||||
expect(isCleanForCommit('')).toBe(true);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,240 @@
|
||||
/**
|
||||
* Integration tests for the three v0.32.6 wire-ups: M1 doctor, M3 MCP op,
|
||||
* M2 synthesize prompt injection.
|
||||
*
|
||||
* Hermetic against PGLite. Doctor + MCP exercised end-to-end; synthesize
|
||||
* exercised at the prompt-builder seam via loadPriorContradictionsBlock.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import { operations, operationsByName, type OperationContext } from '../src/core/operations.ts';
|
||||
import { loadTrend, writeRunRow } from '../src/core/eval-contradictions/trends.ts';
|
||||
import type { ProbeReport } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
/** Minimal OperationContext for hermetic op-handler tests. */
|
||||
function mkCtx(): OperationContext {
|
||||
return {
|
||||
engine,
|
||||
config: {} as OperationContext['config'],
|
||||
logger: { info: () => {}, warn: () => {}, error: () => {}, debug: () => {} } as unknown as OperationContext['logger'],
|
||||
dryRun: false,
|
||||
remote: false,
|
||||
};
|
||||
}
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
function mkReport(opts: Partial<ProbeReport> & {
|
||||
findings?: Array<{ severity: 'low' | 'medium' | 'high'; axis: string; slugA: string; slugB: string }>;
|
||||
} = {}): ProbeReport {
|
||||
const findings = opts.findings ?? [];
|
||||
return {
|
||||
schema_version: 1,
|
||||
run_id: opts.run_id ?? 'test-run',
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
prompt_version: '1',
|
||||
truncation_policy: '1500-chars-utf8-safe',
|
||||
top_k: 5,
|
||||
sampling: 'deterministic',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: findings.length > 0 ? Math.max(1, findings.length) : 0,
|
||||
total_contradictions_flagged: findings.length,
|
||||
calibration: {
|
||||
queries_total: 50,
|
||||
queries_judged_clean: 50 - findings.length,
|
||||
queries_with_contradiction: findings.length > 0 ? Math.max(1, findings.length) : 0,
|
||||
wilson_ci_95: { point: 0.24, lower: 0.14, upper: 0.37 },
|
||||
},
|
||||
judge_errors: { parse_fail: 0, refusal: 0, timeout: 0, http_5xx: 0, unknown: 0, total: 0, note: 'n' },
|
||||
cost_usd: { judge: 1, embedding: 0.01, total: 1.01, estimate_note: 'approx' },
|
||||
cache: { hits: 0, misses: 0, hit_rate: 0 },
|
||||
duration_ms: 45000,
|
||||
source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: findings.length, bulk_vs_bulk: 0, other: 0 },
|
||||
per_query: findings.length > 0 ? [
|
||||
{
|
||||
query: 'what is acme MRR',
|
||||
result_count: 5,
|
||||
pairs_skipped_by_date: 0,
|
||||
pairs_cache_hit: 0,
|
||||
pairs_judged: findings.length,
|
||||
contradictions: findings.map((f, i) => ({
|
||||
kind: 'cross_slug_chunks' as const,
|
||||
a: { slug: f.slugA, chunk_id: i + 1, take_id: null, source_tier: 'curated' as const, holder: null, text: 'a' },
|
||||
b: { slug: f.slugB, chunk_id: i + 100, take_id: null, source_tier: 'bulk' as const, holder: null, text: 'b' },
|
||||
combined_score: 1.0,
|
||||
severity: f.severity,
|
||||
axis: f.axis,
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
resolution_command: `gbrain dream --phase synthesize --slug ${f.slugA}`,
|
||||
})),
|
||||
},
|
||||
] : [],
|
||||
hot_pages: [],
|
||||
...opts,
|
||||
};
|
||||
}
|
||||
|
||||
describe('M3 find_contradictions MCP op', () => {
|
||||
test('op is registered with read scope', () => {
|
||||
const op = operationsByName['find_contradictions'];
|
||||
expect(op).toBeTruthy();
|
||||
expect(op.scope).toBe('read');
|
||||
expect(op.localOnly).toBeFalsy();
|
||||
});
|
||||
|
||||
test('op appears in the operations registry', () => {
|
||||
expect(operations.some((o) => o.name === 'find_contradictions')).toBe(true);
|
||||
});
|
||||
|
||||
test('returns empty contradictions + note when no probe runs exist', async () => {
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const result = await op.handler(mkCtx(), {}) as { contradictions: unknown[]; note?: string };
|
||||
expect(result.contradictions).toEqual([]);
|
||||
expect(result.note).toContain('No probe runs');
|
||||
});
|
||||
|
||||
test('returns findings from latest run', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport({
|
||||
run_id: 'r1',
|
||||
findings: [
|
||||
{ severity: 'high', axis: 'MRR figure', slugA: 'companies/acme', slugB: 'openclaw/chat/x' },
|
||||
{ severity: 'low', axis: 'naming', slugA: 'people/alice', slugB: 'people/alice-smith' },
|
||||
],
|
||||
}),
|
||||
45000,
|
||||
);
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const result = await op.handler(mkCtx(), {}) as { contradictions: unknown[]; total_in_run?: number; run_id?: string };
|
||||
expect(result.contradictions.length).toBe(2);
|
||||
expect(result.total_in_run).toBe(2);
|
||||
expect(result.run_id).toBe('r1');
|
||||
});
|
||||
|
||||
test('severity filter narrows results', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport({
|
||||
findings: [
|
||||
{ severity: 'high', axis: 'a', slugA: 'x/1', slugB: 'y/1' },
|
||||
{ severity: 'low', axis: 'b', slugA: 'x/2', slugB: 'y/2' },
|
||||
],
|
||||
}),
|
||||
1,
|
||||
);
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const result = await op.handler(mkCtx(), { severity: 'high' }) as { contradictions: Array<{ severity: string }> };
|
||||
expect(result.contradictions.length).toBe(1);
|
||||
expect(result.contradictions[0].severity).toBe('high');
|
||||
});
|
||||
|
||||
test('slug filter (substring match) narrows by either side', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport({
|
||||
findings: [
|
||||
{ severity: 'medium', axis: 'a', slugA: 'companies/acme', slugB: 'daily/x' },
|
||||
{ severity: 'medium', axis: 'b', slugA: 'people/alice', slugB: 'openclaw/chat/y' },
|
||||
],
|
||||
}),
|
||||
1,
|
||||
);
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const result = await op.handler(mkCtx(), { slug: 'acme' }) as { contradictions: unknown[] };
|
||||
expect(result.contradictions.length).toBe(1);
|
||||
});
|
||||
|
||||
test('limit caps result count', async () => {
|
||||
const many = Array.from({ length: 30 }, (_, i) => ({
|
||||
severity: 'low' as const,
|
||||
axis: `axis ${i}`,
|
||||
slugA: `x/${i}`,
|
||||
slugB: `y/${i}`,
|
||||
}));
|
||||
await writeRunRow(engine, mkReport({ findings: many }), 1);
|
||||
const op = operationsByName['find_contradictions'];
|
||||
const result = await op.handler(mkCtx(), { limit: 5 }) as { contradictions: unknown[]; total_in_run: number };
|
||||
expect(result.contradictions.length).toBe(5);
|
||||
expect(result.total_in_run).toBe(30);
|
||||
});
|
||||
});
|
||||
|
||||
describe('M1 doctor contradictions check (data-shape contract)', () => {
|
||||
// Doctor's runDoctor calls process.exit; rather than mock that out we
|
||||
// exercise the engine surface the check reads, which is what the check
|
||||
// would see. Full doctor end-to-end is covered by the E2E test in commit 9.
|
||||
test('empty trend looks like the doctor "no probe runs" case', async () => {
|
||||
const rows = await loadTrend(engine, 7);
|
||||
expect(rows).toEqual([]);
|
||||
});
|
||||
|
||||
test('populated trend yields severity-bucketable findings for the check', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport({
|
||||
findings: [
|
||||
{ severity: 'high', axis: 'CFO role', slugA: 'people/alice', slugB: 'companies/acme' },
|
||||
{ severity: 'medium', axis: 'MRR', slugA: 'companies/widget', slugB: 'openclaw/chat/x' },
|
||||
],
|
||||
}),
|
||||
1,
|
||||
);
|
||||
const rows = await loadTrend(engine, 7);
|
||||
expect(rows.length).toBe(1);
|
||||
const findings = rows[0].report_json.per_query.flatMap((q) => q.contradictions);
|
||||
expect(findings.length).toBe(2);
|
||||
const high = findings.filter((f) => f.severity === 'high');
|
||||
expect(high.length).toBe(1);
|
||||
expect(high[0].axis).toBe('CFO role');
|
||||
// Resolution command shape (M7 chain): paste-ready CLI string
|
||||
expect(high[0].resolution_command).toContain('gbrain');
|
||||
});
|
||||
});
|
||||
|
||||
describe('M2 synthesize prompt injection (priorContradictionsBlock)', () => {
|
||||
test('empty trend yields empty block (no impact on existing prompt)', async () => {
|
||||
// loadPriorContradictionsBlock is module-private; we test via the public
|
||||
// buildSynthesisPrompt seam by checking what the orchestrator passes.
|
||||
// Since the helper is private, we exercise the integration end-to-end
|
||||
// through engine.loadContradictionsTrend which returns []; the
|
||||
// synthesize.ts helper handles empty gracefully (silent '').
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows.length).toBe(0);
|
||||
});
|
||||
|
||||
test('populated trend yields findings that the prompt could use', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport({
|
||||
findings: [
|
||||
{ severity: 'high', axis: 'CEO change', slugA: 'people/alice', slugB: 'companies/acme' },
|
||||
],
|
||||
}),
|
||||
1,
|
||||
);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows.length).toBe(1);
|
||||
const report = rows[0].report_json;
|
||||
const findings = report.per_query.flatMap((q) => q.contradictions);
|
||||
expect(findings.length).toBe(1);
|
||||
expect(findings[0].severity).toBe('high');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,89 @@
|
||||
/**
|
||||
* Judge-errors tests — classification + denominator counting.
|
||||
*
|
||||
* The first-class judge_errors output is Codex's fix to the silent-skip
|
||||
* bias. These tests pin the contract: every error class falls into one
|
||||
* of the typed buckets, the total counts up correctly, and the human-
|
||||
* facing `note` field is present so consumers know nothing was hidden.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
classifyError,
|
||||
JudgeErrorCollector,
|
||||
} from '../src/core/eval-contradictions/judge-errors.ts';
|
||||
|
||||
describe('classifyError', () => {
|
||||
test('classifies JSON parse failures', () => {
|
||||
expect(classifyError(new Error('failed to parse JSON output'))).toBe('parse_fail');
|
||||
expect(classifyError(new Error('parseModelJSON: all strategies failed'))).toBe('parse_fail');
|
||||
expect(classifyError(new Error('json repair did not recover'))).toBe('parse_fail');
|
||||
});
|
||||
|
||||
test('classifies model refusals', () => {
|
||||
expect(classifyError(new Error("I can't help with that"))).toBe('refusal');
|
||||
expect(classifyError(new Error('refused to answer'))).toBe('refusal');
|
||||
});
|
||||
|
||||
test('classifies timeouts and aborts', () => {
|
||||
expect(classifyError(new Error('request timeout after 30s'))).toBe('timeout');
|
||||
expect(classifyError(new Error('the operation timed out'))).toBe('timeout');
|
||||
expect(classifyError(new Error('aborted by signal'))).toBe('timeout');
|
||||
});
|
||||
|
||||
test('classifies HTTP 5xx and overload', () => {
|
||||
expect(classifyError(new Error('503 Service Unavailable'))).toBe('http_5xx');
|
||||
expect(classifyError(new Error('Anthropic overloaded'))).toBe('http_5xx');
|
||||
expect(classifyError(new Error('upstream 502'))).toBe('http_5xx');
|
||||
});
|
||||
|
||||
test('falls back to unknown for unrecognized errors', () => {
|
||||
expect(classifyError(new Error('something weird happened'))).toBe('unknown');
|
||||
expect(classifyError(null)).toBe('unknown');
|
||||
expect(classifyError(undefined)).toBe('unknown');
|
||||
expect(classifyError(42)).toBe('unknown');
|
||||
});
|
||||
});
|
||||
|
||||
describe('JudgeErrorCollector', () => {
|
||||
test('starts empty, finalize yields zero counts but a populated note', () => {
|
||||
const c = new JudgeErrorCollector();
|
||||
const counts = c.finalize();
|
||||
expect(counts.total).toBe(0);
|
||||
expect(counts.parse_fail).toBe(0);
|
||||
expect(counts.note.length).toBeGreaterThan(10);
|
||||
});
|
||||
|
||||
test('tallies a mixed set of errors', () => {
|
||||
const c = new JudgeErrorCollector();
|
||||
c.record('pair-1', new Error('JSON parse failed'));
|
||||
c.record('pair-2', new Error('JSON parse failed'));
|
||||
c.record('pair-3', new Error('aborted'));
|
||||
c.record('pair-4', new Error('503 upstream'));
|
||||
c.record('pair-5', new Error('some weird thing'));
|
||||
const counts = c.finalize();
|
||||
expect(counts.parse_fail).toBe(2);
|
||||
expect(counts.timeout).toBe(1);
|
||||
expect(counts.http_5xx).toBe(1);
|
||||
expect(counts.unknown).toBe(1);
|
||||
expect(counts.refusal).toBe(0);
|
||||
expect(counts.total).toBe(5);
|
||||
});
|
||||
|
||||
test('preserves row order and pair ids', () => {
|
||||
const c = new JudgeErrorCollector();
|
||||
c.record('a', new Error('parse'));
|
||||
c.record('b', new Error('timeout'));
|
||||
const rows = c.rowsOut();
|
||||
expect(rows[0].pair_id).toBe('a');
|
||||
expect(rows[1].pair_id).toBe('b');
|
||||
expect(rows[0].kind).toBe('parse_fail');
|
||||
expect(rows[1].kind).toBe('timeout');
|
||||
});
|
||||
|
||||
test('note is stable across runs (no PII leak)', () => {
|
||||
const c1 = new JudgeErrorCollector();
|
||||
const c2 = new JudgeErrorCollector();
|
||||
expect(c1.finalize().note).toBe(c2.finalize().note);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,316 @@
|
||||
/**
|
||||
* Judge wrapper tests — hermetic via direct `chatFn` stub.
|
||||
*
|
||||
* Covers:
|
||||
* - prompt building (query-conditioned, holder included, truncation)
|
||||
* - parseModelJSON 4-strategy fence-stripping
|
||||
* - shape validation (missing/invalid fields throw)
|
||||
* - C1 confidence-floor double-enforcement
|
||||
* - severity classification
|
||||
* - refusal detection
|
||||
* - UTF-8-safe truncation
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
buildJudgePrompt,
|
||||
judgeContradiction,
|
||||
normalizeVerdict,
|
||||
truncateUtf8,
|
||||
DEFAULT_MAX_PAIR_CHARS,
|
||||
} from '../src/core/eval-contradictions/judge.ts';
|
||||
import type { ChatOpts, ChatResult } from '../src/core/ai/gateway.ts';
|
||||
|
||||
function mkResult(text: string, overrides: Partial<ChatResult> = {}): ChatResult {
|
||||
return {
|
||||
text,
|
||||
blocks: [],
|
||||
stopReason: 'end',
|
||||
usage: { input_tokens: 100, output_tokens: 50, cache_read_tokens: 0, cache_creation_tokens: 0 },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
providerId: 'anthropic',
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
function stubChat(response: ChatResult | ((opts: ChatOpts) => ChatResult | Promise<ChatResult>)) {
|
||||
return async (opts: ChatOpts): Promise<ChatResult> => {
|
||||
if (typeof response === 'function') return await response(opts);
|
||||
return response;
|
||||
};
|
||||
}
|
||||
|
||||
describe('truncateUtf8', () => {
|
||||
test('returns unchanged when under limit', () => {
|
||||
expect(truncateUtf8('short', 100)).toBe('short');
|
||||
});
|
||||
|
||||
test('truncates at code-point boundary', () => {
|
||||
const out = truncateUtf8('hello world', 5);
|
||||
expect(out).toBe('hello');
|
||||
});
|
||||
|
||||
test('handles empty string', () => {
|
||||
expect(truncateUtf8('', 100)).toBe('');
|
||||
});
|
||||
|
||||
test('does not split surrogate pairs (4-byte emoji)', () => {
|
||||
// 🚀 = U+1F680 (surrogate pair in UTF-16, length 2 in JS string).
|
||||
const text = 'a🚀b'; // length 4 in JS
|
||||
const out = truncateUtf8(text, 3); // would split the emoji
|
||||
// Should drop the high surrogate, leaving just 'a'.
|
||||
expect(out).toBe('a');
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildJudgePrompt', () => {
|
||||
test('includes user query verbatim (query-conditioned, Codex fix)', () => {
|
||||
const p = buildJudgePrompt({
|
||||
query: 'what is acme MRR',
|
||||
a: { slug: 'companies/acme', text: 'A text' },
|
||||
b: { slug: 'openclaw/chat/1', text: 'B text' },
|
||||
maxPairChars: 1500,
|
||||
});
|
||||
expect(p).toContain("User's query: what is acme MRR");
|
||||
});
|
||||
|
||||
test('truncates per maxPairChars', () => {
|
||||
const longText = 'x'.repeat(5000);
|
||||
const p = buildJudgePrompt({
|
||||
query: 'q',
|
||||
a: { slug: 'a', text: longText },
|
||||
b: { slug: 'b', text: longText },
|
||||
maxPairChars: 500,
|
||||
});
|
||||
// longText was 5000; both sides should appear truncated.
|
||||
expect(p.split('x'.repeat(501)).length).toBe(1); // no 501-x run survives
|
||||
});
|
||||
|
||||
test('includes source-tier label when present', () => {
|
||||
const p = buildJudgePrompt({
|
||||
query: 'q',
|
||||
a: { slug: 'a', text: 'A', source_tier: 'curated' },
|
||||
b: { slug: 'b', text: 'B', source_tier: 'bulk' },
|
||||
maxPairChars: 1500,
|
||||
});
|
||||
expect(p).toContain('source-tier curated');
|
||||
expect(p).toContain('source-tier bulk');
|
||||
});
|
||||
|
||||
test('includes holder for take pairs', () => {
|
||||
const p = buildJudgePrompt({
|
||||
query: 'q',
|
||||
a: { slug: 'a', text: 'A' },
|
||||
b: { slug: 'b', text: 'B', holder: 'garry' },
|
||||
maxPairChars: 1500,
|
||||
});
|
||||
expect(p).toContain('holder garry');
|
||||
});
|
||||
});
|
||||
|
||||
describe('normalizeVerdict', () => {
|
||||
test('valid input passes through', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'MRR figure',
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
});
|
||||
expect(v.contradicts).toBe(true);
|
||||
expect(v.severity).toBe('medium');
|
||||
expect(v.confidence).toBe(0.85);
|
||||
expect(v.resolution_kind).toBe('dream_synthesize');
|
||||
});
|
||||
|
||||
test('throws on missing contradicts field', () => {
|
||||
expect(() => normalizeVerdict({ confidence: 0.9 })).toThrow();
|
||||
});
|
||||
|
||||
test('throws on invalid confidence', () => {
|
||||
expect(() => normalizeVerdict({ contradicts: true, confidence: 'high' })).toThrow();
|
||||
expect(() => normalizeVerdict({ contradicts: true, confidence: NaN })).toThrow();
|
||||
});
|
||||
|
||||
test('throws on missing or non-object input', () => {
|
||||
expect(() => normalizeVerdict(null)).toThrow();
|
||||
expect(() => normalizeVerdict(undefined)).toThrow();
|
||||
expect(() => normalizeVerdict('json string')).toThrow();
|
||||
});
|
||||
|
||||
test('C1 double-enforce: contradicts:true + confidence<0.7 downgrades to false', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: true,
|
||||
severity: 'high',
|
||||
axis: 'something',
|
||||
confidence: 0.6,
|
||||
resolution_kind: 'takes_supersede',
|
||||
});
|
||||
expect(v.contradicts).toBe(false);
|
||||
expect(v.axis).toBe('');
|
||||
expect(v.resolution_kind).toBeNull();
|
||||
});
|
||||
|
||||
test('C1 boundary: confidence exactly 0.7 stays as contradicts:true', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'something',
|
||||
confidence: 0.7,
|
||||
});
|
||||
expect(v.contradicts).toBe(true);
|
||||
expect(v.confidence).toBe(0.7);
|
||||
});
|
||||
|
||||
test('clamps confidence into [0, 1]', () => {
|
||||
const v1 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: -0.5 });
|
||||
expect(v1.confidence).toBe(0);
|
||||
const v2 = normalizeVerdict({ contradicts: false, severity: 'low', confidence: 1.5 });
|
||||
expect(v2.confidence).toBe(1);
|
||||
});
|
||||
|
||||
test('garbage severity defaults to low', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: false,
|
||||
severity: 'critical',
|
||||
confidence: 0.5,
|
||||
});
|
||||
expect(v.severity).toBe('low');
|
||||
});
|
||||
|
||||
test('unknown resolution_kind on contradicts:true falls back to manual_review', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'X',
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'invalid_kind',
|
||||
});
|
||||
expect(v.resolution_kind).toBe('manual_review');
|
||||
});
|
||||
|
||||
test('axis cleared when contradicts:false', () => {
|
||||
const v = normalizeVerdict({
|
||||
contradicts: false,
|
||||
severity: 'low',
|
||||
axis: 'some axis',
|
||||
confidence: 0.4,
|
||||
});
|
||||
expect(v.axis).toBe('');
|
||||
});
|
||||
});
|
||||
|
||||
describe('judgeContradiction', () => {
|
||||
const baseInput = {
|
||||
query: 'what is acme MRR',
|
||||
a: { slug: 'companies/acme', text: 'Acme MRR is $2M (compiled).' },
|
||||
b: { slug: 'openclaw/chat/1', text: 'Acme MRR was $50K back in 2024.' },
|
||||
model: 'anthropic:claude-haiku-4-5',
|
||||
};
|
||||
|
||||
test('happy path: direct-parse JSON response', async () => {
|
||||
const out = await judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult(JSON.stringify({
|
||||
contradicts: true,
|
||||
severity: 'medium',
|
||||
axis: 'MRR figure',
|
||||
confidence: 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
}))),
|
||||
});
|
||||
expect(out.verdict.contradicts).toBe(true);
|
||||
expect(out.verdict.severity).toBe('medium');
|
||||
expect(out.usage.inputTokens).toBe(100);
|
||||
expect(out.usage.outputTokens).toBe(50);
|
||||
});
|
||||
|
||||
test('fence-wrapped JSON: parseModelJSON 4-strategy fallback', async () => {
|
||||
const fenced = '```json\n' + JSON.stringify({
|
||||
contradicts: false,
|
||||
severity: 'low',
|
||||
confidence: 0.3,
|
||||
}) + '\n```';
|
||||
const out = await judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult(fenced)),
|
||||
});
|
||||
expect(out.verdict.contradicts).toBe(false);
|
||||
});
|
||||
|
||||
test('throws on parse failure (counted in judge_errors)', async () => {
|
||||
await expect(
|
||||
judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult('not valid json at all')),
|
||||
})
|
||||
).rejects.toThrow();
|
||||
});
|
||||
|
||||
test('detects refusal via stopReason', async () => {
|
||||
await expect(
|
||||
judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult('Anything', { stopReason: 'refusal' })),
|
||||
})
|
||||
).rejects.toThrow(/refused/i);
|
||||
});
|
||||
|
||||
test('detects refusal via response text', async () => {
|
||||
await expect(
|
||||
judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult("I can't help with that")),
|
||||
})
|
||||
).rejects.toThrow(/refused/i);
|
||||
});
|
||||
|
||||
test('passes maxPairChars through to truncation', async () => {
|
||||
let capturedPrompt = '';
|
||||
await judgeContradiction({
|
||||
...baseInput,
|
||||
a: { slug: 'a', text: 'x'.repeat(5000) },
|
||||
b: { slug: 'b', text: 'y'.repeat(5000) },
|
||||
maxPairChars: 100,
|
||||
chatFn: stubChat(async (opts) => {
|
||||
const userMsg = opts.messages.find((m) => m.role === 'user');
|
||||
capturedPrompt = typeof userMsg?.content === 'string' ? userMsg.content : '';
|
||||
return mkResult(JSON.stringify({
|
||||
contradicts: false, severity: 'low', confidence: 0.5,
|
||||
}));
|
||||
}),
|
||||
});
|
||||
expect(capturedPrompt.split('x'.repeat(101)).length).toBe(1);
|
||||
});
|
||||
|
||||
test('default maxPairChars constant is 1500', () => {
|
||||
expect(DEFAULT_MAX_PAIR_CHARS).toBe(1500);
|
||||
});
|
||||
|
||||
test('C1 enforcement reaches the verdict (low-confidence true → false)', async () => {
|
||||
const out = await judgeContradiction({
|
||||
...baseInput,
|
||||
chatFn: stubChat(mkResult(JSON.stringify({
|
||||
contradicts: true,
|
||||
severity: 'high',
|
||||
axis: 'something',
|
||||
confidence: 0.5,
|
||||
}))),
|
||||
});
|
||||
expect(out.verdict.contradicts).toBe(false);
|
||||
});
|
||||
|
||||
test('query appears in the rendered prompt (Codex fix)', async () => {
|
||||
let capturedQuery = '';
|
||||
await judgeContradiction({
|
||||
...baseInput,
|
||||
query: 'distinctive-query-marker-12345',
|
||||
chatFn: stubChat(async (opts) => {
|
||||
const m = opts.messages[0]?.content;
|
||||
capturedQuery = typeof m === 'string' ? m : '';
|
||||
return mkResult(JSON.stringify({ contradicts: false, severity: 'low', confidence: 0.4 }));
|
||||
}),
|
||||
});
|
||||
expect(capturedQuery).toContain('distinctive-query-marker-12345');
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,435 @@
|
||||
/**
|
||||
* Runner orchestrator tests — hermetic via stubbed judgeFn + searchFn.
|
||||
*
|
||||
* Covers the integration shape: pair generation, date pre-filter wired,
|
||||
* sampling order, cost-cap mid-run stop, pre-flight refusal, judge-error
|
||||
* counting, Wilson CI on the headline, source-tier breakdown, hot pages,
|
||||
* intra-page pairs (P1 batched fetch), and the run-row write integration.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
PreFlightBudgetError,
|
||||
runContradictionProbe,
|
||||
type JudgeFn,
|
||||
} from '../src/core/eval-contradictions/runner.ts';
|
||||
import type { JudgeOutput } from '../src/core/eval-contradictions/judge.ts';
|
||||
import type { SearchResult } from '../src/core/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
/** Seed a page; returns the id. */
|
||||
async function seedPage(slug: string, title: string, body = ''): Promise<number> {
|
||||
await engine.putPage(slug, {
|
||||
title,
|
||||
type: 'concept',
|
||||
frontmatter: {},
|
||||
compiled_truth: body || `body for ${slug}`,
|
||||
timeline: '',
|
||||
});
|
||||
const page = await engine.getPage(slug);
|
||||
return page!.id;
|
||||
}
|
||||
|
||||
/** Build a SearchResult helper for stubbed search. */
|
||||
function mkResult(slug: string, page_id: number, chunk_id: number, text: string, score = 1.0): SearchResult {
|
||||
return {
|
||||
slug, page_id, chunk_id, chunk_index: 0,
|
||||
title: slug,
|
||||
type: 'concept',
|
||||
chunk_text: text,
|
||||
chunk_source: 'compiled_truth',
|
||||
score,
|
||||
stale: false,
|
||||
};
|
||||
}
|
||||
|
||||
/** Stubbed judge that returns a fixed verdict pattern. */
|
||||
function stubJudge(opts: {
|
||||
contradicts?: boolean;
|
||||
severity?: 'low' | 'medium' | 'high';
|
||||
confidence?: number;
|
||||
inputTokens?: number;
|
||||
outputTokens?: number;
|
||||
throwOn?: (i: number) => boolean;
|
||||
}): JudgeFn {
|
||||
let calls = 0;
|
||||
return async (): Promise<JudgeOutput> => {
|
||||
const idx = calls++;
|
||||
if (opts.throwOn && opts.throwOn(idx)) {
|
||||
throw new Error('stub: simulated transient 503');
|
||||
}
|
||||
return {
|
||||
verdict: {
|
||||
contradicts: opts.contradicts ?? true,
|
||||
severity: opts.severity ?? 'medium',
|
||||
axis: 'stub axis',
|
||||
confidence: opts.confidence ?? 0.85,
|
||||
resolution_kind: 'dream_synthesize',
|
||||
},
|
||||
usage: {
|
||||
inputTokens: opts.inputTokens ?? 500,
|
||||
outputTokens: opts.outputTokens ?? 80,
|
||||
},
|
||||
};
|
||||
};
|
||||
}
|
||||
|
||||
describe('runContradictionProbe', () => {
|
||||
test('empty queries returns a report with zero counts', async () => {
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: [],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.queries_evaluated).toBe(0);
|
||||
expect(out.report.total_contradictions_flagged).toBe(0);
|
||||
});
|
||||
|
||||
test('cross-slug pair detection with stubbed search + judge', async () => {
|
||||
const idA = await seedPage('companies/acme', 'Acme');
|
||||
const idB = await seedPage('openclaw/chat/x', 'Chat');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['what is acme MRR'],
|
||||
judgeFn: stubJudge({ contradicts: true, severity: 'medium' }),
|
||||
searchFn: async () => [
|
||||
mkResult('companies/acme', idA, 1, 'Acme MRR is $2M', 1.5),
|
||||
mkResult('openclaw/chat/x', idB, 2, 'Acme MRR is $50K', 0.5),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.total_contradictions_flagged).toBe(1);
|
||||
expect(out.report.queries_with_contradiction).toBe(1);
|
||||
expect(out.report.per_query[0].pairs_judged).toBe(1);
|
||||
});
|
||||
|
||||
test('intra-page chunk-vs-take detection (P1 batched fetch)', async () => {
|
||||
const id1 = await seedPage('people/alice', 'Alice');
|
||||
await engine.addTakesBatch([
|
||||
{
|
||||
page_id: id1, row_num: 1, claim: 'Alice is the CTO', kind: 'fact',
|
||||
holder: 'garry', weight: 1, active: true, superseded_by: null,
|
||||
},
|
||||
]);
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['what is alice role'],
|
||||
judgeFn: stubJudge({ contradicts: true, severity: 'high' }),
|
||||
searchFn: async () => [
|
||||
mkResult('people/alice', id1, 1, 'Alice is the CFO of acme'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.total_contradictions_flagged).toBe(1);
|
||||
const finding = out.report.per_query[0].contradictions[0];
|
||||
expect(finding.kind).toBe('intra_page_chunk_take');
|
||||
expect(finding.b.take_id).not.toBeNull();
|
||||
expect(finding.b.holder).toBe('garry');
|
||||
});
|
||||
|
||||
test('same-slug pairs are NOT generated (cross_slug skip rule)', async () => {
|
||||
const idA = await seedPage('a/page', 'A');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [
|
||||
// Both results from the same page — pair should NOT form.
|
||||
mkResult('a/page', idA, 1, 'chunk one'),
|
||||
mkResult('a/page', idA, 2, 'chunk two'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.per_query[0].pairs_judged).toBe(0);
|
||||
});
|
||||
|
||||
test('date pre-filter rejects quarterly-shape pairs', async () => {
|
||||
const idA = await seedPage('companies/acme', 'Acme');
|
||||
const idB = await seedPage('openclaw/chat/2024', 'Chat');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [
|
||||
mkResult('companies/acme', idA, 1, 'Acme MRR was $50K (2024-08-01)'),
|
||||
mkResult('openclaw/chat/2024', idB, 2, 'Acme MRR is $2M (2026-03-15)'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.per_query[0].pairs_skipped_by_date).toBe(1);
|
||||
expect(out.report.per_query[0].pairs_judged).toBe(0);
|
||||
});
|
||||
|
||||
test('judge throw counts as judge_errors, does not crash run', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
// Throw on the first (and only) call.
|
||||
judgeFn: stubJudge({ throwOn: (i) => i === 0 }),
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'chunk a'),
|
||||
mkResult('b/1', idB, 2, 'chunk b'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.judge_errors.total).toBe(1);
|
||||
expect(out.report.total_contradictions_flagged).toBe(0);
|
||||
expect(out.judgeErrorRows.length).toBe(1);
|
||||
});
|
||||
|
||||
test('cost cap mid-run stop with partial report', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const idC = await seedPage('c/1', 'C');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q1', 'q2', 'q3'],
|
||||
// Huge tokens per call so cap blows fast.
|
||||
judgeFn: stubJudge({ inputTokens: 1_000_000, outputTokens: 200_000 }),
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'a'),
|
||||
mkResult('b/1', idB, 2, 'b'),
|
||||
mkResult('c/1', idC, 3, 'c'),
|
||||
],
|
||||
budgetUsd: 0.001,
|
||||
yesOverride: true, // bypass pre-flight refusal
|
||||
});
|
||||
expect(out.capHitMidRun).toBe(true);
|
||||
expect(out.report.queries_evaluated).toBe(3);
|
||||
// The first query gets some judging; the rest are zero-judged.
|
||||
expect(out.report.per_query.length).toBe(3);
|
||||
});
|
||||
|
||||
test('pre-flight refuses when estimate > budget AND --yes not set', async () => {
|
||||
await expect(
|
||||
runContradictionProbe({
|
||||
engine,
|
||||
queries: Array(1000).fill('q'),
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [],
|
||||
budgetUsd: 0.0000001,
|
||||
})
|
||||
).rejects.toBeInstanceOf(PreFlightBudgetError);
|
||||
});
|
||||
|
||||
test('pre-flight passes when --yes is set', async () => {
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [],
|
||||
budgetUsd: 0.0000001,
|
||||
yesOverride: true,
|
||||
});
|
||||
expect(out.report).toBeTruthy();
|
||||
});
|
||||
|
||||
test('Wilson CI populated on the headline percentage', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({ contradicts: true, severity: 'medium' }),
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'a'),
|
||||
mkResult('b/1', idB, 2, 'b'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.calibration.wilson_ci_95.point).toBe(1); // 1/1 query
|
||||
expect(out.report.calibration.small_sample_note).toBeTruthy();
|
||||
});
|
||||
|
||||
test('source_tier_breakdown computed from observed pairs', async () => {
|
||||
const idA = await seedPage('companies/acme', 'Acme');
|
||||
const idB = await seedPage('openclaw/chat/x', 'Chat');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [
|
||||
mkResult('companies/acme', idA, 1, 'a'),
|
||||
mkResult('openclaw/chat/x', idB, 2, 'b'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
// One pair: curated vs bulk
|
||||
expect(out.report.source_tier_breakdown.curated_vs_bulk).toBe(1);
|
||||
expect(out.report.source_tier_breakdown.curated_vs_curated).toBe(0);
|
||||
});
|
||||
|
||||
test('hot pages roll up across findings', async () => {
|
||||
const id1 = await seedPage('people/alice', 'A');
|
||||
const id2 = await seedPage('companies/acme', 'C');
|
||||
const id3 = await seedPage('openclaw/chat/x', 'X');
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q1', 'q2'],
|
||||
judgeFn: stubJudge({ contradicts: true, severity: 'high' }),
|
||||
searchFn: async (_engine, q) => {
|
||||
// Both queries feature alice; one features acme.
|
||||
if (q === 'q1') {
|
||||
return [
|
||||
mkResult('people/alice', id1, 1, 'a1'),
|
||||
mkResult('companies/acme', id2, 2, 'c'),
|
||||
];
|
||||
}
|
||||
return [
|
||||
mkResult('people/alice', id1, 3, 'a2'),
|
||||
mkResult('openclaw/chat/x', id3, 4, 'x'),
|
||||
];
|
||||
},
|
||||
budgetUsd: 5,
|
||||
});
|
||||
const alice = out.report.hot_pages.find((p) => p.slug === 'people/alice');
|
||||
expect(alice?.appearances).toBe(2);
|
||||
});
|
||||
|
||||
test('cache hit on second probe with same input (cache layer reaches engine)', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const queries = ['q'];
|
||||
const stubSearch = async () => [
|
||||
mkResult('a/1', idA, 1, 'aaa'),
|
||||
mkResult('b/1', idB, 2, 'bbb'),
|
||||
];
|
||||
const first = await runContradictionProbe({
|
||||
engine,
|
||||
queries,
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: stubSearch,
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(first.report.cache.hits).toBe(0);
|
||||
expect(first.report.cache.misses).toBe(1);
|
||||
|
||||
const second = await runContradictionProbe({
|
||||
engine,
|
||||
queries,
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: stubSearch,
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(second.report.cache.hits).toBe(1);
|
||||
});
|
||||
|
||||
test('--no-cache forces every pair to the judge', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'aaa'),
|
||||
mkResult('b/1', idB, 2, 'bbb'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
const second = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'aaa'),
|
||||
mkResult('b/1', idB, 2, 'bbb'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
noCache: true,
|
||||
});
|
||||
expect(second.report.cache.hits).toBe(0);
|
||||
});
|
||||
|
||||
test('abort signal aborts mid-run', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const ctrl = new AbortController();
|
||||
setTimeout(() => ctrl.abort(), 0);
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q1', 'q2', 'q3', 'q4'],
|
||||
judgeFn: async () => {
|
||||
// Yield to let the abort fire.
|
||||
await new Promise((r) => setTimeout(r, 1));
|
||||
return {
|
||||
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.3, resolution_kind: null },
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
};
|
||||
},
|
||||
searchFn: async () => [
|
||||
mkResult('a/1', idA, 1, 'a'),
|
||||
mkResult('b/1', idB, 2, 'b'),
|
||||
],
|
||||
budgetUsd: 5,
|
||||
abortSignal: ctrl.signal,
|
||||
});
|
||||
expect(out.report.queries_evaluated).toBe(4);
|
||||
// Some queries are emitted; aborted ones have zero counts.
|
||||
});
|
||||
|
||||
test('report shape matches schema_version: 1 contract', async () => {
|
||||
const out = await runContradictionProbe({
|
||||
engine,
|
||||
queries: ['q'],
|
||||
judgeFn: stubJudge({}),
|
||||
searchFn: async () => [],
|
||||
budgetUsd: 5,
|
||||
});
|
||||
expect(out.report.schema_version).toBe(1);
|
||||
expect(out.report.prompt_version).toBeTruthy();
|
||||
expect(out.report.truncation_policy).toBeTruthy();
|
||||
expect(out.report.judge_errors.note).toContain('counted');
|
||||
expect(out.report.cost_usd.estimate_note).toContain('soft ceiling');
|
||||
});
|
||||
|
||||
test('deterministic sampling produces stable pair order across runs', async () => {
|
||||
const idA = await seedPage('a/1', 'A');
|
||||
const idB = await seedPage('b/1', 'B');
|
||||
const idC = await seedPage('c/1', 'C');
|
||||
const search = async () => [
|
||||
mkResult('a/1', idA, 1, 'aaa', 1.0),
|
||||
mkResult('b/1', idB, 2, 'bbb', 0.9),
|
||||
mkResult('c/1', idC, 3, 'ccc', 0.8),
|
||||
];
|
||||
let order1: string[] = [];
|
||||
let order2: string[] = [];
|
||||
const recordOrder = (out: string[]): JudgeFn => async (input) => {
|
||||
out.push(`${input.a.slug}|${input.b.slug}`);
|
||||
return {
|
||||
verdict: { contradicts: false, severity: 'low', axis: '', confidence: 0.4, resolution_kind: null },
|
||||
usage: { inputTokens: 1, outputTokens: 1 },
|
||||
};
|
||||
};
|
||||
await runContradictionProbe({
|
||||
engine, queries: ['q'], judgeFn: recordOrder(order1), searchFn: search,
|
||||
budgetUsd: 5, noCache: true,
|
||||
});
|
||||
await runContradictionProbe({
|
||||
engine, queries: ['q'], judgeFn: recordOrder(order2), searchFn: search,
|
||||
budgetUsd: 5, noCache: true,
|
||||
});
|
||||
expect(order1).toEqual(order2);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,157 @@
|
||||
/**
|
||||
* Severity-classify tests — parse, sort, bucket, hot-page rollup.
|
||||
*/
|
||||
|
||||
import { describe, test, expect } from 'bun:test';
|
||||
import {
|
||||
bucketBySeverity,
|
||||
buildHotPages,
|
||||
compareSeverityDesc,
|
||||
parseSeverity,
|
||||
} from '../src/core/eval-contradictions/severity-classify.ts';
|
||||
import type { ContradictionFinding, Severity } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
function mkFinding(opts: {
|
||||
slugA: string;
|
||||
slugB: string;
|
||||
severity: Severity;
|
||||
}): ContradictionFinding {
|
||||
return {
|
||||
kind: 'cross_slug_chunks',
|
||||
a: {
|
||||
slug: opts.slugA,
|
||||
chunk_id: 1,
|
||||
take_id: null,
|
||||
source_tier: 'curated',
|
||||
holder: null,
|
||||
text: 'A',
|
||||
},
|
||||
b: {
|
||||
slug: opts.slugB,
|
||||
chunk_id: 2,
|
||||
take_id: null,
|
||||
source_tier: 'bulk',
|
||||
holder: null,
|
||||
text: 'B',
|
||||
},
|
||||
combined_score: 1,
|
||||
severity: opts.severity,
|
||||
axis: 'test',
|
||||
confidence: 0.9,
|
||||
resolution_kind: 'manual_review',
|
||||
resolution_command: '',
|
||||
};
|
||||
}
|
||||
|
||||
describe('parseSeverity', () => {
|
||||
test('accepts the three valid values', () => {
|
||||
expect(parseSeverity('low')).toBe('low');
|
||||
expect(parseSeverity('medium')).toBe('medium');
|
||||
expect(parseSeverity('high')).toBe('high');
|
||||
});
|
||||
|
||||
test('defaults to low on garbage input', () => {
|
||||
expect(parseSeverity(null)).toBe('low');
|
||||
expect(parseSeverity(undefined)).toBe('low');
|
||||
expect(parseSeverity('critical')).toBe('low');
|
||||
expect(parseSeverity(7)).toBe('low');
|
||||
expect(parseSeverity({})).toBe('low');
|
||||
});
|
||||
});
|
||||
|
||||
describe('compareSeverityDesc', () => {
|
||||
test('high > medium > low', () => {
|
||||
expect(compareSeverityDesc('high', 'low')).toBeLessThan(0);
|
||||
expect(compareSeverityDesc('medium', 'low')).toBeLessThan(0);
|
||||
expect(compareSeverityDesc('high', 'medium')).toBeLessThan(0);
|
||||
expect(compareSeverityDesc('low', 'high')).toBeGreaterThan(0);
|
||||
expect(compareSeverityDesc('medium', 'medium')).toBe(0);
|
||||
});
|
||||
|
||||
test('sorts a list to high-medium-low', () => {
|
||||
const sevs: Severity[] = ['low', 'high', 'medium', 'low', 'high'];
|
||||
sevs.sort(compareSeverityDesc);
|
||||
expect(sevs).toEqual(['high', 'high', 'medium', 'low', 'low']);
|
||||
});
|
||||
});
|
||||
|
||||
describe('bucketBySeverity', () => {
|
||||
test('preserves order within each bucket', () => {
|
||||
const findings = [
|
||||
mkFinding({ slugA: 'a/1', slugB: 'b/1', severity: 'low' }),
|
||||
mkFinding({ slugA: 'a/2', slugB: 'b/2', severity: 'high' }),
|
||||
mkFinding({ slugA: 'a/3', slugB: 'b/3', severity: 'low' }),
|
||||
mkFinding({ slugA: 'a/4', slugB: 'b/4', severity: 'medium' }),
|
||||
];
|
||||
const buckets = bucketBySeverity(findings);
|
||||
expect(buckets.low.length).toBe(2);
|
||||
expect(buckets.medium.length).toBe(1);
|
||||
expect(buckets.high.length).toBe(1);
|
||||
expect(buckets.low[0].a.slug).toBe('a/1');
|
||||
expect(buckets.low[1].a.slug).toBe('a/3');
|
||||
});
|
||||
|
||||
test('empty input yields three empty buckets', () => {
|
||||
const buckets = bucketBySeverity([]);
|
||||
expect(buckets.low).toEqual([]);
|
||||
expect(buckets.medium).toEqual([]);
|
||||
expect(buckets.high).toEqual([]);
|
||||
});
|
||||
});
|
||||
|
||||
describe('buildHotPages', () => {
|
||||
test('counts appearances across both pair ends', () => {
|
||||
const findings = [
|
||||
mkFinding({ slugA: 'people/alice', slugB: 'companies/acme', severity: 'high' }),
|
||||
mkFinding({ slugA: 'people/alice', slugB: 'companies/widget', severity: 'medium' }),
|
||||
mkFinding({ slugA: 'companies/acme', slugB: 'people/bob', severity: 'low' }),
|
||||
];
|
||||
const hot = buildHotPages(findings);
|
||||
const alice = hot.find((p) => p.slug === 'people/alice');
|
||||
const acme = hot.find((p) => p.slug === 'companies/acme');
|
||||
expect(alice?.appearances).toBe(2);
|
||||
expect(acme?.appearances).toBe(2);
|
||||
});
|
||||
|
||||
test('max_severity reflects the worst severity that hit the page', () => {
|
||||
const findings = [
|
||||
mkFinding({ slugA: 'people/alice', slugB: 'x/1', severity: 'low' }),
|
||||
mkFinding({ slugA: 'people/alice', slugB: 'x/2', severity: 'high' }),
|
||||
mkFinding({ slugA: 'people/alice', slugB: 'x/3', severity: 'medium' }),
|
||||
];
|
||||
const hot = buildHotPages(findings);
|
||||
expect(hot[0].slug).toBe('people/alice');
|
||||
expect(hot[0].max_severity).toBe('high');
|
||||
});
|
||||
|
||||
test('does not double-count when both ends share the same slug', () => {
|
||||
const findings = [
|
||||
mkFinding({ slugA: 'same/page', slugB: 'same/page', severity: 'medium' }),
|
||||
];
|
||||
const hot = buildHotPages(findings);
|
||||
expect(hot[0].slug).toBe('same/page');
|
||||
expect(hot[0].appearances).toBe(1);
|
||||
});
|
||||
|
||||
test('sorts by appearances DESC then by max severity DESC', () => {
|
||||
const findings = [
|
||||
mkFinding({ slugA: 'a/1', slugB: 'b/1', severity: 'low' }),
|
||||
mkFinding({ slugA: 'a/2', slugB: 'b/1', severity: 'low' }),
|
||||
mkFinding({ slugA: 'people/star', slugB: 'q/1', severity: 'high' }),
|
||||
];
|
||||
const hot = buildHotPages(findings);
|
||||
// b/1 appears twice; people/star + a/1 + a/2 + q/1 appear once each.
|
||||
// Within ties, max_severity DESC orders people/star above a/1 etc.
|
||||
expect(hot[0].slug).toBe('b/1');
|
||||
expect(hot[0].appearances).toBe(2);
|
||||
expect(hot[1].slug).toBe('people/star');
|
||||
});
|
||||
|
||||
test('respects the limit argument', () => {
|
||||
const findings: ContradictionFinding[] = [];
|
||||
for (let i = 0; i < 30; i++) {
|
||||
findings.push(mkFinding({ slugA: `p/${i}`, slugB: `q/${i}`, severity: 'low' }));
|
||||
}
|
||||
expect(buildHotPages(findings, 5).length).toBe(5);
|
||||
});
|
||||
});
|
||||
@@ -0,0 +1,196 @@
|
||||
/**
|
||||
* Trends helpers tests — write, load, render.
|
||||
*/
|
||||
|
||||
import { describe, test, expect, beforeAll, afterAll, beforeEach } from 'bun:test';
|
||||
import { PGLiteEngine } from '../src/core/pglite-engine.ts';
|
||||
import { resetPgliteState } from './helpers/reset-pglite.ts';
|
||||
import {
|
||||
loadTrend,
|
||||
renderTrendChart,
|
||||
writeRunRow,
|
||||
} from '../src/core/eval-contradictions/trends.ts';
|
||||
import type { ProbeReport } from '../src/core/eval-contradictions/types.ts';
|
||||
|
||||
let engine: PGLiteEngine;
|
||||
|
||||
beforeAll(async () => {
|
||||
engine = new PGLiteEngine();
|
||||
await engine.connect({});
|
||||
await engine.initSchema();
|
||||
});
|
||||
|
||||
afterAll(async () => {
|
||||
await engine.disconnect();
|
||||
});
|
||||
|
||||
beforeEach(async () => {
|
||||
await resetPgliteState(engine);
|
||||
});
|
||||
|
||||
function mkReport(runId: string, overrides: Partial<ProbeReport> = {}): ProbeReport {
|
||||
return {
|
||||
schema_version: 1,
|
||||
run_id: runId,
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
prompt_version: '1',
|
||||
truncation_policy: '1500-chars-utf8-safe',
|
||||
top_k: 5,
|
||||
sampling: 'deterministic',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 12,
|
||||
total_contradictions_flagged: 18,
|
||||
calibration: {
|
||||
queries_total: 50,
|
||||
queries_judged_clean: 38,
|
||||
queries_with_contradiction: 12,
|
||||
wilson_ci_95: { point: 0.24, lower: 0.14, upper: 0.37 },
|
||||
},
|
||||
judge_errors: {
|
||||
parse_fail: 0, refusal: 0, timeout: 0, http_5xx: 0, unknown: 0, total: 0,
|
||||
note: 'errors counted toward denominator',
|
||||
},
|
||||
cost_usd: { judge: 1.0, embedding: 0.005, total: 1.005, estimate_note: 'approx' },
|
||||
cache: { hits: 50, misses: 100, hit_rate: 0.333 },
|
||||
duration_ms: 45000,
|
||||
source_tier_breakdown: { curated_vs_curated: 2, curated_vs_bulk: 11, bulk_vs_bulk: 5, other: 0 },
|
||||
per_query: [],
|
||||
hot_pages: [],
|
||||
...overrides,
|
||||
};
|
||||
}
|
||||
|
||||
describe('writeRunRow', () => {
|
||||
test('persists a run from a ProbeReport', async () => {
|
||||
const inserted = await writeRunRow(engine, mkReport('test-run-1'), 45000);
|
||||
expect(inserted).toBe(true);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows.length).toBe(1);
|
||||
expect(rows[0].run_id).toBe('test-run-1');
|
||||
expect(rows[0].wilson_ci_lower).toBeCloseTo(0.14, 5);
|
||||
});
|
||||
|
||||
test('idempotent on duplicate run_id', async () => {
|
||||
await writeRunRow(engine, mkReport('dup'), 100);
|
||||
const second = await writeRunRow(engine, mkReport('dup'), 100);
|
||||
expect(second).toBe(false);
|
||||
});
|
||||
|
||||
test('flattens nested structures into top-level columns', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport('shape-check', {
|
||||
queries_with_contradiction: 20,
|
||||
calibration: {
|
||||
queries_total: 100,
|
||||
queries_judged_clean: 80,
|
||||
queries_with_contradiction: 20,
|
||||
wilson_ci_95: { point: 0.2, lower: 0.13, upper: 0.29 },
|
||||
},
|
||||
}),
|
||||
777,
|
||||
);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows[0].queries_with_contradiction).toBe(20);
|
||||
expect(rows[0].duration_ms).toBe(777);
|
||||
expect(rows[0].wilson_ci_upper).toBeCloseTo(0.29, 5);
|
||||
});
|
||||
});
|
||||
|
||||
describe('loadTrend', () => {
|
||||
test('newest first', async () => {
|
||||
await writeRunRow(engine, mkReport('old'), 100);
|
||||
await new Promise((r) => setTimeout(r, 10));
|
||||
await writeRunRow(engine, mkReport('new'), 100);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows[0].run_id).toBe('new');
|
||||
expect(rows[1].run_id).toBe('old');
|
||||
});
|
||||
|
||||
test('empty when no runs exist', async () => {
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows).toEqual([]);
|
||||
});
|
||||
|
||||
test('parses source_tier_breakdown back to typed object', async () => {
|
||||
await writeRunRow(
|
||||
engine,
|
||||
mkReport('tier-check', {
|
||||
source_tier_breakdown: { curated_vs_curated: 99, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
|
||||
}),
|
||||
100,
|
||||
);
|
||||
const rows = await loadTrend(engine, 30);
|
||||
expect(rows[0].source_tier_breakdown.curated_vs_curated).toBe(99);
|
||||
});
|
||||
});
|
||||
|
||||
describe('renderTrendChart', () => {
|
||||
test('empty input prints a friendly message, not an empty table', () => {
|
||||
const out = renderTrendChart([]);
|
||||
expect(out).toContain('No contradiction-probe runs');
|
||||
expect(out).toContain('gbrain eval suspected-contradictions');
|
||||
});
|
||||
|
||||
test('single row produces a header + one data row', () => {
|
||||
const out = renderTrendChart([
|
||||
{
|
||||
run_id: 'r1',
|
||||
ran_at: '2026-05-11T00:00:00Z',
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 12,
|
||||
total_contradictions_flagged: 18,
|
||||
wilson_ci_lower: 0.14,
|
||||
wilson_ci_upper: 0.37,
|
||||
judge_errors_total: 0,
|
||||
cost_usd_total: 1.0,
|
||||
duration_ms: 45000,
|
||||
source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
|
||||
report_json: mkReport('r-test'),
|
||||
},
|
||||
]);
|
||||
expect(out).toContain('Date');
|
||||
expect(out).toContain('2026-05-11');
|
||||
expect(out).toContain('claude-haiku-4-5');
|
||||
});
|
||||
|
||||
test('multi-row chart has fully-filled bar for the max-value row', () => {
|
||||
const rows = [
|
||||
{
|
||||
run_id: 'big',
|
||||
ran_at: '2026-05-11T00:00:00Z',
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 25,
|
||||
total_contradictions_flagged: 100,
|
||||
wilson_ci_lower: 0.4, wilson_ci_upper: 0.6,
|
||||
judge_errors_total: 0, cost_usd_total: 5, duration_ms: 60000,
|
||||
source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
|
||||
report_json: mkReport('r-test'),
|
||||
},
|
||||
{
|
||||
run_id: 'small',
|
||||
ran_at: '2026-05-10T00:00:00Z',
|
||||
judge_model: 'anthropic:claude-haiku-4-5',
|
||||
queries_evaluated: 50,
|
||||
queries_with_contradiction: 1,
|
||||
total_contradictions_flagged: 5,
|
||||
wilson_ci_lower: 0.0, wilson_ci_upper: 0.1,
|
||||
judge_errors_total: 0, cost_usd_total: 1, duration_ms: 30000,
|
||||
source_tier_breakdown: { curated_vs_curated: 0, curated_vs_bulk: 0, bulk_vs_bulk: 0, other: 0 },
|
||||
report_json: mkReport('r-test'),
|
||||
},
|
||||
];
|
||||
const out = renderTrendChart(rows);
|
||||
const lines = out.split('\n');
|
||||
const bigLine = lines.find((l) => l.includes('2026-05-11'));
|
||||
const smallLine = lines.find((l) => l.includes('2026-05-10'));
|
||||
expect(bigLine).toBeTruthy();
|
||||
expect(smallLine).toBeTruthy();
|
||||
// Big run gets fully-filled bar; small run gets a near-empty bar.
|
||||
const bigFill = (bigLine!.match(/#/g) ?? []).length;
|
||||
const smallFill = (smallLine!.match(/#/g) ?? []).length;
|
||||
expect(bigFill).toBeGreaterThan(smallFill);
|
||||
});
|
||||
});
|
||||
+5
@@ -0,0 +1,5 @@
|
||||
{"query": "what is acme-example MRR"}
|
||||
{"query": "what role does alice-example hold at acme-example"}
|
||||
{"query": "did widget-co-example raise a Series A"}
|
||||
{"query": "what is the canonical thesis on remote-only startups"}
|
||||
{"query": "what is the latest take on AI agents in 2026"}
|
||||
Reference in New Issue
Block a user