mirror of
https://github.com/garrytan/gbrain.git
synced 2026-08-14 08:53:22 +00:00
Garry TanandClaude Fable 5
f83449be28
Every other Cohere test asserts against a mock response we authored, so a
wire-shape drift would keep CI green while the DEFAULT reranker silently
broke on the query hot path. This adds the only test that can catch that.
Live-verified against api.cohere.com/v2/rerank on 2026-07-28:
- real response shape is {results:[{index, relevance_score}]} — matches the
parser exactly, so the mocks were accidentally correct (now established)
- ranking is semantically right; gateway maps to {index, relevanceScore}
- measured latency ~0.2s, not the ~614ms third-party leaderboards claim
- rerank-v4.0-pro is UNREACHABLE on a trial key (no response, 20s+) while
v3.5 and v4.0-fast answered in 0.2s the same second — excluded from the
assertion and documented, since asserting it would fail for every
trial-tier contributor
Gated on COHERE_API_KEY; skips cleanly (4 skip, 0 fail) without it.
Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>