v0.35.1.1: longmemeval fix wave (adapter + slug + gateway-wire) (#1056)

* docs(designs): 2026-05 embedder shootout eval plan

Adds docs/designs/2026_05_EVAL_PLAN.md — the approved plan + 6 Conductor session
briefs for the OpenAI vs Voyage vs ZeroEntropy embedder comparison.

Why: produce a publishable comparison report for v0.35.x release notes pinning
"which embedder wins, and does zerank-2 carry the win for ZeroEntropy" against
public LongMemEval + in-house BrainBench.

Each session brief is self-contained — repo, branch, commits, verify, ship,
deliverable, hand-off. Stewardable one section per Conductor session.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(pricing): add voyage-4-large + zembed-1 to EMBEDDING_PRICING

v0.35.0.0 shipped ZeroEntropy zembed-1 + zerank-2 reranker support and
expanded the Voyage allow-list to include voyage-4-large. The pricing
table missed both, so `gbrain upgrade`'s post-upgrade reembed prompt
silently fell back to "estimate unavailable" for users on these models.

- voyage:voyage-4-large @ $0.18/MTok (same as voyage-3-large)
- zeroentropyai:zembed-1 @ $0.05/MTok

New test file pins both entries plus the openai/voyage-3-large baselines,
case-insensitive provider matching, bare-model openai-default fallback,
table integrity (lowercase providers, finite non-negative prices), and
the estimateCostFromChars approximation. 11 cases, 46 expect() calls.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(exports): expose gbrain/ai/gateway with canary test

Adds ./ai/gateway to the package.json exports map so external eval
consumers (notably gbrain-evals, the sibling repo running the embedder
shootout in docs/designs/2026_05_EVAL_PLAN.md) can call configureGateway
directly to swap embedding providers per cell.

Why: pre-v0.35.1.0, gbrain-evals adapters hardcoded gbrain/embedding,
which means every retrieval adapter was OpenAI-only. The newly-exposed
gateway lets adapters route through Voyage and ZeroEntropy without
forking gbrain or duplicating the recipe wiring.

- package.json: add "./ai/gateway" -> "./src/core/ai/gateway.ts"
- scripts/check-exports-count.sh: bump expected count 17 -> 18
- test/public-exports.test.ts: add canary pinning configureGateway + embed,
  bump expected count assertion

Pre-existing import-resolution failures in this test file (16 on master)
are unrelated to this change — they're a longstanding Bun package
self-import behavior. The count + EXPECTED_EXPORTS list-match assertions
both pass cleanly.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* feat(eval): add --resume-from <jsonl> to gbrain eval longmemeval

Multi-cell embedder shootouts spend $50+/cell on the gpt-4o judge after
gbrain emits hypotheses. A mid-run abort (rate-limit, cost-cap, OS
interrupt, SIGKILL) previously meant re-paying the full cell. This flag
makes those aborts cheap: re-invoke with --resume-from pointed at the
partial JSONL and only the unanswered question_ids re-run.

Behavior:
- Read question_ids from the file; skip them on this run.
- Rows with non-empty hypothesis count as done.
- Rows with hypothesis="" AND an error field are NOT skipped (retry case
  for per-question failures recorded by the existing try/catch).
- Corrupt trailing lines (SIGKILL'd writer mid-line) are silently skipped
  with a stderr warn.
- When --resume-from path == --output path, the output emitter opens the
  file in append mode instead of truncating, so the existing rows survive.
- Empty resume case (all questions already done) returns immediately
  without spinning up the brain or calling the client.

New exported helper loadResumeSet() makes the parser unit-testable.

6 new test cases pinning:
- File-not-found returns empty set
- Well-formed JSONL load
- Error-row retry semantics (empty hypothesis + error -> not in set)
- Truncated final line recovery
- End-to-end resume against the 5-question mini fixture
- All-done early-return (stub client must NOT be invoked)

All 18 cases in test/eval-longmemeval.test.ts green; bun run typecheck
clean.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.35.1.0

Bumps VERSION + package.json + CHANGELOG entry for the embedder-shootout
prereq release. Three additive changes from the prior 4 commits:

- pricing: voyage-4-large + zembed-1 entries
- exports: gbrain/ai/gateway is now public
- eval: gbrain eval longmemeval --resume-from <jsonl>

Each commit on this branch is independently bisect-friendly and CI-green;
the CHANGELOG entry is the user-facing rollup. No migrations, no breaking
changes — the gateway export expands the surface, the resume-from flag is
additive, the pricing patch only changes "estimate unavailable" -> a real
dollar figure for two specific models.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(eval): longmemeval adapter handles _s split + sanitizes session_id slugs

Three tightly-coupled bugs blocked `gbrain eval longmemeval` against the
public LongMemEval _s split from HuggingFace (the dataset every shootout
cell needs):

1. HAYSTACK SHAPE: the _s split serializes haystack_sessions as
   LongMemEvalTurn[][] (each inner array is one session's turns directly)
   plus a parallel `haystack_session_ids: string[]` field. The
   pre-v0.35.1.1 adapter expected only the oracle `{session_id, turns}`
   shape and crashed with `session.turns is undefined` on every question.
   Fix: new `normalizeSessions` helper accepts both shapes, mirroring the
   proven `normalizeSessions` in gbrain-evals/eval/runner/longmemeval.ts.

2. SLUG VALIDATOR: the _s split's session_ids look like
   `sharegpt_yywfIrx_0` — underscored and mixed-case. The v0.32.7 CJK
   wave's `validatePageSlug` rejects both (allowed set is `[a-z0-9-]`
   case-insensitive, slash-separated). Fix: `sanitizeSessionIdForSlug`
   lowercases and replaces `_` + `.` + any other non-[a-z0-9-] character
   with `-`. The frontmatter `session_id:` keeps the original verbatim
   for downstream JSONL emit; only the SLUG is rewritten.

3. INTERFACE: `LongMemEvalQuestion.haystack_sessions` typed as a union
   of `LongMemEvalSession[] | LongMemEvalTurn[][]` so TypeScript callers
   see both shapes are accepted. New `haystack_session_ids?: string[]`
   field documented as parallel to the array-of-turns shape.

Pre-v0.35.1.1 caught by a fresh smoke pre-spend (3 questions × ZE @ 2560
→ 3 errors). Post-fix: 3/3 OK with non-empty hypotheses, single-session
recall measured (low on a 3-question sample but the pipeline runs).

2 new regression test cases pinning:
- _s split shape normalizes (slugs sanitized + frontmatter preserves
  original session_id + dates flow through)
- _s split with missing haystack_session_ids synthesizes
  `lme_<question_id>_<i>` ids

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* fix(cli): configure AI gateway before running gbrain eval longmemeval

v0.28.8 skipped connectEngine() for `gbrain eval longmemeval` so the
subcommand could run on machines without a configured brain. Side
effect (silent until v0.35.1.0 made it observable via the embedder
shootout): the gateway was never configureGateway()'d either, so the
first embed call inside importFromContent crashed with "AI gateway is
not configured. Call configureGateway() during engine connect."

Fix: call configureGateway() before runEvalLongMemEval, mirroring the
connectEngine() path. Reads `~/.gbrain/config.json` when present; falls
back to env vars (GBRAIN_EMBEDDING_MODEL, GBRAIN_EMBEDDING_DIMENSIONS,
OPENAI_API_KEY, etc.) when there's no config — preserving the v0.28.8
"runs on fresh machine" property.

Gated on the --help short-circuit so `gbrain eval longmemeval --help`
still works without spinning up the gateway.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* chore: v0.35.1.1

Bumps VERSION + package.json + CHANGELOG entry for the longmemeval fix
wave. Three commits this branch:

1. fix(eval): adapter handles _s split + sanitizes session_id slugs
2. fix(cli): configure AI gateway before running gbrain eval longmemeval
3. chore: v0.35.1.1

Each commit independently bisects; CHANGELOG entry is the user-facing
rollup. No schema migration; no breaking change.

Caught pre-spend by smoking Phase 1 of the embedder shootout — would
otherwise have wasted ~$476 in judge tokens across 7 cells.

Co-Authored-By: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

* ci: retrigger workflows

---------

Co-authored-by: Claude Opus 4.7 (1M context) <noreply@anthropic.com>

CHANGELOG.md

@@ -2,6 +2,41 @@
 
 All notable changes to GBrain will be documented in this file.
 
+## [0.35.1.1] - 2026-05-16
+
+**Fix wave: `gbrain eval longmemeval` actually runs against the public _s split.**
+
+A pre-spend smoke for the upcoming embedder shootout caught three tightly-coupled bugs that would have made all 7 cells fail. Shipping the fixes before anyone burns judge tokens.
+
+### What this fixes
+
+**The longmemeval adapter accepts the public _s dataset shape.** Pre-v0.35.1.1 assumed every dataset used the oracle `{session_id, turns}` shape, but the HuggingFace _s split serializes sessions as a parallel `haystack_session_ids: string[]` + a `LongMemEvalTurn[][]` (each inner array is one session's turns directly). The old adapter crashed `session.turns is undefined` on every question. New `normalizeSessions` helper accepts both shapes, mirroring the proven path in `gbrain-evals/eval/runner/longmemeval.ts`.
+
+**Session IDs that contain underscores or uppercase letters now produce valid slugs.** The _s split's IDs look like `sharegpt_yywfIrx_0`, both of which the v0.32.7 CJK-wave slug validator rejects. New `sanitizeSessionIdForSlug` lowercases and rewrites disallowed chars to `-`. The frontmatter `session_id:` line still carries the original verbatim, so downstream JSONL emit + LongMemEval correctness scoring work unchanged — only the slug gets rewritten to satisfy the validator.
+
+**`gbrain eval longmemeval` now configures the AI gateway before running.** v0.28.8 deliberately skipped `connectEngine()` for this subcommand so it would run on machines without a configured brain. Side effect: the gateway never got configured either, so the first embed call inside `importFromContent` crashed with "AI gateway is not configured." Fix: explicit `configureGateway()` before `runEvalLongMemEval`, reading `~/.gbrain/config.json` if present and falling back to env vars (`GBRAIN_EMBEDDING_MODEL`, `GBRAIN_EMBEDDING_DIMENSIONS`, etc.) when there's no config — preserving the "runs on a fresh machine" property.
+
+### Itemized changes
+
+- `src/eval/longmemeval/adapter.ts`: new `normalizeSessions` accepts both oracle (`{session_id, turns}`) and _s (`Turn[][]` + parallel `haystack_session_ids`) shapes; new `sanitizeSessionIdForSlug` rewrites underscores + uppercase + other disallowed chars to `-`; `LongMemEvalQuestion.haystack_sessions` typed as the union, `haystack_session_ids?: string[]` field added with documentation.
+- `src/cli.ts`: gateway configure step added before the `eval longmemeval` dispatch path, gated on `--help` short-circuit so help still works without a configured gateway.
+- `test/eval-longmemeval.test.ts`: 2 new regression cases pinning the _s shape normalization end-to-end (slugs sanitized, frontmatter preserves original session_id, dates flow through) and the missing-haystack_session_ids fallback to synthesized `lme_<question_id>_<i>` ids.
+
+## To take advantage of v0.35.1.1
+
+`gbrain upgrade` handles the fix transparently. Re-run any LongMemEval-against-_s-split commands that had been crashing.
+
+1. **Upgrade:**
+   ```bash
+   gbrain upgrade
+   ```
+2. **(If you hit the prior crash) re-run with `--resume-from` to skip any questions already scored:**
+   ```bash
+   gbrain eval longmemeval ~/datasets/longmemeval/longmemeval_s.json \
+     --output results.jsonl --resume-from results.jsonl --mode tokenmax
+   ```
+3. **No migration, no schema change, no breaking semantics.**
+
 ## [0.35.1.0] - 2026-05-15
 
 **Embedder shootout prereqs: pricing, public gateway export, and resume-from for long eval runs.**

VERSION

@@ -1 +1 @@
-0.35.1.0
+0.35.1.1

package.json

@@ -1,6 +1,6 @@
 {
   "name": "gbrain",
-  "version": "0.35.1.0",
+  "version": "0.35.1.1",
   "description": "Postgres-native personal knowledge brain with hybrid RAG search",
   "type": "module",
   "main": "src/core/index.ts",

src/cli.ts

@@ -964,8 +964,22 @@ async function handleCliOnly(command: string, args: string[]) {
   // v0.28.8: longmemeval brings its own in-memory PGLite. Bypassing
   // connectEngine here keeps `gbrain eval longmemeval --help` and benchmark
   // runs working on machines that have no `~/.gbrain/config.json` configured.
+  //
+  // v0.35.1.1: still need to configureGateway() so the in-memory brain's
+  // import + hybridSearch can embed via the configured provider. Reads
+  // ~/.gbrain/config.json when present; falls back to env vars otherwise
+  // (GBRAIN_EMBEDDING_MODEL / GBRAIN_EMBEDDING_DIMENSIONS).
   if (command === 'eval' && args[0] === 'longmemeval') {
     const { runEvalLongMemEval } = await import('./commands/eval-longmemeval.ts');
+    if (!(args.length > 1 && (args[1] === '--help' || args[1] === '-h'))) {
+      const config = loadConfig() ?? ({
+        embedding_model: process.env.GBRAIN_EMBEDDING_MODEL,
+        embedding_dimensions: process.env.GBRAIN_EMBEDDING_DIMENSIONS
+          ? Number(process.env.GBRAIN_EMBEDDING_DIMENSIONS) : undefined,
+      } as GBrainConfig);
+      const { configureGateway } = await import('./core/ai/gateway.ts');
+      configureGateway(buildGatewayConfig(config));
+    }
     await runEvalLongMemEval(args.slice(1));
     return;
   }

src/eval/longmemeval/adapter.ts

@@ -28,7 +28,18 @@ export interface LongMemEvalQuestion {
   question_type: string;
   question: string;
   answer: string;
-  haystack_sessions: LongMemEvalSession[];
+  /**
+   * Two on-disk shapes are accepted (normalized by `haystackToPages`):
+   *
+   *   1. Oracle/structured: `LongMemEvalSession[]` with `{session_id, turns}`.
+   *   2. _s split (HuggingFace public download as of May 2026):
+   *      `LongMemEvalTurn[][]` — each inner array is the turns of one
+   *      session directly. Session IDs live in a sibling
+   *      `haystack_session_ids: string[]` parallel array.
+   */
+  haystack_sessions: LongMemEvalSession[] | LongMemEvalTurn[][];
+  /** Parallel to haystack_sessions in the _s split. Absent in oracle shape. */
+  haystack_session_ids?: string[];
   /** ISO date strings, parallel to haystack_sessions. Some LongMemEval splits omit this. */
   haystack_dates?: string[];
   /** Ground truth: which haystack sessions actually contain the answer. */
@@ -61,14 +72,66 @@ function renderSession(session: LongMemEvalSession, date?: string): string {
   return fm.join('\n') + body.join('\n');
 }
 
+/**
+ * Normalize the on-disk haystack_sessions shape (oracle OR _s) into the
+ * structured `{session_id, turns}` form `renderSession` consumes.
+ *
+ * v0.35.1.1: the public _s split on HuggingFace uses `LongMemEvalTurn[][]`
+ * for `haystack_sessions` plus a parallel `haystack_session_ids: string[]`
+ * for the IDs. The pre-v0.35.1.1 adapter assumed only the oracle shape
+ * and crashed with `session.turns` undefined on the _s split. This
+ * normalizer accepts both. Mirrors the proven `normalizeSessions` helper
+ * in gbrain-evals/eval/runner/longmemeval.ts.
+ */
+function normalizeSessions(question: LongMemEvalQuestion): LongMemEvalSession[] {
+  const sessions: LongMemEvalSession[] = [];
+  const ids = question.haystack_session_ids ?? [];
+  const raw = question.haystack_sessions;
+  for (let i = 0; i < raw.length; i++) {
+    const item = raw[i] as unknown;
+    if (Array.isArray(item)) {
+      // _s shape: this entry is a turn array directly.
+      const sid = ids[i] ?? `lme_${question.question_id}_${i}`;
+      sessions.push({ session_id: sid, turns: item as LongMemEvalTurn[] });
+    } else if (item && typeof item === 'object' && Array.isArray((item as LongMemEvalSession).turns)) {
+      // Oracle shape: {session_id, turns} object.
+      const sess = item as LongMemEvalSession;
+      sessions.push({
+        session_id: sess.session_id ?? `lme_${question.question_id}_${i}`,
+        turns: sess.turns,
+      });
+    }
+    // Silently skip malformed entries — keeps the run progressing on
+    // mixed/corrupted datasets; the surrounding per-question try/catch
+    // catches whole-question failures anyway.
+  }
+  return sessions;
+}
+
+/**
+ * Normalize an arbitrary session_id into something `validatePageSlug` accepts.
+ *
+ * Validator rules (per v0.32.7 CJK wave): segments are `[a-z0-9CJK\-]+`,
+ * case-insensitive, forward-slash separated. The HuggingFace _s split uses
+ * `sharegpt_yywfIrx_0`-style ids with underscores AND uppercase letters,
+ * both of which are rejected. Lowercase + underscore -> hyphen produces a
+ * stable, validator-passing alias. Collisions are negligible per question
+ * (each question's slug-space is reset per benchmark question by the
+ * harness's resetTables).
+ */
+function sanitizeSessionIdForSlug(sessionId: string): string {
+  return sessionId.toLowerCase().replace(/[_.]/g, '-').replace(/[^a-z0-9-]/g, '-');
+}
+
 export function haystackToPages(question: LongMemEvalQuestion): PageInputForImport[] {
   const pages: PageInputForImport[] = [];
   const dates = question.haystack_dates ?? [];
-  for (let i = 0; i < question.haystack_sessions.length; i++) {
-    const session = question.haystack_sessions[i];
+  const sessions = normalizeSessions(question);
+  for (let i = 0; i < sessions.length; i++) {
+    const session = sessions[i];
     const date = dates[i];
     pages.push({
-      slug: `chat/${session.session_id}`,
+      slug: `chat/${sanitizeSessionIdForSlug(session.session_id)}`,
       content: renderSession(session, date),
     });
   }

test/eval-longmemeval.test.ts

@@ -249,6 +249,57 @@ describe('adapter haystackToPages', () => {
     expect(pages[0].content).toContain('session_id: sess-x');
     expect(pages[0].content).not.toContain('date:');
   });
+
+  // v0.35.1.1 regression: the public LongMemEval _s split uses arrays of
+  // turn-arrays for haystack_sessions plus a parallel haystack_session_ids
+  // string array. The pre-v0.35.1.1 adapter crashed with `session.turns is
+  // undefined` on this shape. Pre-v0.35.1.1 the slug validator also
+  // rejected the underscored, mixed-case session_ids the dataset uses.
+  test('v0.35.1.1: _s split shape (turn-array + parallel ids) normalizes correctly', () => {
+    const q: LongMemEvalQuestion = {
+      question_id: 'q-s-1',
+      question_type: 'single-session-user',
+      question: 'q?',
+      answer: 'a',
+      haystack_dates: ['2025-01-01', '2025-01-02'],
+      answer_session_ids: ['sharegpt_AbC_0'],
+      haystack_session_ids: ['sharegpt_AbC_0', 'sess_DEF_1'],
+      // No {session_id, turns} — turns directly per the _s shape.
+      haystack_sessions: [
+        [{ role: 'user', content: 'hi' }, { role: 'assistant', content: 'hello' }],
+        [{ role: 'user', content: 'bye' }],
+      ],
+    };
+    const pages = haystackToPages(q);
+    expect(pages.length).toBe(2);
+    // Slugs got lowercased + underscores became hyphens (validator-safe).
+    expect(pages[0].slug).toBe('chat/sharegpt-abc-0');
+    expect(pages[1].slug).toBe('chat/sess-def-1');
+    // Frontmatter keeps the ORIGINAL session_id (no sanitization). The
+    // _s ids preserve through the round-trip; only the slug got rewritten.
+    expect(pages[0].content).toContain('session_id: sharegpt_AbC_0');
+    expect(pages[0].content).toContain('date: 2025-01-01');
+    expect(pages[0].content).toContain('**user:** hi');
+    expect(pages[1].content).toContain('**user:** bye');
+  });
+
+  test('v0.35.1.1: missing haystack_session_ids on _s shape synthesizes ids per question', () => {
+    const q: LongMemEvalQuestion = {
+      question_id: 'q-s-2',
+      question_type: 'single-session-user',
+      question: 'q?',
+      answer: 'a',
+      answer_session_ids: [],
+      // _s shape but the parallel ids array is absent. Adapter falls back
+      // to a synthesized `lme_<question_id>_<i>` slug.
+      haystack_sessions: [
+        [{ role: 'user', content: 'turn 1' }],
+      ],
+    };
+    const pages = haystackToPages(q);
+    expect(pages.length).toBe(1);
+    expect(pages[0].slug).toBe('chat/lme-q-s-2-0');
+  });
 });
 
 // ---------------------------------------------------------------------------