fix(evals): harden terminalbench-native harness against tmux death and setup hangs (#536)

Two failure classes hit by a downstream team:

1) tmux/task-env death (TerminalBenchTaskEnv.__enter__, terminalbench_env.py):
   create_session ran inside the spin_up_terminal generator-CM with no
   exception safety — a tmux failure leaked the docker compose project
   (down deferred to GC, never if the env was retained) and surfaced as an
   opaque mid-run death. Now: exception-safe __enter__ with an idempotent
   _teardown(), a tmux/asciinema preflight in the container BEFORE the
   agent loop (TaskEnvironmentError naming the task image + remedy), and
   fail-that-task-cleanly semantics — the failure is recorded as a harness
   error (QueryTrace.error_kind="harness_error"), the container is downed,
   and the run continues.

2) OpenHands in-container SETUP hang misattributed as a model result:
   harness.run() had no bound (terminal-bench runs installed-agent setup
   with max_timeout_sec=inf inside the per-trial agent budget), and the
   summary conversion read the nonexistent results.trial_results attr and
   hardcoded errors=0, folding zero-model-request trials into resolve-rate
   as model misses. Now: global_agent_timeout_sec / global_timeout_multiplier
   are threaded config -> backend -> Harness kwargs (default 1800 s bound on
   SETUP+RUN; configurable per [run]/[[benchmarks]] TOML), and
   summarize_benchmark_results() classifies harness/infra failures out of
   the accuracy denominator keyed on token usage (zero/missing tokens +
   unresolved = the agent never contacted the model), NOT failure_mode —
   terminal-bench 0.2.18 leaves failure_mode UNSET on success AND on
   genuine misses, so a failure_mode-based check would misflag every real
   model miss. Genuine misses (tokens>0, is_resolved=false) stay in the
   denominator.

QueryTrace gains error/error_kind (wired through to_dict/from_dict so the
fields actually reach traces.jsonl; backward-compatible loads), the
AgenticRunner flags zero-model-contact traces and TaskEnvironmentError as
harness errors, and export/summary/console output exclude harness errors
from resolve-rate while reporting them loudly.

terminal-bench stays an undeclared dep on purpose: it requires Python
>=3.12 while this project supports >=3.10,<3.14, so an unmarked extra
would break uv lock. pyproject/uv.lock untouched.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/clone-stats/badge.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": 1,
   "label": "Git Clones",
-  "message": "97,189",
+  "message": "110,366",
   "color": "green",
   "namedLogo": "git"
 }

.github/clone-stats/clone-data.json

@@ -1,6 +1,6 @@
 {
-  "total_clones": 97189,
-  "last_updated": "2026-06-04T07:41:08Z",
+  "total_clones": 110366,
+  "last_updated": "2026-06-11T07:44:18Z",
   "daily": {
     "2026-03-27": 2189,
     "2026-03-28": 1874,
@@ -70,6 +70,13 @@
     "2026-05-31": 1887,
     "2026-06-01": 2072,
     "2026-06-02": 1847,
-    "2026-06-03": 2164
+    "2026-06-03": 2164,
+    "2026-06-04": 2632,
+    "2026-06-05": 2127,
+    "2026-06-06": 2204,
+    "2026-06-07": 1174,
+    "2026-06-08": 2369,
+    "2026-06-09": 1361,
+    "2026-06-10": 1310
   }
 }

.github/workflows/ci.yml

@@ -64,7 +64,7 @@ jobs:
 
       - name: Run tests
         run: |
-          uv run pytest tests/ -v --tb=short -m "not live and not cloud" \
+          uv run pytest tests/ -v --tb=short -m "not live and not cloud and not hub" \
             --cov=openjarvis \
             --cov-report=term-missing \
             --cov-report=xml \

README.md

@@ -8,7 +8,7 @@
     <a href="https://open-jarvis.github.io/OpenJarvis/"><img src="https://img.shields.io/badge/docs-mkdocs-blue" alt="Docs"></a>
     <img src="https://img.shields.io/badge/python-%3E%3D3.10-blue" alt="Python">
     <img src="https://img.shields.io/badge/license-Apache%202.0-green" alt="License">
-    <a href="https://discord.gg/6ZtCB94h5p"><img src="https://img.shields.io/badge/discord-join-7289da?logo=discord&logoColor=white" alt="Discord"></a>
+    <a href="https://discord.gg/CMVBmDQ5Fj"><img src="https://img.shields.io/badge/discord-join-7289da?logo=discord&logoColor=white" alt="Discord"></a>
     <a href="https://x.com/OpenJarvisAI"><img src="https://img.shields.io/badge/X-@OpenJarvisAI-black?logo=x&logoColor=white" alt="X / Twitter"></a>
   </p>
 </div>
@@ -25,6 +25,8 @@
 >
 > **[Project Site](https://scalingintelligence.stanford.edu/blogs/openjarvis/)**
 >
+> **[Paper](https://arxiv.org/abs/2605.17172)**
+>
 > **[Leaderboard](https://open-jarvis.github.io/OpenJarvis/leaderboard/)**
 >
 > **[Roadmap](https://open-jarvis.github.io/OpenJarvis/development/roadmap/)**
@@ -121,7 +123,7 @@ Full documentation — including Docker deployment, cloud engines, development s
 ## Community
 
 - **GitHub:** [github.com/open-jarvis/OpenJarvis](https://github.com/open-jarvis/OpenJarvis)
-- **Discord:** [discord.gg/YZZRxCAhmm](https://discord.gg/YZZRxCAhmm)
+- **Discord:** [discord.gg/CMVBmDQ5Fj](https://discord.gg/CMVBmDQ5Fj)
 - **X / Twitter:** [@OpenJarvisAI](https://x.com/OpenJarvisAI)
 - **Docs:** [open-jarvis.github.io/OpenJarvis](https://open-jarvis.github.io/OpenJarvis/)
 

configs/openjarvis/config.toml

@@ -106,6 +106,11 @@ enabled = true                            # Record traces for analysis
 db_path = "~/.openjarvis/traces.db"
 
 [server]
-host = "0.0.0.0"
+# Bind to loopback by default so the API is not exposed to the local network.
+# To serve other devices on your LAN, set host = "0.0.0.0" AND set an API key
+# (OPENJARVIS_API_KEY / `jarvis auth generate-key`) — startup refuses a
+# non-loopback bind without a key. The "server" security profile also flips
+# this to 0.0.0.0 intentionally.
+host = "127.0.0.1"
 port = 8000
 agent = "native_openhands"

deploy/windows/install.ps1

@@ -5,12 +5,12 @@
 .DESCRIPTION
     Phase-1 of the native-Windows-support RFC (#298). Mirrors the
     behavior of scripts/install/install.sh (the curl-pipe-bash installer
-    for Linux/WSL2/macOS) but for native Windows PowerShell — no WSL,
+    for Linux/WSL2/macOS) but for native Windows PowerShell - no WSL,
     no Docker, no MSYS2.
 
     Steps:
       1. Refuse non-Windows / Windows < 10.
-      2. Check Python 3.10 — 3.13 on PATH (3.14 has no numpy wheels yet,
+      2. Check Python 3.10 - 3.13 on PATH (3.14 has no numpy wheels yet,
          see #432).
       3. Check git on PATH.
       4. Install uv (https://astral.sh/uv) if absent.
@@ -65,7 +65,7 @@ if (-not $Service     -and $env:OPENJARVIS_SERVICE)      { $Service     = $true
 if (-not $Force       -and $env:OPENJARVIS_FORCE)        { $Force       = $true }
 
 # ---------------------------------------------------------------------------
-# Output helpers — coloured but plain enough for Constrained Language Mode.
+# Output helpers - coloured but plain enough for Constrained Language Mode.
 # ---------------------------------------------------------------------------
 
 function Write-Info  ($msg) { Write-Host "[info]  $msg" -ForegroundColor Cyan }
@@ -77,13 +77,13 @@ function Write-Fail  ($msg) {
 }
 
 # ---------------------------------------------------------------------------
-# Shared helpers — winget bootstrap + PATH refresh
+# Shared helpers - winget bootstrap + PATH refresh
 # ---------------------------------------------------------------------------
 
 # Pull the latest Machine + User PATH from the registry into the current
 # PowerShell session. Tools installed by `winget install` (Python, git,
 # Ollama, etc.) update the User PATH, but the running process inherits
-# the parent shell's environment — so without this refresh the just-
+# the parent shell's environment - so without this refresh the just-
 # installed tool stays invisible to subsequent `Get-Command` calls.
 #
 # CRITICAL: registry PATH entries can be REG_EXPAND_SZ (with literal
@@ -157,7 +157,7 @@ function Get-PythonCommand {
 Write-Info "Checking Python (3.10 - 3.13)..."
 $pythonExe = Get-PythonCommand
 if (-not $pythonExe) {
-    Write-Info "Python not on PATH — attempting auto-install via winget..."
+    Write-Info "Python not on PATH - attempting auto-install via winget..."
     $pythonExe = Install-WithWinget -WingetId 'Python.Python.3.13' -CommandName 'python'
     if (-not $pythonExe) {
         Write-Fail @"
@@ -196,7 +196,7 @@ Write-Ok "Python $pyMajor.$pyMinor ($pythonExe)"
 Write-Info "Checking git..."
 $gitExe = (Get-Command git -ErrorAction SilentlyContinue).Source
 if (-not $gitExe) {
-    Write-Info "git not on PATH — attempting auto-install via winget..."
+    Write-Info "git not on PATH - attempting auto-install via winget..."
     $gitExe = Install-WithWinget -WingetId 'Git.Git' -CommandName 'git'
     if (-not $gitExe) {
         Write-Fail @"
@@ -227,7 +227,7 @@ if (-not $uvExe) {
     }
     # The astral installer puts uv at %USERPROFILE%\.local\bin\uv.exe and
     # adds that dir to the User PATH. The current process's PATH isn't
-    # refreshed automatically — prepend the install dir so the rest of
+    # refreshed automatically - prepend the install dir so the rest of
     # this script picks it up.
     $uvDir = Join-Path $env:USERPROFILE '.local\bin'
     if (Test-Path (Join-Path $uvDir 'uv.exe')) {
@@ -295,13 +295,13 @@ try {
 Write-Ok "Dependencies installed"
 
 # ---------------------------------------------------------------------------
-# 7. Ollama — install + start + wait for daemon
+# 7. Ollama - install + start + wait for daemon
 # ---------------------------------------------------------------------------
 
 Write-Info "Checking Ollama..."
 $ollamaExe = (Get-Command ollama -ErrorAction SilentlyContinue).Source
 if (-not $ollamaExe) {
-    Write-Info "  Ollama not on PATH — downloading the official installer (~150 MB)..."
+    Write-Info "  Ollama not on PATH - downloading the official installer (~150 MB)..."
     $ollamaSetup = Join-Path $env:TEMP 'OllamaSetup.exe'
     # SilentlyContinue is load-bearing in PS 5.1: the default progress
     # bar renderer slows Invoke-WebRequest down 30x on large downloads
@@ -340,13 +340,18 @@ Write-Ok "Ollama ($ollamaExe)"
 Write-Info "Waiting for Ollama daemon..."
 $ollamaReady = $false
 for ($i = 0; $i -lt 60; $i++) {
-    & $ollamaExe list 2>&1 | Out-Null
+    # 'ollama list' writes to stderr until the daemon is reachable; under
+    # $ErrorActionPreference='Stop' the 2>&1 merge surfaces that as a
+    # terminating NativeCommandError that would abort the whole install on
+    # the very first probe. Swallow it and rely on $LASTEXITCODE so the
+    # Start-Process serve fallback below actually runs (issue #522).
+    try { & $ollamaExe list 2>&1 | Out-Null } catch { }
     if ($LASTEXITCODE -eq 0) {
         $ollamaReady = $true
         break
     }
     if ($i -eq 5) {
-        # Daemon clearly isn't auto-running — start it ourselves. Ollama
+        # Daemon clearly isn't auto-running - start it ourselves. Ollama
         # for Windows uses the tray app `ollama app.exe`; falling back to
         # `ollama serve` works headless.
         Start-Process -FilePath $ollamaExe -ArgumentList 'serve' -WindowStyle Hidden -ErrorAction SilentlyContinue
@@ -354,11 +359,11 @@ for ($i = 0; $i -lt 60; $i++) {
     Start-Sleep -Seconds 1
 }
 if (-not $ollamaReady) {
-    Write-Warn2 "Ollama daemon didn't become ready in 60s. Continuing — bg-orchestrator will retry later."
+    Write-Warn2 "Ollama daemon didn't become ready in 60s. Continuing - bg-orchestrator will retry later."
 }
 
 # ---------------------------------------------------------------------------
-# 8. Pull a starter model (qwen3.5:2b — ~1.5 GB)
+# 8. Pull a starter model (qwen3.5:2b - ~1.5 GB)
 # ---------------------------------------------------------------------------
 
 $modelPullOk = $false
@@ -372,11 +377,11 @@ if ($ollamaReady) {
         Write-Warn2 "ollama pull failed; the bg-orchestrator will retry once Ollama is reachable."
     }
 } else {
-    Write-Warn2 "Skipping model pull — daemon wasn't ready."
+    Write-Warn2 "Skipping model pull - daemon wasn't ready."
 }
 
 # ---------------------------------------------------------------------------
-# 9. jarvis.cmd shim — so bare `jarvis` works in any new PowerShell
+# 9. jarvis.cmd shim - so bare `jarvis` works in any new PowerShell
 # ---------------------------------------------------------------------------
 
 $binDir = Join-Path $installRoot 'bin'
@@ -387,7 +392,7 @@ if (-not (Test-Path $binDir)) {
 }
 
 # %~dp0 in a .cmd file resolves to the directory containing the script,
-# so the shim is self-locating — moving %LOCALAPPDATA%\OpenJarvis won't
+# so the shim is self-locating - moving %LOCALAPPDATA%\OpenJarvis won't
 # break it as long as the user moves the whole tree. `uv` is resolved
 # from PATH at runtime (astral installer adds it to User PATH); avoids
 # pinning to the install-time uv.exe path which can shift on uv updates.
@@ -400,7 +405,7 @@ uv run --project "%SRC%" jarvis %*
 Set-Content -Path $shimPath -Value $shimContent -Encoding ASCII
 
 # Add %LOCALAPPDATA%\OpenJarvis\bin to User PATH if it isn't already
-# there. The current process won't see it until restart — handled in the
+# there. The current process won't see it until restart - handled in the
 # final banner.
 #
 # Compare against the EXPANDED form: a previous install may have written
@@ -430,7 +435,7 @@ Write-Ok "jarvis shim installed at $shimPath"
 $serviceScript = Join-Path $srcDir 'deploy\windows\jarvis-service.ps1'
 $shouldInstallService = $false
 
-# Pre-check admin if the user wants the service — Register-ScheduledTask
+# Pre-check admin if the user wants the service - Register-ScheduledTask
 # requires elevation. We do this before the prompt so we don't ask "do
 # you want the service?" only to fail with Access Denied after they say
 # yes.
@@ -439,7 +444,7 @@ $isAdmin = ([Security.Principal.WindowsPrincipal] `
 ).IsInRole([Security.Principal.WindowsBuiltInRole]::Administrator)
 
 if ($Service -and -not $isAdmin) {
-    Write-Fail "-Service was requested, but this PowerShell is not elevated. Register-ScheduledTask needs admin rights — re-run from an elevated PowerShell, or drop -Service."
+    Write-Fail "-Service was requested, but this PowerShell is not elevated. Register-ScheduledTask needs admin rights - re-run from an elevated PowerShell, or drop -Service."
 }
 if ($Service) {
     $shouldInstallService = $true
@@ -448,7 +453,7 @@ if ($Service) {
 } elseif (-not $isAdmin) {
     # Default to skip-with-explanation when we can't elevate, rather
     # than prompting and then failing at Register-ScheduledTask.
-    Write-Warn2 "Skipping scheduled-task setup — this PowerShell is not elevated."
+    Write-Warn2 "Skipping scheduled-task setup - this PowerShell is not elevated."
     Write-Warn2 "  Register-ScheduledTask requires admin. To install the service later:"
     Write-Warn2 "    Right-click PowerShell -> Run as administrator, then run:"
     Write-Warn2 "    powershell -ExecutionPolicy Bypass -File `"$serviceScript`" install"
@@ -464,7 +469,7 @@ if ($Service) {
         $reply = Read-Host "Register OpenJarvis as a Windows scheduled task (auto-start at logon, loopback only)? [y/N]"
         $shouldInstallService = ($reply -match '^[yY]')
     } else {
-        Write-Warn2 "Non-interactive install — skipping scheduled-task setup."
+        Write-Warn2 "Non-interactive install - skipping scheduled-task setup."
         Write-Warn2 "To register the service later, run (from an elevated PowerShell):"
         Write-Warn2 "  powershell -ExecutionPolicy Bypass -File `"$serviceScript`" install"
     }
@@ -487,9 +492,9 @@ if ($shouldInstallService) {
 # ---------------------------------------------------------------------------
 
 Write-Host ""
-Write-Host "  ┌──────────────────────────────────┐" -ForegroundColor Green
-Write-Host "  │   OpenJarvis install complete    │" -ForegroundColor Green
-Write-Host "  └──────────────────────────────────┘" -ForegroundColor Green
+Write-Host "  +----------------------------------+" -ForegroundColor Green
+Write-Host "  |   OpenJarvis install complete    |" -ForegroundColor Green
+Write-Host "  +----------------------------------+" -ForegroundColor Green
 Write-Host ""
 Write-Host "  Repo:    $srcDir"
 

docs/assets/showcase/README.md

@@ -0,0 +1,45 @@
+# Showcase screenshots
+
+This directory holds the hero screenshot for each Showcase entry in `docs/showcase/`. Convention is one file per entry, named to match the entry's slug:
+
+| Entry | Screenshot path |
+|---|---|
+| `docs/showcase/morning-brief.md` | `morning-brief.png` |
+| `docs/showcase/persistent-memory.md` | `persistent-memory.png` |
+| `docs/showcase/cost-savings.md` | `cost-savings.png` |
+| `docs/showcase/discord-companion.md` | `discord-companion.png` |
+| `docs/showcase/coding-assistant.md` | `coding-assistant.png` |
+
+## Conventions
+
+| | |
+|---|---|
+| Format | PNG, sRGB, no alpha channel |
+| Size | 1600×1000 (4:2.5 — wider than 16:9, so screenshots don't get letterboxed in the docs grid) |
+| File size | Under 400 KB after `pngquant --quality 70-90 --speed 1` |
+| Loading | All `<img>` and `<figure>` tags in showcase pages use `loading=lazy` — these images are below the fold on the gallery page |
+
+## What to redact
+
+- Real email addresses
+- API keys, OAuth tokens, anything starting with `sk-`, `ghp_`, `xox`, `eyJ`
+- Personal phone numbers
+- Conversation partners' faces or full names (unless they've signed off)
+- File paths that include other people's home directories
+
+## What to keep
+
+- Model names ("llama3.1:8b", "qwen2.5:14b") — they're informative
+- Timestamps — proves the screenshot is recent
+- Dollar amounts on the leaderboard — the whole point
+- Emoji reactions, your own first name, your own avatar
+
+## Placeholder PNGs
+
+This directory ships with no images on the initial PR. The Showcase pages reference image paths that don't exist yet — MkDocs will render a broken-image placeholder, and the figcaption still conveys what should be there. Real screenshots arrive in follow-up PRs as Showcase entries are populated with each contributor's actual setup.
+
+If you're contributing the first real entry, drop your PNG at `docs/assets/showcase/<your-slug>.png` in the same PR that adds your markdown page. The image filename must match the slug used in the showcase page's `<img>` reference.
+
+## Regenerating screenshots in bulk
+
+A future enhancement (tracked as PR #3 in the showcase-tier roadmap) will add `scripts/showcase/regen_screenshots.py` — a Playwright-driven pipeline that boots a demo `jarvis serve` against a sealed config and captures fresh screenshots for every showcase entry on each release tag. Until that lands, screenshots are contributed manually by each Showcase author.

docs/index.md

@@ -14,6 +14,18 @@ OpenJarvis is a research framework for composable, on-device AI systems.
 Build personal AI that runs on your hardware. Cloud APIs are optional.
 </p>
 
+<div class="grid cards" markdown>
+
+-   :material-image-multiple:{ .lg .middle } **See what people use it for**
+
+    ---
+
+    A gallery of real setups — morning briefs that summarize your overnight Slack and email, a Discord companion that knows your calendar, a code reviewer that works at 30,000 feet. Outcome-first, with links to the docs that explain how to build each one.
+
+    [:octicons-arrow-right-24: Browse the Showcase](showcase/index.md)
+
+</div>
+
 ---
 
 ## Why OpenJarvis?
@@ -171,7 +183,7 @@ OpenJarvis is built around five composable layers. Each has a clean interface an
 
     ---
 
-    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), agents, memory, tools, and telemetry.
+    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), [Evaluations](user-guide/evaluations.md), agents, memory, tools, and telemetry.
 
 -   **[Architecture](architecture/overview.md)**
 

docs/javascripts/leaderboard.js

@@ -12,8 +12,14 @@
   // Outlier detection — hide entries with values that are physically
   // implausible relative to their token count.  Thresholds are ~1000x
   // above legitimate per-token values to avoid false positives.
-  var MAX_ENERGY_WH_PER_TOKEN = 10;        // legit ≈ 0.001 Wh/tok
-  var MAX_FLOPS_PER_TOKEN = 1e17;           // legit ≈ 1e12 /tok
+  // Outlier bounds. Set well above realistic upper limits but tight
+  // enough to drop the pre-fix bimodal Group B (1-5 Wh/token, ~3e16
+  // FLOPs/token) — see the leaderboard PR for the full diagnosis.
+  // Realistic per-token rates on a consumer GPU + 10–30B local model:
+  // ~0.001 Wh/token, ~1e10–1e11 FLOPs/token. We allow 500× and 10,000×
+  // headroom respectively for inefficient hardware / larger models.
+  var MAX_ENERGY_WH_PER_TOKEN = 0.5;        // legit ≈ 0.001 Wh/tok
+  var MAX_FLOPS_PER_TOKEN = 1e15;           // legit ≈ 1e11 /tok
   var MAX_DOLLAR_PER_TOKEN = 25.0 / 1e6;   // hard ceiling: $25/1M output
 
   function isOutlier(row) {
@@ -29,6 +35,25 @@
     );
   }
 
+  // Distinguish "user actually has zero work done" from "user's energy /
+  // FLOPs telemetry never landed". The latter happens when the server
+  // submits with valid dollar savings + token counts but the per-record
+  // energy stamp was missing (pre-fix builds, GPU energy meter
+  // unavailable, etc.). Without this check those rows show as "0.00 Wh"
+  // and skew the rankings + headline totals.
+  //
+  // Threshold: 1000 tokens is well above any single chat-turn — if a
+  // user has that many tokens recorded but no measured energy, the
+  // telemetry is incomplete, not legitimately zero.
+  var MIN_TOKENS_FOR_TELEMETRY = 1000;
+
+  function isMissingTelemetry(row) {
+    var tokens = Number(row.total_tokens) || 0;
+    var energy = Number(row.energy_wh_saved) || 0;
+    var flops = Number(row.flops_saved) || 0;
+    return tokens > MIN_TOKENS_FOR_TELEMETRY && energy === 0 && flops === 0;
+  }
+
   function escapeHtml(s) {
     var el = document.createElement("span");
     el.textContent = s;
@@ -62,13 +87,24 @@
       var medal =
         rank === 1 ? "\uD83E\uDD47" : rank === 2 ? "\uD83E\uDD48" : rank === 3 ? "\uD83E\uDD49" : "";
       var row = pageRows[j];
+      // Render "—" for energy / FLOPs columns when telemetry didn't
+      // land (vs the user genuinely having 0). The dollar / request /
+      // token columns are unaffected because those measurements landed
+      // even when energy didn't.
+      var missing = isMissingTelemetry(row);
+      var energyCell = missing
+        ? '<td class="lb-number lb-missing" title="Energy telemetry missing for this entry">—</td>'
+        : '<td class="lb-number">' + Number(row.energy_wh_saved || 0).toFixed(2) + "</td>";
+      var flopsCell = missing
+        ? '<td class="lb-number lb-missing" title="FLOPs telemetry missing for this entry">—</td>'
+        : '<td class="lb-number">' + fmtLarge(Number(row.flops_saved || 0)) + "</td>";
       html +=
         "<tr>" +
         '<td><span class="lb-rank' + rankClass + '">' + (medal || rank) + "</span></td>" +
         '<td class="lb-name">' + escapeHtml(row.display_name) + "</td>" +
         '<td class="lb-number">$' + Number(row.dollar_savings || 0).toFixed(4) + "</td>" +
-        '<td class="lb-number">' + Number(row.energy_wh_saved || 0).toFixed(2) + "</td>" +
-        '<td class="lb-number">' + fmtLarge(Number(row.flops_saved || 0)) + "</td>" +
+        energyCell +
+        flopsCell +
         '<td class="lb-number">' + Number(row.total_calls || 0).toLocaleString() + "</td>" +
         '<td class="lb-number">' + Number(row.total_tokens || 0).toLocaleString() + "</td>" +
         "</tr>";
@@ -116,8 +152,15 @@
     }
 
     fetch(
+      // `methodology_version=gte.1` filter excludes rows that the
+      // leaderboard-correctness migration quarantined (version 0). Rows
+      // written by current and future clients carry version >= 1, so this
+      // is forward-compatible — pre-fix corrupt rows hide at the query
+      // level (fewer bytes over the wire than client-side outlier
+      // filtering), and downstream client-side checks remain as a
+      // belt-and-suspenders second line of defence.
       SUPABASE_URL +
-        "/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&order=dollar_savings.desc&limit=1000",
+        "/rest/v1/savings_entries?select=display_name,dollar_savings,energy_wh_saved,flops_saved,total_calls,total_tokens&methodology_version=gte.1&order=dollar_savings.desc&limit=1000",
       {
         headers: {
           apikey: SUPABASE_ANON_KEY,

docs/showcase/CONTRIBUTING.md

@@ -0,0 +1,104 @@
+---
+title: Contributing a Showcase Entry
+description: How to add your setup to the OpenJarvis Showcase
+---
+
+# Contributing a Showcase Entry
+
+The Showcase exists for one reason: to help a confused, curious, *non-technical* reader figure out whether OpenJarvis is worth their weekend. That goal sets every editorial choice on this page.
+
+## The format
+
+```markdown
+---
+title: <Your Title — short, capitalized>
+description: <One sentence. The hook a stranger sees in search results.>
+---
+
+# <emoji> <One-sentence hook — what it does FOR you, in plain English>
+
+<figure markdown>
+  ![<alt text>](../assets/showcase/<your-image>.png){ .showcase-screenshot loading=lazy }
+  <figcaption>A one-sentence caption that adds context the image can't show on its own.</figcaption>
+</figure>
+
+<2–3 short paragraphs of context: when do you use this, what changed for
+you, what the experience feels like. Concrete > abstract. "I read it on
+my phone before coffee" > "improves morning productivity."
+
+A bulleted list of two or three CONCRETE OUTCOMES works well — your
+calendar, your inbox, your code. Specific verbs and proper nouns.>
+
+## Why it's nice
+
+- **<one-line benefit>.** <one or two sentences of evidence>
+- **<one-line benefit>.** <one or two sentences of evidence>
+- **<one-line benefit>.** <one or two sentences of evidence>
+
+## How I set this up
+
+→ **[Tutorial: <name>](../tutorials/<file>.md)** is the closest match.
+
+→ **[Recipe: <name>](https://github.com/open-jarvis/OpenJarvis/tree/main/src/openjarvis/recipes/data)** if you want the exact config.
+
+→ **[<one more related doc>](../<path>.md)** if the reader is going deeper.
+```
+
+## Editorial conventions
+
+These are guardrails, not rules. Break them if you have a reason.
+
+### Lead with the outcome, not the technology
+
+❌ "Multi-channel routing with MCP-backed memory and an orchestrator agent."<br>
+✅ "Jarvis answers my Discord messages while I sleep."
+
+The reader doesn't know what an "orchestrator agent" is yet. They know what a Discord message is.
+
+### Show one screenshot. Make it the headline.
+
+A single, large, *interesting* screenshot beats five small ones. Crop it to show the result, not the UI chrome. If you can convey it in an image, don't write the paragraph.
+
+**Screenshot specs:**
+
+- 1600×1000 PNG, sRGB, no alpha
+- File path: `docs/assets/showcase/<your-slug>.png`
+- Redact: real email addresses, API keys, personal phone numbers, conversation partners' faces or full names (unless they've signed off)
+- Keep: model names, timestamps, dollar amounts, emoji reactions, your own first name
+
+### Specific over impressive
+
+❌ "Saves significant time every morning."<br>
+✅ "Cut my morning catch-up from 25 minutes to 2."
+
+Numbers, durations, dollar amounts, and named tools build trust. Adjectives don't.
+
+### Three paragraphs is plenty
+
+A reader who wants more clicks the "How I set this up →" link at the bottom. Showcase pages are a funnel into the docs, not a replacement for them. If you find yourself explaining configuration in the showcase entry, that material belongs in the linked tutorial.
+
+### "Why it's nice" is for the experience, not the architecture
+
+The bullets under **Why it's nice** should answer "what's different *for you*?" — not "what's different about how the framework works?". Save the architecture talk for the linked docs.
+
+❌ "Uses local SQLite for state with WAL mode for concurrent reads."<br>
+✅ "I can read my own memory file in a text editor. I can delete a line and the memory is gone."
+
+### Every entry must end with at least one "How I set this up →" link
+
+If there isn't a relevant tutorial yet, link to the closest [User Guide](../user-guide/cli.md) and open an issue noting that the tutorial is missing. We will write it.
+
+## Submitting
+
+1. **Fork** the repo and create a branch: `docs/showcase-<your-slug>`.
+2. **Add** your markdown file at `docs/showcase/<your-slug>.md` and screenshot at `docs/assets/showcase/<your-slug>.png`.
+3. **Add a tile** to the grid in `docs/showcase/index.md` (matches the existing pattern — emoji + title + 1-sentence summary + `[:octicons-arrow-right-24: See it](<your-slug>.md)`).
+4. **Open a PR** with the title `docs(showcase): <your title>`. Tag a maintainer if you'd like editorial feedback before merge.
+
+## Where this goes after merge
+
+Hannah and the docs team post merged showcase entries to **`#config-showcase`** in [the OpenJarvis Discord](https://discord.gg/openjarvis). You'll get tagged in the post — you don't have to do it yourself.
+
+## Questions, drafts, half-finished ideas
+
+Drop them in **`#config-showcase`** on Discord *before* opening a PR. Editorial feedback is faster on chat than in a PR review, and you'll save yourself a round of revisions.

docs/showcase/coding-assistant.md

@@ -0,0 +1,36 @@
+---
+title: Offline Code Reviewer
+description: Review a pull request on a transatlantic flight, no internet required
+---
+
+# 🛠️ Offline Code Reviewer — code review on an airplane
+
+<figure markdown>
+  ![Jarvis reviewing a diff with no internet connection](../assets/showcase/coding-assistant.png){ .showcase-screenshot loading=lazy }
+  <figcaption>Airplane mode in the menu bar. Jarvis reading a `git diff`, the surrounding files, and producing a code review at gate-level Wi-Fi (i.e., none).</figcaption>
+</figure>
+
+Earlier this month I was on a flight from SFO to FRA — eleven hours, no usable Wi-Fi. I had a teammate's pull request open in VS Code. I asked Jarvis to review it. It read the diff, read the three files the diff touched, read the project's `CLAUDE.md` for conventions, and produced a review with five comments — two of which caught real bugs.
+
+The review took about 40 seconds on the laptop's built-in GPU. No API call. No "you're offline" error. By the time we landed I'd dropped the comments into GitHub and the PR was merging.
+
+The same setup handles:
+
+- **Code review** — diff + context files + conventions, structured comments.
+- **Debugging** — paste a traceback, Jarvis reads the stack, opens the relevant files, suggests fixes.
+- **Test generation** — point at a function, get back a `pytest` file with edge cases.
+- **Documentation** — generate docstrings that actually match the code, because Jarvis has the file open.
+
+## Why it's nice
+
+- **It works on a plane.** Or a train, or a hotel with bad Wi-Fi, or your couch when Comcast is having a day. Same speed every time.
+- **It sees your repo, not a sanitized chunk.** Cloud coding assistants make you upload a context window. The local one just reads `git status` and the files you're working on.
+- **No "we trained on your code" question.** Your code never leaves your laptop. Period.
+
+## How I set this up
+
+→ **[Tutorial: Code Companion](../tutorials/code-companion.md)** walks through the ReAct-agent + git/file/shell tool stack this uses end-to-end.
+
+→ **[User Guide: Code Assistant](../user-guide/code-assistant.md)** is the focused recipe walkthrough for daily-driver code review.
+
+→ **[OpenAI-compatible server](../getting-started/quickstart.md)** — point your editor's existing AI integration (Cursor, Continue, Cody, Aider) at `localhost:8000`. They mostly don't know they're not talking to OpenAI.

docs/showcase/cost-savings.md

@@ -0,0 +1,38 @@
+---
+title: Track Your Savings
+description: A leaderboard that tells you exactly how much you saved by running locally
+---
+
+# 💸 Track Your Savings — the leaderboard that makes local-first feel real
+
+<figure markdown>
+  ![OpenJarvis savings leaderboard with personal row highlighted](../assets/showcase/cost-savings.png){ .showcase-screenshot loading=lazy }
+  <figcaption>The public leaderboard. The bar on the right is what a month of my Jarvis usage would have cost on the cloud — measured per-query, not estimated.</figcaption>
+</figure>
+
+OpenJarvis tracks every inference call you make — the tokens, the latency, the GPU energy — and computes what that same call *would have cost* on OpenAI, Anthropic, Google, and Bedrock. There's a public leaderboard at **[/leaderboard](../leaderboard.md)** where anyone running Jarvis can opt in and watch their savings rack up.
+
+My current month is roughly:
+
+| | |
+|---|---|
+| Local inference cost | **`$0.00`** |
+| Cloud-equivalent cost | **`$342.18`** (Claude Sonnet 4.6 baseline) |
+| Energy used | **`1.4 kWh`** (~12¢ of grid power) |
+| Prompts sent to a third party | **`0`** |
+
+The dollar number is the hook. The bottom row is the actual reason I run Jarvis.
+
+## Why it's nice
+
+- **You can see what each query costs you.** Not estimated, not "roughly" — measured. Watt-hours per token, FLOPs per token, latency. Every primitive in OpenJarvis treats compute cost as a first-class quantity alongside accuracy.
+- **It makes "local-first" stop being abstract.** Watching a bar chart accumulate `$X` a week that *didn't* leave your hands is a different kind of motivating than "your data is private" claims that you can't verify.
+- **Privacy stops being an act of faith.** Every prompt I send to Jarvis can be traced through the codebase to local-only paths. No "cloud failover" hiding behind a switch.
+
+## How I set this up
+
+You don't, really — it's on by default. Every `jarvis ask`, `jarvis serve` request, and channel-routed message is metered by the [telemetry system](../telemetry.md). To opt your savings into the public leaderboard:
+
+→ **[Leaderboard guide](../leaderboard.md)** — one command to opt in, one command to opt out. Telemetry is local-only by default.
+
+→ **[Telemetry overview](../telemetry.md)** — what's measured, where it's stored, and how to inspect it yourself with `jarvis telemetry`.

docs/showcase/discord-companion.md

@@ -0,0 +1,34 @@
+---
+title: Discord Companion
+description: Jarvis answers questions in your private Discord while you sleep — reads your notes, checks your calendar, schedules things
+---
+
+# 💬 Discord Companion — a personal assistant that lives in my Discord
+
+<figure markdown>
+  ![Jarvis answering a Discord DM about the user's calendar and notes](../assets/showcase/discord-companion.png){ .showcase-screenshot loading=lazy }
+  <figcaption>I DM'd Jarvis from my phone at midnight. It checked my Google Calendar, cross-referenced a note from last week, and answered — running on the Mac mini in my closet.</figcaption>
+</figure>
+
+I have a private Discord server with two channels and one user (me). Jarvis lives there. I can DM it from my phone, my laptop, or my watch — anywhere Discord runs. Sample things I've asked it this week:
+
+- "What's the address of the place I had that meeting last Tuesday?" → Jarvis searches my calendar + meeting notes, replies in 4 seconds.
+- "Reply to Mom's text from earlier saying I'll call tomorrow at 7." → drafts a reply, asks me to confirm, sends.
+- "Add 'Sam's birthday is March 12' to my long-term memory." → updates `MEMORY.md`, confirms.
+- "Summarize the last hour of conversation in `#deploys-prod`." → reads the Slack channel via MCP, summarizes.
+
+I used to use my phone's voice assistant for this. The two differences that matter: **Jarvis answers in three sentences, not one,** and **it actually has my context** — my notes, my calendar, my projects, my history.
+
+## Why it's nice
+
+- **Latency feels like talking to a person.** Local inference on a modest GPU is 5–10× faster than round-tripping to a cloud API. Question to answer in 3 seconds.
+- **The Discord interface is multi-device for free.** Same conversation thread on my phone, laptop, watch — no special app to install.
+- **It's already private.** A Discord server I run, talking to a model on a machine I own. The data trail is two endpoints I control.
+
+## How I set this up
+
+→ **[Tutorial: Messaging Hub](../tutorials/messaging-hub.md)** is the closest match — same channel-adapter + orchestrator-agent pattern, with Discord substituted for Slack.
+
+→ **[Channel docs](../user-guide/cli.md)** walks through Discord/Slack/Telegram/WhatsApp setup. Discord is two environment variables and a bot token.
+
+→ **[MCP integration guide](../user-guide/cli.md)** if you want Jarvis to reach into Notion, Linear, Gmail, etc.

docs/showcase/index.md

@@ -0,0 +1,72 @@
+---
+title: Showcase
+description: What people actually do with OpenJarvis — outcomes first, scripts later
+---
+
+# Showcase
+
+These are stories from people who use OpenJarvis day to day. Each entry shows the **result** — a screenshot, a paragraph of context, and a short link to the docs that explain how to build it. If you're trying to figure out whether OpenJarvis is worth a weekend of your time, start here.
+
+!!! tip "New here?"
+    The Showcase answers *"what's possible?"*. When you find something you want for yourself, follow the **How I set this up** link at the bottom of each page — it lands on a [Tutorial](../tutorials/index.md) that walks through the build.
+
+<div class="grid cards" markdown>
+
+- :material-coffee:{ .lg .middle } **Morning Brief**
+
+    ---
+
+    Slack, email, GitHub, and calendar — read overnight, summarized into 5 bullets in your phone by 7am. Cuts the daily "what did I miss" tax to zero.
+
+    [:octicons-arrow-right-24: See it](morning-brief.md)
+
+- :material-brain:{ .lg .middle } **Memory That Doesn't Reset**
+
+    ---
+
+    Tell Jarvis you're allergic to shellfish once. Three months later it brings it up when you're restaurant-planning. Plain markdown files, no vector-DB tricks.
+
+    [:octicons-arrow-right-24: See it](persistent-memory.md)
+
+- :material-piggy-bank-outline:{ .lg .middle } **Track Your Savings**
+
+    ---
+
+    A leaderboard that tells you exactly how much you saved by running locally — and reminds you that none of your prompts ever left your house.
+
+    [:octicons-arrow-right-24: See it](cost-savings.md)
+
+- :material-message-text:{ .lg .middle } **Discord Companion**
+
+    ---
+
+    Jarvis answers questions in your private Discord while you sleep. Reads your notes, checks your calendar, schedules things, replies in your voice.
+
+    [:octicons-arrow-right-24: See it](discord-companion.md)
+
+- :material-code-tags-check:{ .lg .middle } **Offline Code Reviewer**
+
+    ---
+
+    Review a pull request on a transatlantic flight. Jarvis reads the diff, the surrounding files, and the project conventions — without an internet connection.
+
+    [:octicons-arrow-right-24: See it](coding-assistant.md)
+
+</div>
+
+---
+
+## Share your setup
+
+The Showcase grows from real users. If you've built something interesting on top of OpenJarvis — or just have a configuration you're proud of — the format is simple and the bar is low:
+
+1. **One-sentence hook**: what does this *do for you*?
+2. **A screenshot or 15-second screen recording**: the visible result.
+3. **2–3 short paragraphs**: when you use it, why it's nice (cost, privacy, speed, calm).
+4. **"How I set this up →"**: a link to the relevant [Tutorial](../tutorials/index.md), [User Guide](../user-guide/cli.md), or [Recipe](https://github.com/open-jarvis/OpenJarvis/tree/main/src/openjarvis/recipes/data).
+
+See [Contributing a Showcase Entry](CONTRIBUTING.md) for the template and the editorial conventions (screenshot sizing, what to redact, tone).
+
+## Want to talk to other people doing this?
+
+The **`#config-showcase`** channel in the [OpenJarvis Discord](https://discord.gg/openjarvis) is where people post and discuss personal setups. Drop a screenshot, ask "how would I do X?", or browse what others have shared.

docs/showcase/morning-brief.md

@@ -0,0 +1,39 @@
+---
+title: Morning Brief
+description: Slack, email, GitHub, and calendar — summarized into a 5-bullet brief on your phone by 7am
+---
+
+# ☕ Morning Brief — Jarvis reads everything overnight so I don't have to
+
+<figure markdown>
+  ![Morning brief in Discord](../assets/showcase/morning-brief.png){ .showcase-screenshot loading=lazy }
+  <figcaption>The 7am brief that arrives in my private Discord — 5 bullets, two minutes to read, written by an agent that ran on my desk while I slept.</figcaption>
+</figure>
+
+Every morning at 7am, before my first coffee, a message appears in my private Discord with five bullets:
+
+- what shipped at work overnight (GitHub releases + merged PRs)
+- the two emails I actually need to act on (with one-line summaries)
+- anything mentioned in my team's `#general` Slack channel
+- today's calendar with the next 24 hours of meetings
+- one thing I asked Jarvis to track for me ("did Tuesday's deploy roll out cleanly?")
+
+It's the first thing I read on my phone, while I'm still in bed. The brief used to take me 25 minutes — opening four apps, scrolling, deciding what mattered. Now it's two minutes of reading and I'm done.
+
+## Why it's nice
+
+- **Costs me nothing per month.** It runs on a Mac mini in my closet. Same prompt-volume on the OpenAI API would be `~$18/month` based on the leaderboard's estimates.
+- **Nothing leaves my house.** My inbox, my Slack DMs, my calendar — Jarvis reads them locally and writes the digest locally. The only network call is the Discord webhook to my own private server.
+- **It learns my taste.** Over a few weeks Jarvis figured out that PR titles starting with `chore:` aren't worth surfacing and that I don't want to see calendar holds I created myself. The summarizer has a `MEMORY.md` it updates when I react with 👎 to a bullet.
+
+## What you'd need
+
+A laptop or mini-PC that stays on overnight, an inference engine (Ollama is the easy default), accounts on whichever surfaces you want summarized (Slack, Gmail, GitHub, Google Calendar), and a Discord (or Slack, or Telegram, or email) destination to post the brief to.
+
+## How I set this up
+
+→ **[Tutorial: Scheduled Personal Ops](../tutorials/scheduled-ops.md)** walks through the cron-scheduled agent pattern this uses. The morning-brief flavour is `orchestrator` agent + the channel adapters + the scheduler primitive — three primitives, one TOML recipe.
+
+→ **[User Guide: Morning Digest](../user-guide/morning-digest.md)** is the focused recipe walkthrough if you only want this one workflow.
+
+→ **[User Guide: Channels](../user-guide/cli.md)** for connecting Discord/Slack/Telegram as the destination.

docs/showcase/persistent-memory.md

@@ -0,0 +1,34 @@
+---
+title: Memory That Doesn't Reset
+description: Tell Jarvis something once. It remembers — three months later, across every conversation
+---
+
+# 🧠 Memory That Doesn't Reset — Jarvis actually knows me
+
+<figure markdown>
+  ![Jarvis remembering a user preference three months later](../assets/showcase/persistent-memory.png){ .showcase-screenshot loading=lazy }
+  <figcaption>Three months after I mentioned the allergy in passing, Jarvis brings it up — unprompted — while helping me pick a birthday-dinner restaurant.</figcaption>
+</figure>
+
+I mentioned to Jarvis once, in a throwaway sentence in April, that I'm allergic to shellfish. In July, when I asked it to help me pick a restaurant for my partner's birthday, it volunteered "you'll want to filter for menus that have non-shellfish options" — without being reminded, in a totally different conversation, on a different topic.
+
+That's not magic. The trick is that Jarvis writes to three plain markdown files in my home directory whenever it learns something worth remembering:
+
+- `SOUL.md` — how I want it to behave (tone, length, what to push back on)
+- `MEMORY.md` — facts about me, my projects, my preferences
+- `USER.md` — who I am: my role, my team, my context
+
+Every new conversation starts by reading those three files. I can open them in any text editor. I can delete a line and the memory is gone. The whole thing is `~6 KB` of markdown. No vector DB, no embedding cache, no opaque "personalization layer."
+
+## Why it's nice
+
+- **It's auditable.** I can read what Jarvis "knows" about me in 30 seconds. Most personal-AI products literally can't tell you.
+- **It's portable.** I keep my three files in iCloud Drive. When I set up Jarvis on a new machine, my memory comes with me — without re-onboarding.
+- **It compounds.** After two weeks Jarvis stopped re-asking what my code style is. After six weeks it stopped re-asking who's on my team. The conversations get shorter because the context is already there.
+- **It can't drift.** Vector retrieval can confidently surface the wrong "memory" and you'd never know. Plain markdown that I can read can't lie about what it contains.
+
+## How I set this up
+
+→ **[User Guide: Agents](../user-guide/agents.md)** explains the persistent-agent pattern, including how `SOUL.md` / `MEMORY.md` / `USER.md` are loaded at conversation start.
+
+→ **[Tutorial: Deep Research Assistant](../tutorials/deep-research.md)** uses the same persistent-memory primitive — a good place to see it in action with code.

docs/stylesheets/extra.css

@@ -440,6 +440,13 @@
   font-family: var(--md-code-font-family, monospace);
   font-size: 13px;
 }
+/* Placeholder for rows where energy / FLOPs telemetry didn't land.
+   Distinguishes "telemetry missing" from "user genuinely had 0 work
+   done" without making the row visually pop more than data rows. */
+.lb-missing {
+  color: var(--md-default-fg-color--light, #999);
+  font-style: italic;
+}
 
 /* ── DocSearch ───────────────────────────────────────────────────────── */
 #docsearch {
@@ -562,3 +569,18 @@
     display: none !important;
   }
 }
+
+/* ---------------------------------------------------------------------------
+ * Showcase screenshots
+ *
+ * Hero images on docs/showcase/* pages. Constrains width on wide screens and
+ * adds a subtle border so placeholder/broken-image states still look intentional
+ * before community-contributed screenshots populate docs/assets/showcase/.
+ * ------------------------------------------------------------------------- */
+.showcase-screenshot {
+  max-width: 100%;
+  height: auto;
+  border-radius: 8px;
+  border: 1px solid var(--md-default-fg-color--lightest, rgba(0, 0, 0, 0.08));
+  box-shadow: 0 2px 8px rgba(0, 0, 0, 0.06);
+}

docs/user-guide/evaluations.md

@@ -1,14 +1,14 @@
 # Evaluations
 
-The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correctness and accuracy** on academic datasets. It is a separate package from the main OpenJarvis library and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
+The OpenJarvis evaluation framework (`openjarvis.evals`) measures model **correctness and accuracy** on academic datasets. It ships inside the main `openjarvis` package (at `src/openjarvis/evals/`) and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
 
 !!! info "Evals vs. Benchmarks"
     OpenJarvis has two distinct measurement systems that complement each other:
 
-    | System | Package | Measures | Entry Point |
-    |--------|---------|----------|-------------|
-    | **Evaluations** | `openjarvis-evals` | Correctness on academic datasets (accuracy, pass rate) | `openjarvis-eval` |
-    | **Benchmarks** | `openjarvis` | Engine performance (latency, throughput) | `jarvis bench` |
+    | System | Module | Measures | Entry Point |
+    |--------|--------|----------|-------------|
+    | **Evaluations** | `openjarvis.evals` | Correctness on academic datasets (accuracy, pass rate) | `jarvis eval` |
+    | **Benchmarks** | `openjarvis.bench` | Engine performance (latency, throughput) | `jarvis bench` |
 
     Use evaluations to answer "does this model get the right answer?" and benchmarks to answer "how fast does this model respond?". See the [Benchmarks guide](benchmarks.md) for the performance measurement system.
 
@@ -18,22 +18,38 @@ The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correc
 
 ## Installation
 
-The evaluation framework is a standalone package in the `evals/` directory. Install it alongside OpenJarvis:
+The evaluation framework is part of the main `openjarvis` package — no separate install or extra is required. The standard dev setup is enough:
 
 ```bash
-uv sync --extra eval
+uv sync --extra dev
 ```
 
-This installs the `openjarvis-eval` CLI entry point and all required dependencies (`datasets`, `huggingface-hub`, `tqdm`, `rich`).
+The framework's core dependencies (`click`, `datasets`, `rich`) are base dependencies of `openjarvis`. Two optional extras enable experiment tracking integrations:
+
+```bash
+uv sync --extra dev --extra eval-wandb     # Weights & Biases run tracking
+uv sync --extra dev --extra eval-sheets    # Google Sheets results export
+```
 
 !!! note "Python version requirement"
-    Python 3.10 requires the `tomli` package for TOML config parsing. The `evals/pyproject.toml` includes this as a conditional dependency, so it is installed automatically.
+    Python 3.10 requires the `tomli` package for TOML config parsing. `openjarvis` declares it as a conditional dependency, so it is installed automatically.
+
+## Entry Points
+
+Two equivalent entry points expose the framework:
+
+| Command | Surface |
+|---------|---------|
+| `jarvis eval {list,run,compare,report}` | Canonical CLI. `run` covers the common options; `compare` and `report` post-process result files. |
+| `python -m openjarvis.evals {list,run,run-all,summarize,reparse-judge}` | Full research surface, including judge configuration, the agentic runner, and episode mode. |
+
+The `openjarvis-eval` console script is an alias for `python -m openjarvis.evals` — same commands, same options. This guide uses `jarvis eval` wherever its option set suffices and the module form for research-only options.
 
 ---
 
 ## Datasets
 
-The framework ships with **30+ datasets** covering academic reasoning, agentic tasks, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below.
+The framework ships with **40 registered benchmarks** covering academic reasoning, agentic tasks, coding, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below; `uv run python -m openjarvis.evals list` prints the authoritative registry.
 
 ### Use-Case Benchmarks
 
@@ -64,6 +80,7 @@ These benchmarks measure reasoning and knowledge on established academic dataset
 | **MATH-500** | `math500` | reasoning | Competition-level math problems |
 | **NaturalReasoning** | `natural-reasoning` | reasoning | Natural language reasoning |
 | **HLE** | `hle` | reasoning | Humanity's Last Exam hard challenges |
+| **LiveResearchBench** | `liveresearchbench` | reasoning | Recent research comprehension (Salesforce) |
 | **SimpleQA** | `simpleqa` | chat | Short-form factual question answering |
 | **IPW** | `ipw` | chat | Intelligence Per Watt mixed benchmark |
 
@@ -79,6 +96,11 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **TerminalBench** | `terminalbench` | agentic | Terminal-based task completion |
 | **TerminalBench Native** | `terminalbench-native` | agentic | TerminalBench with native Docker execution |
 | **TerminalBench V2.1** | `terminalbench-v2.1` | agentic | TB v2.1 Harbor-style Docker tasks |
+| **PinchBench** | `pinchbench` | agentic | Real-world agent tasks |
+| **TauBench** | `taubench` | agentic | Multi-turn customer service |
+| **DeepResearchBench** | `liveresearch` | agentic | Deep research report generation |
+| **DeepResearchBench (alias)** | `deepresearch` | agentic | Same benchmark as `liveresearch` |
+| **ToolCall-15** | `toolcall15` | agentic | Tool calling benchmark |
 | **LifelongAgent** | `lifelong-agent` | agentic | Sequential task learning across sessions |
 | **PaperArena** | `paperarena` | agentic | Scientific paper analysis |
 | **DeepPlanning** | `deepplanning` | agentic | Shopping constraint planning |
@@ -87,6 +109,14 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **WebChoreArena** | `webchorearena` | agentic | Web chore tasks |
 | **WorkArena** | `workarena` | agentic | WorkArena++ enterprise workflows |
 
+Both `liveresearch` and `deepresearch` are registered keys for the DeepResearchBench report-generation benchmark.
+
+### Coding Benchmarks
+
+| Dataset | Key | Category | Description |
+|---------|-----|----------|-------------|
+| **LiveCodeBench** | `livecodebench` | coding | Competitive programming |
+
 ### Retrieval Benchmarks
 
 | Dataset | Key | Category | Description |
@@ -123,7 +153,7 @@ The framework includes two pre-built configs for evaluating models on the five c
 ### Cloud models
 
 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
 ```
 
 This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, GPT-5.4, GPT-5 Mini) against all 5 use-case benchmarks with 30 samples each, producing a 6x5 = 30-run matrix. Results are written to `results/use-cases-v2-cloud/`.
@@ -131,7 +161,7 @@ This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gem
 ### Local models
 
 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_local.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_local.toml
 ```
 
 This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS 120B, GLM4, Qwen3.5 35B-A3B, GLM-4.7-Flash) against the same 5 benchmarks, producing a 5x5 = 25-run matrix. Uses 2 workers (suitable for single-GPU setups). Results are written to `results/use-cases-v2-local/`.
@@ -143,15 +173,22 @@ This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS
 
 ## Inference Backends
 
-Every evaluation run routes model calls through one of two backends:
+Every evaluation run routes model calls through one of four backends:
 
 | Backend | Key | Description |
 |---------|-----|-------------|
 | **jarvis-direct** | `jarvis-direct` | Engine-level inference via `SystemBuilder`. Works for local (Ollama, vLLM, llama.cpp) and cloud models. |
 | **jarvis-agent** | `jarvis-agent` | Agent-level inference with tool calling. Uses `JarvisSystem.ask()` with the specified agent and tools. |
+| **hermes** | `hermes` | Real Hermes Agent (Nous Research) via subprocess. Requires `--base-url` and `--api-key`. |
+| **openclaw** | `openclaw` | Real OpenClaw via Node subprocess. Requires `--base-url` and `--api-key`. |
 
 Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark requires tool use — for example, GAIA tasks that reference files that must be read with `file_read`, or arithmetic tasks that benefit from `calculator`.
 
+The `hermes` and `openclaw` backends shell out to external agent frameworks and need an OpenAI-compatible endpoint for their model calls: pass `--base-url`/`--api-key`, set the `JARVIS_BACKEND_BASE_URL`/`JARVIS_BACKEND_API_KEY` environment variables, or add a `[backend.external]` section to your config (see [Config Reference](#backendexternal)).
+
+!!! note "TerminalBench Native"
+    `jarvis eval run --backend` additionally accepts `terminalbench-native`, a Docker-based execution backend used by the TerminalBench Native benchmark.
+
 ---
 
 ## CLI Usage
@@ -159,73 +196,106 @@ Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark
 ### List available benchmarks and backends
 
 ```bash
-openjarvis-eval list
+uv run python -m openjarvis.evals list
 ```
 
-Output:
+Abridged output (40 benchmarks, 4 backends):
 
 ```
-Benchmarks:
-  supergpqa    [reasoning  ] SuperGPQA multiple-choice
-  gaia         [agentic    ] GAIA agentic benchmark
-  frames       [rag        ] FRAMES multi-hop RAG
-  wildchat     [chat       ] WildChat conversation quality
-
-Backends:
-  jarvis-direct    Engine-level inference (local or cloud)
-  jarvis-agent     Agent-level inference with tool calling
+                         Available Benchmarks
+┌──────────────────────┬───────────┬───────────────────────────────────┐
+│ Name                 │ Category  │ Description                       │
+├──────────────────────┼───────────┼───────────────────────────────────┤
+│ supergpqa            │ reasoning │ SuperGPQA multiple-choice         │
+│ gpqa                 │ reasoning │ GPQA graduate-level MCQ           │
+│ ...                  │ ...       │ ...                               │
+│ livecodebench        │ coding    │ LiveCodeBench competitive progr.  │
+│ toolcall15           │ agentic   │ ToolCall-15 tool calling benchmark│
+└──────────────────────┴───────────┴───────────────────────────────────┘
+                         Available Backends
+┌───────────────┬──────────────────────────────────────────────────┐
+│ jarvis-direct │ Engine-level inference (local or cloud)          │
+│ jarvis-agent  │ Agent-level inference with tool calling          │
+│ hermes        │ Real Hermes Agent (Nous Research) via subprocess │
+│ openclaw      │ Real OpenClaw via Node subprocess                │
+└───────────────┴──────────────────────────────────────────────────┘
 ```
 
+`jarvis eval list` prints a similar table but currently shows a curated subset of the registry; the module form above is the authoritative listing.
+
 ### Run a single benchmark
 
 ```bash
-# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples default)
-openjarvis-eval run -b supergpqa -m qwen3:8b
+# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples)
+uv run jarvis eval run -b supergpqa -m qwen3:8b -n 10
 
-# Evaluate GPT-4o on GAIA using the agent backend with tools
-openjarvis-eval run -b gaia -m gpt-4o --backend jarvis-agent \
+# Evaluate GPT-5 Mini on GAIA using the agent backend with tools
+uv run jarvis eval run -b gaia -m gpt-5-mini --backend jarvis-agent \
     --agent orchestrator --tools calculator,file_read -n 50
 
-# Run FRAMES with vLLM engine, write output to a file
-openjarvis-eval run -b frames -m llama3:70b -e vllm \
+# Run FRAMES with the vLLM engine, write output to a file
+uv run jarvis eval run -b frames -m llama3:70b -e vllm \
     -o results/frames_llama70b.jsonl
 
 # Run WildChat with a higher temperature for chat quality
-openjarvis-eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
+uv run jarvis eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
 ```
 
-#### Full option reference
+#### `jarvis eval run` option reference
 
 | Option | Short | Type | Default | Description |
 |--------|-------|------|---------|-------------|
 | `--config` | `-c` | path | — | TOML config file; when provided, `-b` and `-m` are not required |
-| `--benchmark` | `-b` | choice | required* | `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `--backend` | | choice | `jarvis-direct` | `jarvis-direct` or `jarvis-agent` |
-| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-4o`) |
-| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
-| `--agent` | | str | `orchestrator` | Agent name for `jarvis-agent` backend |
-| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--benchmark` | `-b` | str | required* | Any registered benchmark key (see `... list`) |
+| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-5-mini`) |
 | `--max-samples` | `-n` | int | all | Limit the number of samples evaluated |
-| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
-| `--judge-model` | | str | `gpt-4o` | LLM used for judge-based scoring |
-| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--backend` | | choice | `jarvis-direct` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
+| `--base-url` | | str | — | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `--api-key` | | str | — | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
+| `--agent` | | str | — | Agent name for `jarvis-agent` backend (e.g., `orchestrator`) |
+| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
+| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--telemetry/--no-telemetry` | | flag | off | Enable telemetry collection during eval |
+| `--gpu-metrics/--no-gpu-metrics` | | flag | off | Enable GPU metric polling |
 | `--seed` | | int | `42` | Random seed for dataset shuffling |
-| `--split` | | str | dataset default | Override the dataset split |
 | `--temperature` | | float | `0.0` | Generation temperature |
 | `--max-tokens` | | int | `2048` | Maximum output tokens |
+| `--model-filter` | | str | — | Filter models by name substring (multi-model configs) |
+| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--wandb-project` / `--wandb-entity` / `--wandb-tags` / `--wandb-group` | | str | `""` | Weights & Biases tracking (requires `eval-wandb` extra) |
+| `--sheets-id` / `--sheets-worksheet` / `--sheets-creds` | | str | `""` | Google Sheets export (requires `eval-sheets` extra) |
 | `--verbose` | `-v` | flag | off | Enable debug logging |
 
 *Required when `--config` is not provided.
 
+#### Research-only options (`python -m openjarvis.evals run`)
+
+The module CLI accepts everything above plus research-grade options that `jarvis eval run` does not expose:
+
+| Option | Short | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
+| `--judge-model` | | str | `gpt-5-mini-2025-08-07` | LLM used for judge-based scoring (see `--help` for the current default) |
+| `--judge-engine` | | str | `cloud` | Engine key for the LLM judge; use `vllm` to judge locally |
+| `--split` | | str | dataset default | Override the dataset split |
+| `--compact` | | flag | off | Dense single-table output |
+| `--trace-detail` | | flag | off | Full per-step trace listing |
+| `--agentic` | | flag | off | Use `AgenticRunner` for multi-turn agent execution |
+| `--episode-mode` | | flag | off | Sequential episode processing with lifelong learning (required for `lifelong-agent` and similar benchmarks) |
+| `--concurrency` | | int | `1` | Parallel query execution (AgenticRunner only) |
+| `--query-timeout` | | float | — | Per-query wall-clock timeout in seconds (AgenticRunner only) |
+
+Note: the module CLI's `--backend` choice covers `jarvis-direct`, `jarvis-agent`, `hermes`, and `openclaw`; `terminalbench-native` as a backend is available via `jarvis eval run` and TOML configs.
+
 ### Run all benchmarks at once
 
-The `run-all` command evaluates a single model against all four benchmarks sequentially and writes results to an output directory:
+The `run-all` command (module CLI only) evaluates a single model against **every registered benchmark** sequentially and writes results to an output directory:
 
 ```bash
-openjarvis-eval run-all -m qwen3:8b
+uv run python -m openjarvis.evals run-all -m qwen3:8b
 
 # With options
-openjarvis-eval run-all -m gpt-4o -n 100 --output-dir results/gpt4o/
+uv run python -m openjarvis.evals run-all -m gpt-5-mini -n 100 --output-dir results/gpt5mini/
 ```
 
 Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The model slug replaces `/` and `:` with `-`, so `qwen3:8b` becomes `qwen3-8b`.
@@ -235,7 +305,7 @@ Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The m
 After a run, inspect a JSONL results file:
 
 ```bash
-openjarvis-eval summarize results/supergpqa_qwen3-8b.jsonl
+uv run python -m openjarvis.evals summarize results/supergpqa_qwen3-8b.jsonl
 ```
 
 Output:
@@ -251,6 +321,55 @@ Accuracy:  0.7222
 Errors:    2
 ```
 
+The module CLI also provides `reparse-judge`, which re-parses stored judge output in a results file and recovers records whose judge verdicts initially failed to parse — useful after improving the judge-output parser without re-running inference.
+
+### Compare and report
+
+`jarvis eval` adds two post-processing commands for result files:
+
+```bash
+# Side-by-side metric comparison across runs
+uv run jarvis eval compare results/supergpqa_qwen3-8b.jsonl results/supergpqa_gpt-5-mini.jsonl
+
+# Detailed report (accuracy, latency, cost, per-subject breakdown) for one run
+uv run jarvis eval report results/supergpqa_qwen3-8b.jsonl
+```
+
+---
+
+## Evaluating an Already-Running Endpoint
+
+If you already have an OpenAI-compatible server running — `jarvis serve`, vLLM, SGLang, llama.cpp's server, or a hosted endpoint — point an eval directly at it with `--base-url` and `--api-key`:
+
+```bash
+# A vLLM server is already serving Qwen/Qwen3-8B on a GPU node:
+#   vllm serve Qwen/Qwen3-8B --port 8000
+uv run jarvis eval run -b supergpqa -m Qwen/Qwen3-8B \
+    --base-url http://gpu-node:8000/v1 \
+    --api-key local-key \
+    -n 50
+```
+
+The `-m` value must match a model id the server reports at `GET /v1/models`. Both flags fall back to the `JARVIS_BACKEND_BASE_URL` and `JARVIS_BACKEND_API_KEY` environment variables, so CI jobs can set them once:
+
+```bash
+export JARVIS_BACKEND_BASE_URL=http://gpu-node:8000/v1
+export JARVIS_BACKEND_API_KEY=local-key
+uv run jarvis eval run -b gaia -m Qwen/Qwen3-8B --backend jarvis-agent -n 25
+```
+
+For the external `hermes` and `openclaw` backends these values are **required** (the foreign frameworks need an endpoint to send model calls to).
+
+!!! tip "Engine-level alternative for vLLM"
+    The vLLM engine also honors the `VLLM_HOST` environment variable (default `http://localhost:8000`):
+
+    ```bash
+    VLLM_HOST=http://gpu-node:8000 uv run python -m openjarvis.evals run \
+        -b supergpqa -m Qwen/Qwen3-8B -e vllm -n 50
+    ```
+
+    `VLLM_HOST` is process-global — if the candidate and the judge both use the `vllm` engine, they share the same endpoint. Prefer `--base-url` when you need them separate.
+
 ---
 
 ## TOML Config System
@@ -260,7 +379,7 @@ For research workflows that compare multiple models across multiple benchmarks,
 ### Running from a config
 
 ```bash
-openjarvis-eval run --config src/openjarvis/evals/configs/full-suite.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/full-suite.toml
 ```
 
 When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options are not required. All settings come from the config file. The CLI expands the matrix, prints a progress table, and writes results to the configured `output_dir`.
@@ -269,7 +388,7 @@ When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options a
 
 A config file has six sections: `[meta]`, `[defaults]`, `[judge]`, `[run]`, `[[models]]`, and `[[benchmarks]]`. Only `[[models]]` and `[[benchmarks]]` are required — all other sections are optional and fall back to built-in defaults.
 
-```toml title="evals/configs/full-suite.toml"
+```toml title="src/openjarvis/evals/configs/full-suite.toml"
 # Suite-level metadata (optional)
 [meta]
 name = "full-suite-v1"
@@ -353,7 +472,7 @@ For example, `temperature` is resolved as: use `[defaults].temperature` (0.0), t
 
 A config requires only one `[[models]]` and one `[[benchmarks]]` entry:
 
-```toml title="evals/configs/minimal.toml"
+```toml title="src/openjarvis/evals/configs/minimal.toml"
 [[models]]
 name = "qwen3:8b"
 
@@ -365,7 +484,7 @@ This runs SuperGPQA against qwen3:8b with all default settings. Use this as a st
 
 ### Single-run config with full options
 
-```toml title="evals/configs/single-run.toml"
+```toml title="src/openjarvis/evals/configs/single-run.toml"
 [meta]
 name = "single-run-example"
 description = "Evaluate SuperGPQA with a single model and full configuration"
@@ -425,7 +544,8 @@ Configuration for the LLM used as a judge in GAIA, FRAMES, and WildChat scoring.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `model` | str | `"gpt-4o"` | Judge model identifier |
+| `model` | str | `"gpt-5-mini-2025-08-07"` | Judge model identifier |
+| `engine` | str | `None` | Engine key for the judge (e.g., `"vllm"` to judge locally; defaults to cloud) |
 | `provider` | str | `None` | Provider override (e.g., `"openai"`) |
 | `temperature` | float | `0.0` | Judge sampling temperature |
 | `max_tokens` | int | `1024` | Maximum judge output tokens |
@@ -444,6 +564,20 @@ Execution settings that apply to the entire suite.
 | `seed` | int | `42` | Random seed for dataset shuffling |
 | `telemetry` | bool | `false` | Enable GPU telemetry capture (energy, power, utilization, throughput) |
 | `gpu_metrics` | bool | `false` | Enable GPU metric polling via `pynvml` (requires `pynvml` or `nvidia-ml-py`) |
+| `warmup_samples` | int | `0` | Untimed warmup samples before measurement |
+| `energy_vendor` | str | `""` | GPU energy vendor override |
+| `max_turns` | int | `None` | Maximum agent turns per query |
+| `wandb_project` / `wandb_entity` / `wandb_tags` / `wandb_group` | str | `""` | Weights & Biases tracking |
+| `sheets_spreadsheet_id` / `sheets_worksheet` / `sheets_credentials_path` | str | `""` / `"Results"` / `""` | Google Sheets export |
+
+### `[backend.external]`
+
+Endpoint settings for the `hermes` and `openclaw` backends. Environment variables override TOML values.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `base_url` | str | `None` | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `api_key` | str | `None` | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
 
 ### `[[models]]`
 
@@ -451,7 +585,7 @@ One block per model. The `name` field is required.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-4o"`) |
+| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-5-mini"`) |
 | `engine` | str | `None` | Engine key to use (`"ollama"`, `"vllm"`, `"cloud"`, ...) |
 | `provider` | str | `None` | Provider override for cloud models (e.g., `"openai"`) |
 | `temperature` | float | `None` | Override `[defaults].temperature` for this model |
@@ -468,10 +602,12 @@ One block per benchmark. The `name` field is required.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Benchmark key: `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `backend` | str | `"jarvis-direct"` | Inference backend: `jarvis-direct` or `jarvis-agent` |
+| `name` | str | required | Any registered benchmark key (see `uv run python -m openjarvis.evals list`) |
+| `backend` | str | `"jarvis-direct"` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
 | `max_samples` | int | `None` | Limit number of samples; `None` evaluates the full dataset |
 | `split` | str | `None` | Override the default dataset split |
+| `subset` | str | `None` | Dataset subset/variant (benchmark-specific) |
+| `record_ids` | list[str] | `None` | Evaluate only these record ids |
 | `agent` | str | `None` | Agent name for `jarvis-agent` backend (e.g., `"orchestrator"`) |
 | `tools` | list[str] | `[]` | Tool names for `jarvis-agent` backend |
 | `judge_model` | str | `None` | Override `[judge].model` for this benchmark only |
@@ -647,7 +783,7 @@ The `EvalRunner` processes samples concurrently using a `ThreadPoolExecutor`. Re
 
 ```bash
 # Use more workers for faster evaluation (if the engine supports concurrent requests)
-openjarvis-eval run -b supergpqa -m qwen3:8b -w 8 -n 500
+uv run python -m openjarvis.evals run -b supergpqa -m qwen3:8b -w 8 -n 500
 ```
 
 !!! warning "Worker count and engine load"

frontend/src-tauri/src/lib.rs

@@ -1540,19 +1540,89 @@ async fn fetch_models(api_url: String) -> Result<serde_json::Value, String> {
 
 #[tauri::command]
 async fn run_jarvis_command(args: Vec<String>) -> Result<String, String> {
-    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
-    cmd_args.extend(args);
     let uv_bin = resolve_bin("uv");
-    let output = tokio::process::Command::new(&uv_bin)
-        .args(&cmd_args)
-        .output()
-        .await
-        .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
 
-    if output.status.success() {
-        Ok(String::from_utf8_lossy(&output.stdout).to_string())
-    } else {
-        Err(String::from_utf8_lossy(&output.stderr).to_string())
+    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
+    cmd_args.extend(args.iter().cloned());
+
+    let mut cmd = tokio::process::Command::new(&uv_bin);
+    cmd.args(&cmd_args);
+    // Run from the project root so `uv run jarvis` resolves the OpenJarvis
+    // project regardless of the app's launch cwd. In a packaged install the
+    // cwd isn't the checkout, so without this `jarvis` isn't found and the
+    // backend never starts — the UI then shows "Failed to get response"
+    // (see #531).
+    if let Some(ref root) = find_project_root() {
+        cmd.current_dir(root);
+    }
+
+    let is_serve = args.first().map(|a| a.as_str() == "serve").unwrap_or(false);
+
+    if !is_serve {
+        // Short-lived command (e.g. `stop`, `status`): wait for it and return
+        // its captured output.
+        let output = cmd
+            .output()
+            .await
+            .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
+        return if output.status.success() {
+            Ok(String::from_utf8_lossy(&output.stdout).to_string())
+        } else {
+            Err(String::from_utf8_lossy(&output.stderr).to_string())
+        };
+    }
+
+    // `jarvis serve` is a long-running server that never exits. The old code
+    // used `.output()`, which waits for the process to exit and so hung this
+    // command forever — the "Start" button never resolved (#531). Spawn it
+    // detached instead, drain stderr (a full 4 KB Windows pipe can otherwise
+    // stall the child mid-startup, #309), and poll /health for readiness.
+    cmd.stdout(std::process::Stdio::null())
+        .stderr(std::process::Stdio::piped());
+    let mut child = cmd
+        .spawn()
+        .map_err(|e| format!("Failed to launch jarvis serve: {}", e))?;
+
+    let tail: StderrTail = Arc::new(Mutex::new(Vec::new()));
+    if let Some(stderr) = child.stderr.take() {
+        spawn_jarvis_stderr_drainer(stderr, tail.clone());
+    }
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(2))
+        .build()
+        .map_err(|e| format!("Failed to build HTTP client: {}", e))?;
+    let url = format!("http://127.0.0.1:{}/health", JARVIS_PORT);
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(120);
+
+    loop {
+        // Surface an early crash (bad venv, missing Rust ext, etc.) right away
+        // instead of waiting out the full readiness timeout.
+        if let Ok(Some(status)) = child.try_wait() {
+            let stderr = String::from_utf8_lossy(tail.lock().await.as_slice()).into_owned();
+            return Err(format!(
+                "jarvis serve exited (code {:?}) before becoming healthy:\n{}",
+                status.code(),
+                stderr.trim()
+            ));
+        }
+        if let Ok(resp) = client.get(&url).send().await {
+            if resp.status().is_success() {
+                // Leave the server running (the Child is detached on drop —
+                // kill_on_drop defaults to false); `stop` tears it down.
+                return Ok(format!(
+                    "jarvis serve is ready on http://127.0.0.1:{}",
+                    JARVIS_PORT
+                ));
+            }
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!(
+                "jarvis serve did not become healthy on port {} within 120s.",
+                JARVIS_PORT
+            ));
+        }
+        tokio::time::sleep(Duration::from_millis(500)).await;
     }
 }
 

frontend/src/lib/api.ts

@@ -948,12 +948,31 @@ export interface MemoryStats {
 
 export interface MemoryConfig {
   backend: string;
+  // Set by the server when the native `openjarvis_rust` extension is missing,
+  // so the UI can show the real cause instead of a healthy-looking config.
+  available?: boolean;
+  detail?: string | null;
   context_from_memory: boolean;
   context_top_k: number;
   context_min_score: number;
   context_max_tokens: number;
 }
 
+/**
+ * Extract the server's `detail` message from a failed JSON response so the UI
+ * surfaces the real cause (e.g. "openjarvis_rust extension is not installed")
+ * instead of a blanket fallback string (#502).
+ */
+async function memoryErrorDetail(res: Response, fallback: string): Promise<string> {
+  try {
+    const data = await res.json();
+    if (data && typeof data.detail === 'string' && data.detail) return data.detail;
+  } catch {
+    // Non-JSON body — fall through to the generic message below.
+  }
+  return fallback;
+}
+
 export async function getMemoryStats(): Promise<MemoryStats> {
   const res = await apiFetch(`/v1/memory/stats`);
   if (!res.ok) throw new Error('Failed to fetch memory stats');
@@ -977,16 +996,16 @@ export async function storeMemory(content: string, metadata?: Record<string, unk
     headers: { 'Content-Type': 'application/json' },
     body: JSON.stringify({ content, metadata }),
   });
-  if (!res.ok) throw new Error('Failed to store memory');
+  if (!res.ok) throw new Error(await memoryErrorDetail(res, 'Failed to store memory'));
 }
 
-export async function indexMemoryPath(path: string): Promise<{ chunks_indexed: number }> {
+export async function indexMemoryPath(path: string): Promise<{ chunks_indexed: number; note?: string }> {
   const res = await apiFetch(`/v1/memory/index`, {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
     body: JSON.stringify({ path }),
   });
-  if (!res.ok) throw new Error('Failed to index path');
+  if (!res.ok) throw new Error(await memoryErrorDetail(res, 'Failed to index path'));
   return res.json();
 }
 

mkdocs.yml

@@ -150,6 +150,14 @@ nav:
       - Quick Start: getting-started/quickstart.md
       - Code Snippets: getting-started/snippets.md
       - Configuration: getting-started/configuration.md
+  - Showcase:
+      - Overview: showcase/index.md
+      - Morning Brief: showcase/morning-brief.md
+      - Memory That Doesn't Reset: showcase/persistent-memory.md
+      - Track Your Savings: showcase/cost-savings.md
+      - Discord Companion: showcase/discord-companion.md
+      - Offline Code Reviewer: showcase/coding-assistant.md
+      - Contributing: showcase/CONTRIBUTING.md
   - Tutorials:
       - Overview: tutorials/index.md
       - Deep Research Assistant: tutorials/deep-research.md
@@ -185,6 +193,8 @@ nav:
       - External MCP Servers: user-guide/mcp-external-servers.md
       - Scheduler: user-guide/scheduler.md
       - Telemetry: user-guide/telemetry.md
+      - Evaluations: user-guide/evaluations.md
+      - Benchmarks: user-guide/benchmarks.md
       - Security: user-guide/security.md
       - LLM-guided spec search: user-guide/llm-guided-spec-search.md
   - Leaderboard: leaderboard.md

pyproject.toml

@@ -4,7 +4,7 @@ build-backend = "hatchling.build"
 
 [project]
 name = "OpenJarvis"
-version = "0.1.1"
+version = "1.0.2"
 description = "OpenJarvis — modular AI assistant backend with composable intelligence primitives"
 readme = "README.md"
 # Upper bound: numpy 2.2.x (pinned transitively via datasets/pandas) ships no
@@ -152,6 +152,7 @@ Issues = "https://github.com/open-jarvis/OpenJarvis/issues"
 
 [project.scripts]
 jarvis = "openjarvis.cli:main"
+openjarvis-eval = "openjarvis.evals.cli:main"
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/openjarvis"]
@@ -174,6 +175,7 @@ markers = [
     "live_channel: requires real channel credentials (env vars)",
     "nvidia: requires NVIDIA GPU",
     "slow: long-running test",
+    "hub: downloads real datasets from the HuggingFace Hub at runtime; excluded from the default CI lane (run with -m hub)",
     "live_external: requires HERMES_AGENT_PATH and OPENCLAW_PATH; spawns real foreign-framework subprocesses",
     "modal: requires Modal token + network; runs real swebench harness on Modal",
 ]

scripts/install/build-extension.sh

@@ -28,6 +28,22 @@ fi
 
 cd "$SRC_DIR"
 if uv run maturin develop -m "$MANIFEST" >>"$LOG" 2>&1; then
+    # Verify the extension actually imports from THIS venv before declaring
+    # success. `maturin develop` can report success while installing the .so
+    # into a different venv than the one that runs the server, which leaves
+    # memory silently broken at runtime (#502). Only the import check below
+    # proves the serving venv can load it.
+    if ! uv run python -c "import openjarvis_rust" >>"$LOG" 2>&1; then
+        rc=$?
+        {
+            echo "build-extension.sh: maturin succeeded but 'import openjarvis_rust'"
+            echo "failed in the serving venv ($SRC_DIR/.venv) — the extension was"
+            echo "not installed where the server runs. (exit=$rc)"
+            tail -n 50 "$LOG" 2>/dev/null || true
+        } > "$FAILED"
+        rm -f "$BUILT"
+        exit "$rc"
+    fi
     tmp="$BUILT.tmp"
     date -u +"%Y-%m-%dT%H:%M:%SZ" > "$tmp"
     mv "$tmp" "$BUILT"

src/openjarvis/channels/_stubs.py

@@ -60,7 +60,27 @@ class BaseChannel(ABC):
         conversation_id: str = "",
         metadata: Dict[str, Any] | None = None,
     ) -> bool:
-        """Send a message to a specific channel. Returns True on success."""
+        """Send a message to a specific channel. Returns True on success.
+
+        Canonical send contract shared by **every** channel adapter:
+
+        ``channel``
+            The DESTINATION identifier — the per-adapter native id of the
+            place the message goes (Discord/Slack channel id, Telegram chat
+            id, email recipient address, ...).  This is *not* the channel
+            TYPE label.  An incoming :class:`ChannelMessage` carries that
+            destination in its ``conversation_id`` field (``channel`` there
+            is only the type label such as ``"discord"``), so dispatch code
+            replying to a message must pass ``cm.conversation_id`` here.
+        ``conversation_id``
+            An optional reply/thread reference — the native id of the
+            message being replied to (Discord ``message_reference``, Slack
+            ``thread_ts``, Telegram ``reply_to_message_id``, email
+            ``In-Reply-To``, ...).  When replying to an inbound message this
+            should be ``cm.message_id``, never the channel id.  Passing a
+            channel id here yields broken references (e.g. Discord
+            ``MESSAGE_REFERENCE_UNKNOWN_MESSAGE``).
+        """
 
     @abstractmethod
     def status(self) -> ChannelStatus:

src/openjarvis/channels/telegram.py

@@ -112,7 +112,15 @@ class TelegramChannel(BaseChannel):
 
             _TELEGRAM_MAX_LEN = 4096
             url = f"https://api.telegram.org/bot{self._token}/sendMessage"
-            chat_id = conversation_id or channel
+            # Canonical channel send contract (see BaseChannel.send): the first
+            # positional ``channel`` arg is the DESTINATION (the Telegram chat
+            # id).  ``conversation_id`` is the inbound message id used as a
+            # reply/thread reference (``reply_to_message_id``).  We fall back to
+            # ``conversation_id`` as the chat id only when ``channel`` is empty,
+            # for backwards compatibility with legacy callers that passed the
+            # chat id via ``conversation_id``.
+            chat_id = channel or conversation_id
+            reply_to = conversation_id if (channel and conversation_id) else ""
             chunks = textwrap.wrap(
                 content,
                 width=_TELEGRAM_MAX_LEN,
@@ -126,6 +134,8 @@ class TelegramChannel(BaseChannel):
                 }
                 if self._parse_mode:
                     payload["parse_mode"] = self._parse_mode
+                if reply_to:
+                    payload["reply_to_message_id"] = reply_to
 
                 resp = httpx.post(url, json=payload, timeout=10.0)
                 if resp.status_code >= 300:

src/openjarvis/cli/ask.py

@@ -714,7 +714,13 @@ def ask(
     register_builtin_models()
 
     effective_engine_key = engine_key or config.intelligence.preferred_engine or None
-    resolved = get_engine(config, effective_engine_key)
+    # Pass the model we intend to run so engine selection can skip an engine
+    # that can't actually serve it (e.g. the cloud fallback when the local
+    # engine is down but only a non-OpenAI key is set — see #532). This is the
+    # -m flag or the configured default; when neither is set we leave it None
+    # and a model is chosen per-engine below.
+    selection_model = model_name or config.intelligence.default_model or None
+    resolved = get_engine(config, effective_engine_key, model=selection_model)
     if resolved is None:
         console.print(
             "[red bold]No inference engine available.[/red bold]\n\n"

src/openjarvis/cli/chat_cmd.py

@@ -28,12 +28,22 @@ def _read_input(prompt: str = "You> ") -> Optional[str]:
 @click.option("-a", "--agent", "agent_name", default=None, help="Agent type.")
 @click.option("--tools", default=None, help="Comma-separated tool names.")
 @click.option("--system", "system_prompt", default=None, help="Custom system prompt.")
+@click.option(
+    "--persona",
+    "persona_name",
+    default=None,
+    help=(
+        "Named persona dir under ~/.openjarvis/personas/<name>/ "
+        "(overrides config). Pass 'none' to disable all persona files."
+    ),
+)
 def chat(
     engine_key: str | None,
     model_name: str | None,
     agent_name: str | None,
     tools: str | None,
     system_prompt: str | None,
+    persona_name: str | None,
 ) -> None:
     """Start an interactive multi-turn chat session.
 
@@ -48,6 +58,14 @@ def chat(
 
     config = load_config()
 
+    import dataclasses as _dc
+
+    effective_mf = (
+        _dc.replace(config.memory_files, persona_name=persona_name)
+        if persona_name is not None
+        else config.memory_files
+    )
+
     # Resolve engine
     from openjarvis.engine import get_engine
     from openjarvis.intelligence import register_builtin_models
@@ -121,6 +139,21 @@ def chat(
 
                     kwargs["interactive"] = True
                     kwargs["confirm_callback"] = _confirm
+
+                import inspect as _inspect
+
+                if (
+                    "prompt_builder"
+                    in _inspect.signature(agent_cls.__init__).parameters
+                ):
+                    from openjarvis.prompt.builder import SystemPromptBuilder
+
+                    kwargs["prompt_builder"] = SystemPromptBuilder(
+                        agent_template=config.agent.default_system_prompt or "",
+                        memory_files_config=effective_mf,
+                        system_prompt_config=config.system_prompt,
+                    )
+
                 agent = agent_cls(engine, model, **kwargs)
         except Exception as exc:
             console.print(f"[yellow]Agent '{agent_key}' failed: {exc}[/yellow]")
@@ -147,6 +180,16 @@ def chat(
     _notifications = NotificationDispatcher(get_status())
 
     # Conversation state
+    if not system_prompt:
+        from openjarvis.prompt.builder import SystemPromptBuilder
+
+        builder = SystemPromptBuilder(
+            agent_template=config.agent.default_system_prompt or "",
+            memory_files_config=effective_mf,
+            system_prompt_config=config.system_prompt,
+        )
+        system_prompt = builder.build()
+
     history: List[Message] = []
     if system_prompt:
         history.append(Message(role=Role.SYSTEM, content=system_prompt))

src/openjarvis/cli/compose_cmd.py

@@ -332,7 +332,7 @@ def compose_bench(
         for i, rc in enumerate(run_configs, 1):
             console.print(f"\n[bold]Run {i}/{len(run_configs)}:[/bold] {rc.benchmark}")
             try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                 results_table.add_row(
                     rc.benchmark,
                     f"{summary.accuracy:.4f}",

src/openjarvis/cli/eval_cmd.py

@@ -61,6 +61,12 @@ KNOWN_BENCHMARKS = {
 KNOWN_BACKENDS = {
     "jarvis-direct": "Engine-level inference (local or cloud)",
     "jarvis-agent": "Agent-level inference with tool calling",
+    "hermes": "Real Hermes Agent (Nous Research) via subprocess",
+    "openclaw": "Real OpenClaw via Node subprocess",
+    "terminalbench-native": (
+        "TerminalBench V2.1 via terminal-bench Harness "
+        "(selected with -b terminalbench-native)"
+    ),
 }
 
 
@@ -146,7 +152,9 @@ def eval_list() -> None:
     "base_url",
     default=None,
     help=(
-        "OpenAI-compat endpoint URL for hermes/openclaw backends "
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
         "(env: JARVIS_BACKEND_BASE_URL)."
     ),
 )
@@ -154,7 +162,11 @@ def eval_list() -> None:
     "--api-key",
     "api_key",
     default=None,
-    help=("API key for the hermes/openclaw endpoint (env: JARVIS_BACKEND_API_KEY)."),
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
 @click.option(
     "--agent",
@@ -347,7 +359,7 @@ def eval_run(
                 f"{rc.benchmark} / {rc.model}"
             )
             try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                 console.print(
                     f"  [green]{summary.accuracy:.4f}[/green] "
                     f"({summary.correct}/{summary.scored_samples})"
@@ -399,8 +411,10 @@ def eval_run(
         sheets_spreadsheet_id=sheets_spreadsheet_id,
         sheets_worksheet=sheets_worksheet,
         sheets_credentials_path=sheets_credentials_path,
-        # Spec §6.2 — for hermes/openclaw external backends. Falls back to env vars
-        # so users can also set JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
+        # OpenAI-compatible endpoint for the model under eval. Required for
+        # hermes/openclaw (Spec §6.2); honored by first-party backends too on
+        # this CLI path. Falls back to env vars so users can also set
+        # JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
         base_url=base_url or os.environ.get("JARVIS_BACKEND_BASE_URL"),
         api_key=api_key or os.environ.get("JARVIS_BACKEND_API_KEY"),
     )

src/openjarvis/cli/serve.py

@@ -24,6 +24,61 @@ from openjarvis.intelligence import (
 logger = logging.getLogger(__name__)
 
 
+def _unique_model_ids(model_ids: list[str]) -> list[str]:
+    """Return model ids in first-seen order without duplicates."""
+    unique: list[str] = []
+    seen: set[str] = set()
+    for model_id in model_ids:
+        if model_id and model_id not in seen:
+            seen.add(model_id)
+            unique.append(model_id)
+    return unique
+
+
+def _safe_list_models(engine: object) -> list[str]:
+    try:
+        list_models = getattr(engine, "list_models")
+        return list(list_models())
+    except Exception as exc:
+        logger.debug("Failed to list models for selected server engine: %s", exc)
+        return []
+
+
+def _resolve_server_model(
+    requested_model: str | None,
+    *,
+    config: object,
+    engine_name: str,
+    engine: object,
+    all_models: dict[str, list[str]],
+) -> str:
+    """Pick a startup model that is present on the active server engine.
+
+    CLI ``--model`` remains authoritative. For config-driven startup, prefer the
+    configured server/default model only when the active engine can actually
+    serve it; otherwise use ``intelligence.fallback_model`` or the first
+    reachable model. This prevents MLX-preferred configs from hiding a healthy
+    Ollama fallback behind an empty/incorrect model map.
+    """
+    if requested_model:
+        return requested_model
+
+    candidates = [
+        getattr(config.server, "model", ""),
+        getattr(config.intelligence, "default_model", ""),
+        getattr(config.intelligence, "fallback_model", ""),
+    ]
+    available = _unique_model_ids(
+        _safe_list_models(engine) + list(all_models.get(engine_name, []))
+    )
+
+    for candidate in candidates:
+        if candidate and (not available or candidate in available):
+            return candidate
+
+    return available[0] if available else ""
+
+
 @click.command()
 @click.option("--host", default=None, help="Bind address (default: config).")
 @click.option(
@@ -91,7 +146,13 @@ def serve(
         except Exception as exc:
             logger.debug("Telemetry store init failed: %s", exc)
 
-    resolved = get_engine(config, engine_key)
+    # Select with the model we'll actually serve so an engine that can't
+    # serve it (e.g. the cloud fallback without the matching provider key) is
+    # skipped rather than chosen and failing per-request later (see #532).
+    selection_model = (
+        model_name or config.server.model or config.intelligence.default_model or None
+    )
+    resolved = get_engine(config, engine_key, model=selection_model)
     if resolved is None:
         console.print(
             "[red bold]No inference engine available.[/red bold]\n\n"
@@ -107,10 +168,12 @@ def serve(
     sec = setup_security(config, engine, bus)
     engine = sec.engine
 
-    # If cloud API keys are set, wrap with MultiEngine so both local
-    # and cloud models appear in the model list and can be used.
+    # If cloud API keys are set, prepare a cloud engine. We build the
+    # MultiEngine after local discovery so healthy local fallbacks such as
+    # Ollama stay visible even when the configured preferred engine is MLX.
     import os
 
+    cloud_engine = None
     _has_cloud = (
         os.environ.get("OPENAI_API_KEY")
         or os.environ.get("ANTHROPIC_API_KEY")
@@ -121,12 +184,9 @@ def serve(
     if _has_cloud and engine_name != "cloud":
         try:
             from openjarvis.engine.cloud import CloudEngine
-            from openjarvis.engine.multi import MultiEngine
 
-            cloud = CloudEngine()
-            engine = MultiEngine([(engine_name, engine), ("cloud", cloud)])
-            engine_name = "multi"
-            if cloud.health():
+            cloud_engine = CloudEngine()
+            if cloud_engine.health():
                 console.print("  Cloud:  [cyan]enabled[/cyan] (API keys detected)")
             else:
                 console.print(
@@ -163,20 +223,55 @@ def serve(
     for ek, model_ids in all_models.items():
         merge_discovered_models(ek, model_ids)
 
+    multi_entries = [(engine_name, engine)]
+    for discovered_name, discovered_engine in all_engines:
+        if discovered_name != engine_name:
+            multi_entries.append((discovered_name, discovered_engine))
+    if cloud_engine is not None:
+        multi_entries.append(("cloud", cloud_engine))
+
+    if len(multi_entries) > 1:
+        from openjarvis.engine.multi import MultiEngine
+
+        engine = MultiEngine(multi_entries)
+        engine_name = "multi"
+        all_models[engine_name] = engine.list_models()
+        merge_discovered_models(engine_name, all_models[engine_name])
+
     # Resolve model
-    if model_name is None:
-        model_name = config.server.model or config.intelligence.default_model
+    configured_model = (
+        model_name or config.server.model or config.intelligence.default_model
+    )
+    model_name = _resolve_server_model(
+        model_name,
+        config=config,
+        engine_name=engine_name,
+        engine=engine,
+        all_models=all_models,
+    )
+    if configured_model and model_name and model_name != configured_model:
+        console.print(
+            "[yellow]Configured model "
+            f"{configured_model!r} is not reachable; using {model_name!r}.[/yellow]"
+        )
     if not model_name:
-        engine_models = all_models.get(engine_name, [])
-        if engine_models:
-            model_name = engine_models[0]
-        else:
-            console.print("[red]No model available on engine.[/red]")
-            sys.exit(1)
+        console.print(
+            "[red]No model available on any reachable engine.[/red]\n\n"
+            "Start an inference backend and make sure it lists at least one model.\n"
+            "For Ollama: [cyan]ollama serve[/cyan] and "
+            "[cyan]ollama pull qwen3.5:9b[/cyan].\n"
+            "For MLX: start the MLX OpenAI-compatible server on the configured host."
+        )
+        sys.exit(1)
 
     # Resolve agent
     agent = None
     agent_key = agent_name or config.server.agent
+    # Tool instances resolved for the primary agent are reused below to build
+    # the scheduler's ToolExecutor — avoiding a second full SystemBuilder.build()
+    # (which would re-discover the engine, re-resolve tools, re-open the channel,
+    # etc.). See the scheduler block near the bottom of this function (#263).
+    resolved_tools: list = []
     if agent_key:
         try:
             import openjarvis.agents  # noqa: F401
@@ -244,6 +339,8 @@ def serve(
 
                     if tools:
                         agent_kwargs["tools"] = tools
+                    # Reuse these for the scheduler's ToolExecutor (#263).
+                    resolved_tools = tools
 
                 if getattr(agent_cls, "accepts_tools", False):
                     agent_kwargs["max_turns"] = config.agent.max_turns
@@ -377,6 +474,24 @@ def serve(
     # Create app
     from openjarvis.server.app import create_app
 
+    # Set up memory backend for context injection. Built before the scheduler
+    # block so the executor's JarvisSystem can reference it (#263).
+    memory_backend = None
+    if config.agent.context_from_memory:
+        try:
+            import openjarvis.tools.storage  # noqa: F401
+            from openjarvis.core.registry import MemoryRegistry
+
+            mem_key = config.memory.default_backend
+            if MemoryRegistry.contains(mem_key):
+                memory_backend = MemoryRegistry.create(
+                    mem_key,
+                    db_path=config.memory.db_path,
+                )
+                console.print("  Memory:    [cyan]active[/cyan]")
+        except Exception as exc:
+            logger.debug("Memory backend init failed: %s", exc)
+
     # Set up agent manager
     agent_manager = None
     if config.agent_manager.enabled:
@@ -416,9 +531,55 @@ def serve(
                 event_bus=bus,
                 trace_store=_trace_store,
             )
-            from openjarvis.system import SystemBuilder
+            # Reuse the components already built inline above instead of a
+            # second full SystemBuilder.build() — the original double-build
+            # re-discovered the engine, re-instrumented it, re-resolved tools,
+            # re-opened the channel and re-created the agent manager, costing
+            # ~30-40s on top of an already-paid startup (#263). The executor
+            # only reads engine/model/config/memory_backend/tool_executor/
+            # session_store/channel_backend from the system (see
+            # AgentExecutor), all of which are wired here.
+            from openjarvis.sessions.session import SessionStore
+            from openjarvis.system import JarvisSystem
+            from openjarvis.tools._stubs import ToolExecutor
 
-            system = SystemBuilder(config).build()
+            _sched_session_store = None
+            if config.sessions.enabled:
+                try:
+                    from pathlib import Path as _SchedPath
+
+                    _sched_session_store = SessionStore(
+                        db_path=_SchedPath(config.sessions.db_path).expanduser(),
+                        max_age_hours=config.sessions.max_age_hours,
+                        consolidation_threshold=(
+                            config.sessions.consolidation_threshold
+                        ),
+                    )
+                except Exception as exc:
+                    logger.debug("Scheduler session store init failed: %s", exc)
+
+            _sched_tool_executor = (
+                ToolExecutor(resolved_tools, bus) if resolved_tools else None
+            )
+
+            system = JarvisSystem(
+                config=config,
+                bus=bus,
+                engine=engine,
+                engine_key=engine_name,
+                model=model_name,
+                agent=agent,
+                agent_name=agent_key or "",
+                tools=resolved_tools,
+                tool_executor=_sched_tool_executor,
+                memory_backend=memory_backend,
+                telemetry_store=telem_store,
+                trace_store=_trace_store,
+                session_store=_sched_session_store,
+                capability_policy=sec.capability_policy,
+                agent_manager=agent_manager,
+                agent_executor=executor,
+            )
             executor.set_system(system)
 
             agent_scheduler = AgentScheduler(
@@ -438,23 +599,6 @@ def serve(
         except Exception as exc:
             logger.debug("Agent scheduler init failed: %s", exc)
 
-    # Set up memory backend for context injection
-    memory_backend = None
-    if config.agent.context_from_memory:
-        try:
-            import openjarvis.tools.storage  # noqa: F401
-            from openjarvis.core.registry import MemoryRegistry
-
-            mem_key = config.memory.default_backend
-            if MemoryRegistry.contains(mem_key):
-                memory_backend = MemoryRegistry.create(
-                    mem_key,
-                    db_path=config.memory.db_path,
-                )
-                console.print("  Memory:    [cyan]active[/cyan]")
-        except Exception as exc:
-            logger.debug("Memory backend init failed: %s", exc)
-
     # --- Channel Gateway: API key, sessions, ChannelBridge ---
     import os as _os
 

src/openjarvis/core/types.py

@@ -125,6 +125,16 @@ class ToolResult:
     metadata: Dict[str, Any] = field(default_factory=dict)
 
 
+# Bump when token-counting methodology changes so the leaderboard can
+# distinguish entries computed under different rules.
+#   v1 = original (Ollama prompt_eval_count, may under-count due to KV cache)
+#   v2 = full prompt token count, no KV-cache assumption, system prompt
+#        always counted
+# Lives here (not in server/savings) so the telemetry layer can read it
+# without crossing the server → telemetry layering.
+TOKEN_COUNTING_VERSION: int = 2
+
+
 @dataclass(slots=True)
 class TelemetryRecord:
     """Single telemetry observation recorded after an inference call."""
@@ -167,6 +177,14 @@ class TelemetryRecord:
     gpu_energy_joules: float = 0.0
     dram_energy_joules: float = 0.0
     tokens_per_joule: float = 0.0
+    # Version tag for the token-counting methodology used when this record
+    # was produced. `None` (= legacy) means the record predates per-record
+    # versioning; the leaderboard aggregator filters those out to avoid
+    # mixing pre-fix and post-fix records in the same per-token efficiency
+    # metric — they were the dominant source of the bimodal Wh/token
+    # distribution on the public leaderboard. New records always write
+    # `TOKEN_COUNTING_VERSION` from `server/savings.py`.
+    token_counting_version: Optional[int] = None
     mining_session_id: Optional[str] = None
     metadata: Dict[str, Any] = field(default_factory=dict)
 

src/openjarvis/engine/_discovery.py

@@ -156,12 +156,26 @@ def discover_models(
 
 
 def get_engine(
-    config: JarvisConfig, engine_key: str | None = None
+    config: JarvisConfig,
+    engine_key: str | None = None,
+    model: str | None = None,
 ) -> Tuple[str, InferenceEngine] | None:
     """Get a specific engine by key, or the default with fallback.
 
+    When *model* is given, an engine is selected only if it can actually
+    serve that model (``engine.can_serve(model)``). This stops the cloud
+    fallback from being chosen — when the local engine is down — for a model
+    whose provider client is missing, which otherwise surfaces as a confusing
+    "OpenAI client not available" instead of a helpful "start your local
+    engine" message (see #532). When *model* is ``None`` selection stays
+    model-agnostic (unchanged behaviour).
+
     Returns ``(key, engine_instance)`` or ``None`` if no engine is available.
     """
+
+    def _usable(engine: InferenceEngine) -> bool:
+        return engine.health() and (model is None or engine.can_serve(model))
+
     # Build an ordered list of keys to try, then fall back to full discovery.
     keys_to_try: list[str] = []
     if engine_key:
@@ -176,14 +190,16 @@ def get_engine(
             continue
         try:
             engine = _make_engine(key, config)
-            if engine.health():
+            if _usable(engine):
                 return (key, engine)
         except Exception as exc:
             logger.debug("Engine %r health check failed: %s", key, exc)
 
-    # Fallback to any healthy engine
-    healthy = discover_engines(config)
-    return healthy[0] if healthy else None
+    # Fallback to the first healthy engine that can serve the model.
+    for key, engine in discover_engines(config):
+        if model is None or engine.can_serve(model):
+            return (key, engine)
+    return None
 
 
 __all__ = ["discover_engines", "discover_models", "get_engine"]

src/openjarvis/engine/_openai_compat.py

@@ -28,12 +28,31 @@ class _OpenAICompatibleEngine(InferenceEngine):
     _default_host: str = "http://localhost:8000"
     _api_prefix: str = "/v1"
 
-    def __init__(self, host: str | None = None, *, timeout: float = 600.0) -> None:
+    def __init__(
+        self,
+        host: str | None = None,
+        *,
+        api_key: str | None = None,
+        timeout: float = 600.0,
+    ) -> None:
         import os
 
-        env_key = f"{self.engine_id.upper()}_HOST"
-        self._host = (host or os.environ.get(env_key) or self._default_host).rstrip("/")
-        self._client = httpx.Client(base_url=self._host, timeout=timeout)
+        # Sanitize the engine id for env-var lookup ("openai-compat" ->
+        # "OPENAI_COMPAT_..."); shells cannot set hyphenated variable names.
+        env_prefix = self.engine_id.upper().replace("-", "_")
+        self._host = (
+            host or os.environ.get(f"{env_prefix}_HOST") or self._default_host
+        ).rstrip("/")
+        # Bearer auth for endpoints started with e.g. ``vllm serve --api-key``.
+        # Setting it on the client covers generate/stream/stream_full/
+        # list_models/health alike; ``None`` keeps requests header-free.
+        self._api_key = api_key or os.environ.get(f"{env_prefix}_API_KEY") or None
+        headers = (
+            {"Authorization": f"Bearer {self._api_key}"} if self._api_key else None
+        )
+        self._client = httpx.Client(
+            base_url=self._host, timeout=timeout, headers=headers
+        )
 
     # -- InferenceEngine interface ------------------------------------------
 
@@ -69,6 +88,23 @@ class _OpenAICompatibleEngine(InferenceEngine):
             raise EngineConnectionError(
                 f"{self.engine_id} engine not reachable at {self._host}"
             ) from exc
+        except httpx.HTTPStatusError as exc:
+            error_detail = exc.response.text.strip()
+            if exc.response.status_code == 404:
+                detail_suffix = (
+                    f" Response body: {error_detail}" if error_detail else ""
+                )
+                raise EngineConnectionError(
+                    f"{self.engine_id} engine at {self._host} returned 404 for "
+                    f"{self._api_prefix}/chat/completions. Make sure this port "
+                    "is running an OpenAI-compatible chat server, not another "
+                    f"local web service.{detail_suffix}"
+                ) from exc
+            detail_suffix = f": {error_detail}" if error_detail else ""
+            raise EngineConnectionError(
+                f"{self.engine_id} engine at {self._host} returned HTTP "
+                f"{exc.response.status_code}{detail_suffix}"
+            ) from exc
         data = resp.json()
         choices = data.get("choices", [])
         if not choices:

src/openjarvis/engine/_stubs.py

@@ -119,6 +119,17 @@ class InferenceEngine(ABC):
     def health(self) -> bool:
         """Return ``True`` when the engine is reachable and healthy."""
 
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` if this engine can serve *model*.
+
+        Defaults to ``True``: local engines accept any model id (whether a
+        specific model is *installed* is a separate concern from engine
+        selection). Engines that multiplex provider-specific clients (e.g.
+        the cloud engine) override this so selection can skip an engine whose
+        client for the model's provider isn't configured (see #532).
+        """
+        return True
+
     def close(self) -> None:
         """Release resources (HTTP clients, connections, threads, etc.)."""
 

src/openjarvis/engine/cloud.py

@@ -893,6 +893,13 @@ class CloudEngine(InferenceEngine):
             "max_tokens": max_tokens,
             "temperature": temperature,
         }
+        # Forward tools / tool_choice (OpenRouter is OpenAI-compatible).
+        tools = kwargs.pop("tools", None)
+        if tools:
+            create_kwargs["tools"] = tools
+        tool_choice = kwargs.pop("tool_choice", None)
+        if tool_choice is not None:
+            create_kwargs["tool_choice"] = tool_choice
         t0 = time.monotonic()
         resp = self._openrouter_client.chat.completions.create(**create_kwargs)
         elapsed = time.monotonic() - t0
@@ -900,7 +907,7 @@ class CloudEngine(InferenceEngine):
         usage = resp.usage
         prompt_tokens = usage.prompt_tokens if usage else 0
         completion_tokens = usage.completion_tokens if usage else 0
-        return {
+        result: Dict[str, Any] = {
             "content": choice.message.content or "",
             "usage": {
                 "prompt_tokens": prompt_tokens,
@@ -911,6 +918,19 @@ class CloudEngine(InferenceEngine):
             "finish_reason": choice.finish_reason or "stop",
             "ttft": elapsed,
         }
+        if getattr(choice.message, "tool_calls", None):
+            result["tool_calls"] = [
+                {
+                    "id": tc.id,
+                    "type": tc.type,
+                    "function": {
+                        "name": tc.function.name,
+                        "arguments": tc.function.arguments,
+                    },
+                }
+                for tc in choice.message.tool_calls
+            ]
+        return result
 
     def _generate_minimax(
         self,
@@ -1195,6 +1215,13 @@ class CloudEngine(InferenceEngine):
             "temperature": temperature,
             "stream": True,
         }
+        # Forward tools / tool_choice (OpenRouter is OpenAI-compatible).
+        tools = kwargs.pop("tools", None)
+        if tools:
+            create_kwargs["tools"] = tools
+        tool_choice = kwargs.pop("tool_choice", None)
+        if tool_choice is not None:
+            create_kwargs["tool_choice"] = tool_choice
         resp = self._openrouter_client.chat.completions.create(**create_kwargs)
         for chunk in resp:
             delta = chunk.choices[0].delta if chunk.choices else None
@@ -1450,6 +1477,34 @@ class CloudEngine(InferenceEngine):
             models.extend(_CODEX_MODELS)
         return models
 
+    def _client_for_model(self, model: str) -> Any:
+        """Return the provider client ``generate``/``stream`` will dispatch to
+        for *model* (mirrors the routing in those methods)."""
+        if _is_codex_model(model):
+            return self._codex_client
+        if _is_openrouter_model(model):
+            return self._openrouter_client
+        if _is_minimax_model(model):
+            return self._minimax_client
+        if _is_anthropic_model(model):
+            return self._anthropic_client
+        if _is_google_model(model):
+            return self._google_client
+        return self._openai_client
+
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` only if the provider client for *model* exists.
+
+        ``health()`` is ``True`` whenever *any* provider client is configured,
+        but a request for, say, a ``gpt-*`` model still needs the OpenAI
+        client specifically. Without this check the cloud engine gets picked
+        as a fallback (when the local engine is down) for a model it can't
+        serve, then dies at call time with "<provider> client not available"
+        instead of the user getting a helpful "start your local engine"
+        message (see #532).
+        """
+        return self._client_for_model(model) is not None
+
     def health(self) -> bool:
         return (
             self._openai_client is not None

src/openjarvis/engine/openai_compat_engines.py

@@ -1,5 +1,7 @@
 """Data-driven registration of OpenAI-compatible inference engines."""
 
+from __future__ import annotations
+
 from openjarvis.core.registry import EngineRegistry
 from openjarvis.engine._openai_compat import _OpenAICompatibleEngine
 
@@ -25,4 +27,35 @@ for _key, (_cls_name, _default_host, _api_prefix) in _ENGINES.items():
     EngineRegistry.register(_key)(_cls)
     globals()[_cls_name] = _cls
 
-__all__ = [name for name, _, _ in _ENGINES.values()]
+
+def normalize_openai_base_url(url: str) -> str:
+    """Strip a single trailing ``/v1`` segment from a user-supplied base URL.
+
+    Users habitually pass ``http://host:8000/v1`` (the full OpenAI-compatible
+    prefix); the engine's ``_api_prefix`` re-appends ``/v1`` to every request
+    path, so a trailing copy would double up as ``/v1/v1``. Only a literal
+    trailing ``/v1`` is stripped — proxy/gateway path prefixes are preserved.
+    """
+    base = url.rstrip("/")
+    if base.endswith("/v1"):
+        base = base[: -len("/v1")]
+    return base
+
+
+class OpenAICompatEngine(_OpenAICompatibleEngine):
+    """Generic engine for an explicitly-provided OpenAI-compatible endpoint.
+
+    Deliberately NOT registered in ``EngineRegistry``: it is only ever
+    constructed with an explicit host (e.g. ``jarvis eval --base-url``), so
+    registering it would just add a useless localhost discovery probe and
+    interact with the per-test registry wipe.
+    """
+
+    engine_id = "openai-compat"
+    _api_prefix = "/v1"
+
+
+__all__ = [name for name, _, _ in _ENGINES.values()] + [
+    "OpenAICompatEngine",
+    "normalize_openai_base_url",
+]

src/openjarvis/evals/backends/_endpoint_util.py

@@ -0,0 +1,52 @@
+"""Shared helper for targeting an explicit OpenAI-compatible endpoint.
+
+Used by the first-party eval backends (jarvis-direct, jarvis-agent) when
+``--base-url`` is given: the eval must use exactly that endpoint, with no
+silent fallback to whatever other engine discovery happens to find.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+
+def build_endpoint_engine(
+    base_url: str,
+    api_key: Optional[str] = None,
+    engine_key: Optional[str] = None,
+):
+    """Construct an :class:`OpenAICompatEngine` pinned to ``base_url``.
+
+    Pre-flight health-checks the endpoint and raises a loud, actionable
+    error when it is unreachable — engine discovery is never consulted.
+    """
+    from openjarvis.engine.openai_compat_engines import (
+        OpenAICompatEngine,
+        normalize_openai_base_url,
+    )
+
+    if engine_key:
+        logger.warning(
+            "Both an engine key (%r) and base_url (%r) were given; "
+            "base_url wins — targeting the endpoint directly.",
+            engine_key,
+            base_url,
+        )
+    host = normalize_openai_base_url(base_url)
+    engine = OpenAICompatEngine(host=host, api_key=api_key)
+    if not engine.health():
+        engine.close()
+        raise RuntimeError(
+            f"--base-url endpoint not reachable: {base_url} "
+            f"(GET {host}/v1/models failed). Is an OpenAI-compatible server "
+            "(e.g. `vllm serve`) running at that address? If it requires "
+            "authentication (HTTP 401), pass --api-key or set "
+            "JARVIS_BACKEND_API_KEY."
+        )
+    return engine
+
+
+__all__ = ["build_endpoint_engine"]

src/openjarvis/evals/backends/jarvis_agent.py

@@ -31,6 +31,8 @@ class JarvisAgentBackend(InferenceBackend):
         max_turns: Optional[int] = None,
         skills_enabled: bool = True,
         overlay_dir: Optional[Path] = None,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
     ) -> None:
         from openjarvis.system import SystemBuilder
 
@@ -40,7 +42,17 @@ class JarvisAgentBackend(InferenceBackend):
         self._gpu_metrics = gpu_metrics
 
         builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
             builder.engine(engine_key)
         if model:
             builder.model(model)

src/openjarvis/evals/backends/jarvis_direct.py

@@ -24,6 +24,8 @@ class JarvisDirectBackend(InferenceBackend):
         engine_key: Optional[str] = None,
         telemetry: bool = False,
         gpu_metrics: bool = False,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
     ) -> None:
         from openjarvis.system import SystemBuilder
 
@@ -31,7 +33,17 @@ class JarvisDirectBackend(InferenceBackend):
         self._gpu_metrics = gpu_metrics
 
         builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
             builder.engine(engine_key)
         # Propagate gpu_metrics to the runtime config so SystemBuilder
         # creates an EnergyMonitor / GpuMonitor for the InstrumentedEngine.

src/openjarvis/evals/backends/terminalbench_native.py

@@ -5,11 +5,13 @@ Uses Harness for Docker-based execution and scoring.
 
 from __future__ import annotations
 
+import inspect
 import logging
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Tuple
 
 from openjarvis.evals.core.backend import InferenceBackend
+from openjarvis.evals.core.types import RunSummary
 
 LOGGER = logging.getLogger(__name__)
 
@@ -20,6 +22,97 @@ try:
 except ImportError:
     _HAS_TB = False
 
+# terminal-bench FailureMode values that are definitionally infrastructure
+# failures (the harness broke before/while driving the agent), never a
+# judgment on the model's answer. NOTE: clean trials leave failure_mode
+# "unset" in terminal-bench 0.2.18 — both on success AND on genuine
+# unresolved misses — so failure_mode alone can NOT be used to detect
+# harness errors (it would misflag every real model miss).
+_INFRA_FAILURE_MODES = frozenset({"agent_installation_failed", "unknown_agent_error"})
+
+# Harness kwargs that older terminal-bench versions may not support.
+_TIMEOUT_KWARGS = ("global_agent_timeout_sec", "global_timeout_multiplier")
+
+
+def summarize_benchmark_results(
+    results: Any,
+    *,
+    model: str,
+    benchmark: str = "terminalbench-native",
+) -> Tuple[RunSummary, List[Dict[str, str]]]:
+    """Convert terminal-bench ``BenchmarkResults`` into a ``RunSummary``.
+
+    Trials are classified into three buckets:
+
+    - resolved: ``is_resolved`` is True -> counted correct.
+    - model miss: unresolved, but the model was actually contacted ->
+      counted in the accuracy denominator.
+    - harness/infra failure: excluded from the accuracy denominator and
+      reported in ``RunSummary.errors`` plus the returned failure list.
+
+    Zero-model-contact signal choice: terminal-bench 0.2.18 leaves
+    ``failure_mode`` UNSET both on clean success and on genuine unresolved
+    misses, so failure_mode cannot distinguish "the model tried and failed"
+    from "the agent never called the model". Token usage can: this backend
+    always runs terminus-2, which reports real LiteLLM usage, so an
+    unresolved trial with zero/missing input+output tokens means no model
+    request ever completed — an infrastructure failure (in-container setup
+    hang/death, tmux failure), not a model miss. CAVEAT: terminal-bench
+    "installed agents" (openhands, claude-code, ...) hardcode 0 tokens even
+    on success; if this backend ever honors ``agent_name`` for installed
+    agents, this heuristic must be gated on the agent type.
+    """
+    trials = list(getattr(results, "results", None) or [])
+
+    harness_failures: List[Dict[str, str]] = []
+    scored = 0
+    correct = 0
+
+    for tr in trials:
+        task_id = getattr(tr, "task_id", None) or getattr(tr, "trial_name", "unknown")
+        is_resolved = getattr(tr, "is_resolved", None) is True
+        fm = getattr(tr, "failure_mode", None)
+        fm_value = str(getattr(fm, "value", fm) or "unset").lower()
+        tokens = (getattr(tr, "total_input_tokens", None) or 0) + (
+            getattr(tr, "total_output_tokens", None) or 0
+        )
+
+        zero_model_contact = not is_resolved and tokens == 0
+        infra_failure_mode = fm_value in _INFRA_FAILURE_MODES
+
+        if zero_model_contact or infra_failure_mode:
+            harness_failures.append(
+                {
+                    "task_id": str(task_id),
+                    "failure_mode": fm_value,
+                    "reason": (
+                        "zero_model_requests" if zero_model_contact else fm_value
+                    ),
+                }
+            )
+            continue
+
+        scored += 1
+        if is_resolved:
+            correct += 1
+
+    return (
+        RunSummary(
+            benchmark=benchmark,
+            category="agentic",
+            backend="terminalbench-native",
+            model=model,
+            total_samples=len(trials),
+            scored_samples=scored,
+            correct=correct,
+            accuracy=correct / scored if scored else 0.0,
+            errors=len(harness_failures),
+            mean_latency_seconds=0.0,
+            total_cost_usd=0.0,
+        ),
+        harness_failures,
+    )
+
 
 class TerminalBenchNativeBackend(InferenceBackend):
     """Runs terminal-bench tasks natively via Harness with Docker execution.
@@ -44,7 +137,22 @@ class TerminalBenchNativeBackend(InferenceBackend):
         system_prompt: str = "",
         max_tokens: int = 16384,
         n_concurrent: int = 4,
+        global_agent_timeout_sec: Optional[float] = 1800.0,
+        global_timeout_multiplier: Optional[float] = None,
     ) -> None:
+        """Args of note:
+
+        global_agent_timeout_sec: Hard wall-clock bound for each trial's
+            agent phase. terminal-bench runs installed-agent SETUP inside
+            this same budget with an infinite tmux timeout, so this bounds
+            SETUP+RUN together (a setup-only timeout needs an upstream
+            terminal-bench change). When set, it REPLACES each task's own
+            ``max_agent_timeout_sec``. Set ``None`` or ``0`` to fall back
+            to per-task budgets.
+        global_timeout_multiplier: Scales per-task budgets when
+            ``global_agent_timeout_sec`` is not set. ``None`` keeps
+            terminal-bench's default (1.0).
+        """
         if not _HAS_TB:
             raise ImportError("terminal-bench is required: pip install terminal-bench")
 
@@ -59,6 +167,8 @@ class TerminalBenchNativeBackend(InferenceBackend):
         self._system_prompt = system_prompt
         self._max_tokens = max_tokens
         self._n_concurrent = n_concurrent
+        self._global_agent_timeout_sec = global_agent_timeout_sec
+        self._global_timeout_multiplier = global_timeout_multiplier
         self._results: Optional[BenchmarkResults] = None
 
     def run_harness(self, run_id: str) -> BenchmarkResults:
@@ -91,10 +201,53 @@ class TerminalBenchNativeBackend(InferenceBackend):
         if self._max_samples is not None:
             harness_kwargs["n_tasks"] = self._max_samples
 
+        # Bound each trial's agent phase. Without this, an in-container
+        # installed-agent SETUP hang runs with an infinite tmux timeout,
+        # bounded only by whatever budget the task happens to declare.
+        if self._global_agent_timeout_sec:
+            harness_kwargs["global_agent_timeout_sec"] = float(
+                self._global_agent_timeout_sec
+            )
+        if self._global_timeout_multiplier is not None:
+            harness_kwargs["global_timeout_multiplier"] = float(
+                self._global_timeout_multiplier
+            )
+
+        self._check_timeout_kwargs_supported(harness_kwargs)
+
         harness = Harness(**harness_kwargs)
         self._results = harness.run()
         return self._results
 
+    @staticmethod
+    def _check_timeout_kwargs_supported(harness_kwargs: Dict[str, Any]) -> None:
+        """Fail loudly if this terminal-bench build lacks the timeout kwargs.
+
+        terminal-bench is an undeclared, unpinned dependency, so installs may
+        predate the global timeout kwargs (added by 0.2.x). Passing an
+        unknown kwarg raises an opaque TypeError; dropping it silently would
+        re-create the unbounded-setup hang. Detect and explain instead.
+        """
+        try:
+            params = inspect.signature(Harness.__init__).parameters
+        except (TypeError, ValueError):
+            return
+        if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()):
+            return
+        unsupported = [
+            key
+            for key in _TIMEOUT_KWARGS
+            if key in harness_kwargs and key not in params
+        ]
+        if unsupported:
+            raise RuntimeError(
+                "The installed terminal-bench does not support "
+                f"{', '.join(unsupported)} (requires terminal-bench >= "
+                "0.2.18). Upgrade terminal-bench, or disable the bound by "
+                "setting global_agent_timeout_sec = 0 in the eval config "
+                "[run] section."
+            )
+
     def generate(
         self,
         prompt: str,
@@ -121,4 +274,4 @@ class TerminalBenchNativeBackend(InferenceBackend):
         pass
 
 
-__all__ = ["TerminalBenchNativeBackend"]
+__all__ = ["TerminalBenchNativeBackend", "summarize_benchmark_results"]

src/openjarvis/evals/cli.py

@@ -183,14 +183,28 @@ def _build_backend(
     max_turns: Optional[int] = None,
     base_url: Optional[str] = None,
     api_key: Optional[str] = None,
+    first_party_endpoint: bool = True,
 ):
     """Construct the appropriate backend.
 
-    For "hermes" and "openclaw" backends, ``base_url`` and ``api_key`` are
-    REQUIRED — these foreign frameworks need an OpenAI-compatible endpoint
-    to send model calls to. Pass them via the eval config's
-    ``[backend.external]`` section or env vars.
+    ``base_url``/``api_key`` point at the OpenAI-compatible endpoint serving
+    the model under eval:
+
+    - For "hermes" and "openclaw" they are REQUIRED — these foreign
+      frameworks always call out to an external endpoint.
+    - "jarvis-direct" and "jarvis-agent" honor them when
+      ``first_party_endpoint`` is True (the CLI ``--base-url`` path): the
+      eval targets exactly that endpoint — no engine-discovery fallback —
+      and fails fast if it is unreachable. Suite mode passes
+      ``first_party_endpoint=False`` so the suite TOML's
+      ``[backend.external]`` section stays scoped to hermes/openclaw
+      (extending it to first-party backends is explicitly deferred).
     """
+    if not first_party_endpoint:
+        fp_base_url = fp_api_key = None
+    else:
+        fp_base_url, fp_api_key = base_url, api_key
+
     if backend_name == "jarvis-agent":
         from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
 
@@ -202,6 +216,8 @@ def _build_backend(
             gpu_metrics=gpu_metrics,
             model=model,
             max_turns=max_turns,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
         )
     elif backend_name == "jarvis-direct":
         from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
@@ -210,6 +226,8 @@ def _build_backend(
             engine_key=engine_key,
             telemetry=telemetry,
             gpu_metrics=gpu_metrics,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
         )
     elif backend_name == "hermes":
         from openjarvis.evals.backends.external import HermesBackend
@@ -656,25 +674,53 @@ def _build_trackers(config) -> list:
     return trackers
 
 
-def _run_terminalbench_native(config, console: Console) -> object:
-    """Run TerminalBench V2.1 natively via terminal-bench Harness."""
+def _run_terminalbench_native(
+    config,
+    console: Console,
+    *,
+    base_url: Optional[str] = None,
+    api_key: Optional[str] = None,
+) -> object:
+    """Run TerminalBench V2.1 natively via terminal-bench Harness.
+
+    ``base_url`` (from ``--base-url`` / JARVIS_BACKEND_BASE_URL) targets an
+    already-running OpenAI-compatible endpoint; when unset, the legacy local
+    vLLM default (http://localhost:8000/v1) is used.
+    """
+    from openjarvis.engine.openai_compat_engines import normalize_openai_base_url
     from openjarvis.evals.backends.terminalbench_native import (
         TerminalBenchNativeBackend,
+        summarize_benchmark_results,
     )
-    from openjarvis.evals.core.types import RunSummary
 
     model = config.model
     # LiteLLM expects "openai/<model>" for OpenAI-compatible servers
     litellm_model = f"openai/{model}"
     output_dir = getattr(config, "output_path", None) or "results/terminalbench-native/"
 
+    # Harness budgets: only forward explicit config values so the backend
+    # defaults (global_agent_timeout_sec=1800) apply otherwise.
+    timeout_kwargs = {}
+    if getattr(config, "global_agent_timeout_sec", None) is not None:
+        timeout_kwargs["global_agent_timeout_sec"] = config.global_agent_timeout_sec
+    if getattr(config, "global_timeout_multiplier", None) is not None:
+        timeout_kwargs["global_timeout_multiplier"] = config.global_timeout_multiplier
+
+    # Normalize to exactly one trailing "/v1" — LiteLLM's api_base wants the
+    # full OpenAI-compatible prefix, and users pass both forms of the URL.
+    if base_url:
+        api_base = normalize_openai_base_url(base_url) + "/v1"
+    else:
+        api_base = "http://localhost:8000/v1"
+
     backend = TerminalBenchNativeBackend(
         model=litellm_model,
-        api_base="http://localhost:8000/v1",
+        api_base=api_base,
         temperature=config.temperature,
         max_samples=config.max_samples,
         output_dir=output_dir,
         n_concurrent=config.max_workers or 4,
+        **timeout_kwargs,
     )
 
     import re
@@ -683,46 +729,83 @@ def _run_terminalbench_native(config, console: Console) -> object:
     model_slug = re.sub(r"[^a-z0-9_-]", "-", model.lower().replace("/", "-"))
     run_id = f"tb21-{model_slug}"
     console.print(f"  Running TerminalBench V2.1 natively: {model}")
+    console.print(f"  API base: {api_base}")
     console.print(f"  Harness run_id: {run_id}")
 
-    results = backend.run_harness(run_id)
+    if api_key:
+        # terminus-2 routes model calls through LiteLLM with the "openai/"
+        # prefix, which reads OPENAI_API_KEY from the environment. The
+        # harness runs in-process, so set the var for the duration of the
+        # run and restore the previous value afterwards.
+        prev_key = os.environ.get("OPENAI_API_KEY")
+        os.environ["OPENAI_API_KEY"] = api_key
+        try:
+            results = backend.run_harness(run_id)
+        finally:
+            if prev_key is None:
+                os.environ.pop("OPENAI_API_KEY", None)
+            else:
+                os.environ["OPENAI_API_KEY"] = prev_key
+    else:
+        results = backend.run_harness(run_id)
 
-    # Convert BenchmarkResults to RunSummary
-    total = len(results.trial_results) if hasattr(results, "trial_results") else 0
-    correct = 0
-    if hasattr(results, "trial_results"):
-        for tr in results.trial_results:
-            if getattr(tr, "is_resolved", False):
-                correct += 1
-
-    accuracy = correct / total if total > 0 else 0.0
-    return RunSummary(
-        benchmark="terminalbench-native",
-        category="agentic",
-        backend="terminalbench-native",
-        model=model,
-        total_samples=total,
-        scored_samples=total,
-        correct=correct,
-        accuracy=accuracy,
-        errors=0,
-        mean_latency_seconds=0.0,
-        total_cost_usd=0.0,
-    )
+    # Convert BenchmarkResults to RunSummary, classifying harness/infra
+    # failures (e.g. zero-model-contact setup hangs) out of the resolve-rate.
+    summary, harness_failures = summarize_benchmark_results(results, model=model)
+    if harness_failures:
+        console.print(
+            f"  [red bold]{len(harness_failures)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for failure in harness_failures:
+            console.print(
+                f"  [red]- {failure['task_id']}: {failure['reason']} "
+                f"(failure_mode={failure['failure_mode']})[/red]"
+            )
+    return summary
 
 
-def _run_single(config, console: Optional[Console] = None) -> object:
-    """Run a single eval from a RunConfig and return the summary."""
+def _run_single(
+    config,
+    console: Optional[Console] = None,
+    *,
+    suite_mode: bool = False,
+) -> object:
+    """Run a single eval from a RunConfig and return the summary.
+
+    ``suite_mode=True`` (TOML-suite drivers) scopes ``config.base_url`` /
+    ``config.api_key`` — stamped from the suite's ``[backend.external]``
+    section onto every RunConfig — to the hermes/openclaw backends only;
+    extending suite-level endpoint targeting to first-party backends is
+    explicitly deferred. The CLI single-run path (``suite_mode=False``)
+    honors ``--base-url``/``--api-key`` for every backend.
+    """
     from openjarvis.evals.core.runner import EvalRunner
 
     if console is None:
         console = Console()
 
+    _metadata = getattr(config, "metadata", None) or {}
+    base_url = (
+        getattr(config, "base_url", None)
+        or _metadata.get("base_url")
+        or os.environ.get("JARVIS_BACKEND_BASE_URL")
+    )
+    api_key = (
+        getattr(config, "api_key", None)
+        or _metadata.get("api_key")
+        or os.environ.get("JARVIS_BACKEND_API_KEY")
+    )
+
     # TerminalBench V2.1 native: use terminal-bench Harness directly
     if config.benchmark == "terminalbench-native":
-        return _run_terminalbench_native(config, console)
+        return _run_terminalbench_native(
+            config,
+            console,
+            base_url=None if suite_mode else base_url,
+            api_key=None if suite_mode else api_key,
+        )
 
-    _metadata = getattr(config, "metadata", None) or {}
     eval_backend = _build_backend(
         config.backend,
         config.engine_key,
@@ -732,16 +815,9 @@ def _run_single(config, console: Optional[Console] = None) -> object:
         gpu_metrics=getattr(config, "gpu_metrics", False),
         model=config.model,
         max_turns=getattr(config, "max_turns", None),
-        base_url=(
-            getattr(config, "base_url", None)
-            or _metadata.get("base_url")
-            or os.environ.get("JARVIS_BACKEND_BASE_URL")
-        ),
-        api_key=(
-            getattr(config, "api_key", None)
-            or _metadata.get("api_key")
-            or os.environ.get("JARVIS_BACKEND_API_KEY")
-        ),
+        base_url=base_url,
+        api_key=api_key,
+        first_party_endpoint=not suite_mode,
     )
     dataset = _build_dataset(config.benchmark)
     # Inject engine config for benchmarks that run their own simulation
@@ -964,7 +1040,9 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
     from rich.table import Table
 
     completed = sum(1 for t in traces if t.completed)
-    resolved = sum(1 for t in traces if t.is_resolved is True)
+    harness_errors = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = [t for t in traces if t.error_kind != "harness_error"]
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
     timed_out = sum(1 for t in traces if t.timed_out)
     total_turns = sum(t.num_turns for t in traces)
     total_tool_calls = sum(t.total_tool_calls for t in traces)
@@ -992,8 +1070,11 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
     table.add_row("Queries", str(len(traces)))
     table.add_row("Completed", f"{completed}/{len(traces)}")
     if any(t.is_resolved is not None for t in traces):
-        table.add_row("Resolved", f"{resolved}/{len(traces)}")
+        # Harness errors are excluded from the resolve-rate denominator:
+        # they are infra failures, not model misses.
+        table.add_row("Resolved", f"{resolved}/{len(model_traces)}")
     table.add_row("Timed out", str(timed_out))
+    table.add_row("Harness errors", str(len(harness_errors)))
     table.add_row("Total turns", str(total_turns))
     avg_t = f"{total_turns / len(traces):.1f}" if traces else "0"
     table.add_row("Avg turns/query", avg_t)
@@ -1020,6 +1101,19 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
 
     console.print(table)
 
+    if harness_errors:
+        console.print(
+            f"[red bold]{len(harness_errors)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for t in harness_errors[:5]:
+            console.print(f"[red]  {t.query_id}: {(t.error or '')[:300]}[/red]")
+        if len(harness_errors) > 5:
+            console.print(
+                f"[red]  ... and {len(harness_errors) - 5} more "
+                "(see traces.jsonl)[/red]"
+            )
+
 
 def _run_from_config(
     config_path: str,
@@ -1070,7 +1164,7 @@ def _run_from_config(
             f"Run {i}/{len(run_configs)}: {rc.benchmark} / {rc.model}",
         )
         try:
-            summary = _run_single(rc, console=console)
+            summary = _run_single(rc, console=console, suite_mode=True)
             summaries.append(summary)
             console.print(
                 f"  [green]{summary.accuracy:.4f}[/green] "
@@ -1115,12 +1209,21 @@ def main():
 @click.option(
     "--base-url",
     default=None,
-    help="OpenAI-compat endpoint for hermes/openclaw",
+    help=(
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
+        "(env: JARVIS_BACKEND_BASE_URL)."
+    ),
 )
 @click.option(
     "--api-key",
     default=None,
-    help="API key for hermes/openclaw endpoint",
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
 @click.option("-m", "--model", default=None, help="Model identifier")
 @click.option(
@@ -1526,8 +1629,7 @@ def summarize(jsonl_path):
     default=None,
     type=click.Path(),
     help=(
-        "Output JSONL path. Defaults to <jsonl>.reparsed when "
-        "--in-place is not set."
+        "Output JSONL path. Defaults to <jsonl>.reparsed when --in-place is not set."
     ),
 )
 @click.option(
@@ -1642,9 +1744,7 @@ def reparse_judge(jsonl_path, out_path, in_place, summary_out):
         _json.dump(summary, f, indent=2)
 
     old_cont = [float(s) for s in old_scores if s is not None]
-    old_acc = (
-        sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
-    )
+    old_acc = sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
     old_mean = sum(old_cont) / len(old_cont) if old_cont else 0.0
     new_mean = sum(cont) / len(cont) if cont else 0.0
     mean_shift = new_mean - old_mean

src/openjarvis/evals/core/agentic_runner.py

@@ -20,6 +20,7 @@ from contextlib import nullcontext
 from pathlib import Path
 from typing import Any, Callable, Optional
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.core.event_recorder import AgentEvent, EventRecorder, EventType
 from openjarvis.evals.core.trace import QueryTrace, TurnTrace
 
@@ -176,9 +177,12 @@ class AgenticRunner:
                 )
             self._traces.append(trace)
 
-            status = (
-                "TIMEOUT" if trace.timed_out else ("OK" if trace.completed else "FAIL")
-            )
+            if trace.timed_out:
+                status = "TIMEOUT"
+            elif trace.error_kind == "harness_error":
+                status = "HARNESS_ERROR"
+            else:
+                status = "OK" if trace.completed else "FAIL"
             LOGGER.info(
                 "Task %s: %s in %.1fs",
                 query_id,
@@ -260,11 +264,12 @@ class AgenticRunner:
                         is_resolved=record.metadata.get("is_resolved"),
                     )
 
-                status = (
-                    "TIMEOUT"
-                    if trace.timed_out
-                    else ("OK" if trace.completed else "FAIL")
-                )
+                if trace.timed_out:
+                    status = "TIMEOUT"
+                elif trace.error_kind == "harness_error":
+                    status = "HARNESS_ERROR"
+                else:
+                    status = "OK" if trace.completed else "FAIL"
                 LOGGER.info(
                     "Task %s: %s in %.1fs",
                     query_id,
@@ -444,7 +449,23 @@ class AgenticRunner:
                 _run_body()
 
         except Exception as exc:
-            LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
+            # Distinguish infrastructure breakage (task env failed to start,
+            # Docker/tmux death) from agent failures: harness errors must be
+            # recorded distinctly so scoring excludes them from resolve-rate
+            # instead of silently counting a model miss. Either way the run
+            # continues with the next record (fail THIS task, not the run).
+            is_harness_error = isinstance(exc, TaskEnvironmentError) or bool(
+                record.metadata.get("harness_error")
+            )
+            if is_harness_error:
+                LOGGER.error(
+                    "Harness/environment failure on query %s (record %s): %s",
+                    query_id,
+                    getattr(record, "record_id", "?"),
+                    exc,
+                )
+            else:
+                LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
             end_time = time.time()
             # Unsubscribe EventBus relays
             if agent_bus is not None:
@@ -461,6 +482,8 @@ class AgenticRunner:
                 total_wall_clock_s=end_time - start_time,
                 completed=False,
                 is_resolved=record.metadata.get("is_resolved"),
+                error=str(exc),
+                error_kind="harness_error" if is_harness_error else "agent_error",
             )
 
         # Unsubscribe EventBus relays
@@ -534,6 +557,48 @@ class AgenticRunner:
                     model, turn.input_tokens, turn.output_tokens
                 )
 
+        # --- Zero-model-contact sanity check ----------------------------
+        # Failed in-container setups (e.g. an installed agent's SETUP phase
+        # hanging or dying) historically produced traces that looked like
+        # model results: completed=True, a synthetic 1-event turn,
+        # is_resolved=False from run_tests, and zero tokens — silently
+        # dragging resolve-rate down as a fake model miss. Signal choice:
+        # token usage plus LM inference events is the reliable discriminator
+        # here — a genuine model miss has token usage (and/or LM events),
+        # while "the agent never called the model" has neither. We do NOT
+        # key off completion status or is_resolved, which are identical in
+        # both cases. run_agent_loop envs drive the model directly
+        # (bypassing usage reporting), so their turn_wall_clocks count as
+        # model contact.
+        had_lm_events = any(
+            e.event_type in (EventType.LM_INFERENCE_START, EventType.LM_INFERENCE_END)
+            for e in events
+        )
+        had_loop_turns = bool(
+            task_env is not None and getattr(task_env, "turn_wall_clocks", None)
+        )
+        turn_tokens = sum(t.input_tokens + t.output_tokens for t in turns)
+        error: Optional[str] = None
+        error_kind: Optional[str] = None
+        if (
+            not had_lm_events
+            and not had_loop_turns
+            and in_tok + out_tok == 0
+            and turn_tokens == 0
+        ):
+            error = (
+                "zero_model_requests: the agent produced no LM inference "
+                "events and reported zero token usage — the model was never "
+                "contacted. This is a harness/infrastructure failure (e.g. "
+                "in-container agent setup hang/death), not a model miss, and "
+                "is excluded from resolve-rate. If your agent genuinely "
+                "contacted the model, make it report token usage or emit "
+                "LM_INFERENCE events. Response tail: "
+                f"{(response_text or '')[-500:]!r}"
+            )
+            error_kind = "harness_error"
+            LOGGER.error("Query %s: %s", query_id, error)
+
         # Query-level energy from telemetry window
         query_gpu_energy = _compute_energy_delta(readings, "gpu_energy_j")
         query_cpu_energy = _compute_energy_delta(readings, "cpu_energy_j")
@@ -563,6 +628,8 @@ class AgenticRunner:
             is_resolved=record.metadata.get("is_resolved"),
             query_mbu_avg_pct=query_mbu_avg,
             query_mbu_max_pct=query_mbu_max,
+            error=error,
+            error_kind=error_kind,
         )
 
         # Correlate energy with trace

src/openjarvis/evals/core/config.py

@@ -143,6 +143,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
         sheets_worksheet=run_raw.get("sheets_worksheet", "Results"),
         sheets_credentials_path=run_raw.get("sheets_credentials_path", ""),
         max_turns=(int(run_raw["max_turns"]) if "max_turns" in run_raw else None),
+        global_agent_timeout_sec=(
+            float(run_raw["global_agent_timeout_sec"])
+            if "global_agent_timeout_sec" in run_raw
+            else None
+        ),
+        global_timeout_multiplier=(
+            float(run_raw["global_timeout_multiplier"])
+            if "global_timeout_multiplier" in run_raw
+            else None
+        ),
     )
 
     # Parse [[models]]
@@ -212,6 +222,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
                 max_tokens=int(b["max_tokens"]) if "max_tokens" in b else None,
                 subset=b.get("subset"),
                 record_ids=record_ids,
+                global_agent_timeout_sec=(
+                    float(b["global_agent_timeout_sec"])
+                    if "global_agent_timeout_sec" in b
+                    else None
+                ),
+                global_timeout_multiplier=(
+                    float(b["global_timeout_multiplier"])
+                    if "global_timeout_multiplier" in b
+                    else None
+                ),
             )
         )
 
@@ -275,6 +295,14 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
             if bench.judge_model is not None:
                 judge_model = bench.judge_model
 
+            # terminal-bench harness budgets: benchmark > [run]
+            global_agent_timeout_sec = suite.run.global_agent_timeout_sec
+            if bench.global_agent_timeout_sec is not None:
+                global_agent_timeout_sec = bench.global_agent_timeout_sec
+            global_timeout_multiplier = suite.run.global_timeout_multiplier
+            if bench.global_timeout_multiplier is not None:
+                global_timeout_multiplier = bench.global_timeout_multiplier
+
             # Auto-generate output path
             model_slug = model.name.replace("/", "-").replace(":", "-")
             output_path = f"{output_dir}/{bench.name}_{model_slug}.jsonl"
@@ -328,6 +356,8 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
                     base_url=suite.backend_external_base_url,
                     api_key=suite.backend_external_api_key,
                     record_ids=bench.record_ids,
+                    global_agent_timeout_sec=global_agent_timeout_sec,
+                    global_timeout_multiplier=global_timeout_multiplier,
                 )
             )
 

src/openjarvis/evals/core/environment.py

@@ -8,6 +8,18 @@ from typing import Any, Dict, Tuple
 from openjarvis.evals.core.types import EvalRecord
 
 
+class TaskEnvironmentError(RuntimeError):
+    """A task execution environment failed to start or operate.
+
+    Raised when infrastructure backing a task (Docker container, docker
+    compose project, tmux session, recording binaries, ...) breaks. This is
+    a harness/environment failure, **not** a model failure: runners record
+    it distinctly (``QueryTrace.error_kind == "harness_error"``) so scoring
+    can exclude the sample from resolve-rate instead of silently counting
+    it as a model miss.
+    """
+
+
 class EnvironmentProvider(ABC):
     """Manages an external environment for evaluation benchmarks.
 
@@ -50,3 +62,6 @@ class EnvironmentProvider(ABC):
     @abstractmethod
     def teardown(self) -> None:
         """Stop the environment and release resources."""
+
+
+__all__ = ["EnvironmentProvider", "TaskEnvironmentError"]

src/openjarvis/evals/core/export.py

@@ -26,13 +26,22 @@ def _agg_stats(values: Sequence[Optional[float]]) -> dict[str, Optional[float]]:
     }
 
 
+def _model_attributable(traces: list[QueryTrace]) -> list[QueryTrace]:
+    """Traces whose outcome is attributable to the model.
+
+    Harness errors (infra/setup failures, zero-model-contact runs) are
+    excluded so they never count as model misses in resolve-rate.
+    """
+    return [t for t in traces if t.error_kind != "harness_error"]
+
+
 def _compute_efficiency(
     traces: list[QueryTrace],
     total_gpu_energy: Optional[float],
     total_cpu_energy: Optional[float],
 ) -> dict[str, Optional[float]]:
     """Compute efficiency metrics from traces and aggregate energy."""
-    scored = [t for t in traces if t.is_resolved is not None]
+    scored = [t for t in _model_attributable(traces) if t.is_resolved is not None]
     resolved = sum(1 for t in scored if t.is_resolved is True)
     accuracy = resolved / len(scored) if scored else None
     gpu_powers = [
@@ -247,8 +256,12 @@ def export_summary_json(
             cpu_energy_values.append(sum(cpu_vals))
     total_cpu_energy = sum(cpu_energy_values) if cpu_energy_values else None
 
-    resolved = sum(1 for t in traces if t.is_resolved is True)
-    unresolved = sum(1 for t in traces if t.is_resolved is False)
+    # Harness errors (infra failures, zero-model-contact runs) are excluded
+    # from the resolve-rate denominator: they are not model misses.
+    harness_error_traces = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = _model_attributable(traces)
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
+    unresolved = sum(1 for t in model_traces if t.is_resolved is False)
 
     cost_values = [t.total_cost_usd for t in traces if t.total_cost_usd is not None]
     total_cost = sum(cost_values) if cost_values else None
@@ -345,6 +358,7 @@ def export_summary_json(
             "completed": completed,
             "resolved": resolved,
             "unresolved": unresolved,
+            "harness_errors": len(harness_error_traces),
             "accuracy": accuracy,
             "turns": total_turns,
             "tool_calls": total_tool_calls,
@@ -365,6 +379,12 @@ def export_summary_json(
         "efficiency": efficiency,
     }
 
+    if harness_error_traces:
+        summary["harness_error_details"] = [
+            {"query_id": t.query_id, "error": (t.error or "")[:500]}
+            for t in harness_error_traces
+        ]
+
     if action_totals:
         summary["action_energy_summary"] = action_totals
 
@@ -399,7 +419,7 @@ def export_summary_json(
 
     accuracy_vals: list[float] = [
         1.0 if t.is_resolved is True else 0.0
-        for t in traces
+        for t in _model_attributable(traces)
         if t.is_resolved is not None
     ]
     latency_vals = [t.total_wall_clock_s for t in traces if t.total_wall_clock_s > 0]

src/openjarvis/evals/core/trace.py

@@ -91,6 +91,13 @@ class QueryTrace:
     is_resolved: Optional[bool] = None
     query_mbu_avg_pct: Optional[float] = None
     query_mbu_max_pct: Optional[float] = None
+    # Error taxonomy. ``error_kind`` distinguishes infrastructure failures
+    # ("harness_error": task env / Docker / tmux broke, or the agent never
+    # contacted the model) from agent failures ("agent_error"). Harness
+    # errors are excluded from resolve-rate by export/summary code so they
+    # are never silently counted as model misses.
+    error: Optional[str] = None
+    error_kind: Optional[str] = None
 
     @property
     def num_turns(self) -> int:
@@ -196,6 +203,8 @@ class QueryTrace:
             "is_resolved": self.is_resolved,
             "query_mbu_avg_pct": self.query_mbu_avg_pct,
             "query_mbu_max_pct": self.query_mbu_max_pct,
+            "error": self.error,
+            "error_kind": self.error_kind,
         }
 
     @classmethod
@@ -216,6 +225,8 @@ class QueryTrace:
             is_resolved=d.get("is_resolved"),
             query_mbu_avg_pct=d.get("query_mbu_avg_pct"),
             query_mbu_max_pct=d.get("query_mbu_max_pct"),
+            error=d.get("error"),
+            error_kind=d.get("error_kind"),
         )
 
     def save_jsonl(self, path: Path) -> None:

src/openjarvis/evals/core/types.py

@@ -102,6 +102,15 @@ class RunConfig:
     # specific records (e.g. recovering silent-fake records without
     # re-running the entire benchmark).
     record_ids: Optional[List[str]] = None
+    # terminal-bench harness budgets (terminalbench-native backend).
+    # global_agent_timeout_sec bounds each trial's agent phase — SETUP+RUN
+    # together, since terminal-bench runs installed-agent setup inside the
+    # agent budget with an infinite tmux timeout. When set it REPLACES the
+    # per-task max_agent_timeout_sec; 0 disables the bound (per-task budgets
+    # apply); None uses the backend default (1800 s).
+    global_agent_timeout_sec: Optional[float] = None
+    # Scales per-task budgets when global_agent_timeout_sec is not set.
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)
@@ -232,6 +241,9 @@ class ExecutionConfig:
     # to JarvisConfig.agent.max_turns (default 10). Bump to 30-50 for
     # thinking/reasoning models on agentic benchmarks (GAIA, LiveResearch).
     max_turns: Optional[int] = None
+    # terminal-bench harness budgets (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)
@@ -265,6 +277,10 @@ class BenchmarkConfig:
     max_tokens: Optional[int] = None
     subset: Optional[str] = None
     record_ids: Optional[List[str]] = None
+    # Per-benchmark override of the terminal-bench harness budgets
+    # (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)

src/openjarvis/evals/execution/terminalbench_env.py

@@ -8,6 +8,8 @@ from pathlib import Path
 from types import TracebackType
 from typing import Any, MutableMapping, Optional, Type
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
+
 LOGGER = logging.getLogger(__name__)
 
 
@@ -29,8 +31,6 @@ class TerminalBenchTaskEnv:
     # ------------------------------------------------------------------
 
     def __enter__(self) -> TerminalBenchTaskEnv:
-        from terminal_bench.terminal.terminal import spin_up_terminal
-
         task = self._metadata.get("task")
         task_paths = self._metadata.get("task_paths")
         task_id = self._metadata.get("task_id", "unknown")
@@ -41,6 +41,8 @@ class TerminalBenchTaskEnv:
                 "Use the 'terminalbench-native' dataset."
             )
 
+        from terminal_bench.terminal.terminal import spin_up_terminal
+
         docker_image_prefix = f"tb__{task_id}".replace(".", "-")
         client_image_name = f"{docker_image_prefix}__client"
         client_container_name = f"oj-{task_id}".replace(".", "-")
@@ -48,19 +50,59 @@ class TerminalBenchTaskEnv:
         self._logs_tmpdir = tempfile.TemporaryDirectory(prefix="oj_tb_logs_")
         logs_path = Path(self._logs_tmpdir.name)
 
-        self._terminal_cm = spin_up_terminal(
-            client_container_name=client_container_name,
-            client_image_name=client_image_name,
-            docker_compose_path=task_paths.docker_compose_path,
-            docker_image_name_prefix=docker_image_prefix,
-            sessions_logs_path=logs_path,
-            disable_recording=task.disable_asciinema,
-        )
-        self._terminal = self._terminal_cm.__enter__()
+        # Everything below is exception-safe: a failure mid-startup (docker
+        # compose, tmux, asciinema) tears the spun-up terminal back down
+        # immediately instead of leaking the docker compose project until GC
+        # (or forever, when the env object is retained), and re-raises as a
+        # loud TaskEnvironmentError naming the task image so the runner can
+        # record a harness error for THIS task and continue with the rest.
+        try:
+            self._terminal_cm = spin_up_terminal(
+                client_container_name=client_container_name,
+                client_image_name=client_image_name,
+                docker_compose_path=task_paths.docker_compose_path,
+                docker_image_name_prefix=docker_image_prefix,
+                sessions_logs_path=logs_path,
+                disable_recording=task.disable_asciinema,
+            )
+            self._terminal = self._terminal_cm.__enter__()
 
-        session = self._terminal.create_session(
-            "agent", is_active_stream=False, as_configured_user=True
-        )
+            # Preflight BEFORE the agent loop: terminal-bench drives the
+            # agent through tmux (and records via asciinema unless the task
+            # disables it). A missing binary otherwise surfaces mid-run as
+            # an opaque RuntimeError or a fake TimeoutError.
+            self._preflight_container_binaries(
+                task, task_id, client_image_name, client_container_name
+            )
+
+            session = self._terminal.create_session(
+                "agent", is_active_stream=False, as_configured_user=True
+            )
+        except BaseException as exc:
+            self._teardown(type(exc), exc, exc.__traceback__)
+            if not isinstance(exc, Exception):
+                # KeyboardInterrupt / SystemExit: clean up but never mask.
+                raise
+            if isinstance(exc, TaskEnvironmentError):
+                self._metadata["harness_error"] = str(exc)
+                raise
+            message = (
+                f"Task '{task_id}': failed to start the task environment "
+                f"(image '{client_image_name}', container "
+                f"'{client_container_name}'): {exc}. This is a harness/"
+                "environment failure, not a model failure. Check that the "
+                "Docker daemon is healthy and that tmux is installed in the "
+                "task image."
+            )
+            # docker compose stderr is only logged at DEBUG by
+            # terminal-bench; surface it here so the failure is actionable.
+            stderr = getattr(exc, "stderr", None)
+            if stderr:
+                if isinstance(stderr, bytes):
+                    stderr = stderr.decode("utf-8", errors="replace")
+                message += f"\ndocker compose stderr (tail):\n{stderr[-2000:]}"
+            self._metadata["harness_error"] = message
+            raise TaskEnvironmentError(message) from exc
 
         self._metadata["terminal"] = self._terminal
         self._metadata["session"] = session
@@ -68,24 +110,99 @@ class TerminalBenchTaskEnv:
 
         return self
 
+    def _preflight_container_binaries(
+        self,
+        task: Any,
+        task_id: str,
+        client_image_name: str,
+        client_container_name: str,
+    ) -> None:
+        """Verify tmux (and asciinema if recording) exist in the container.
+
+        Raises:
+            TaskEnvironmentError: naming the task image and the missing
+                binary, with the remedy, before any agent work starts.
+        """
+        container = getattr(self._terminal, "container", None)
+        if container is None:
+            # Terminal implementation without a container handle (e.g. a
+            # future terminal-bench version); fall through to terminal-bench's
+            # own checks rather than guessing.
+            return
+
+        checks: list[tuple[str, list[str], str]] = [
+            (
+                "tmux",
+                ["tmux", "-V"],
+                f"install tmux in the task image '{client_image_name}'",
+            ),
+        ]
+        if not getattr(task, "disable_asciinema", False):
+            checks.append(
+                (
+                    "asciinema",
+                    ["asciinema", "--version"],
+                    f"install asciinema in the task image '{client_image_name}' "
+                    "or set disable_asciinema in task.yaml",
+                )
+            )
+
+        for binary, cmd, remedy in checks:
+            result = container.exec_run(cmd)
+            exit_code = getattr(result, "exit_code", 0)
+            if exit_code == 0:
+                continue
+            output = getattr(result, "output", b"")
+            if isinstance(output, bytes):
+                output = output.decode("utf-8", errors="replace")
+            raise TaskEnvironmentError(
+                f"Task '{task_id}': required binary '{binary}' is not usable "
+                f"in task image '{client_image_name}' (container "
+                f"'{client_container_name}'): exec exit code {exit_code}, "
+                f"output {str(output).strip()!r}. Remedy: {remedy}. This is "
+                "a harness/environment failure, not a model failure."
+            )
+
+    def _teardown(
+        self,
+        exc_type: Optional[Type[BaseException]] = None,
+        exc_val: Optional[BaseException] = None,
+        exc_tb: Optional[TracebackType] = None,
+    ) -> None:
+        """Idempotent cleanup shared by ``__exit__`` and failed ``__enter__``.
+
+        Secondary cleanup errors are logged, never raised, so they cannot
+        mask the original failure.
+        """
+        self._metadata.pop("terminal", None)
+        self._metadata.pop("session", None)
+        self._metadata.pop("container", None)
+
+        terminal_cm, self._terminal_cm, self._terminal = self._terminal_cm, None, None
+        if terminal_cm is not None:
+            try:
+                terminal_cm.__exit__(exc_type, exc_val, exc_tb)
+            except Exception:
+                LOGGER.exception(
+                    "Secondary error while tearing down the terminal for "
+                    "task %s (original error, if any, is re-raised)",
+                    self._metadata.get("task_id", "unknown"),
+                )
+
+        logs_tmpdir, self._logs_tmpdir = self._logs_tmpdir, None
+        if logs_tmpdir is not None:
+            try:
+                logs_tmpdir.cleanup()
+            except Exception:
+                LOGGER.exception("Failed to clean up session logs tmpdir")
+
     def __exit__(
         self,
         exc_type: Optional[Type[BaseException]],
         exc_val: Optional[BaseException],
         exc_tb: Optional[TracebackType],
     ) -> None:
-        self._metadata.pop("terminal", None)
-        self._metadata.pop("session", None)
-        self._metadata.pop("container", None)
-
-        if self._terminal_cm is not None:
-            self._terminal_cm.__exit__(exc_type, exc_val, exc_tb)
-            self._terminal_cm = None
-            self._terminal = None
-
-        if self._logs_tmpdir is not None:
-            self._logs_tmpdir.cleanup()
-            self._logs_tmpdir = None
+        self._teardown(exc_type, exc_val, exc_tb)
 
     # ------------------------------------------------------------------
     # Test execution
@@ -93,12 +210,6 @@ class TerminalBenchTaskEnv:
 
     def run_tests(self) -> tuple[bool, dict[str, Any]]:
         """Copy test scripts into container, execute, parse results."""
-        from terminal_bench.parsers.base_parser import UnitTestStatus
-        from terminal_bench.parsers.parser_factory import ParserFactory
-        from terminal_bench.terminal.docker_compose_manager import (
-            DockerComposeManager,
-        )
-
         task = self._metadata["task"]
         task_paths = self._metadata["task_paths"]
         terminal = self._terminal
@@ -110,6 +221,12 @@ class TerminalBenchTaskEnv:
             self._metadata["test_results"] = results
             return False, results
 
+        from terminal_bench.parsers.base_parser import UnitTestStatus
+        from terminal_bench.parsers.parser_factory import ParserFactory
+        from terminal_bench.terminal.docker_compose_manager import (
+            DockerComposeManager,
+        )
+
         try:
             paths_to_copy = [task_paths.run_tests_path]
             if task_paths.test_dir.exists():

src/openjarvis/evals/tests/test_backends.py

@@ -4,6 +4,27 @@ from __future__ import annotations
 
 from unittest.mock import MagicMock, patch
 
+import httpx
+import pytest
+import respx
+
+
+def _mock_builder() -> MagicMock:
+    """A SystemBuilder mock whose fluent methods chain like the real one."""
+    builder = MagicMock()
+    for method in (
+        "engine",
+        "engine_instance",
+        "model",
+        "agent",
+        "tools",
+        "telemetry",
+        "traces",
+    ):
+        getattr(builder, method).return_value = builder
+    builder.build.return_value = MagicMock()
+    return builder
+
 
 class TestJarvisDirectBackend:
     @patch("openjarvis.system.SystemBuilder")
@@ -137,3 +158,130 @@ class TestJarvisAgentBackend:
         assert result["content"] == "The answer is 4."
         assert result["turns"] == 2
         assert len(result["tool_results"]) == 1
+
+
+class TestJarvisDirectBackendBaseUrl:
+    """--base-url targeting for the jarvis-direct backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        # Trailing /v1 is normalized away so request paths don't double up.
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        # The discovery path must not be engaged at all.
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisDirectBackend(base_url="http://127.0.0.1:18998")
+
+        # No silent engine substitution: the system is never built.
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisDirectBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_wins_over_engine_key(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(engine_key="vllm", base_url="http://127.0.0.1:18999")
+
+        mock_builder.engine.assert_not_called()
+        mock_builder.engine_instance.assert_called_once()
+        # The engine key is kept as the label for the injected engine.
+        assert mock_builder.engine_instance.call_args.kwargs["key"] == "vllm"
+
+
+class TestJarvisAgentBackendBaseUrl:
+    """--base-url targeting for the jarvis-agent backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisAgentBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisAgentBackend(base_url="http://127.0.0.1:18998")
+
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisAgentBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()

src/openjarvis/evals/tests/test_cli_endpoint.py

@@ -0,0 +1,190 @@
+"""--base-url/--api-key forwarding through the eval CLI plumbing.
+
+Covers the fix for the eval-CLI endpoint gap: the flags used to be silently
+dropped for jarvis-direct/jarvis-agent and ignored by terminalbench-native
+(which hardcoded api_base="http://localhost:8000/v1").
+"""
+
+from __future__ import annotations
+
+import io
+import os
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import click
+import pytest
+from rich.console import Console
+
+from openjarvis.evals.cli import _build_backend, _run_terminalbench_native
+from openjarvis.evals.core.types import RunConfig
+
+
+def _quiet_console() -> Console:
+    return Console(file=io.StringIO())
+
+
+def _tb_config(**overrides) -> RunConfig:
+    defaults = dict(
+        benchmark="terminalbench-native",
+        backend="jarvis-direct",
+        model="my-model",
+        max_samples=1,
+        max_workers=1,
+        temperature=0.2,
+    )
+    defaults.update(overrides)
+    return RunConfig(**defaults)
+
+
+class TestBuildBackendForwardsEndpoint:
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_jarvis_direct_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_agent.JarvisAgentBackend")
+    def test_jarvis_agent_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-agent",
+            "vllm",
+            "orchestrator",
+            ["calculator"],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_suite_mode_scopes_endpoint_to_external_backends(self, mock_cls):
+        """[backend.external] suite semantics stay hermes/openclaw-only:
+        first_party_endpoint=False must not forward to first-party."""
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+            first_party_endpoint=False,
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] is None
+        assert kwargs["api_key"] is None
+
+    def test_hermes_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="hermes"):
+            _build_backend("hermes", None, "orchestrator", [])
+
+    def test_openclaw_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="openclaw"):
+            _build_backend("openclaw", None, "orchestrator", [])
+
+
+class TestTerminalBenchNativeApiBase:
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_passed_through_as_api_base(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_without_v1_gets_single_v1_suffix(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_default_api_base_unchanged_without_base_url(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(_tb_config(), _quiet_console())
+        assert mock_cls.call_args.kwargs["api_base"] == "http://localhost:8000/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_api_key_exported_as_openai_api_key_during_run(self, mock_cls, monkeypatch):
+        """terminus-2 reads OPENAI_API_KEY via LiteLLM; the var must be set
+        during harness.run() and restored afterwards."""
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+        seen: dict = {}
+
+        def fake_run_harness(run_id):
+            seen["openai_api_key"] = os.environ.get("OPENAI_API_KEY")
+            return SimpleNamespace(trial_results=[])
+
+        mock_backend = MagicMock()
+        mock_backend.run_harness.side_effect = fake_run_harness
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert seen["openai_api_key"] == "sk-tb"
+        assert "OPENAI_API_KEY" not in os.environ  # restored
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_preexisting_openai_api_key_restored(self, mock_cls, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "sk-original")
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert os.environ["OPENAI_API_KEY"] == "sk-original"
+
+
+class TestRunSingleSuiteModeGating:
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_suite_mode_drops_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console(), suite_mode=True)
+        assert mock_tb.call_args.kwargs["base_url"] is None
+        assert mock_tb.call_args.kwargs["api_key"] is None
+
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_cli_mode_forwards_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console())
+        assert mock_tb.call_args.kwargs["base_url"] == "http://node7:8123/v1"
+        assert mock_tb.call_args.kwargs["api_key"] == "sk-k"

src/openjarvis/learning/intelligence/orchestrator/sft_trainer.py

@@ -141,10 +141,18 @@ class OrchestratorSFTDataset:
             return_tensors="pt",
         )
 
+        input_ids = encoding["input_ids"].squeeze(0)
+        attention_mask = encoding["attention_mask"].squeeze(0)
+
+        # Exclude padding positions from the loss (-100 is the
+        # cross-entropy ignore_index).
+        labels = input_ids.clone()
+        labels[attention_mask == 0] = -100
+
         return {
-            "input_ids": encoding["input_ids"].squeeze(0),
-            "attention_mask": encoding["attention_mask"].squeeze(0),
-            "labels": encoding["input_ids"].squeeze(0).clone(),
+            "input_ids": input_ids,
+            "attention_mask": attention_mask,
+            "labels": labels,
         }
 
     def _format_conversation(self, conversations: List[Dict[str, str]]) -> str:

src/openjarvis/learning/training/lora.py

@@ -389,10 +389,15 @@ class LoRATrainer:
             [item["attention_mask"] for item in batch_items]
         ).to(self.device)
 
+        # Exclude padding positions from the loss (-100 is the
+        # cross-entropy ignore_index; pad_token == eos_token here).
+        labels = input_ids.clone()
+        labels[attention_mask == 0] = -100
+
         outputs = self.model(
             input_ids=input_ids,
             attention_mask=attention_mask,
-            labels=input_ids,
+            labels=labels,
         )
         loss = outputs.loss
 

src/openjarvis/prompt/builder.py

@@ -229,6 +229,11 @@ class SystemPromptBuilder:
             )
 
     def _load_file(self, path_str: str, max_chars: int) -> str:
+        # An empty path means "no file" (e.g. the persona "none" opt-out, which
+        # resolves to empty paths). Guard before Path("") — which becomes "." —
+        # so reading it does not raise IsADirectoryError.
+        if not path_str:
+            return ""
         path = Path(path_str).expanduser()
         if not path.exists():
             return ""

src/openjarvis/server/agent_manager_routes.py

@@ -111,24 +111,35 @@ def _make_lightweight_system(
     model: str,
     config: Any = None,
 ) -> _LightweightSystem:
-    """Build a minimal system with a plain OllamaEngine.
+    """Build a minimal system with a fresh inference engine.
 
     The server's ``app.state.engine`` is heavily wrapped
     (MultiEngine -> InstrumentedEngine -> GuardrailsEngine) and can
-    return empty content from background threads.  Create a fresh
-    OllamaEngine directly (no health checks or model discovery that
-    could interfere with in-flight Ollama requests).
+    return empty content from background threads. Create a fresh
+    engine directly (no health checks or model discovery that
+    could interfere with in-flight requests).
     """
     try:
-        from openjarvis.engine.ollama import OllamaEngine
+        from openjarvis.engine._discovery import get_engine
 
         cfg = config
         if cfg is None:
             from openjarvis.core.config import load_config
 
             cfg = load_config()
-        host = cfg.engine.ollama.host if cfg else ""
-        plain_engine = OllamaEngine(host=host) if host else OllamaEngine()
+
+        pref = cfg.intelligence.preferred_engine
+        key = pref or cfg.engine.default
+        resolved = get_engine(cfg, key)
+
+        if resolved is not None:
+            plain_engine = resolved[1]
+        else:
+            from openjarvis.engine.ollama import OllamaEngine
+
+            host = cfg.engine.ollama.host if cfg else ""
+            plain_engine = OllamaEngine(host=host) if host else OllamaEngine()
+
         # Wrap with InstrumentedEngine so agent ticks are recorded
         # in telemetry (FLOPs, energy, cost savings).
         try:
@@ -842,9 +853,7 @@ async def _stream_managed_agent(
 
         app_config = load_config()
 
-    final_system_prompt = _build_managed_system_prompt(
-        system_prompt or "", app_config
-    )
+    final_system_prompt = _build_managed_system_prompt(system_prompt or "", app_config)
 
     if final_system_prompt and final_system_prompt.strip():
         llm_messages.append(

src/openjarvis/server/api_routes.py

@@ -153,14 +153,30 @@ memory_router = APIRouter(prefix="/v1/memory", tags=["memory"])
 
 
 def _get_memory_backend(request: Request):
-    """Return the app-level memory backend, falling back to a fresh SQLiteMemory."""
+    """Return the app-level memory backend, falling back to a fresh SQLiteMemory.
+
+    Raises ``HTTPException(503)`` with an actionable message when the backend
+    cannot be built because the mandatory ``openjarvis_rust`` extension is not
+    installed in the serving venv. This is deliberately distinct from a benign
+    "memory not configured" case (which returns ``None``): a missing native
+    extension must fail loudly, never silently degrade (#502).
+    """
     backend = getattr(request.app.state, "memory_backend", None)
     if backend is None:
+        from openjarvis.tools.storage._stubs import MemoryBackendUnavailable
+
         try:
             from openjarvis.tools.storage.sqlite import SQLiteMemory
 
             backend = SQLiteMemory()
+        except MemoryBackendUnavailable as exc:
+            # The native extension is missing — surface a loud, actionable error
+            # rather than a misleading "no backend" / silent no-op.
+            logger.error("%s", exc)
+            raise HTTPException(status_code=503, detail=str(exc)) from exc
         except Exception:
+            # Memory is genuinely unconfigured for a benign reason — preserve
+            # the existing graceful "no backend" behaviour.
             return None
     return backend
 
@@ -170,7 +186,9 @@ async def memory_store(req: MemoryStoreRequest, request: Request):
     """Store content in memory."""
     backend = _get_memory_backend(request)
     if backend is None:
-        return {"status": "stored", "note": "no backend available"}
+        # Memory is intentionally disabled; report it honestly instead of a
+        # 200 that silently discards the write (#502).
+        raise HTTPException(status_code=503, detail="Memory is not configured")
     try:
         backend.store(req.content, metadata=req.metadata or {})
         return {"status": "stored"}
@@ -216,7 +234,13 @@ async def memory_stats(request: Request):
 
 @memory_router.get("/config")
 async def memory_config(request: Request):
-    """Return current memory configuration."""
+    """Return current memory configuration.
+
+    Reports memory as *unavailable* (rather than falsely claiming
+    ``backend_type: sqlite``) when the native ``openjarvis_rust`` extension is
+    missing, so the UI can show the real cause instead of a healthy-looking
+    config that backs a silent no-op (#502).
+    """
     try:
         config = getattr(request.app.state, "config", None)
         if config is None:
@@ -224,12 +248,30 @@ async def memory_config(request: Request):
 
             config = load_config()
         backend = getattr(request.app.state, "memory_backend", None)
+        available = True
+        detail: Optional[str] = None
+        if backend is None:
+            from openjarvis.tools.storage._stubs import MemoryBackendUnavailable
+
+            try:
+                from openjarvis.tools.storage.sqlite import SQLiteMemory
+
+                backend = SQLiteMemory()
+            except MemoryBackendUnavailable as exc:
+                available = False
+                detail = str(exc)
+            except Exception:
+                # Benign: cannot construct a probe backend here, but the
+                # configured default is still what would be used.
+                pass
         return {
             "backend_type": (
                 backend.backend_id
                 if backend is not None
                 else config.memory.default_backend
             ),
+            "available": available,
+            "detail": detail,
             "context_top_k": config.memory.context_top_k,
             "context_min_score": config.memory.context_min_score,
             "context_max_tokens": config.memory.context_max_tokens,
@@ -243,17 +285,42 @@ async def memory_config(request: Request):
 async def memory_index(req: MemoryIndexRequest, request: Request):
     """Index files from a path into memory."""
     try:
+        import os
         from pathlib import Path
 
+        from openjarvis.security.file_policy import is_sensitive_file
         from openjarvis.tools.storage.ingest import ingest_path
 
         target = Path(req.path).expanduser().resolve()
         if not target.exists():
             raise HTTPException(status_code=404, detail=f"Path not found: {req.path}")
 
+        # Sandbox: when workspace roots are configured via OPENJARVIS_WORKSPACE
+        # (os.pathsep-separated), only allow indexing inside them. This endpoint
+        # must not become an arbitrary-filesystem read primitive over the API.
+        workspace = os.environ.get("OPENJARVIS_WORKSPACE", "").strip()
+        if workspace:
+            roots = [
+                Path(d).expanduser().resolve()
+                for d in workspace.split(os.pathsep)
+                if d.strip()
+            ]
+            if not any(
+                target == root or root in target.parents for root in roots
+            ):
+                raise HTTPException(
+                    status_code=403,
+                    detail="Path is outside the allowed workspace directories.",
+                )
+        # Never ingest sensitive files (.env, private keys, credentials, ...).
+        if target.is_file() and is_sensitive_file(target):
+            raise HTTPException(
+                status_code=403, detail="Refusing to index a sensitive file."
+            )
+
         backend = _get_memory_backend(request)
         if backend is None:
-            raise HTTPException(status_code=503, detail="No memory backend available")
+            raise HTTPException(status_code=503, detail="Memory is not configured")
 
         chunks = ingest_path(target)
         stored = 0
@@ -264,7 +331,16 @@ async def memory_index(req: MemoryIndexRequest, request: Request):
             backend.store(chunk.content, metadata=metadata)
             stored += 1
 
-        return {"status": "indexed", "chunks_indexed": stored}
+        result = {"status": "indexed", "chunks_indexed": stored}
+        if stored == 0:
+            # "indexed" must never silently mean "stored nothing". Surface why
+            # so a folder of short notes doesn't look like a successful no-op
+            # (#502 follow-up).
+            result["note"] = (
+                "no content was indexed — the path contained no readable "
+                "documents with indexable text"
+            )
+        return result
     except HTTPException:
         raise
     except Exception as exc:
@@ -554,6 +630,30 @@ async def prometheus_metrics(request: Request):
 websocket_router = APIRouter(tags=["websocket"])
 
 
+def _record_ws_trace(
+    trace_store,
+    *,
+    query: str,
+    result: str,
+    model: str,
+    started_at: float,
+    ended_at: float,
+) -> None:
+    """Record a trace for a completed WebSocket chat (best-effort)."""
+    if trace_store is None or not result:
+        return
+    from openjarvis.traces.collector import record_response_trace
+
+    record_response_trace(
+        trace_store,
+        query=query,
+        result=result,
+        model=model,
+        started_at=started_at,
+        ended_at=ended_at,
+    )
+
+
 @websocket_router.websocket("/v1/chat/stream")
 async def websocket_chat_stream(websocket: WebSocket):
     """Stream chat responses over a WebSocket connection.
@@ -608,6 +708,14 @@ async def websocket_chat_stream(websocket: WebSocket):
 
             messages = [{"role": "user", "content": message}]
 
+            # This WS path streams straight from the engine (no agent /
+            # TraceCollector), so record the interaction directly once it
+            # finishes — otherwise WebSocket chats never reach traces.db.
+            import time as _time
+
+            trace_store = getattr(websocket.app.state, "trace_store", None)
+            _ws_started_at = _time.time()
+
             try:
                 # Prefer streaming if the engine supports it
                 stream_fn = getattr(engine, "stream", None)
@@ -651,6 +759,14 @@ async def websocket_chat_stream(websocket: WebSocket):
                     await websocket.send_json(
                         {"type": "done", "content": full_content},
                     )
+                    _record_ws_trace(
+                        trace_store,
+                        query=message,
+                        result=full_content,
+                        model=model,
+                        started_at=_ws_started_at,
+                        ended_at=_time.time(),
+                    )
                 else:
                     # No stream method — single-shot generate
                     result = engine.generate(messages, model=model)
@@ -668,6 +784,14 @@ async def websocket_chat_stream(websocket: WebSocket):
                     await websocket.send_json(
                         {"type": "done", "content": content},
                     )
+                    _record_ws_trace(
+                        trace_store,
+                        query=message,
+                        result=content,
+                        model=model,
+                        started_at=_ws_started_at,
+                        ended_at=_time.time(),
+                    )
             except WebSocketDisconnect:
                 raise
             except Exception as exc:

src/openjarvis/server/app.py

@@ -229,7 +229,16 @@ def create_app(
     # AuthMiddleware never sees WS upgrade requests). Empty = auth disabled.
     app.state.api_key = api_key
 
-    # Wire up trace store if traces are enabled
+    # Wire up trace store if traces are enabled.
+    #
+    # We deliberately do NOT subscribe the trace store to the bus. The chat
+    # endpoints persist through a TraceCollector that calls store.save()
+    # directly (mirroring system/orchestrator.py), and the collector ALSO
+    # publishes TRACE_COMPLETE. A store subscribed to that same bus would
+    # therefore save every agent trace twice — the second INSERT hitting the
+    # UNIQUE constraint on trace_id (a 500 on every completion). Keeping the
+    # collector the single writer is what makes the dual code path safe; only
+    # the telemetry store is bus-subscribed (see system/builder.py).
     app.state.trace_store = None
     try:
         from openjarvis.core.config import load_config
@@ -237,11 +246,7 @@ def create_app(
 
         cfg = config if config is not None else load_config()
         if cfg.traces.enabled:
-            _trace_store = TraceStore(db_path=cfg.traces.db_path)
-            app.state.trace_store = _trace_store
-            _bus = getattr(app.state, "bus", None)
-            if _bus is not None:
-                _trace_store.subscribe_to_bus(_bus)
+            app.state.trace_store = TraceStore(db_path=cfg.traces.db_path)
     except Exception:
         pass  # traces are optional; don't block server startup
 

src/openjarvis/server/auth_middleware.py

@@ -33,7 +33,10 @@ class AuthMiddleware(BaseHTTPMiddleware):
                     status_code=401,
                 )
             scheme, _, token = auth.partition(" ")
-            if scheme.lower() != "bearer" or token != self._api_key:
+            # Constant-time comparison to avoid leaking the key via timing.
+            if scheme.lower() != "bearer" or not secrets.compare_digest(
+                token, self._api_key
+            ):
                 return JSONResponse(
                     {"detail": "Invalid API key"},
                     status_code=401,
@@ -42,8 +45,18 @@ class AuthMiddleware(BaseHTTPMiddleware):
 
     @staticmethod
     def _requires_auth(path: str) -> bool:
-        """Only protect API routes, not the frontend UI or static assets."""
-        return path.startswith("/v1/") or path.startswith("/api/")
+        """Protect API routes and operational metrics; leave the UI/health open.
+
+        ``/metrics`` exposes request/token counters that should not be readable
+        by unauthenticated clients, so it is gated alongside ``/v1`` and
+        ``/api``. ``/health`` stays open for liveness probes.
+        """
+        return (
+            path.startswith("/v1/")
+            or path.startswith("/api/")
+            or path == "/metrics"
+            or path.startswith("/metrics/")
+        )
 
 
 

src/openjarvis/server/routes.py

@@ -151,7 +151,13 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request
             return await _handle_stream_tools(
                 engine, model, request_body, complexity_info
             )
-        return await _handle_stream(engine, model, request_body, complexity_info)
+        return await _handle_stream(
+            engine,
+            model,
+            request_body,
+            complexity_info,
+            trace_store=getattr(request.app.state, "trace_store", None),
+        )
 
     # Non-streaming: use agent if available, otherwise direct engine call.
     #
@@ -170,7 +176,14 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request
     # the agent to execute them), add an explicit opt-in header rather
     # than removing this guard — silent re-routing is what produced #414.
     if agent is not None and not request_body.tools:
-        return _handle_agent(agent, model, request_body, complexity_info)
+        return _handle_agent(
+            agent,
+            model,
+            request_body,
+            complexity_info,
+            trace_store=getattr(request.app.state, "trace_store", None),
+            bus=getattr(request.app.state, "bus", None),
+        )
 
     bus = getattr(request.app.state, "bus", None)
     return _handle_direct(
@@ -195,17 +208,50 @@ def _handle_direct(
     if req.tools:
         kwargs["tools"] = req.tools
     if bus:
+        from openjarvis.telemetry.instrumented_engine import InstrumentedEngine
         from openjarvis.telemetry.wrapper import instrumented_generate
 
-        result = instrumented_generate(
-            engine,
-            messages,
-            model=model,
-            bus=bus,
-            temperature=req.temperature,
-            max_tokens=req.max_tokens,
-            **kwargs,
-        )
+        # `app.state.engine` may already be an InstrumentedEngine (the
+        # common case when telemetry is wired in). If we then wrap it
+        # with `instrumented_generate`, BOTH layers fire a
+        # TELEMETRY_RECORD per call:
+        #
+        #   - InstrumentedEngine.generate() publishes a FULL record
+        #     (energy_joules, GPU stats, token_counting_version, ...).
+        #   - instrumented_generate() publishes a BARE record (timing +
+        #     tokens only; no energy meter, no version stamp).
+        #
+        # The doubled count was the dominant driver of the bimodal
+        # Wh/token distribution on the public leaderboard.
+        #
+        # The fix below is NOT "unwrap and call instrumented_generate":
+        # that would have replaced "doubled records" with "every
+        # request emits only a bare record with no energy / no version",
+        # which the leaderboard's `current_methodology_only=True` filter
+        # would then drop entirely. Instead, when the engine is already
+        # an InstrumentedEngine, skip the wrapper and call `generate`
+        # directly — InstrumentedEngine publishes the full per-record
+        # event itself with energy + version intact. Only fall back to
+        # the lightweight wrapper for engines that aren't already
+        # instrumented.
+        if isinstance(engine, InstrumentedEngine):
+            result = engine.generate(
+                messages,
+                model=model,
+                temperature=req.temperature,
+                max_tokens=req.max_tokens,
+                **kwargs,
+            )
+        else:
+            result = instrumented_generate(
+                engine,
+                messages,
+                model=model,
+                bus=bus,
+                temperature=req.temperature,
+                max_tokens=req.max_tokens,
+                **kwargs,
+            )
     else:
         result = engine.generate(
             messages,
@@ -255,8 +301,19 @@ def _handle_agent(
     model: str,
     req: ChatCompletionRequest,
     complexity_info=None,
+    *,
+    trace_store=None,
+    bus=None,
 ) -> ChatCompletionResponse:
-    """Run through agent."""
+    """Run through agent.
+
+    When *trace_store* is set, the agent run is wrapped in a
+    ``TraceCollector`` (mirroring ``system/orchestrator.py``) so every
+    completion records a ``Trace`` to ``traces.db``. Previously this endpoint
+    called ``agent.run()`` raw, so the server never produced traces:
+    ``traces.db`` stayed empty and spec_search's cold-start gate
+    (``check_readiness``, min 20 traces) could never open.
+    """
     from openjarvis.agents._stubs import AgentContext
 
     # Build context from prior messages
@@ -274,7 +331,13 @@ def _handle_agent(
     if model:
         agent._model = model
     try:
-        result = agent.run(input_text, context=ctx)
+        if trace_store is not None:
+            from openjarvis.traces.collector import TraceCollector
+
+            collector = TraceCollector(agent, store=trace_store, bus=bus)
+            result = collector.run(input_text, context=ctx)
+        else:
+            result = agent.run(input_text, context=ctx)
     finally:
         agent._model = original_model
 
@@ -426,8 +489,19 @@ async def _handle_stream(
     model: str,
     req: ChatCompletionRequest,
     complexity_info=None,
+    *,
+    trace_store=None,
 ):
-    """Stream response using SSE format."""
+    """Stream response using SSE format.
+
+    This path streams straight from the engine, bypassing the agent /
+    ``TraceCollector``. When *trace_store* is set we accumulate the streamed
+    tokens and record a minimal ``Trace`` once the stream completes
+    successfully — otherwise streamed chats (the desktop GUI's main path)
+    would never populate ``traces.db``.
+    """
+    import time
+
     from openjarvis.server.cloud_router import (
         is_cloud_model,
         stream_cloud,
@@ -437,11 +511,20 @@ async def _handle_stream(
     messages = _to_messages(req.messages)
     chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
 
+    # Last user message — recorded as the trace query.
+    query_text = ""
+    for _m in reversed(req.messages):
+        if _m.role == "user" and _m.content:
+            query_text = _m.content
+            break
+
     # Route directly to the right backend — bypasses engine routing entirely
     # so broken MultiEngine state can never misdirect requests.
     use_cloud = is_cloud_model(model)
 
     async def generate():
+        started_at = time.time()
+        full_content = ""
         # Send role chunk first
         first_chunk = ChatCompletionChunk(
             id=chunk_id,
@@ -494,6 +577,7 @@ async def _handle_stream(
                         max_tokens=req.max_tokens,
                     )
             async for token in token_iter:
+                full_content += token
                 chunk = ChatCompletionChunk(
                     id=chunk_id,
                     model=model,
@@ -530,6 +614,22 @@ async def _handle_stream(
             yield "data: [DONE]\n\n"
             return
 
+        # Record a trace for the completed stream (best-effort; never breaks
+        # the response). Mirrors the agent path so streamed chats also
+        # populate traces.db.
+        if trace_store is not None and full_content:
+            from openjarvis.traces.collector import record_response_trace
+
+            record_response_trace(
+                trace_store,
+                query=query_text,
+                result=full_content,
+                model=model,
+                engine="cloud" if use_cloud else "ollama",
+                started_at=started_at,
+                ended_at=time.time(),
+            )
+
         # Send finish chunk with usage data if available
         import json as _json
 
@@ -736,7 +836,11 @@ async def savings(request: Request):
 
     agg = TelemetryAggregator(db_path)
     try:
-        summary = agg.summary(since=session_start)
+        # current_methodology_only excludes pre-fix legacy rows from
+        # the leaderboard's per-token efficiency numerator/denominator
+        # — see the comment on _time_filter for the bimodal-Wh/token
+        # background.
+        summary = agg.summary(since=session_start, current_methodology_only=True)
         # Exclude cloud model tokens from savings — only local
         # inference counts toward cost savings.
         _cloud_prefixes = (

src/openjarvis/server/savings.py

@@ -10,11 +10,12 @@ import time
 from dataclasses import asdict, dataclass, field
 from typing import Any, Dict, List
 
-# Bump when token-counting methodology changes so that the leaderboard
-# can distinguish entries computed under different rules.
-#   v1 = original (Ollama prompt_eval_count, may under-count due to KV cache)
-#   v2 = full prompt token count, no KV-cache assumption, system prompt always counted
-TOKEN_COUNTING_VERSION: int = 2
+# Source of truth for the methodology-version constant lives in
+# `openjarvis.core.types` so the telemetry layer can read it without
+# crossing the server → telemetry layering. Re-exported here for
+# backward compatibility with existing imports of `from
+# openjarvis.server.savings import TOKEN_COUNTING_VERSION`.
+from openjarvis.core.types import TOKEN_COUNTING_VERSION  # noqa: E402,F401
 
 # ---------------------------------------------------------------------------
 # Cloud provider pricing (USD per 1M tokens)
@@ -104,11 +105,22 @@ def compute_savings(
       served from KV cache.  Used for **FLOPs** and **energy**
       calculations since these reflect actual compute.
 
-    If ``prompt_tokens_evaluated`` is 0 (e.g. old telemetry without the
-    column), it falls back to ``prompt_tokens``.
+    When ``prompt_tokens_evaluated`` is 0 we used to fall back to
+    ``prompt_tokens``. That's wrong in multi-turn: routes.py aggregates
+    by summing each turn's full prompt — which counts the system prompt
+    N times for an N-turn conversation — so the fallback turned the FLOPs
+    and energy estimates into N×-too-high numbers. That was the dominant
+    contributor to the bimodal Wh/token distribution on the leaderboard.
+
+    Conservative behaviour now: when the KV-cache-aware count is
+    missing, treat `prompt_tokens_evaluated` as 0 — so the FLOPs/energy
+    denominator becomes just `completion_tokens`. That under-estimates
+    rather than over-estimates compute, and (intentionally) cascades
+    into the leaderboard's `isMissingTelemetry` UI render so those
+    rows show `—` instead of a misleading zero.
     """
     if prompt_tokens_evaluated <= 0:
-        prompt_tokens_evaluated = prompt_tokens
+        prompt_tokens_evaluated = 0
     total_tokens = prompt_tokens + completion_tokens
     total_tokens_evaluated = prompt_tokens_evaluated + completion_tokens
     providers: List[ProviderSavings] = []

src/openjarvis/server/webhook_routes.py

@@ -113,7 +113,14 @@ def create_webhook_router(
         signature = request.headers.get("X-Twilio-Signature", "")
         url = str(request.url)
 
-        if twilio_auth_token and not _validate_twilio_signature(
+        # Fail closed: an unconfigured token means we cannot verify the sender,
+        # so reject rather than trust unsigned input.
+        if not twilio_auth_token:
+            logger.error(
+                "Twilio webhook rejected: TWILIO_AUTH_TOKEN not configured."
+            )
+            return Response("Webhook signature verification not configured", 403)
+        if not _validate_twilio_signature(
             twilio_auth_token, url, params, signature
         ):
             return Response("Invalid signature", status_code=403)
@@ -257,7 +264,13 @@ def create_webhook_router(
         request: Request,
     ) -> Response:
         auth = request.headers.get("Authorization", "")
-        if bluebubbles_password and auth != bluebubbles_password:
+        # Fail closed when no password is configured.
+        if not bluebubbles_password:
+            logger.error(
+                "BlueBubbles webhook rejected: password not configured."
+            )
+            return Response("Webhook authentication not configured", 403)
+        if not hmac.compare_digest(auth, bluebubbles_password):
             return Response("Invalid password", status_code=403)
 
         payload = await request.json()
@@ -292,7 +305,11 @@ def create_webhook_router(
         token = request.query_params.get("hub.verify_token", "")
         challenge = request.query_params.get("hub.challenge", "")
 
-        if mode == "subscribe" and token == whatsapp_verify_token:
+        # Fail closed: never echo the challenge if no verify token is set,
+        # otherwise an empty token would match an empty query value.
+        if not whatsapp_verify_token:
+            return Response("Forbidden", status_code=403)
+        if mode == "subscribe" and hmac.compare_digest(token, whatsapp_verify_token):
             return PlainTextResponse(challenge)
         return Response("Forbidden", status_code=403)
 
@@ -302,19 +319,23 @@ def create_webhook_router(
     ) -> Response:
         body_bytes = await request.body()
 
-        # Verify signature
-        if whatsapp_app_secret:
-            signature = request.headers.get("X-Hub-Signature-256", "")
-            expected = (
-                "sha256="
-                + hmac.new(
-                    whatsapp_app_secret.encode(),
-                    body_bytes,
-                    hashlib.sha256,
-                ).hexdigest()
+        # Fail closed: reject when no app secret is configured to verify HMAC.
+        if not whatsapp_app_secret:
+            logger.error(
+                "WhatsApp webhook rejected: app secret not configured."
             )
-            if not hmac.compare_digest(signature, expected):
-                return Response("Invalid signature", status_code=403)
+            return Response("Webhook signature verification not configured", 403)
+        signature = request.headers.get("X-Hub-Signature-256", "")
+        expected = (
+            "sha256="
+            + hmac.new(
+                whatsapp_app_secret.encode(),
+                body_bytes,
+                hashlib.sha256,
+            ).hexdigest()
+        )
+        if not hmac.compare_digest(signature, expected):
+            return Response("Invalid signature", status_code=403)
 
         payload = json.loads(body_bytes)
         for entry in payload.get("entry", []):
@@ -349,16 +370,16 @@ def create_webhook_router(
         # Get the SendBlue channel — may be passed at init or set later
         sb = sendblue_channel or getattr(request.app.state, "sendblue_channel", None)
 
-        # Verify webhook secret if configured
-        if sb and sb.webhook_secret:
-            header_secret = request.headers.get("x-sendblue-secret", "")
-            if header_secret != sb.webhook_secret:
-                return Response("Invalid secret", status_code=403)
-        elif sb:
-            logger.warning(
-                "SendBlue webhook received without secret verification. "
-                "Set webhook_secret for HMAC validation."
+        # Fail closed: require a configured channel + webhook secret to verify
+        # the sender before processing any inbound message.
+        if sb is None or not getattr(sb, "webhook_secret", ""):
+            logger.error(
+                "SendBlue webhook rejected: webhook_secret not configured."
             )
+            return Response("Webhook secret not configured", status_code=403)
+        header_secret = request.headers.get("x-sendblue-secret", "")
+        if not hmac.compare_digest(header_secret, sb.webhook_secret):
+            return Response("Invalid secret", status_code=403)
 
         # Ignore outbound status callbacks
         if payload.get("is_outbound", False):

src/openjarvis/system/builder.py

@@ -33,6 +33,8 @@ class SystemBuilder:
             self._config = load_config()
 
         self._engine_key: Optional[str] = None
+        self._engine_instance: Optional[InferenceEngine] = None
+        self._engine_instance_key: Optional[str] = None
         self._model: Optional[str] = None
         self._agent_name: Optional[str] = None
         self._tool_names: Optional[List[str]] = None
@@ -50,6 +52,20 @@ class SystemBuilder:
         self._engine_key = key
         return self
 
+    def engine_instance(
+        self, engine: InferenceEngine, key: str = "openai-compat"
+    ) -> SystemBuilder:
+        """Inject a pre-built engine instance, bypassing engine discovery.
+
+        Used by callers that must target one exact endpoint (e.g.
+        ``jarvis eval --base-url``). ``build()`` health-checks the instance
+        and raises a loud error if it is unreachable — it never silently
+        substitutes a different discovered engine.
+        """
+        self._engine_instance = engine
+        self._engine_instance_key = key
+        return self
+
     def model(self, name: str) -> SystemBuilder:
         self._model = name
         return self
@@ -303,6 +319,23 @@ class SystemBuilder:
         return system
 
     def _resolve_engine(self, config: JarvisConfig):
+        # An explicitly injected engine instance always wins and is never
+        # silently replaced: when the caller pinned an endpoint (e.g.
+        # ``jarvis eval --base-url``) and it is down, substituting whatever
+        # other engine discovery finds would silently run against the wrong
+        # model server. Fail loudly instead.
+        if self._engine_instance is not None:
+            engine = self._engine_instance
+            key = self._engine_instance_key or "openai-compat"
+            if not engine.health():
+                host = getattr(engine, "_host", "<unknown host>")
+                raise RuntimeError(
+                    f"Injected engine {key!r} is not reachable at {host} — "
+                    "is the endpoint running and serving GET /v1/models? "
+                    "Refusing to fall back to engine discovery."
+                )
+            return engine, key
+
         from openjarvis.engine._discovery import get_engine
 
         pref = config.intelligence.preferred_engine
@@ -313,7 +346,18 @@ class SystemBuilder:
                 "No inference engine available. "
                 "Make sure an engine is running (e.g. ollama serve)."
             )
-        return resolved[1], resolved[0]
+        resolved_key, engine = resolved
+        if self._engine_key and resolved_key != self._engine_key:
+            # get_engine() falls back to any healthy discovered engine; make
+            # the substitution visible when the caller asked for a specific
+            # engine (observed: requested vllm, silently got ollama@11434).
+            logger.warning(
+                "Requested engine %r is unavailable; using %r at %s instead",
+                self._engine_key,
+                resolved_key,
+                getattr(engine, "_host", "<unknown host>"),
+            )
+        return engine, resolved_key
 
     def _resolve_model(self, config: JarvisConfig, engine: InferenceEngine) -> str:
         if self._model:

src/openjarvis/system/core.py

@@ -268,10 +268,20 @@ class JarvisSystem:
 
             if reply:
                 try:
+                    # Canonical channel send contract (see BaseChannel.send):
+                    # the first positional arg is the DESTINATION id, and the
+                    # `conversation_id=` kwarg is the inbound message id used as
+                    # a reply/thread reference.  ``cm.conversation_id`` holds the
+                    # real per-adapter destination (Discord/Slack channel id,
+                    # Telegram chat id, ...) while ``cm.channel`` is only the
+                    # channel TYPE label ("discord", "telegram", ...).  Passing
+                    # the type label as the destination produced HTTP 400s
+                    # (#515) and using the channel id as a reply reference
+                    # produced MESSAGE_REFERENCE_UNKNOWN_MESSAGE (#516).
                     channel_bridge.send(
-                        cm.channel,
+                        cm.conversation_id,
                         reply,
-                        conversation_id=cm.conversation_id,
+                        conversation_id=getattr(cm, "message_id", ""),
                     )
                 except Exception:
                     logger.exception("Channel send error")

src/openjarvis/telemetry/aggregator.py

@@ -92,12 +92,22 @@ class TelemetryAggregator:
         self._conn = sqlite3.connect(self._db_path, check_same_thread=False)
         self._conn.row_factory = sqlite3.Row
 
-    @staticmethod
     def _time_filter(
+        self,
         since: Optional[float] = None,
         until: Optional[float] = None,
+        current_methodology_only: bool = False,
     ) -> tuple[str, list[Any]]:
-        """Build a WHERE clause fragment for time-range filtering."""
+        """Build a WHERE clause fragment for time + methodology filtering.
+
+        ``current_methodology_only`` is opt-in (default False) because the
+        local dashboard still wants to render historical aggregates that
+        predate the per-record version stamp. The leaderboard ingest path
+        in ``server/routes.savings`` flips it to True so legacy rows (the
+        ones with NULL token_counting_version) don't pollute the public
+        per-token efficiency metric — they were the dominant source of
+        the bimodal Wh/token distribution.
+        """
         clauses: list[str] = []
         params: list[Any] = []
         if since is not None:
@@ -106,6 +116,11 @@ class TelemetryAggregator:
         if until is not None:
             clauses.append("timestamp <= ?")
             params.append(until)
+        if current_methodology_only and self._safe_col("token_counting_version"):
+            from openjarvis.core.types import TOKEN_COUNTING_VERSION
+
+            clauses.append("token_counting_version = ?")
+            params.append(TOKEN_COUNTING_VERSION)
         if clauses:
             return " WHERE " + " AND ".join(clauses), params
         return "", params
@@ -124,8 +139,11 @@ class TelemetryAggregator:
         *,
         since: Optional[float] = None,
         until: Optional[float] = None,
+        current_methodology_only: bool = False,
     ) -> List[ModelStats]:
-        where, params = self._time_filter(since, until)
+        where, params = self._time_filter(
+            since, until, current_methodology_only=current_methodology_only
+        )
 
         # Build optional columns for new fields (graceful on old DBs)
         extra_cols = ""
@@ -215,8 +233,11 @@ class TelemetryAggregator:
         *,
         since: Optional[float] = None,
         until: Optional[float] = None,
+        current_methodology_only: bool = False,
     ) -> List[EngineStats]:
-        where, params = self._time_filter(since, until)
+        where, params = self._time_filter(
+            since, until, current_methodology_only=current_methodology_only
+        )
 
         extra_cols = ""
         has_tpj = self._safe_col("tokens_per_joule")
@@ -305,9 +326,14 @@ class TelemetryAggregator:
         *,
         since: Optional[float] = None,
         until: Optional[float] = None,
+        current_methodology_only: bool = False,
     ) -> AggregatedStats:
-        model_stats = self.per_model_stats(since=since, until=until)
-        engine_stats = self.per_engine_stats(since=since, until=until)
+        model_stats = self.per_model_stats(
+            since=since, until=until, current_methodology_only=current_methodology_only
+        )
+        engine_stats = self.per_engine_stats(
+            since=since, until=until, current_methodology_only=current_methodology_only
+        )
         total_calls = sum(m.call_count for m in model_stats)
 
         def _weighted_avg(attr: str) -> float:

src/openjarvis/telemetry/instrumented_engine.py

@@ -8,7 +8,7 @@ from collections.abc import AsyncIterator
 from typing import Any, Dict, List, Optional, Sequence
 
 from openjarvis.core.events import EventBus, EventType
-from openjarvis.core.types import Message, TelemetryRecord
+from openjarvis.core.types import TOKEN_COUNTING_VERSION, Message, TelemetryRecord
 from openjarvis.engine._stubs import InferenceEngine, StreamChunk
 from openjarvis.telemetry.gpu_monitor import GpuSample
 
@@ -233,6 +233,11 @@ class InstrumentedEngine(InferenceEngine):
             gpu_energy_joules=gpu_energy_joules,
             dram_energy_joules=dram_energy_joules,
             tokens_per_joule=tokens_per_joule,
+            # Stamp every new record with the current methodology version
+            # so the leaderboard aggregator can drop legacy rows (those
+            # left NULL by pre-fix builds) cleanly. Source of truth for
+            # the integer is `server/savings.py`.
+            token_counting_version=TOKEN_COUNTING_VERSION,
         )
 
         event_data = {
@@ -454,6 +459,8 @@ class InstrumentedEngine(InferenceEngine):
             gpu_energy_joules=gpu_energy_joules,
             dram_energy_joules=dram_energy_joules,
             tokens_per_joule=tokens_per_joule,
+            # Stamp the methodology version on streaming records too.
+            token_counting_version=TOKEN_COUNTING_VERSION,
         )
 
         event_data = {

src/openjarvis/telemetry/store.py

@@ -55,6 +55,11 @@ CREATE TABLE IF NOT EXISTS telemetry (
     p99_itl_ms           REAL NOT NULL DEFAULT 0.0,
     std_itl_ms           REAL NOT NULL DEFAULT 0.0,
     is_streaming         INTEGER NOT NULL DEFAULT 0,
+    -- token_counting_version: nullable on purpose. Records inserted by
+    -- pre-fix builds left this NULL; the leaderboard aggregator treats
+    -- NULL as legacy and excludes those rows from per-token efficiency
+    -- sums (so the bimodal-Wh/token leaderboard population disappears).
+    token_counting_version INTEGER,
     mining_session_id    TEXT,
     metadata        TEXT    NOT NULL DEFAULT '{}'
 );
@@ -91,13 +96,14 @@ INSERT INTO telemetry (
     prefill_energy_joules, decode_energy_joules,
     mean_itl_ms, median_itl_ms, p90_itl_ms, p95_itl_ms, p99_itl_ms, std_itl_ms,
     is_streaming,
+    token_counting_version,
     mining_session_id,
     metadata
 ) VALUES (
     ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
     ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
     ?, ?, ?, ?, ?, ?, ?, ?, ?, ?,
-    ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
+    ?, ?, ?, ?, ?, ?, ?, ?, ?, ?, ?
 )
 """
 
@@ -128,6 +134,10 @@ _MIGRATE_COLUMNS = [
     ("std_itl_ms", "REAL NOT NULL DEFAULT 0.0"),
     ("is_streaming", "INTEGER NOT NULL DEFAULT 0"),
     ("prompt_tokens_evaluated", "INTEGER NOT NULL DEFAULT 0"),
+    # `token_counting_version` is nullable on purpose — rows that existed
+    # before this migration ran predate per-record versioning and the
+    # aggregator filter treats them as legacy.
+    ("token_counting_version", "INTEGER"),
     ("mining_session_id", "TEXT"),
 ]
 
@@ -197,6 +207,7 @@ class TelemetryStore:
                 rec.p99_itl_ms,
                 rec.std_itl_ms,
                 1 if rec.is_streaming else 0,
+                rec.token_counting_version,
                 rec.mining_session_id,
                 json.dumps(rec.metadata),
             ),

src/openjarvis/tools/http_request.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import logging
 import os
 import time
+import urllib.parse
 from typing import Any
 
 import httpx
@@ -21,6 +22,13 @@ _MAX_RESPONSE_BYTES = 1_048_576
 
 _ALLOWED_METHODS = frozenset({"GET", "POST", "PUT", "DELETE", "PATCH", "HEAD"})
 
+# Cap redirect chains so a malicious server cannot loop us indefinitely.
+_MAX_REDIRECTS = 5
+
+
+class _SSRFRedirectError(Exception):
+    """Raised when a redirect target fails the SSRF check."""
+
 
 @ToolRegistry.register("http_request")
 class HttpRequestTool(BaseTool):
@@ -136,13 +144,11 @@ class HttpRequestTool(BaseTool):
 
         try:
             t0 = time.time()
-            response = httpx.request(
-                method,
-                url,
-                headers=headers,
-                content=body,
-                timeout=float(timeout),
-                follow_redirects=True,
+            # Follow redirects manually so each hop is re-checked for SSRF — an
+            # allowed public URL must not be able to 30x-redirect us to an
+            # internal/metadata address.
+            response = self._request_following_redirects(
+                method, url, headers=headers, content=body, timeout=float(timeout)
             )
             elapsed_ms = (time.time() - t0) * 1000
 
@@ -178,6 +184,12 @@ class HttpRequestTool(BaseTool):
                 content=f"Request timed out after {timeout}s: {exc}",
                 success=False,
             )
+        except _SSRFRedirectError as exc:
+            return ToolResult(
+                tool_name="http_request",
+                content=f"SSRF protection blocked redirect: {exc}",
+                success=False,
+            )
         except httpx.RequestError as exc:
             return ToolResult(
                 tool_name="http_request",
@@ -191,5 +203,52 @@ class HttpRequestTool(BaseTool):
                 success=False,
             )
 
+    @staticmethod
+    def _request_following_redirects(
+        method: str,
+        url: str,
+        *,
+        headers: dict,
+        content: Any,
+        timeout: float,
+    ) -> httpx.Response:
+        """Issue the request, re-checking SSRF on every redirect hop.
+
+        httpx's built-in ``follow_redirects`` would chase a 30x ``Location``
+        without re-validating it, letting a public URL bounce us to an internal
+        host. We follow manually and run :func:`check_ssrf` on each target.
+        """
+        current_url = url
+        current_method = method
+        body = content
+        # Use module-level ``httpx.request`` (not a private Client) so the SSRF
+        # re-check seam stays patchable by callers' tests, with redirects
+        # disabled so we control every hop ourselves.
+        for _ in range(_MAX_REDIRECTS + 1):
+            response = httpx.request(
+                current_method,
+                current_url,
+                headers=headers,
+                content=body,
+                timeout=timeout,
+                follow_redirects=False,
+            )
+            if response.status_code not in (301, 302, 303, 307, 308):
+                return response
+            location = response.headers.get("location", "")
+            if not location:
+                return response
+            # Resolve relative redirects against the URL we just fetched.
+            current_url = urllib.parse.urljoin(str(response.url), location)
+            ssrf_error = check_ssrf(current_url)
+            if ssrf_error:
+                raise _SSRFRedirectError(ssrf_error)
+            # Per RFC 7231, 301/302/303 turn the method into GET and drop
+            # the body (except for HEAD).
+            if response.status_code in (301, 302, 303) and current_method != "HEAD":
+                current_method = "GET"
+                body = None
+        raise _SSRFRedirectError(f"Exceeded maximum of {_MAX_REDIRECTS} redirects.")
+
 
 __all__ = ["HttpRequestTool"]

src/openjarvis/tools/storage/_stubs.py

@@ -10,6 +10,32 @@ from abc import ABC, abstractmethod
 from dataclasses import dataclass, field
 from typing import Any, Dict, List, Optional
 
+#: Actionable message surfaced whenever a Rust-backed memory backend cannot be
+#: constructed because the mandatory ``openjarvis_rust`` extension is missing
+#: from the *current* venv. Kept as a single constant so the server routes, the
+#: SDK and the regression tests all surface exactly the same wording.
+RUST_MISSING_HINT = (
+    "Memory backend unavailable: the native `openjarvis_rust` extension is not "
+    "installed in this environment. Build it into the venv that runs the server "
+    "with `uv run maturin develop -m rust/crates/openjarvis-python/Cargo.toml` "
+    "(needs rustc >= 1.88), then restart. Verify with "
+    '`python -c "from openjarvis._rust_bridge import RUST_AVAILABLE; '
+    'print(RUST_AVAILABLE)"`.'
+)
+
+
+class MemoryBackendUnavailable(RuntimeError):
+    """Raised when a memory backend cannot be built because the mandatory
+    ``openjarvis_rust`` extension is missing from the current environment.
+
+    This is deliberately distinct from "memory is intentionally disabled": a
+    missing native extension is an environment/install error that must be
+    surfaced loudly and actionably, never swallowed into a silent no-op.
+    """
+
+    def __init__(self, message: str = RUST_MISSING_HINT) -> None:
+        super().__init__(message)
+
 
 @dataclass(slots=True)
 class RetrievalResult:
@@ -59,4 +85,9 @@ class MemoryBackend(ABC):
         """Remove all stored documents."""
 
 
-__all__ = ["MemoryBackend", "RetrievalResult"]
+__all__ = [
+    "RUST_MISSING_HINT",
+    "MemoryBackend",
+    "MemoryBackendUnavailable",
+    "RetrievalResult",
+]

src/openjarvis/tools/storage/chunking.py

@@ -135,10 +135,17 @@ def chunk_text(
         current_tokens.extend(para_tokens)
         current_offset += len(para_tokens)
 
-    # Flush remaining tokens
+    # Flush remaining tokens.
+    #
+    # ``min_chunk_size`` exists to discard tiny *trailing* fragments once a
+    # document has already produced at least one chunk. It must NOT silently
+    # drop an entire short document: indexing a folder of short notes would
+    # otherwise report success while storing nothing (#502 follow-up). So if no
+    # chunk has been emitted yet, keep the remaining content regardless of the
+    # floor.
     if current_tokens:
         chunk_content = " ".join(current_tokens)
-        if _count_tokens(chunk_content) >= cfg.min_chunk_size:
+        if not chunks or _count_tokens(chunk_content) >= cfg.min_chunk_size:
             chunks.append(
                 Chunk(
                     content=chunk_content,

src/openjarvis/tools/storage/sqlite.py

@@ -9,7 +9,11 @@ from typing import Any, Dict, List, Optional
 
 from openjarvis.core.events import EventType, get_event_bus
 from openjarvis.core.registry import MemoryRegistry
-from openjarvis.tools.storage._stubs import MemoryBackend, RetrievalResult
+from openjarvis.tools.storage._stubs import (
+    MemoryBackend,
+    MemoryBackendUnavailable,
+    RetrievalResult,
+)
 
 
 def _check_fts5(conn: sqlite3.Connection) -> bool:
@@ -40,7 +44,15 @@ class SQLiteMemory(MemoryBackend):
 
         from openjarvis._rust_bridge import get_rust_module
 
-        _rust = get_rust_module()
+        # The Rust backend is mandatory and there is no Python fallback. When
+        # the extension is missing from *this* venv, ``get_rust_module`` raises
+        # ImportError; translate it into a clear, actionable error so callers
+        # never degrade to a misleading "Failed to index path" or a silent
+        # no-op (see #502).
+        try:
+            _rust = get_rust_module()
+        except ImportError as exc:
+            raise MemoryBackendUnavailable() from exc
         self._rust_impl = _rust.SQLiteMemory(self._db_path)
         self._conn = None  # type: ignore[assignment]
 

src/openjarvis/traces/collector.py

@@ -221,4 +221,59 @@ class TraceCollector:
         )
 
 
-__all__ = ["TraceCollector"]
+def record_response_trace(
+    store: Optional[TraceStore],
+    *,
+    query: str,
+    result: str,
+    model: str = "",
+    engine: str = "",
+    agent: str = "server",
+    started_at: float,
+    ended_at: float,
+) -> Optional[Trace]:
+    """Persist a minimal single-step ``Trace`` for a non-agent response.
+
+    The streaming SSE and WebSocket chat paths stream straight from the
+    engine, bypassing the agent (and therefore ``TraceCollector``). They call
+    this so those interactions still land in ``traces.db`` — otherwise streamed
+    chats, which are the desktop GUI's main path, would never produce traces.
+
+    Best-effort: returns the saved ``Trace`` or ``None`` (when *store* is
+    ``None`` or persistence raised), and never propagates an exception into the
+    caller's response path.
+    """
+    if store is None:
+        return None
+    try:
+        duration = max(0.0, ended_at - started_at)
+        trace = Trace(
+            query=query,
+            agent=agent,
+            model=model,
+            engine=engine,
+            result=result,
+            started_at=started_at,
+            ended_at=ended_at,
+            steps=[
+                TraceStep(
+                    step_type=StepType.RESPOND,
+                    timestamp=ended_at,
+                    duration_seconds=duration,
+                    output={"content": result},
+                )
+            ],
+        )
+        trace.total_latency_seconds = duration
+        store.save(trace)
+        return trace
+    except Exception:
+        import logging
+
+        logging.getLogger("openjarvis.traces").debug(
+            "record_response_trace failed", exc_info=True
+        )
+        return None
+
+
+__all__ = ["TraceCollector", "record_response_trace"]

src/openjarvis/traces/store.py

@@ -106,7 +106,15 @@ class TraceStore:
         self._conn.commit()
 
     def save(self, trace: Trace) -> None:
-        """Persist a complete trace with all its steps."""
+        """Persist a complete trace with all its steps.
+
+        ``trace_id`` is a primary key: saving a second, different trace under
+        an existing id raises ``sqlite3.IntegrityError`` (the external-corpus
+        adapter relies on this to surface duplicate record ids). The server
+        avoids re-saving the same trace by keeping the ``TraceCollector`` the
+        single writer — see ``server/app.py`` — rather than swallowing
+        collisions here.
+        """
         self._conn.execute(
             _INSERT_TRACE,
             (

tests/channels/test_discord_channel.py

@@ -130,3 +130,63 @@ class TestStatus:
         ch = DiscordChannel()
         ch.connect()
         assert ch.status() == ChannelStatus.ERROR
+
+
+class TestWireChannelEndToEnd:
+    """Regression for #515/#516 — the full inbound→reply path through
+    JarvisSystem.wire_channel must call the real Discord REST API with the
+    numeric channel id (not "discord") and a message_reference equal to the
+    inbound message id (not the channel id).
+    """
+
+    def test_reply_hits_real_channel_id_and_message_reference(self, tmp_path):
+        from openjarvis.channels._stubs import ChannelMessage
+        from openjarvis.core.config import JarvisConfig
+        from openjarvis.core.events import EventBus
+        from openjarvis.system import JarvisSystem
+
+        config = JarvisConfig()
+        config.sessions.db_path = str(tmp_path / "sessions.db")
+        from unittest.mock import MagicMock as _MM
+
+        system = JarvisSystem(
+            config=config,
+            bus=EventBus(record_history=False),
+            engine=_MM(),
+            engine_key="mock",
+            model="test-model",
+            agent_name="",
+        )
+        system.ask = _MM(return_value={"content": "pong"})
+
+        channel = DiscordChannel(bot_token="my-bot-token")
+        system.wire_channel(channel)
+
+        # Exactly the ChannelMessage shape DiscordChannel._gateway_loop emits:
+        # channel = "discord" (TYPE label), conversation_id = numeric channel
+        # id, message_id = numeric message id.
+        cm = ChannelMessage(
+            channel="discord",
+            sender="user-1",
+            content="hello",
+            message_id="111122223333444455",
+            conversation_id="987654321098765432",
+        )
+
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+        with patch("httpx.post", return_value=mock_response) as mock_post:
+            # Invoke the handler wire_channel registered on the channel.
+            for handler in channel._handlers:
+                handler(cm)
+
+        mock_post.assert_called_once()
+        url = mock_post.call_args[0][0]
+        # #515: destination is the numeric channel id, not the "discord" label.
+        assert "discord.com/api/v10/channels/987654321098765432/messages" in url
+        assert "channels/discord/messages" not in url
+        payload = mock_post.call_args[1]["json"]
+        assert payload["content"] == "pong"
+        # #516: message_reference is the inbound message id, NOT the channel id.
+        assert payload["message_reference"] == {"message_id": "111122223333444455"}
+        assert payload["message_reference"]["message_id"] != "987654321098765432"

tests/channels/test_telegram.py

@@ -105,6 +105,41 @@ class TestSend:
         event_types = [e.event_type for e in bus.history]
         assert EventType.CHANNEL_MESSAGE_SENT in event_types
 
+    def test_send_uses_channel_as_chat_id_under_unified_contract(self):
+        """Canonical contract (#515/#516): the first positional ``channel``
+        arg is the chat destination, and ``conversation_id`` is the inbound
+        message id used as ``reply_to_message_id`` — not the chat id."""
+        ch = TelegramChannel(bot_token="123:ABC")
+
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+
+        with patch("httpx.post", return_value=mock_response) as mock_post:
+            result = ch.send("12345678", "Reply!", conversation_id="55")
+            assert result is True
+            payload = mock_post.call_args[1]["json"]
+            # Destination is the chat id from the positional channel arg.
+            assert payload["chat_id"] == "12345678"
+            # conversation_id becomes the reply reference, not the chat id.
+            assert payload["reply_to_message_id"] == "55"
+
+    def test_send_legacy_conversation_id_only_still_targets_chat(self):
+        """Backwards compatibility: a legacy caller passing the chat id via
+        ``conversation_id`` (with an empty ``channel``) still delivers."""
+        ch = TelegramChannel(bot_token="123:ABC")
+
+        mock_response = MagicMock()
+        mock_response.status_code = 200
+
+        with patch("httpx.post", return_value=mock_response) as mock_post:
+            result = ch.send("", "Hello!", conversation_id="12345678")
+            assert result is True
+            payload = mock_post.call_args[1]["json"]
+            assert payload["chat_id"] == "12345678"
+            # When channel is empty, conversation_id is the chat id, so it must
+            # not also be used as a self-referential reply id.
+            assert "reply_to_message_id" not in payload
+
 
 class TestStatus:
     def test_no_token_connect_error(self):

tests/cli/test_serve_channel_wiring.py

@@ -79,10 +79,14 @@ class TestWireChannelWithAgent:
 
         system.ask.assert_called_once()
         assert system.ask.call_args[0][0] == "ping"
+        # Canonical send contract (#515/#516): the destination is the real
+        # per-adapter id (carried in ChannelMessage.conversation_id), not the
+        # channel TYPE label, and the conversation_id kwarg is the inbound
+        # message id used as a reply reference, not the channel id.
         mock_channel.send.assert_called_once_with(
-            "telegram",
+            "42",
             "pong",
-            conversation_id="42",
+            conversation_id="1",
         )
 
     def test_session_store_created_lazily(self, tmp_path):
@@ -126,13 +130,101 @@ class TestWireChannelWithEngine:
         handler = mock_channel.on_message.call_args[0][0]
         handler(_make_channel_message(content="hi"))
 
+        # Canonical send contract (#515/#516): destination = real channel id
+        # (from ChannelMessage.conversation_id), reply ref = inbound message id.
         mock_channel.send.assert_called_once_with(
-            "telegram",
+            "42",
             "raw reply",
-            conversation_id="42",
+            conversation_id="1",
         )
 
 
+class TestWireChannelCanonicalContract:
+    """Regression for #515/#516 — wire_channel must dispatch the canonical
+    send contract so each adapter receives the right destination/reply ids,
+    regardless of the channel TYPE label.
+    """
+
+    def test_discord_uses_real_channel_id_not_type_label(self, tmp_path):
+        """A Discord ChannelMessage (channel="discord" TYPE label,
+        conversation_id=<numeric channel id>, message_id=<numeric msg id>)
+        must reply to the numeric channel id, with the message id as the
+        reply reference — never "discord" as destination (#515) and never the
+        channel id as the message reference (#516).
+        """
+        system = _make_system(tmp_path=tmp_path)
+        system.ask = MagicMock(return_value={"content": "pong"})
+
+        mock_channel = MagicMock()
+        system.wire_channel(mock_channel)
+        handler = mock_channel.on_message.call_args[0][0]
+
+        cm = ChannelMessage(
+            channel="discord",
+            sender="user-1",
+            content="hello",
+            message_id="111122223333444455",
+            conversation_id="987654321098765432",
+        )
+        handler(cm)
+
+        args, kwargs = mock_channel.send.call_args
+        # Destination is the real Discord channel id, NOT the type label.
+        assert args[0] == "987654321098765432"
+        assert args[0] != "discord"
+        # Reply reference is the inbound message id, NOT the channel id.
+        assert kwargs["conversation_id"] == "111122223333444455"
+        assert kwargs["conversation_id"] != "987654321098765432"
+
+    def test_telegram_uses_chat_id_and_message_id(self, tmp_path):
+        """Telegram must keep working under the unified contract: destination
+        is the chat id (conversation_id), reply ref is the message id."""
+        system = _make_system(tmp_path=tmp_path)
+        system.ask = MagicMock(return_value={"content": "pong"})
+
+        mock_channel = MagicMock()
+        system.wire_channel(mock_channel)
+        handler = mock_channel.on_message.call_args[0][0]
+
+        cm = ChannelMessage(
+            channel="telegram",
+            sender="42",
+            content="ping",
+            message_id="55",
+            conversation_id="12345678",
+        )
+        handler(cm)
+
+        mock_channel.send.assert_called_once_with(
+            "12345678",
+            "pong",
+            conversation_id="55",
+        )
+
+    def test_session_key_still_uses_conversation_id(self, tmp_path):
+        """The fix must not change session isolation keying, which still uses
+        ``<channel>:<conversation_id>``."""
+        system = _make_system(tmp_path=tmp_path)
+        system.ask = MagicMock(return_value={"content": "ok"})
+
+        mock_channel = MagicMock()
+        system.wire_channel(mock_channel)
+        handler = mock_channel.on_message.call_args[0][0]
+
+        cm = ChannelMessage(
+            channel="discord",
+            sender="u1",
+            content="hi",
+            message_id="msg-9",
+            conversation_id="chan-7",
+        )
+        handler(cm)
+
+        # Session was created under the channel:conversation_id key.
+        session = system.session_store.get_or_create("discord:chan-7")
+        assert any(m.content == "hi" for m in session.messages)
+
+
 class TestWireChannelSessionIsolation:
     """Separate conversation_ids get independent sessions."""
 

tests/cli/test_serve_model_resolution.py

@@ -0,0 +1,67 @@
+"""Tests for API server model selection."""
+
+from __future__ import annotations
+
+from openjarvis.cli.serve import _resolve_server_model
+from openjarvis.core.config import JarvisConfig
+
+
+class _FakeEngine:
+    def __init__(self, models: list[str]) -> None:
+        self._models = models
+
+    def list_models(self) -> list[str]:
+        return self._models
+
+
+def test_server_model_falls_back_to_reachable_ollama_model() -> None:
+    cfg = JarvisConfig()
+    cfg.server.model = "mlx-community/Qwen2.5-7B-Instruct-4bit"
+    cfg.intelligence.default_model = "mlx-community/Qwen2.5-7B-Instruct-4bit"
+    cfg.intelligence.fallback_model = "qwen3.5:9b"
+
+    model = _resolve_server_model(
+        None,
+        config=cfg,
+        engine_name="multi",
+        engine=_FakeEngine(["qwen3.5:9b"]),
+        all_models={"multi": ["qwen3.5:9b"]},
+    )
+
+    assert model == "qwen3.5:9b"
+
+
+def test_server_model_prefers_reachable_configured_model() -> None:
+    cfg = JarvisConfig()
+    cfg.server.model = "mlx-community/Qwen2.5-7B-Instruct-4bit"
+    cfg.intelligence.default_model = "qwen3.5:9b"
+    cfg.intelligence.fallback_model = "qwen3.5:9b"
+
+    model = _resolve_server_model(
+        None,
+        config=cfg,
+        engine_name="multi",
+        engine=_FakeEngine(
+            ["mlx-community/Qwen2.5-7B-Instruct-4bit", "qwen3.5:9b"]
+        ),
+        all_models={"multi": ["mlx-community/Qwen2.5-7B-Instruct-4bit"]},
+    )
+
+    assert model == "mlx-community/Qwen2.5-7B-Instruct-4bit"
+
+
+def test_server_model_keeps_explicit_cli_model() -> None:
+    cfg = JarvisConfig()
+    cfg.server.model = "configured-model"
+    cfg.intelligence.fallback_model = "fallback-model"
+
+    model = _resolve_server_model(
+        "explicit-model",
+        config=cfg,
+        engine_name="multi",
+        engine=_FakeEngine(["fallback-model"]),
+        all_models={"multi": ["fallback-model"]},
+    )
+
+    assert model == "explicit-model"
+

tests/cli/test_serve_single_build.py

@@ -0,0 +1,210 @@
+"""Regression tests for #263 — ``jarvis serve`` must build the system once.
+
+serve.py used to construct all heavy components inline and then call
+``SystemBuilder(config).build()`` a second time inside the scheduler block,
+re-discovering the engine, re-instrumenting it, re-resolving tools, re-opening
+the channel and re-creating the agent manager — ~30-40s of redundant work.
+
+These tests pin the fix:
+
+1. ``SystemBuilder.build`` is never called during ``jarvis serve`` startup
+   (the duplicate build is gone).
+2. The ``AgentExecutor`` still receives a system exposing the attributes it
+   actually reads: ``tool_executor``, ``session_store``, ``memory_backend``,
+   plus ``engine`` / ``model`` / ``config``.
+"""
+
+from __future__ import annotations
+
+import importlib
+from unittest.mock import MagicMock, patch
+
+import pytest
+from click.testing import CliRunner
+
+from openjarvis.cli import cli
+
+pytest.importorskip("fastapi")
+pytest.importorskip("uvicorn")
+
+# ``openjarvis.cli.serve`` as an attribute resolves to the click *command*
+# (re-exported on the package); grab the real module to monkeypatch its globals.
+serve_mod = importlib.import_module("openjarvis.cli.serve")
+
+
+def _fake_engine() -> MagicMock:
+    engine = MagicMock()
+    engine.list_models.return_value = ["test-model"]
+    engine.health.return_value = True
+    engine.name = "mock"
+    return engine
+
+
+def _repopulate_registries() -> None:
+    """Re-run the @register decorators wiped by the autouse conftest fixture.
+
+    The tool/memory modules are import-cached, so a plain ``import`` inside
+    serve.py is a no-op after the registries are cleared per-test. Reload the
+    individual submodules so ToolRegistry/MemoryRegistry are populated exactly
+    as they would be on a fresh process — otherwise serve would resolve an
+    empty tool list and no memory backend, masking the very wiring under test.
+    """
+    import importlib
+    import sys
+
+    import openjarvis.agents  # noqa: F401
+    import openjarvis.tools  # noqa: F401
+    import openjarvis.tools.storage  # noqa: F401
+    from openjarvis.core.registry import (
+        AgentRegistry,
+        MemoryRegistry,
+        ToolRegistry,
+    )
+
+    if not AgentRegistry.keys():
+        for mod_name in list(sys.modules):
+            if mod_name.startswith("openjarvis.agents.") and not mod_name.endswith(
+                "_stubs"
+            ):
+                try:
+                    importlib.reload(sys.modules[mod_name])
+                except Exception:
+                    pass
+
+    if not ToolRegistry.keys():
+        for mod_name in list(sys.modules):
+            if (
+                mod_name.startswith("openjarvis.tools.")
+                and not mod_name.endswith("_stubs")
+                and not mod_name.endswith("agent_tools")
+            ):
+                try:
+                    importlib.reload(sys.modules[mod_name])
+                except Exception:
+                    pass
+
+    if not MemoryRegistry.keys():
+        for mod_name in list(sys.modules):
+            if mod_name.startswith(
+                "openjarvis.tools.storage."
+            ) and not mod_name.endswith("_stubs"):
+                try:
+                    importlib.reload(sys.modules[mod_name])
+                except Exception:
+                    pass
+
+
+def _run_serve(tmp_path, monkeypatch, *, build_spy, set_system_spy):
+    """Invoke ``jarvis serve`` with all heavy/blocking pieces stubbed out.
+
+    Returns the CliRunner result. The server is never actually started
+    (``uvicorn.run`` is a no-op) and no real engine is contacted.
+    """
+    from openjarvis.core.config import JarvisConfig
+
+    _repopulate_registries()
+
+    config = JarvisConfig()
+    # Keep the scheduler block alive (it owns the executor wiring under test)
+    # while pointing every store at the temp dir.
+    config.agent_manager.enabled = True
+    config.agent_manager.db_path = str(tmp_path / "agents.db")
+    config.sessions.enabled = True
+    config.sessions.db_path = str(tmp_path / "sessions.db")
+    config.memory.db_path = str(tmp_path / "memory.db")
+    config.telemetry.enabled = False
+    config.traces.enabled = False
+    config.channel.enabled = False
+    config.skills.enabled = False
+    config.server.host = "127.0.0.1"
+    config.server.port = 8123
+    # Resolve a model without contacting a real engine / discovery.
+    config.intelligence.default_model = "test-model"
+
+    engine = _fake_engine()
+
+    monkeypatch.setattr(serve_mod, "load_config", lambda *a, **k: config)
+    monkeypatch.setattr(serve_mod, "get_engine", lambda *a, **k: ("mock", engine))
+    monkeypatch.setattr(serve_mod, "discover_engines", lambda *a, **k: {})
+    monkeypatch.setattr(serve_mod, "discover_models", lambda *a, **k: {})
+
+    # setup_security returns its own context; pass the engine straight through
+    # so we don't need real guardrails wired up.
+    sec = MagicMock()
+    sec.engine = engine
+    sec.capability_policy = None
+    sec.audit_logger = None
+    monkeypatch.setattr("openjarvis.security.setup_security", lambda *a, **k: sec)
+
+    with (
+        patch(
+            "openjarvis.system.builder.SystemBuilder.build",
+            build_spy,
+        ),
+        patch(
+            "openjarvis.agents.executor.AgentExecutor.set_system",
+            set_system_spy,
+        ),
+        patch("uvicorn.run", lambda *a, **k: None),
+    ):
+        return CliRunner().invoke(cli, ["serve"], catch_exceptions=False)
+
+
+def test_serve_does_not_call_systembuilder_build(tmp_path, monkeypatch):
+    """The redundant second full build is gone (#263)."""
+    build_spy = MagicMock(
+        side_effect=AssertionError(
+            "SystemBuilder.build() must not run during `jarvis serve` startup "
+            "— it is the duplicate build #263 removed."
+        )
+    )
+    set_system_spy = MagicMock()
+
+    result = _run_serve(
+        tmp_path,
+        monkeypatch,
+        build_spy=build_spy,
+        set_system_spy=set_system_spy,
+    )
+
+    assert result.exit_code == 0, result.output
+    build_spy.assert_not_called()
+
+
+def test_executor_receives_required_system_attrs(tmp_path, monkeypatch):
+    """The executor still gets a system exposing the attributes it reads.
+
+    AgentExecutor reads engine/model/config/memory_backend/tool_executor/
+    session_store off ``self._system``; the de-dup must not strip any of them.
+    """
+    build_spy = MagicMock()
+    captured: dict = {}
+
+    def _capture_set_system(self, system):  # noqa: ANN001
+        captured["system"] = system
+        # Preserve real behaviour so the executor is usable afterwards.
+        self._system = system
+
+    result = _run_serve(
+        tmp_path,
+        monkeypatch,
+        build_spy=build_spy,
+        set_system_spy=_capture_set_system,
+    )
+
+    assert result.exit_code == 0, result.output
+    # Built once, from the inline components — not via SystemBuilder.build().
+    build_spy.assert_not_called()
+
+    system = captured.get("system")
+    assert system is not None, "executor.set_system was never called"
+
+    # Correctness constraint from the verifier: these must survive the de-dup.
+    assert system.tool_executor is not None
+    assert system.session_store is not None
+    assert system.memory_backend is not None
+
+    # And the basics the executor resolves engine/model from.
+    assert system.engine is not None
+    assert system.model == "test-model"
+    assert system.config is not None

tests/engine/test_cloud.py

@@ -381,3 +381,95 @@ class TestCodexGenerate:
         engine._codex_client = {"token": "t", "url": "http://test"}
         engine.close()
         assert engine._codex_client is None
+
+
+class TestOpenRouterToolForwarding:
+    """Regression for #511: the OpenRouter engine must forward tools/tool_choice
+    to the (OpenAI-compatible) API and parse tool_calls back out of the response.
+    Pre-fix, both were dropped, silently breaking function-calling via OpenRouter.
+    """
+
+    def test_generate_forwards_tools_and_parses_tool_calls(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+
+        fake_tc = SimpleNamespace(
+            id="call_1",
+            type="function",
+            function=SimpleNamespace(name="get_weather", arguments='{"city": "NYC"}'),
+        )
+        fake_choice = SimpleNamespace(
+            message=SimpleNamespace(content=None, tool_calls=[fake_tc]),
+            finish_reason="tool_calls",
+        )
+        fake_resp = SimpleNamespace(
+            choices=[fake_choice],
+            usage=SimpleNamespace(prompt_tokens=3, completion_tokens=2, total_tokens=5),
+            model="openai/gpt-4o",
+        )
+        fake_client = mock.MagicMock()
+        fake_client.chat.completions.create.return_value = fake_resp
+
+        EngineRegistry.register_value("cloud", CloudEngine)
+        engine = CloudEngine()
+        engine._openrouter_client = fake_client
+
+        tools = [
+            {
+                "type": "function",
+                "function": {"name": "get_weather", "parameters": {}},
+            }
+        ]
+        result = engine.generate(
+            [Message(role=Role.USER, content="weather in NYC?")],
+            model="openrouter/openai/gpt-4o",
+            tools=tools,
+            tool_choice="auto",
+        )
+
+        # tools / tool_choice are forwarded to the API call
+        sent = fake_client.chat.completions.create.call_args.kwargs
+        assert sent["tools"] == tools
+        assert sent["tool_choice"] == "auto"
+
+        # tool_calls from the response are parsed back into the result
+        assert result["tool_calls"][0]["id"] == "call_1"
+        assert result["tool_calls"][0]["function"]["name"] == "get_weather"
+        assert result["tool_calls"][0]["function"]["arguments"] == '{"city": "NYC"}'
+
+
+class TestCloudEngineCanServe:
+    """#532: can_serve gates on the per-provider client, not just health().
+
+    health() is True whenever *any* provider client is configured, but a
+    request for a gpt-* model still needs the OpenAI client specifically — so
+    engine selection must not pick the cloud engine for a model whose provider
+    client is missing.
+    """
+
+    @staticmethod
+    def _engine(**clients: object) -> CloudEngine:
+        eng = CloudEngine.__new__(CloudEngine)  # bypass real client init
+        for name in (
+            "_openai_client",
+            "_anthropic_client",
+            "_google_client",
+            "_openrouter_client",
+            "_minimax_client",
+            "_codex_client",
+        ):
+            setattr(eng, name, clients.get(name))
+        return eng
+
+    def test_openai_only_serves_openai_models(self) -> None:
+        eng = self._engine(_openai_client=object())
+        assert eng.can_serve("gpt-4o") is True
+        assert eng.can_serve("claude-sonnet-4") is False
+        assert eng.can_serve("gemini-2.5-pro") is False
+        assert eng.can_serve("openrouter/openai/gpt-4o") is False
+
+    def test_anthropic_only_serves_anthropic_models(self) -> None:
+        eng = self._engine(_anthropic_client=object())
+        assert eng.can_serve("claude-sonnet-4") is True
+        assert eng.can_serve("gpt-4o") is False

tests/engine/test_discovery.py

@@ -174,6 +174,51 @@ class TestGetEngine:
         assert result is not None
         assert result[0] == "running"
 
+    def test_skips_engine_that_cannot_serve_model(self) -> None:
+        """#532: a healthy engine that can't serve the requested model is
+        skipped for one that can — this is what stops the cloud fallback being
+        chosen (when the local engine is down) for a model whose provider
+        client is missing.
+        """
+        _reg("picky", "picky")
+        _reg("local", "local")
+
+        class _Picky(_FakeEngine):
+            def can_serve(self, model: str) -> bool:
+                return model == "servable"
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "picky"
+
+        def _make(k, c):  # noqa: ANN001
+            if k == "picky":
+                return _Picky(healthy=True)
+            return _FakeEngine(healthy=(k == "local"))
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=_make,
+        ):
+            # "picky" is healthy but cannot serve "other" -> fall back to "local"
+            result = get_engine(cfg, model="other")
+        assert result is not None
+        assert result[0] == "local"
+
+    def test_model_none_preserves_model_agnostic_selection(self) -> None:
+        """model=None keeps the legacy behaviour: first healthy engine wins."""
+        _reg("primary", "primary")
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "primary"
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=lambda k, c: _FakeEngine(healthy=True),  # noqa: ANN001
+        ):
+            result = get_engine(cfg, model=None)
+        assert result is not None
+        assert result[0] == "primary"
+
 
 class TestMiningSidecarEngineHandoff:
     """Engine discovery picks up (or ignores) a mining sidecar at runtime."""

tests/engine/test_openai_compat_api_key.py

@@ -0,0 +1,109 @@
+"""API-key (Authorization header) support in the OpenAI-compat engine base."""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+import respx
+
+from openjarvis.core.types import Message, Role
+from openjarvis.engine.openai_compat_engines import (
+    OpenAICompatEngine,
+    VLLMEngine,
+    normalize_openai_base_url,
+)
+
+_CHAT_RESPONSE = {
+    "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}],
+    "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+    "model": "m",
+}
+
+
+class TestAuthorizationHeader:
+    def test_bearer_header_sent_when_api_key_set(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_no_authorization_header_without_api_key(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert "authorization" not in route.calls.last.request.headers
+
+    def test_health_check_sends_bearer_header(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_env_var_fallback_sanitizes_hyphen(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        # engine_id "openai-compat" must map to OPENAI_COMPAT_API_KEY —
+        # shells cannot set hyphenated env-var names.
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-env"
+
+    def test_vllm_env_var_fallback(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        monkeypatch.setenv("VLLM_API_KEY", "sk-vllm")
+        engine = VLLMEngine(host="http://testhost:8000")
+        with respx.mock:
+            route = respx.get("http://testhost:8000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-vllm"
+
+    def test_explicit_api_key_beats_env_var(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
+        assert engine._api_key == "sk-explicit"
+
+
+class TestNormalizeOpenAIBaseUrl:
+    @pytest.mark.parametrize(
+        ("url", "expected"),
+        [
+            ("http://h:8000", "http://h:8000"),
+            ("http://h:8000/", "http://h:8000"),
+            ("http://h:8000/v1", "http://h:8000"),
+            ("http://h:8000/v1/", "http://h:8000"),
+            ("http://h:8000/gateway/v1", "http://h:8000/gateway"),
+            # Only a literal trailing "/v1" is stripped — never other paths.
+            ("http://h:8000/v1x", "http://h:8000/v1x"),
+            ("http://h:8000/v2", "http://h:8000/v2"),
+        ],
+    )
+    def test_normalization(self, url: str, expected: str) -> None:
+        assert normalize_openai_base_url(url) == expected
+
+    def test_engine_requests_have_single_v1_prefix(self) -> None:
+        """End to end: a user-supplied .../v1 URL must not produce /v1/v1."""
+        host = normalize_openai_base_url("http://testhost:9000/v1")
+        engine = OpenAICompatEngine(host=host)
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.url.path == "/v1/models"

tests/engine/test_vllm_models.py

@@ -204,10 +204,14 @@ class TestVLLMErrors:
         respx_mock.post(f"{VLLM_HOST}/v1/chat/completions").mock(
             return_value=httpx.Response(404, json={"error": "model not found"})
         )
-        with pytest.raises(httpx.HTTPStatusError):
+        # The OpenAI-compatible engine wraps upstream HTTP errors (incl. 404)
+        # in EngineConnectionError with an actionable message (see #463); the
+        # raw httpx.HTTPStatusError is the chained cause.
+        with pytest.raises(EngineConnectionError) as exc_info:
             engine.generate(
                 [Message(role=Role.USER, content="Hi")], model="nonexistent"
             )
+        assert isinstance(exc_info.value.__cause__, httpx.HTTPStatusError)
 
     def test_timeout_raises_connection_error(self) -> None:
         engine = _make_engine()

tests/evals/core/test_config_timeouts.py

@@ -0,0 +1,72 @@
+"""Tests for terminal-bench harness timeout plumbing through TOML configs."""
+
+from __future__ import annotations
+
+import textwrap
+
+from openjarvis.evals.core.config import expand_suite, load_eval_config
+
+
+def _write(tmp_path, body: str):
+    path = tmp_path / "suite.toml"
+    path.write_text(textwrap.dedent(body))
+    return path
+
+
+BASE = """
+    [meta]
+    name = "timeouts"
+
+    [run]
+    output_dir = "results/"
+    {run_extra}
+
+    [[models]]
+    name = "test-model"
+
+    [[benchmarks]]
+    name = "terminalbench-native"
+    backend = "terminalbench-native"
+    {bench_extra}
+"""
+
+
+class TestTimeoutConfigPlumbing:
+    def test_run_level_timeouts_parse_and_expand(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra=(
+                    "global_agent_timeout_sec = 1200\n"
+                    "    global_timeout_multiplier = 1.5"
+                ),
+                bench_extra="",
+            ),
+        )
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec == 1200.0
+        assert suite.run.global_timeout_multiplier == 1.5
+
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec == 1200.0
+        assert rc.global_timeout_multiplier == 1.5
+
+    def test_benchmark_override_wins(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra="global_agent_timeout_sec = 1200",
+                bench_extra="global_agent_timeout_sec = 300",
+            ),
+        )
+        (rc,) = expand_suite(load_eval_config(path))
+        assert rc.global_agent_timeout_sec == 300.0
+
+    def test_defaults_are_none(self, tmp_path):
+        path = _write(tmp_path, BASE.format(run_extra="", bench_extra=""))
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec is None
+        assert suite.run.global_timeout_multiplier is None
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec is None
+        assert rc.global_timeout_multiplier is None

tests/evals/datasets/test_external_agent_datasets.py

@@ -10,6 +10,14 @@ These tests download each dataset once to ~/.cache/huggingface and
 verify the provider loads, iterates, and honours the split kwarg.
 The ModuleNotFoundError skip branch is defensive — currently
 unreachable since all three provider modules exist.
+
+Marked ``hub``: they hit the live HuggingFace Hub, so they are excluded
+from the default CI lane (which runs ``-m "not live and not cloud and not
+hub"``) — a transient Hub outage or rate-limit must not redden ``main``.
+Run them on demand with ``pytest -m hub``. The ADP provider swallows
+per-config download errors and returns 0 records on a network failure
+(see ``adp.py``), so a Hub outage surfaces here as ``assert 1 <= 0``
+rather than an exception — another reason these can't run unguarded in CI.
 """
 
 from __future__ import annotations
@@ -18,6 +26,8 @@ import importlib
 
 import pytest
 
+pytestmark = pytest.mark.hub
+
 PROVIDERS = [
     ("openjarvis.evals.datasets.adp", "ADPDataset"),
     ("openjarvis.evals.datasets.toolorchestra", "ToolOrchestraDataset"),

tests/evals/test_agentic_runner.py

@@ -9,6 +9,7 @@ from typing import Any, Dict, List
 import pytest
 
 from openjarvis.evals.core.agentic_runner import AgenticRunner, _extract_patch
+from openjarvis.evals.core.environment import TaskEnvironmentError
 
 # ---------------------------------------------------------------------------
 # Mock objects
@@ -50,6 +51,66 @@ class MockFailingAgent:
         raise RuntimeError("Agent error")
 
 
+class MockZeroContactAgent:
+    """Agent that returns without ever contacting the model.
+
+    Mirrors the downstream failure signature: a hung/dead in-container
+    setup yields zero LM events and zero token usage, while run_tests
+    still stamps is_resolved=False.
+    """
+
+    def ask(self, query: str) -> dict:
+        return {"content": "setup log tail ...", "usage": {}}
+
+
+class FailingTaskEnv:
+    """Task env whose __enter__ fails like a tmux/compose breakage."""
+
+    def __init__(self, metadata: Dict[str, Any]) -> None:
+        self._metadata = metadata
+
+    def __enter__(self) -> "FailingTaskEnv":
+        message = (
+            "Task 't1': required binary 'tmux' is not usable in task image "
+            "'tb__t1__client'"
+        )
+        self._metadata["harness_error"] = message
+        raise TaskEnvironmentError(message)
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+
+class ResolvingTaskEnv:
+    """Task env that stamps is_resolved into metadata like run_tests does."""
+
+    def __init__(self, metadata: Dict[str, Any], is_resolved: bool) -> None:
+        self._metadata = metadata
+        self._is_resolved = is_resolved
+
+    def __enter__(self) -> "ResolvingTaskEnv":
+        return self
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+    def run_tests(self):
+        self._metadata["is_resolved"] = self._is_resolved
+        return self._is_resolved, {}
+
+
+class EnvDataset(MockDataset):
+    """Dataset whose create_task_env is configurable per record id."""
+
+    def __init__(self, records: List[MockRecord], env_factories: Dict[str, Any]):
+        super().__init__(records)
+        self._env_factories = env_factories
+
+    def create_task_env(self, record: MockRecord):
+        factory = self._env_factories.get(record.record_id)
+        return factory(record.metadata) if factory is not None else None
+
+
 # ---------------------------------------------------------------------------
 # Tests
 # ---------------------------------------------------------------------------
@@ -99,6 +160,59 @@ class TestAgenticRunner:
         assert len(traces) == 1
         assert not traces[0].completed
         assert "Agent error" in traces[0].response_text
+        assert traces[0].error_kind == "agent_error"
+        assert "Agent error" in (traces[0].error or "")
+
+    def test_harness_failure_recorded_and_run_continues(self):
+        """(c) one task's env breakage doesn't kill the run."""
+        records = [
+            MockRecord(record_id="r1", problem="broken env"),
+            MockRecord(record_id="r2", problem="healthy"),
+        ]
+        dataset = EnvDataset(records, {"r1": FailingTaskEnv})
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert len(traces) == 2
+        # Failed task: recorded distinctly as a harness error, not a miss.
+        assert traces[0].error_kind == "harness_error"
+        assert not traces[0].completed
+        assert "tmux" in (traces[0].error or "")
+        assert traces[0].is_resolved is None
+        # Healthy task still ran to completion.
+        assert traces[1].completed
+        assert traces[1].error_kind is None
+        assert "Response to: healthy" in traces[1].response_text
+
+    def test_zero_model_contact_flagged_as_harness_error(self):
+        """(e) zero model requests -> harness_error, not a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockZeroContactAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].error_kind == "harness_error"
+        assert "zero_model_requests" in (traces[0].error or "")
+        assert traces[0].total_input_tokens == 0
+        assert traces[0].total_output_tokens == 0
+
+    def test_genuine_model_miss_not_flagged(self):
+        """(e) control: tokens>0 + is_resolved=False is a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].is_resolved is False
+        assert traces[0].error_kind is None
+        assert traces[0].error is None
+        assert traces[0].total_input_tokens > 0
 
     def test_synthetic_turn_created(self):
         records = [MockRecord(record_id="r1", problem="test")]

tests/evals/test_dataset_splits_integration.py

@@ -1,4 +1,13 @@
-"""Integration test: each provider's split kwarg produces disjoint train/test slices."""
+"""Integration test: each provider's split kwarg produces disjoint train/test slices.
+
+Marked ``hub``: every provider here downloads a real corpus from the
+HuggingFace Hub, so the module is excluded from the default CI lane (which
+runs ``-m "not live and not cloud and not hub"``). A transient Hub outage
+or rate-limit raises connectivity errors (``LocalEntryNotFoundError``,
+``HfHubHTTPError``) that the gated/not-found skip branch below does not
+catch — so running these unguarded made ``main`` flaky-red. Run on demand
+with ``pytest -m hub``.
+"""
 
 from __future__ import annotations
 
@@ -6,6 +15,8 @@ import importlib
 
 import pytest
 
+pytestmark = pytest.mark.hub
+
 PROVIDERS = [
     ("openjarvis.evals.datasets.pinchbench", "PinchBenchDataset"),
     ("openjarvis.evals.datasets.liveresearch", "LiveResearchBenchDataset"),

tests/evals/test_export.py

@@ -127,6 +127,84 @@ class TestExportSummaryJson:
         assert summary["totals"]["queries"] == 0
 
 
+class TestHarnessErrorExclusion:
+    """Harness errors must never count as model misses in resolve-rate."""
+
+    def _traces(self):
+        return [
+            QueryTrace(
+                query_id="q0000",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=True,
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0001",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=False,  # genuine model miss: stays in denominator
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0002",
+                workload_type="agentic",
+                completed=False,
+                is_resolved=False,  # stamped by run_tests despite zero contact
+                error="zero_model_requests: the agent never contacted the model",
+                error_kind="harness_error",
+            ),
+        ]
+
+    def test_summary_excludes_harness_errors_from_accuracy(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(self._traces(), {"model": "m"}, path)
+        summary = json.loads(path.read_text())
+
+        totals = summary["totals"]
+        assert totals["harness_errors"] == 1
+        assert totals["resolved"] == 1
+        assert totals["unresolved"] == 1  # NOT 2: harness error excluded
+        assert totals["accuracy"] == 0.5  # NOT 1/3
+        # Flat table_gen metrics exclude the harness error too.
+        assert summary["metrics"]["accuracy"]["n"] == 2
+        assert summary["metrics"]["accuracy"]["mean"] == 0.5
+        # Details surfaced for diagnosis.
+        details = summary["harness_error_details"]
+        assert details[0]["query_id"] == "q0002"
+        assert "zero_model_requests" in details[0]["error"]
+
+    def test_efficiency_excludes_harness_errors(self):
+        result = _compute_efficiency(self._traces(), None, None)
+        assert result["accuracy"] == 0.5
+
+    def test_no_harness_errors_key_absent(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(_make_traces(), {}, path)
+        summary = json.loads(path.read_text())
+        assert summary["totals"]["harness_errors"] == 0
+        assert "harness_error_details" not in summary
+
+
+class TestTraceErrorFieldRoundTrip:
+    def test_round_trip_preserves_error_fields(self):
+        trace = QueryTrace(
+            query_id="q0",
+            workload_type="agentic",
+            error="zero_model_requests: ...",
+            error_kind="harness_error",
+        )
+        restored = QueryTrace.from_dict(trace.to_dict())
+        assert restored.error == trace.error
+        assert restored.error_kind == "harness_error"
+
+    def test_old_trace_dicts_still_load(self):
+        """Backward compat: traces.jsonl written before the schema change."""
+        restored = QueryTrace.from_dict({"query_id": "q0", "workload_type": "agentic"})
+        assert restored.error is None
+        assert restored.error_kind is None
+
+
 class TestExportArtifactsManifest:
     def test_no_artifacts_dir(self, tmp_path):
         result = export_artifacts_manifest(tmp_path)

tests/evals/test_terminalbench_env.py

@@ -1,15 +1,104 @@
-"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency)."""
+"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency).
+
+These tests install a fake ``terminal_bench`` module tree into
+``sys.modules`` so they run without the real package or a Docker daemon
+(terminal-bench is an undeclared optional dep that CI never installs).
+"""
 
 from __future__ import annotations
 
+import sys
+import types
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from types import SimpleNamespace
+from typing import Any, Dict, List
+
 import pytest
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.execution.terminalbench_env import TerminalBenchTaskEnv
 
-# terminal_bench is an optional dep — skip all tests if unavailable
-terminal_bench = pytest.importorskip(
-    "terminal_bench", reason="terminal_bench not installed"
-)
+# ---------------------------------------------------------------------------
+# Fake terminal_bench seam
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class FakeExecResult:
+    exit_code: int = 0
+    output: bytes = b""
+
+
+@dataclass
+class FakeContainer:
+    """Container stub whose exec_run results are configurable per binary."""
+
+    exec_results: Dict[str, FakeExecResult] = field(default_factory=dict)
+    exec_calls: List[List[str]] = field(default_factory=list)
+
+    def exec_run(self, cmd: List[str]) -> FakeExecResult:
+        self.exec_calls.append(list(cmd))
+        return self.exec_results.get(cmd[0], FakeExecResult())
+
+
+class FakeTerminal:
+    def __init__(self, events: List[str], container: FakeContainer) -> None:
+        self._events = events
+        self.container = container
+        self.create_session_error: Exception | None = None
+
+    def create_session(self, name: str, **_kwargs: Any) -> str:
+        self._events.append(f"create_session({name})")
+        if self.create_session_error is not None:
+            raise self.create_session_error
+        return f"session-{name}"
+
+
+@pytest.fixture()
+def fake_tb(monkeypatch, tmp_path):
+    """Install a fake terminal_bench tree; return the shared test state."""
+    events: List[str] = []
+    container = FakeContainer()
+    terminal = FakeTerminal(events, container)
+    state = SimpleNamespace(
+        events=events,
+        container=container,
+        terminal=terminal,
+        spin_up_error=None,
+    )
+
+    @contextmanager
+    def spin_up_terminal(**kwargs: Any):
+        events.append("compose_up")
+        try:
+            if state.spin_up_error is not None:
+                raise state.spin_up_error
+            yield terminal
+        finally:
+            events.append("compose_down")
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_terminal_pkg = types.ModuleType("terminal_bench.terminal")
+    mod_terminal = types.ModuleType("terminal_bench.terminal.terminal")
+    mod_terminal.spin_up_terminal = spin_up_terminal
+    mod_tb.terminal = mod_terminal_pkg
+    mod_terminal_pkg.terminal = mod_terminal
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal", mod_terminal_pkg)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal.terminal", mod_terminal)
+
+    state.metadata = {
+        "task_id": "hello.world",
+        "task": SimpleNamespace(disable_asciinema=True),
+        "task_paths": SimpleNamespace(docker_compose_path=tmp_path / "compose.yaml"),
+    }
+    return state
+
+
+# ---------------------------------------------------------------------------
+# Existing behavior (now running without the real terminal_bench package)
+# ---------------------------------------------------------------------------
 
 
 class TestTerminalBenchTaskEnv:
@@ -46,3 +135,113 @@ class TestTerminalBenchTaskEnv:
         assert is_resolved is False
         assert results["error"] == "terminal_not_running"
         assert metadata["is_resolved"] is False
+
+
+# ---------------------------------------------------------------------------
+# Exception-safe __enter__ / preflight / teardown
+# ---------------------------------------------------------------------------
+
+
+class TestEnterExceptionSafety:
+    def test_success_path(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            assert fake_tb.metadata["terminal"] is fake_tb.terminal
+            assert fake_tb.metadata["session"] == "session-agent"
+            assert fake_tb.metadata["container"] == "oj-hello-world"
+            assert "compose_down" not in fake_tb.events
+        assert fake_tb.events.count("compose_down") == 1
+        assert "terminal" not in fake_tb.metadata
+
+    def test_create_session_failure_tears_down_terminal(self, fake_tb):
+        """(a) tmux failure in __enter__ -> terminal torn down, loud error."""
+        fake_tb.terminal.create_session_error = RuntimeError(
+            "tmux is not installed in the container."
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        # No leak: compose project downed BEFORE the exception escaped.
+        assert "compose_down" in fake_tb.events
+        # Actionable: names the task, the image, and the failure.
+        message = str(excinfo.value)
+        assert "hello.world" in message
+        assert "tb__hello-world__client" in message
+        assert "tmux is not installed" in message
+        # Recorded for the runner / scorers; handles cleared.
+        assert fake_tb.metadata["harness_error"] == message
+        assert "terminal" not in fake_tb.metadata
+        assert "session" not in fake_tb.metadata
+        assert "container" not in fake_tb.metadata
+        assert env._terminal is None
+        assert env._terminal_cm is None
+        assert env._logs_tmpdir is None
+
+    def test_preflight_catches_missing_tmux(self, fake_tb):
+        """(b) preflight catches missing tmux, naming the task image."""
+        fake_tb.container.exec_results["tmux"] = FakeExecResult(
+            exit_code=127,
+            output=b'exec: "tmux": executable file not found in $PATH',
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        message = str(excinfo.value)
+        assert "tmux" in message
+        assert "tb__hello-world__client" in message  # task image named
+        assert "127" in message
+        # Fired BEFORE the agent session was created.
+        assert not any(e.startswith("create_session") for e in fake_tb.events)
+        assert "compose_down" in fake_tb.events
+        assert fake_tb.metadata["harness_error"] == message
+
+    def test_preflight_checks_asciinema_when_recording(self, fake_tb):
+        fake_tb.metadata["task"] = SimpleNamespace(disable_asciinema=False)
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError, match="asciinema"):
+            env.__enter__()
+        assert "disable_asciinema" in fake_tb.metadata["harness_error"]
+        assert "compose_down" in fake_tb.events
+
+    def test_preflight_skips_asciinema_when_disabled(self, fake_tb):
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            pass
+        assert ["asciinema", "--version"] not in fake_tb.container.exec_calls
+
+    def test_compose_up_failure_includes_stderr(self, fake_tb):
+        import subprocess
+
+        fake_tb.spin_up_error = subprocess.CalledProcessError(
+            returncode=1,
+            cmd=["docker", "compose", "up"],
+            stderr="no space left on device",
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+        assert "no space left on device" in str(excinfo.value)
+
+    def test_keyboard_interrupt_not_masked(self, fake_tb):
+        fake_tb.terminal.create_session_error = KeyboardInterrupt()
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(KeyboardInterrupt):
+            env.__enter__()
+        # Still cleaned up, but the interrupt is not wrapped.
+        assert "compose_down" in fake_tb.events
+
+    def test_teardown_is_idempotent(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        env.__enter__()
+        env.__exit__(None, None, None)
+        env.__exit__(None, None, None)
+        assert fake_tb.events.count("compose_down") == 1

tests/evals/test_terminalbench_native_backend.py

@@ -0,0 +1,302 @@
+"""Tests for the TerminalBench native backend (mocked terminal_bench).
+
+Covers: timeout kwargs threading (config -> backend -> Harness kwargs),
+loud failure on terminal-bench builds without the timeout kwargs, and the
+harness-error classification in ``summarize_benchmark_results`` —
+including the zero-model-contact vs genuine-model-miss distinction.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+from types import SimpleNamespace
+from typing import Any, Dict, List, Optional
+
+import pytest
+
+import openjarvis.evals.backends.terminalbench_native as tbn
+from openjarvis.evals.backends.terminalbench_native import (
+    summarize_benchmark_results,
+)
+
+# ---------------------------------------------------------------------------
+# Helpers / fakes
+# ---------------------------------------------------------------------------
+
+
+def make_trial(
+    task_id: str,
+    *,
+    is_resolved: bool,
+    failure_mode: str = "unset",
+    input_tokens: Optional[int] = None,
+    output_tokens: Optional[int] = None,
+) -> SimpleNamespace:
+    """Build a duck-typed terminal-bench 0.2.18 TrialResults."""
+    return SimpleNamespace(
+        task_id=task_id,
+        trial_name=f"{task_id}.1-of-1",
+        is_resolved=is_resolved,
+        failure_mode=SimpleNamespace(value=failure_mode),
+        total_input_tokens=input_tokens,
+        total_output_tokens=output_tokens,
+    )
+
+
+class FakeHarness:
+    """Records constructor kwargs; run() returns the configured results."""
+
+    captured_kwargs: Dict[str, Any] = {}
+    results: Any = SimpleNamespace(results=[])
+
+    def __init__(self, **kwargs: Any) -> None:
+        type(self).captured_kwargs = kwargs
+
+    def run(self) -> Any:
+        return type(self).results
+
+
+class OldFakeHarness:
+    """A pre-timeout-kwargs Harness signature (no **kwargs)."""
+
+    def __init__(
+        self,
+        output_path: Any = None,
+        run_id: Any = None,
+        dataset_name: Any = None,
+        dataset_version: Any = None,
+        model_name: Any = None,
+        n_concurrent_trials: Any = None,
+        cleanup: Any = None,
+        agent_name: Any = None,
+        agent_kwargs: Any = None,
+        n_tasks: Any = None,
+    ) -> None:
+        pass
+
+    def run(self) -> Any:
+        return SimpleNamespace(results=[])
+
+
+@pytest.fixture()
+def fake_tb_backend(monkeypatch):
+    """Enable the backend without the real terminal_bench package."""
+    FakeHarness.captured_kwargs = {}
+    FakeHarness.results = SimpleNamespace(results=[])
+    monkeypatch.setattr(tbn, "_HAS_TB", True)
+    monkeypatch.setattr(tbn, "Harness", FakeHarness, raising=False)
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_agents = types.ModuleType("terminal_bench.agents")
+    mod_agent_name = types.ModuleType("terminal_bench.agents.agent_name")
+    mod_agent_name.AgentName = lambda name: name
+    mod_tb.agents = mod_agents
+    mod_agents.agent_name = mod_agent_name
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents", mod_agents)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents.agent_name", mod_agent_name)
+    return FakeHarness
+
+
+# ---------------------------------------------------------------------------
+# Timeout kwargs threading
+# ---------------------------------------------------------------------------
+
+
+class TestTimeoutKwargs:
+    def test_default_bound_reaches_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1800.0
+        assert "global_timeout_multiplier" not in kwargs
+
+    def test_explicit_values_reach_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path),
+            global_agent_timeout_sec=1234.0,
+            global_timeout_multiplier=2.0,
+        )
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1234.0
+        assert kwargs["global_timeout_multiplier"] == 2.0
+
+    def test_zero_disables_bound(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path), global_agent_timeout_sec=0
+        )
+        backend.run_harness("run-1")
+        assert "global_agent_timeout_sec" not in fake_tb_backend.captured_kwargs
+
+    def test_old_terminal_bench_fails_loud(
+        self, fake_tb_backend, monkeypatch, tmp_path
+    ):
+        """An old Harness without the kwargs must not hang silently."""
+        monkeypatch.setattr(tbn, "Harness", OldFakeHarness, raising=False)
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        with pytest.raises(RuntimeError, match="global_agent_timeout_sec"):
+            backend.run_harness("run-1")
+
+
+# ---------------------------------------------------------------------------
+# Harness-error classification (zero-model-contact detection)
+# ---------------------------------------------------------------------------
+
+
+class TestSummarizeBenchmarkResults:
+    def test_genuine_model_miss_is_not_flagged(self):
+        """MANDATORY: a real miss (tokens>0, failure_mode unset) stays a miss.
+
+        terminal-bench 0.2.18 leaves failure_mode UNSET on success and on
+        genuine unresolved misses — classifying on failure_mode would flag
+        every real miss as a harness error and inflate accuracy.
+        """
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-ok", is_resolved=True, input_tokens=900, output_tokens=100
+                ),
+                make_trial(
+                    "t-miss", is_resolved=False, input_tokens=800, output_tokens=50
+                ),
+                make_trial(
+                    "t-setup-dead",
+                    is_resolved=False,
+                    failure_mode="agent_installation_failed",
+                    input_tokens=0,
+                    output_tokens=0,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2  # genuine miss stays in denominator
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5  # not 1.0 (miss kept), not 1/3 (infra out)
+        assert summary.errors == 1
+        assert [f["task_id"] for f in failures] == ["t-setup-dead"]
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_zero_contact_flagged_even_with_unset_failure_mode(self):
+        """Setup hang signature: unresolved, zero requests, failure_mode unset."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-hang", is_resolved=False, input_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert summary.scored_samples == 0
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_missing_token_fields_treated_as_zero_contact(self):
+        results = SimpleNamespace(results=[make_trial("t-none", is_resolved=False)])
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+
+    def test_infra_failure_mode_flagged_despite_tokens(self):
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-crash",
+                    is_resolved=False,
+                    failure_mode="unknown_agent_error",
+                    input_tokens=500,
+                    output_tokens=20,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert failures[0]["reason"] == "unknown_agent_error"
+
+    def test_resolved_with_zero_tokens_not_flagged(self):
+        """Installed agents report 0 tokens on success — never flag resolved."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-ok", is_resolved=True, input_tokens=0, output_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 0
+        assert summary.correct == 1
+        assert summary.accuracy == 1.0
+
+    def test_empty_results(self):
+        summary, failures = summarize_benchmark_results(
+            SimpleNamespace(results=[]), model="m"
+        )
+        assert summary.total_samples == 0
+        assert summary.accuracy == 0.0
+        assert failures == []
+
+
+# ---------------------------------------------------------------------------
+# CLI wiring: config -> backend -> harness kwargs -> RunSummary
+# ---------------------------------------------------------------------------
+
+
+class TestRunTerminalbenchNativeWiring:
+    def _run(self, fake_tb_backend, tmp_path, trials: List[Any], **config_kwargs):
+        from rich.console import Console
+
+        from openjarvis.evals.cli import _run_terminalbench_native
+        from openjarvis.evals.core.types import RunConfig
+
+        fake_tb_backend.results = SimpleNamespace(results=trials)
+        config = RunConfig(
+            benchmark="terminalbench-native",
+            backend="terminalbench-native",
+            model="test-model",
+            output_path=str(tmp_path / "out"),
+            **config_kwargs,
+        )
+        console = Console(record=True, width=120)
+        summary = _run_terminalbench_native(config, console)
+        return summary, console.export_text()
+
+    def test_config_timeouts_reach_harness_kwargs(self, fake_tb_backend, tmp_path):
+        """(d) timeout kwargs travel config -> backend -> harness_kwargs."""
+        self._run(
+            fake_tb_backend,
+            tmp_path,
+            [],
+            global_agent_timeout_sec=901.0,
+            global_timeout_multiplier=1.5,
+        )
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 901.0
+        assert kwargs["global_timeout_multiplier"] == 1.5
+
+    def test_config_defaults_use_backend_bound(self, fake_tb_backend, tmp_path):
+        self._run(fake_tb_backend, tmp_path, [])
+        assert fake_tb_backend.captured_kwargs["global_agent_timeout_sec"] == 1800.0
+
+    def test_summary_counts_real_trials(self, fake_tb_backend, tmp_path):
+        """Regression: results field is ``results``, not ``trial_results``.
+
+        The old conversion read the nonexistent ``trial_results`` attribute
+        and hardcoded errors=0, rendering every run as 0 samples / 0.0.
+        """
+        trials = [
+            make_trial("t-ok", is_resolved=True, input_tokens=10, output_tokens=10),
+            make_trial("t-miss", is_resolved=False, input_tokens=10, output_tokens=2),
+            make_trial(
+                "t-hang",
+                is_resolved=False,
+                failure_mode="agent_timeout",
+                input_tokens=0,
+                output_tokens=0,
+            ),
+        ]
+        summary, output = self._run(fake_tb_backend, tmp_path, trials)
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5
+        assert summary.errors == 1
+        # The harness failure is reported loudly with its task id.
+        assert "t-hang" in output
+        assert "zero_model_requests" in output

tests/evals/test_use_case_benchmarks.py

@@ -424,11 +424,22 @@ class TestSavings:
         assert "total_calls" in d
 
     def test_energy_scales_linearly(self) -> None:
-        """Energy should scale linearly with evaluated tokens (KV-cache model)."""
+        """Energy should scale linearly with evaluated tokens (KV-cache model).
+
+        The test must pass `prompt_tokens_evaluated` explicitly — under the
+        leaderboard-correctness fix, `compute_savings` no longer falls back
+        from a missing `prompt_tokens_evaluated` to the (multi-turn-inflated)
+        `prompt_tokens` sum. When the KV-cache-aware count is missing, FLOPs
+        derive from `completion_tokens` only. With completion=0, that path
+        produces zero energy and divides by zero. Passing the evaluated
+        count explicitly is the API contract that lets this invariant be
+        tested without depending on the (now intentionally conservative)
+        fallback behaviour.
+        """
         from openjarvis.server.savings import compute_savings
 
-        s1 = compute_savings(1000, 0)
-        s10 = compute_savings(10000, 0)
+        s1 = compute_savings(1000, 0, prompt_tokens_evaluated=1000)
+        s10 = compute_savings(10000, 0, prompt_tokens_evaluated=10000)
         # FLOPs = 2*P*T (linear), so 10x tokens => 10x FLOPs => 10x energy
         for p1, p10 in zip(s1.per_provider, s10.per_provider):
             ratio = p10.energy_wh / p1.energy_wh
@@ -437,10 +448,16 @@ class TestSavings:
             )
 
     def test_energy_wh_matches_direct_formula(self) -> None:
-        """Energy must equal flops * wh_per_flop for known constants."""
+        """Energy must equal flops * wh_per_flop for known constants.
+
+        Pass `prompt_tokens_evaluated` explicitly for the same reason as
+        `test_energy_scales_linearly` — the conservative-fallback fix in
+        savings.py means an unset `prompt_tokens_evaluated` produces 0
+        FLOPs and would make this test pass trivially with 0 == 0.
+        """
         from openjarvis.server.savings import CLOUD_PRICING, compute_savings
 
-        summary = compute_savings(10000, 0)
+        summary = compute_savings(10000, 0, prompt_tokens_evaluated=10000)
         for p in summary.per_provider:
             pricing = CLOUD_PRICING[p.provider]
             wh_per_flop = pricing["energy_wh_per_1k_tokens"] / (
@@ -450,6 +467,12 @@ class TestSavings:
             assert abs(p.energy_wh - expected) < 1e-6, (
                 f"{p.provider}: energy_wh={p.energy_wh}, expected={expected}"
             )
+            # Sanity: assert the formula isn't matching trivially with both
+            # sides equal to zero (would happen if FLOPs collapse to 0).
+            assert p.flops > 0, (
+                f"{p.provider}: FLOPs must be positive when "
+                f"prompt_tokens_evaluated is explicitly set"
+            )
 
     def test_energy_not_zero(self) -> None:
         """Energy must be positive for non-zero token counts."""

tests/learning/training/test_lora.py

@@ -2,6 +2,8 @@
 
 from __future__ import annotations
 
+from unittest.mock import MagicMock
+
 import pytest
 
 from openjarvis.learning.training.lora import HAS_TORCH, LoRATrainer, LoRATrainingConfig
@@ -145,3 +147,55 @@ class TestLoRATrainerWithTorch:
 
         assert result["status"] == "skipped"
         assert "reason" in result
+
+
+@pytest.mark.skipif(not HAS_TORCH, reason="torch not installed")
+class TestLoRATrainStepMasking:
+    """Regression for #521: _train_step must exclude padding from the loss labels.
+
+    The pre-fix code passed ``labels=input_ids`` (the *same* tensor), so the loss
+    counted every padded EOS position and an in-place mask would also corrupt
+    ``input_ids``. ``_train_step`` must build a masked clone: ``-100`` wherever
+    ``attention_mask == 0``, equal to ``input_ids`` elsewhere, leaving
+    ``input_ids`` untouched.
+    """
+
+    def test_train_step_masks_padding_in_labels(self) -> None:
+        import torch
+
+        # Bypass __init__ (which loads a real model) and inject the minimal
+        # attributes _train_step touches before the forward pass.
+        trainer = LoRATrainer.__new__(LoRATrainer)
+        trainer.device = "cpu"
+
+        captured: dict = {}
+
+        class _StopForward(Exception):
+            pass
+
+        def _capture_model(*, input_ids, attention_mask, labels):
+            captured["labels"] = labels
+            raise _StopForward  # stop before backward()/optimizer.step()
+
+        trainer.model = _capture_model
+
+        batch_items = [
+            {
+                "input_ids": torch.tensor([11, 12, 0, 0]),
+                "attention_mask": torch.tensor([1, 1, 0, 0]),
+            },
+            {
+                "input_ids": torch.tensor([13, 14, 15, 0]),
+                "attention_mask": torch.tensor([1, 1, 1, 0]),
+            },
+        ]
+        with pytest.raises(_StopForward):
+            trainer._train_step(batch_items, optimizer=MagicMock())
+
+        ids = torch.stack([b["input_ids"] for b in batch_items])
+        mask = torch.stack([b["attention_mask"] for b in batch_items])
+        labels = captured["labels"]
+
+        assert (labels[mask == 0] == -100).all()  # padded -> ignored by loss
+        assert (labels[mask == 1] == ids[mask == 1]).all()  # real -> unchanged
+        assert (ids[mask == 0] != -100).all()  # input_ids not mutated in place

tests/memory/test_chunking.py

@@ -66,13 +66,40 @@ def test_custom_config():
     assert len(chunks) >= 3
 
 
-def test_min_chunk_size_filters_tiny():
+def test_short_only_document_not_dropped():
+    """A whole document below min_chunk_size must still produce a chunk.
+
+    Regression for #502 follow-up: previously a folder of short notes indexed
+    to ``chunks_indexed: 0`` (HTTP 200), silently storing nothing. ``min_chunk_size``
+    should only discard tiny *trailing fragments*, never an entire short doc.
+    """
     cfg = ChunkConfig(chunk_size=100, chunk_overlap=0, min_chunk_size=50)
-    # 30 words is below min_chunk_size=50
+    # 30 words is below min_chunk_size=50, but it's the entire document.
     words = [f"w{i}" for i in range(30)]
     text = " ".join(words)
     chunks = chunk_text(text, config=cfg)
-    assert len(chunks) == 0
+    assert len(chunks) == 1
+    assert chunks[0].content == text
+
+
+def test_short_real_world_note_not_dropped():
+    """The exact repro from the issue: a ~4-word note must not vanish."""
+    chunks = chunk_text("hello world\nsome content\n", source="a.txt")
+    assert len(chunks) == 1
+    assert "hello world" in chunks[0].content
+
+
+def test_min_chunk_size_filters_tiny_trailing_fragment():
+    """A tiny fragment trailing a real chunk is still dropped by the floor."""
+    cfg = ChunkConfig(chunk_size=50, chunk_overlap=0, min_chunk_size=10)
+    # Two paragraphs: the first fills a real chunk, the second is a tiny tail.
+    para1 = " ".join(f"a{i}" for i in range(50))
+    para2 = " ".join(f"b{i}" for i in range(3))  # 3 words < min_chunk_size=10
+    text = f"{para1}\n\n{para2}"
+    chunks = chunk_text(text, config=cfg)
+    # The 3-word trailing fragment is discarded; only the real chunk remains.
+    assert len(chunks) == 1
+    assert "b0" not in chunks[0].content
 
 
 def test_source_propagated():

tests/prompt/test_persona_scope.py

@@ -31,3 +31,23 @@ def test_named_persona_resolves_to_personas_dir():
 def test_path_traversal_rejected(bad):
     with pytest.raises(ValueError):
         SystemPromptBuilder._resolve_persona(MemoryFilesConfig(persona_name=bad))
+
+
+def test_none_persona_build_does_not_raise():
+    """Regression (#497): `--persona none` resolves to empty file paths; building
+    the prompt must not raise IsADirectoryError when those empty paths are read
+    (Path("") is "." — reading a directory raised before the empty-path guard).
+    """
+    import dataclasses
+
+    from openjarvis.core.config import load_config
+
+    cfg = load_config()
+    mf = dataclasses.replace(cfg.memory_files, persona_name="none")
+    builder = SystemPromptBuilder(
+        agent_template=cfg.agent.default_system_prompt or "",
+        memory_files_config=mf,
+        system_prompt_config=cfg.system_prompt,
+    )
+    out = builder.build()
+    assert isinstance(out, str)

tests/sdk/test_system.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch
 
 import pytest
 
@@ -327,6 +327,85 @@ class TestSystemBuilder:
         assert builder._engine_key == "ollama"
 
 
+class TestSystemBuilderEngineInstance:
+    """Explicit engine injection (jarvis eval --base-url path)."""
+
+    @staticmethod
+    def _fake_engine(healthy: bool = True) -> MagicMock:
+        engine = MagicMock(
+            spec=["health", "can_serve", "generate", "list_models", "close"]
+        )
+        engine.health.return_value = healthy
+        engine._host = "http://127.0.0.1:18999"
+        return engine
+
+    def test_engine_instance_is_fluent(self):
+        builder = SystemBuilder(JarvisConfig())
+        engine = self._fake_engine()
+        result = builder.engine_instance(engine, key="my-endpoint")
+        assert result is builder
+        assert builder._engine_instance is engine
+        assert builder._engine_instance_key == "my-endpoint"
+
+    def test_resolve_engine_returns_injected_instance(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        resolved_engine, resolved_key = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        assert resolved_key == "endpoint"
+
+    def test_unhealthy_injected_instance_raises_naming_host(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18999"):
+            builder._resolve_engine(config)
+
+    def test_unhealthy_injected_instance_never_consults_discovery(self):
+        """The observed failure mode: an explicit endpoint must NOT be
+        silently replaced by whatever other engine discovery finds."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine)
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            with pytest.raises(RuntimeError, match="Refusing to fall back"):
+                builder._resolve_engine(config)
+        mock_get_engine.assert_not_called()
+
+    def test_healthy_injected_instance_never_consults_discovery(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            resolved_engine, _ = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        mock_get_engine.assert_not_called()
+
+    def test_build_wires_injected_engine(self):
+        """build() must use the injected engine (possibly behind security
+        wrappers) instead of running discovery."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        engine.list_models.return_value = ["stub-model"]
+        builder = (
+            SystemBuilder(config)
+            .engine_instance(engine, key="endpoint")
+            .model("stub-model")
+            .telemetry(False)
+            .traces(False)
+        )
+        system = builder.build()
+        try:
+            inner = system.engine
+            while hasattr(inner, "_engine"):
+                inner = inner._engine
+            assert inner is engine
+            assert system.engine_key == "endpoint"
+        finally:
+            system.close()
+
+
 class TestJarvisSystemClose:
     def test_close_with_scheduler_store(self):
         engine = MagicMock()

tests/server/conftest.py

@@ -0,0 +1,34 @@
+"""Shared fixtures for server route tests.
+
+Server tests build apps via ``create_app``, which (with traces enabled by
+default) wires a ``TraceStore`` at the real ``~/.openjarvis/traces.db``. Now
+that the chat endpoints actually *write* traces, an unguarded run would
+pollute the developer's real trace DB and make tests non-hermetic. This
+autouse fixture redirects the traces DB to a per-test temp path.
+"""
+
+from __future__ import annotations
+
+import pytest
+
+
+@pytest.fixture(autouse=True)
+def _isolate_traces_db(tmp_path, monkeypatch):
+    """Point ``config.traces.db_path`` at a temp file for every server test.
+
+    ``load_config`` returns a fresh ``JarvisConfig`` per call (no caching), so
+    wrapping it to rewrite ``traces.db_path`` only affects calls made during
+    the test — there is no global leak.
+    """
+    from openjarvis.core import config as _config
+
+    real_load_config = _config.load_config
+    db_path = str(tmp_path / "traces.db")
+
+    def _patched_load_config(*args, **kwargs):
+        cfg = real_load_config(*args, **kwargs)
+        cfg.traces.db_path = db_path
+        return cfg
+
+    monkeypatch.setattr(_config, "load_config", _patched_load_config)
+    return db_path

tests/server/test_agent_manager_routes.py

@@ -5,6 +5,7 @@ from __future__ import annotations
 import json
 import tempfile
 from pathlib import Path
+from types import SimpleNamespace
 from unittest.mock import MagicMock
 
 import pytest
@@ -488,3 +489,51 @@ class TestResolveToolSpecs:
 
         assert _resolve_tool_specs(None) == []
         assert _resolve_tool_specs([]) == []
+
+
+class TestLightweightSystemEngineResolution:
+    """Regression for #477 / #514: the managed-agent lightweight system must
+    resolve the user's *configured* engine (preferred_engine, else
+    engine.default), not a hardcoded OllamaEngine. We assert the key passed to
+    ``get_engine`` — captured before the system is built — rather than the final
+    (telemetry-wrapped) engine object.
+    """
+
+    @staticmethod
+    def _cfg(preferred, default):
+        # context_from_memory absent on .agent -> memory backend resolves to None
+        return SimpleNamespace(
+            intelligence=SimpleNamespace(preferred_engine=preferred),
+            engine=SimpleNamespace(default=default, ollama=SimpleNamespace(host="")),
+            agent=SimpleNamespace(),
+        )
+
+    def _capture_get_engine(self, monkeypatch):
+        captured = {}
+
+        def fake_get_engine(cfg, key):
+            captured["key"] = key
+            return ("resolved", MagicMock())
+
+        monkeypatch.setattr("openjarvis.engine._discovery.get_engine", fake_get_engine)
+        return captured
+
+    def test_resolves_preferred_engine_over_default(self, monkeypatch):
+        pytest.importorskip("fastapi")
+        from openjarvis.server import agent_manager_routes as amr
+
+        captured = self._capture_get_engine(monkeypatch)
+        amr._make_lightweight_system(
+            engine=MagicMock(), model="m", config=self._cfg("vllm", "ollama")
+        )
+        assert captured["key"] == "vllm"
+
+    def test_falls_back_to_engine_default_without_preference(self, monkeypatch):
+        pytest.importorskip("fastapi")
+        from openjarvis.server import agent_manager_routes as amr
+
+        captured = self._capture_get_engine(monkeypatch)
+        amr._make_lightweight_system(
+            engine=MagicMock(), model="m", config=self._cfg(None, "llamacpp")
+        )
+        assert captured["key"] == "llamacpp"

tests/server/test_api_routes.py

@@ -49,6 +49,54 @@ class TestMemoryRoutes:
         assert resp.status_code in (200, 500)
 
 
+class TestMemoryRustMissing:
+    """Regression for #502: when the native ``openjarvis_rust`` extension is
+    missing from the serving venv, memory ops must surface a CLEAR, ACTIONABLE
+    error — never the misleading "Failed to index path" or a 200 silent no-op.
+    """
+
+    @staticmethod
+    def _client(monkeypatch):
+        # Force the same failure mode as a venv without the compiled extension.
+        def _boom():
+            raise ImportError("No module named 'openjarvis_rust'")
+
+        import openjarvis._rust_bridge as bridge
+
+        monkeypatch.setattr(bridge, "get_rust_module", _boom)
+        return TestClient(_make_app())
+
+    def test_store_is_not_a_silent_noop(self, monkeypatch):
+        client = self._client(monkeypatch)
+        resp = client.post("/v1/memory/store", json={"content": "hi"})
+        # Must NOT return the old 200 {"status":"stored","note":"no backend..."}.
+        assert resp.status_code == 503
+        detail = resp.json()["detail"]
+        assert "openjarvis_rust" in detail
+        assert "maturin develop" in detail
+
+    def test_index_surfaces_actionable_detail(self, monkeypatch, tmp_path):
+        (tmp_path / "note.txt").write_text("hello world some content here")
+        client = self._client(monkeypatch)
+        resp = client.post("/v1/memory/index", json={"path": str(tmp_path)})
+        assert resp.status_code == 503
+        detail = resp.json()["detail"]
+        # The frontend reads this `detail`; it must point at the real cause,
+        # not blame the indexed path.
+        assert "openjarvis_rust" in detail
+        assert detail != "Failed to index path"
+        assert detail != "No memory backend available"
+
+    def test_config_reports_unavailable(self, monkeypatch):
+        client = self._client(monkeypatch)
+        resp = client.get("/v1/memory/config")
+        assert resp.status_code == 200
+        data = resp.json()
+        # Must not falsely report a healthy backend when none could be built.
+        assert data["available"] is False
+        assert "openjarvis_rust" in (data["detail"] or "")
+
+
 class TestBudgetRoutes:
     def test_get_budget(self):
         client = TestClient(_make_app())