feat(vision): add image + screen capture input for vision models (#486 )

OpenJarvis can run vision-capable local models (gemma3, qwen2.5-vl), but the CLI had no way to send them a picture -- the Ollama engine only serialized text. This adds end-to-end image input. What's new - `jarvis ask -i/--image <file>` attaches one or more images to the query. - `jarvis ask -S/--screen` captures the primary monitor (dependency-free on Windows via .NET; mss/Pillow fallback elsewhere). - Vision auto-routes to direct-to-engine mode; with an explicit --agent it warns rather than silently dropping the image. - Privacy guard: warns before sending an image to a non-local engine, keeping OpenJarvis local-first by default. - Context-window default raised 8k -> 16k (JARVIS_NUM_CTX) so an image plus a conversation fit. Implementation - Message.images carries base64 data; messages_to_dicts() forwards it to Ollama's /api/chat "images" field. Text-only messages are unchanged. - GuardrailsEngine preserves images when it rewrites a flagged message. Tests (tests/test_vision.py, 6/6 pass, ruff-clean) - payload forwarding, text path untouched, num_ctx override, guardrail image preservation. Verified on AMD RX 9070 XT (Ollama/Vulkan, 100% GPU) with gemma3:4b: solid-color image, file image, and live screen capture all described. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>
fix(gui): inject bearer token into streaming chat/research routes (#499 )
2026-08-14 17:02:01 +00:00 · 2026-06-14 18:19:00 -07:00 · 2026-06-14 17:32:43 -07:00 · 2026-06-14 07:38:47 +00:00 · 2026-06-13 07:21:38 +00:00 · 2026-06-12 07:41:03 +00:00
50 changed files with 3030 additions and 206 deletions
@@ -1,7 +1,7 @@
 {
  "schemaVersion": 1,
  "label": "Git Clones",
-  "message": "107,695",
+  "message": "117,047",
  "color": "green",
  "namedLogo": "git"
 }
@@ -1,6 +1,6 @@
 {
-  "total_clones": 107695,
-  "last_updated": "2026-06-10T07:32:26Z",
+  "total_clones": 117047,
+  "last_updated": "2026-06-14T07:38:47Z",
  "daily": {
    "2026-03-27": 2189,
    "2026-03-28": 1874,
@@ -75,6 +75,11 @@
    "2026-06-05": 2127,
    "2026-06-06": 2204,
    "2026-06-07": 1174,
-    "2026-06-08": 2369
+    "2026-06-08": 2369,
+    "2026-06-09": 1361,
+    "2026-06-10": 1310,
+    "2026-06-11": 2564,
+    "2026-06-12": 1313,
+    "2026-06-13": 2804
  }
 }
@@ -8,6 +8,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).

 ## [Unreleased]

+### Added
+
+**Vision input for `jarvis ask`** — attach images to a query with
+`-i`/`--image` (repeatable) or capture the current screen with
+`-S`/`--screen`, for vision-capable models such as `gemma3:4b`. Images flow
+through `Message.images` into Ollama's `/api/chat` `images` field; text-only
+requests are unaffected. A privacy guard warns before any image is sent to a
+non-local engine, and the security guardrail now preserves images when it
+sanitizes a flagged prompt. Screen capture uses the built-in Windows .NET
+stack with `mss`/`Pillow` fallbacks on other platforms. Adds the
+`JARVIS_NUM_CTX` environment variable to tune the Ollama context window
+(default `16384`).
+
 ## [1.0.2] - 2026-05-24

 A patch release that fixes a packaging bug which broke the v1.0.1
@@ -183,7 +183,7 @@ OpenJarvis is built around five composable layers. Each has a clean interface an

    ---

-    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), agents, memory, tools, and telemetry.
+    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), [Evaluations](user-guide/evaluations.md), agents, memory, tools, and telemetry.

 -   **[Architecture](architecture/overview.md)**

@@ -66,6 +66,8 @@ jarvis ask "What is the capital of France?"
 | `--no-context`                | flag    | off        | Disable memory context injection                       |
 | `-a`, `--agent AGENT`         | string  | none       | Agent to use (`simple`, `orchestrator`)                |
 | `--tools TOOLS`               | string  | none       | Comma-separated tool names to enable                   |
+| `-i`, `--image PATH`          | path    | none       | Image file for a vision model (e.g. `gemma3:4b`); repeatable |
+| `-S`, `--screen`              | flag    | off        | Capture the current screen and send it to the vision model  |

 ### Direct Mode vs Agent Mode

@@ -105,6 +107,39 @@ jarvis ask --no-context "Tell me about Python"
 jarvis ask --max-tokens 2048 "Write a detailed essay about AI"
 ```

+### Vision Input
+
+Vision-capable models (such as `gemma3:4b`) can read images alongside your
+text prompt. Attach one or more image files with `-i`/`--image`, or capture
+the current screen with `-S`/`--screen`:
+
+```bash
+# Ask about a local image
+jarvis ask -i screenshot.png "What is shown in this image?"
+
+# Send multiple images (the flag is repeatable)
+jarvis ask -i chart-a.png -i chart-b.png "Compare these two charts"
+
+# Capture the current screen and ask about it
+jarvis ask --screen "Summarize what's on my screen"
+```
+
+Vision runs in **direct mode** only. If you also pass `--agent`, the image is
+ignored and a note is printed — re-run with `--agent ""` to force direct mode.
+
+The Ollama context window can be tuned for large images or long prompts with
+the `JARVIS_NUM_CTX` environment variable (default `16384`):
+
+```bash
+JARVIS_NUM_CTX=8192 jarvis ask --screen "What's on my screen?"
+```
+
+!!! note "Keep vision on-device"
+    Images are sensitive. OpenJarvis prints a privacy warning before sending
+    an image to a non-local engine, so a screenshot never leaves your machine
+    unnoticed. Use a local engine (e.g. `ollama` with `gemma3:4b`) to keep
+    vision fully local.
+
 ### JSON Output Format

 When using `--json` in **direct mode**, the output includes:
@@ -1,14 +1,14 @@
 # Evaluations

-The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correctness and accuracy** on academic datasets. It is a separate package from the main OpenJarvis library and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
+The OpenJarvis evaluation framework (`openjarvis.evals`) measures model **correctness and accuracy** on academic datasets. It ships inside the main `openjarvis` package (at `src/openjarvis/evals/`) and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.

 !!! info "Evals vs. Benchmarks"
    OpenJarvis has two distinct measurement systems that complement each other:

-    | System | Package | Measures | Entry Point |
-    |--------|---------|----------|-------------|
-    | **Evaluations** | `openjarvis-evals` | Correctness on academic datasets (accuracy, pass rate) | `openjarvis-eval` |
-    | **Benchmarks** | `openjarvis` | Engine performance (latency, throughput) | `jarvis bench` |
+    | System | Module | Measures | Entry Point |
+    |--------|--------|----------|-------------|
+    | **Evaluations** | `openjarvis.evals` | Correctness on academic datasets (accuracy, pass rate) | `jarvis eval` |
+    | **Benchmarks** | `openjarvis.bench` | Engine performance (latency, throughput) | `jarvis bench` |

    Use evaluations to answer "does this model get the right answer?" and benchmarks to answer "how fast does this model respond?". See the [Benchmarks guide](benchmarks.md) for the performance measurement system.

@@ -18,22 +18,38 @@ The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correc

 ## Installation

-The evaluation framework is a standalone package in the `evals/` directory. Install it alongside OpenJarvis:
+The evaluation framework is part of the main `openjarvis` package — no separate install or extra is required. The standard dev setup is enough:

 ```bash
-uv sync --extra eval
+uv sync --extra dev
 ```

-This installs the `openjarvis-eval` CLI entry point and all required dependencies (`datasets`, `huggingface-hub`, `tqdm`, `rich`).
+The framework's core dependencies (`click`, `datasets`, `rich`) are base dependencies of `openjarvis`. Two optional extras enable experiment tracking integrations:
+
+```bash
+uv sync --extra dev --extra eval-wandb     # Weights & Biases run tracking
+uv sync --extra dev --extra eval-sheets    # Google Sheets results export
+```

 !!! note "Python version requirement"
-    Python 3.10 requires the `tomli` package for TOML config parsing. The `evals/pyproject.toml` includes this as a conditional dependency, so it is installed automatically.
+    Python 3.10 requires the `tomli` package for TOML config parsing. `openjarvis` declares it as a conditional dependency, so it is installed automatically.
+
+## Entry Points
+
+Two equivalent entry points expose the framework:
+
+| Command | Surface |
+|---------|---------|
+| `jarvis eval {list,run,compare,report}` | Canonical CLI. `run` covers the common options; `compare` and `report` post-process result files. |
+| `python -m openjarvis.evals {list,run,run-all,summarize,reparse-judge}` | Full research surface, including judge configuration, the agentic runner, and episode mode. |
+
+The `openjarvis-eval` console script is an alias for `python -m openjarvis.evals` — same commands, same options. This guide uses `jarvis eval` wherever its option set suffices and the module form for research-only options.

 ---

 ## Datasets

-The framework ships with **30+ datasets** covering academic reasoning, agentic tasks, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below.
+The framework ships with **40 registered benchmarks** covering academic reasoning, agentic tasks, coding, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below; `uv run python -m openjarvis.evals list` prints the authoritative registry.

 ### Use-Case Benchmarks

@@ -64,6 +80,7 @@ These benchmarks measure reasoning and knowledge on established academic dataset
 | **MATH-500** | `math500` | reasoning | Competition-level math problems |
 | **NaturalReasoning** | `natural-reasoning` | reasoning | Natural language reasoning |
 | **HLE** | `hle` | reasoning | Humanity's Last Exam hard challenges |
+| **LiveResearchBench** | `liveresearchbench` | reasoning | Recent research comprehension (Salesforce) |
 | **SimpleQA** | `simpleqa` | chat | Short-form factual question answering |
 | **IPW** | `ipw` | chat | Intelligence Per Watt mixed benchmark |

@@ -79,6 +96,11 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **TerminalBench** | `terminalbench` | agentic | Terminal-based task completion |
 | **TerminalBench Native** | `terminalbench-native` | agentic | TerminalBench with native Docker execution |
 | **TerminalBench V2.1** | `terminalbench-v2.1` | agentic | TB v2.1 Harbor-style Docker tasks |
+| **PinchBench** | `pinchbench` | agentic | Real-world agent tasks |
+| **TauBench** | `taubench` | agentic | Multi-turn customer service |
+| **DeepResearchBench** | `liveresearch` | agentic | Deep research report generation |
+| **DeepResearchBench (alias)** | `deepresearch` | agentic | Same benchmark as `liveresearch` |
+| **ToolCall-15** | `toolcall15` | agentic | Tool calling benchmark |
 | **LifelongAgent** | `lifelong-agent` | agentic | Sequential task learning across sessions |
 | **PaperArena** | `paperarena` | agentic | Scientific paper analysis |
 | **DeepPlanning** | `deepplanning` | agentic | Shopping constraint planning |
@@ -87,6 +109,14 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **WebChoreArena** | `webchorearena` | agentic | Web chore tasks |
 | **WorkArena** | `workarena` | agentic | WorkArena++ enterprise workflows |

+Both `liveresearch` and `deepresearch` are registered keys for the DeepResearchBench report-generation benchmark.
+
+### Coding Benchmarks
+
+| Dataset | Key | Category | Description |
+|---------|-----|----------|-------------|
+| **LiveCodeBench** | `livecodebench` | coding | Competitive programming |
+
 ### Retrieval Benchmarks

 | Dataset | Key | Category | Description |
@@ -123,7 +153,7 @@ The framework includes two pre-built configs for evaluating models on the five c
 ### Cloud models

 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
 ```

 This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, GPT-5.4, GPT-5 Mini) against all 5 use-case benchmarks with 30 samples each, producing a 6x5 = 30-run matrix. Results are written to `results/use-cases-v2-cloud/`.
@@ -131,7 +161,7 @@ This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gem
 ### Local models

 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_local.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_local.toml
 ```

 This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS 120B, GLM4, Qwen3.5 35B-A3B, GLM-4.7-Flash) against the same 5 benchmarks, producing a 5x5 = 25-run matrix. Uses 2 workers (suitable for single-GPU setups). Results are written to `results/use-cases-v2-local/`.
@@ -143,15 +173,22 @@ This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS

 ## Inference Backends

-Every evaluation run routes model calls through one of two backends:
+Every evaluation run routes model calls through one of four backends:

 | Backend | Key | Description |
 |---------|-----|-------------|
 | **jarvis-direct** | `jarvis-direct` | Engine-level inference via `SystemBuilder`. Works for local (Ollama, vLLM, llama.cpp) and cloud models. |
 | **jarvis-agent** | `jarvis-agent` | Agent-level inference with tool calling. Uses `JarvisSystem.ask()` with the specified agent and tools. |
+| **hermes** | `hermes` | Real Hermes Agent (Nous Research) via subprocess. Requires `--base-url` and `--api-key`. |
+| **openclaw** | `openclaw` | Real OpenClaw via Node subprocess. Requires `--base-url` and `--api-key`. |

 Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark requires tool use — for example, GAIA tasks that reference files that must be read with `file_read`, or arithmetic tasks that benefit from `calculator`.

+The `hermes` and `openclaw` backends shell out to external agent frameworks and need an OpenAI-compatible endpoint for their model calls: pass `--base-url`/`--api-key`, set the `JARVIS_BACKEND_BASE_URL`/`JARVIS_BACKEND_API_KEY` environment variables, or add a `[backend.external]` section to your config (see [Config Reference](#backendexternal)).
+
+!!! note "TerminalBench Native"
+    `jarvis eval run --backend` additionally accepts `terminalbench-native`, a Docker-based execution backend used by the TerminalBench Native benchmark.
+
 ---

 ## CLI Usage
@@ -159,73 +196,106 @@ Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark
 ### List available benchmarks and backends

 ```bash
-openjarvis-eval list
+uv run python -m openjarvis.evals list
 ```

-Output:
+Abridged output (40 benchmarks, 4 backends):

 ```
-Benchmarks:
-  supergpqa    [reasoning  ] SuperGPQA multiple-choice
-  gaia         [agentic    ] GAIA agentic benchmark
-  frames       [rag        ] FRAMES multi-hop RAG
-  wildchat     [chat       ] WildChat conversation quality
-
-Backends:
-  jarvis-direct    Engine-level inference (local or cloud)
-  jarvis-agent     Agent-level inference with tool calling
+                         Available Benchmarks
+┌──────────────────────┬───────────┬───────────────────────────────────┐
+│ Name                 │ Category  │ Description                       │
+├──────────────────────┼───────────┼───────────────────────────────────┤
+│ supergpqa            │ reasoning │ SuperGPQA multiple-choice         │
+│ gpqa                 │ reasoning │ GPQA graduate-level MCQ           │
+│ ...                  │ ...       │ ...                               │
+│ livecodebench        │ coding    │ LiveCodeBench competitive progr.  │
+│ toolcall15           │ agentic   │ ToolCall-15 tool calling benchmark│
+└──────────────────────┴───────────┴───────────────────────────────────┘
+                         Available Backends
+┌───────────────┬──────────────────────────────────────────────────┐
+│ jarvis-direct │ Engine-level inference (local or cloud)          │
+│ jarvis-agent  │ Agent-level inference with tool calling          │
+│ hermes        │ Real Hermes Agent (Nous Research) via subprocess │
+│ openclaw      │ Real OpenClaw via Node subprocess                │
+└───────────────┴──────────────────────────────────────────────────┘
 ```

+`jarvis eval list` prints a similar table but currently shows a curated subset of the registry; the module form above is the authoritative listing.
+
 ### Run a single benchmark

 ```bash
-# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples default)
-openjarvis-eval run -b supergpqa -m qwen3:8b
+# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples)
+uv run jarvis eval run -b supergpqa -m qwen3:8b -n 10

-# Evaluate GPT-4o on GAIA using the agent backend with tools
-openjarvis-eval run -b gaia -m gpt-4o --backend jarvis-agent \
+# Evaluate GPT-5 Mini on GAIA using the agent backend with tools
+uv run jarvis eval run -b gaia -m gpt-5-mini --backend jarvis-agent \
    --agent orchestrator --tools calculator,file_read -n 50

-# Run FRAMES with vLLM engine, write output to a file
-openjarvis-eval run -b frames -m llama3:70b -e vllm \
+# Run FRAMES with the vLLM engine, write output to a file
+uv run jarvis eval run -b frames -m llama3:70b -e vllm \
    -o results/frames_llama70b.jsonl

 # Run WildChat with a higher temperature for chat quality
-openjarvis-eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
+uv run jarvis eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
 ```

-#### Full option reference
+#### `jarvis eval run` option reference

 | Option | Short | Type | Default | Description |
 |--------|-------|------|---------|-------------|
 | `--config` | `-c` | path | — | TOML config file; when provided, `-b` and `-m` are not required |
-| `--benchmark` | `-b` | choice | required* | `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `--backend` | | choice | `jarvis-direct` | `jarvis-direct` or `jarvis-agent` |
-| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-4o`) |
-| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
-| `--agent` | | str | `orchestrator` | Agent name for `jarvis-agent` backend |
-| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--benchmark` | `-b` | str | required* | Any registered benchmark key (see `... list`) |
+| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-5-mini`) |
 | `--max-samples` | `-n` | int | all | Limit the number of samples evaluated |
-| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
-| `--judge-model` | | str | `gpt-4o` | LLM used for judge-based scoring |
-| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--backend` | | choice | `jarvis-direct` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
+| `--base-url` | | str | — | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `--api-key` | | str | — | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
+| `--agent` | | str | — | Agent name for `jarvis-agent` backend (e.g., `orchestrator`) |
+| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
+| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--telemetry/--no-telemetry` | | flag | off | Enable telemetry collection during eval |
+| `--gpu-metrics/--no-gpu-metrics` | | flag | off | Enable GPU metric polling |
 | `--seed` | | int | `42` | Random seed for dataset shuffling |
-| `--split` | | str | dataset default | Override the dataset split |
 | `--temperature` | | float | `0.0` | Generation temperature |
 | `--max-tokens` | | int | `2048` | Maximum output tokens |
+| `--model-filter` | | str | — | Filter models by name substring (multi-model configs) |
+| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--wandb-project` / `--wandb-entity` / `--wandb-tags` / `--wandb-group` | | str | `""` | Weights & Biases tracking (requires `eval-wandb` extra) |
+| `--sheets-id` / `--sheets-worksheet` / `--sheets-creds` | | str | `""` | Google Sheets export (requires `eval-sheets` extra) |
 | `--verbose` | `-v` | flag | off | Enable debug logging |

 *Required when `--config` is not provided.

+#### Research-only options (`python -m openjarvis.evals run`)
+
+The module CLI accepts everything above plus research-grade options that `jarvis eval run` does not expose:
+
+| Option | Short | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
+| `--judge-model` | | str | `gpt-5-mini-2025-08-07` | LLM used for judge-based scoring (see `--help` for the current default) |
+| `--judge-engine` | | str | `cloud` | Engine key for the LLM judge; use `vllm` to judge locally |
+| `--split` | | str | dataset default | Override the dataset split |
+| `--compact` | | flag | off | Dense single-table output |
+| `--trace-detail` | | flag | off | Full per-step trace listing |
+| `--agentic` | | flag | off | Use `AgenticRunner` for multi-turn agent execution |
+| `--episode-mode` | | flag | off | Sequential episode processing with lifelong learning (required for `lifelong-agent` and similar benchmarks) |
+| `--concurrency` | | int | `1` | Parallel query execution (AgenticRunner only) |
+| `--query-timeout` | | float | — | Per-query wall-clock timeout in seconds (AgenticRunner only) |
+
+Note: the module CLI's `--backend` choice covers `jarvis-direct`, `jarvis-agent`, `hermes`, and `openclaw`; `terminalbench-native` as a backend is available via `jarvis eval run` and TOML configs.
+
 ### Run all benchmarks at once

-The `run-all` command evaluates a single model against all four benchmarks sequentially and writes results to an output directory:
+The `run-all` command (module CLI only) evaluates a single model against **every registered benchmark** sequentially and writes results to an output directory:

 ```bash
-openjarvis-eval run-all -m qwen3:8b
+uv run python -m openjarvis.evals run-all -m qwen3:8b

 # With options
-openjarvis-eval run-all -m gpt-4o -n 100 --output-dir results/gpt4o/
+uv run python -m openjarvis.evals run-all -m gpt-5-mini -n 100 --output-dir results/gpt5mini/
 ```

 Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The model slug replaces `/` and `:` with `-`, so `qwen3:8b` becomes `qwen3-8b`.
@@ -235,7 +305,7 @@ Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The m
 After a run, inspect a JSONL results file:

 ```bash
-openjarvis-eval summarize results/supergpqa_qwen3-8b.jsonl
+uv run python -m openjarvis.evals summarize results/supergpqa_qwen3-8b.jsonl
 ```

 Output:
@@ -251,6 +321,55 @@ Accuracy:  0.7222
 Errors:    2
 ```

+The module CLI also provides `reparse-judge`, which re-parses stored judge output in a results file and recovers records whose judge verdicts initially failed to parse — useful after improving the judge-output parser without re-running inference.
+
+### Compare and report
+
+`jarvis eval` adds two post-processing commands for result files:
+
+```bash
+# Side-by-side metric comparison across runs
+uv run jarvis eval compare results/supergpqa_qwen3-8b.jsonl results/supergpqa_gpt-5-mini.jsonl
+
+# Detailed report (accuracy, latency, cost, per-subject breakdown) for one run
+uv run jarvis eval report results/supergpqa_qwen3-8b.jsonl
+```
+
+---
+
+## Evaluating an Already-Running Endpoint
+
+If you already have an OpenAI-compatible server running — `jarvis serve`, vLLM, SGLang, llama.cpp's server, or a hosted endpoint — point an eval directly at it with `--base-url` and `--api-key`:
+
+```bash
+# A vLLM server is already serving Qwen/Qwen3-8B on a GPU node:
+#   vllm serve Qwen/Qwen3-8B --port 8000
+uv run jarvis eval run -b supergpqa -m Qwen/Qwen3-8B \
+    --base-url http://gpu-node:8000/v1 \
+    --api-key local-key \
+    -n 50
+```
+
+The `-m` value must match a model id the server reports at `GET /v1/models`. Both flags fall back to the `JARVIS_BACKEND_BASE_URL` and `JARVIS_BACKEND_API_KEY` environment variables, so CI jobs can set them once:
+
+```bash
+export JARVIS_BACKEND_BASE_URL=http://gpu-node:8000/v1
+export JARVIS_BACKEND_API_KEY=local-key
+uv run jarvis eval run -b gaia -m Qwen/Qwen3-8B --backend jarvis-agent -n 25
+```
+
+For the external `hermes` and `openclaw` backends these values are **required** (the foreign frameworks need an endpoint to send model calls to).
+
+!!! tip "Engine-level alternative for vLLM"
+    The vLLM engine also honors the `VLLM_HOST` environment variable (default `http://localhost:8000`):
+
+    ```bash
+    VLLM_HOST=http://gpu-node:8000 uv run python -m openjarvis.evals run \
+        -b supergpqa -m Qwen/Qwen3-8B -e vllm -n 50
+    ```
+
+    `VLLM_HOST` is process-global — if the candidate and the judge both use the `vllm` engine, they share the same endpoint. Prefer `--base-url` when you need them separate.
+
 ---

 ## TOML Config System
@@ -260,7 +379,7 @@ For research workflows that compare multiple models across multiple benchmarks,
 ### Running from a config

 ```bash
-openjarvis-eval run --config src/openjarvis/evals/configs/full-suite.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/full-suite.toml
 ```

 When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options are not required. All settings come from the config file. The CLI expands the matrix, prints a progress table, and writes results to the configured `output_dir`.
@@ -269,7 +388,7 @@ When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options a

 A config file has six sections: `[meta]`, `[defaults]`, `[judge]`, `[run]`, `[[models]]`, and `[[benchmarks]]`. Only `[[models]]` and `[[benchmarks]]` are required — all other sections are optional and fall back to built-in defaults.

-```toml title="evals/configs/full-suite.toml"
+```toml title="src/openjarvis/evals/configs/full-suite.toml"
 # Suite-level metadata (optional)
 [meta]
 name = "full-suite-v1"
@@ -353,7 +472,7 @@ For example, `temperature` is resolved as: use `[defaults].temperature` (0.0), t

 A config requires only one `[[models]]` and one `[[benchmarks]]` entry:

-```toml title="evals/configs/minimal.toml"
+```toml title="src/openjarvis/evals/configs/minimal.toml"
 [[models]]
 name = "qwen3:8b"

@@ -365,7 +484,7 @@ This runs SuperGPQA against qwen3:8b with all default settings. Use this as a st

 ### Single-run config with full options

-```toml title="evals/configs/single-run.toml"
+```toml title="src/openjarvis/evals/configs/single-run.toml"
 [meta]
 name = "single-run-example"
 description = "Evaluate SuperGPQA with a single model and full configuration"
@@ -425,7 +544,8 @@ Configuration for the LLM used as a judge in GAIA, FRAMES, and WildChat scoring.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `model` | str | `"gpt-4o"` | Judge model identifier |
+| `model` | str | `"gpt-5-mini-2025-08-07"` | Judge model identifier |
+| `engine` | str | `None` | Engine key for the judge (e.g., `"vllm"` to judge locally; defaults to cloud) |
 | `provider` | str | `None` | Provider override (e.g., `"openai"`) |
 | `temperature` | float | `0.0` | Judge sampling temperature |
 | `max_tokens` | int | `1024` | Maximum judge output tokens |
@@ -444,6 +564,20 @@ Execution settings that apply to the entire suite.
 | `seed` | int | `42` | Random seed for dataset shuffling |
 | `telemetry` | bool | `false` | Enable GPU telemetry capture (energy, power, utilization, throughput) |
 | `gpu_metrics` | bool | `false` | Enable GPU metric polling via `pynvml` (requires `pynvml` or `nvidia-ml-py`) |
+| `warmup_samples` | int | `0` | Untimed warmup samples before measurement |
+| `energy_vendor` | str | `""` | GPU energy vendor override |
+| `max_turns` | int | `None` | Maximum agent turns per query |
+| `wandb_project` / `wandb_entity` / `wandb_tags` / `wandb_group` | str | `""` | Weights & Biases tracking |
+| `sheets_spreadsheet_id` / `sheets_worksheet` / `sheets_credentials_path` | str | `""` / `"Results"` / `""` | Google Sheets export |
+
+### `[backend.external]`
+
+Endpoint settings for the `hermes` and `openclaw` backends. Environment variables override TOML values.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `base_url` | str | `None` | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `api_key` | str | `None` | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |

 ### `[[models]]`

@@ -451,7 +585,7 @@ One block per model. The `name` field is required.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-4o"`) |
+| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-5-mini"`) |
 | `engine` | str | `None` | Engine key to use (`"ollama"`, `"vllm"`, `"cloud"`, ...) |
 | `provider` | str | `None` | Provider override for cloud models (e.g., `"openai"`) |
 | `temperature` | float | `None` | Override `[defaults].temperature` for this model |
@@ -468,10 +602,12 @@ One block per benchmark. The `name` field is required.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Benchmark key: `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `backend` | str | `"jarvis-direct"` | Inference backend: `jarvis-direct` or `jarvis-agent` |
+| `name` | str | required | Any registered benchmark key (see `uv run python -m openjarvis.evals list`) |
+| `backend` | str | `"jarvis-direct"` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
 | `max_samples` | int | `None` | Limit number of samples; `None` evaluates the full dataset |
 | `split` | str | `None` | Override the default dataset split |
+| `subset` | str | `None` | Dataset subset/variant (benchmark-specific) |
+| `record_ids` | list[str] | `None` | Evaluate only these record ids |
 | `agent` | str | `None` | Agent name for `jarvis-agent` backend (e.g., `"orchestrator"`) |
 | `tools` | list[str] | `[]` | Tool names for `jarvis-agent` backend |
 | `judge_model` | str | `None` | Override `[judge].model` for this benchmark only |
@@ -647,7 +783,7 @@ The `EvalRunner` processes samples concurrently using a `ThreadPoolExecutor`. Re

 ```bash
 # Use more workers for faster evaluation (if the engine supports concurrent requests)
-openjarvis-eval run -b supergpqa -m qwen3:8b -w 8 -n 500
+uv run python -m openjarvis.evals run -b supergpqa -m qwen3:8b -w 8 -n 500
 ```

 !!! warning "Worker count and engine load"
@@ -1540,19 +1540,89 @@ async fn fetch_models(api_url: String) -> Result<serde_json::Value, String> {

 #[tauri::command]
 async fn run_jarvis_command(args: Vec<String>) -> Result<String, String> {
-    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
-    cmd_args.extend(args);
    let uv_bin = resolve_bin("uv");
-    let output = tokio::process::Command::new(&uv_bin)
-        .args(&cmd_args)
-        .output()
-        .await
-        .map_err(|e| format!("Failed to launch jarvis: {}", e))?;

-    if output.status.success() {
-        Ok(String::from_utf8_lossy(&output.stdout).to_string())
-    } else {
-        Err(String::from_utf8_lossy(&output.stderr).to_string())
+    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
+    cmd_args.extend(args.iter().cloned());
+
+    let mut cmd = tokio::process::Command::new(&uv_bin);
+    cmd.args(&cmd_args);
+    // Run from the project root so `uv run jarvis` resolves the OpenJarvis
+    // project regardless of the app's launch cwd. In a packaged install the
+    // cwd isn't the checkout, so without this `jarvis` isn't found and the
+    // backend never starts — the UI then shows "Failed to get response"
+    // (see #531).
+    if let Some(ref root) = find_project_root() {
+        cmd.current_dir(root);
+    }
+
+    let is_serve = args.first().map(|a| a.as_str() == "serve").unwrap_or(false);
+
+    if !is_serve {
+        // Short-lived command (e.g. `stop`, `status`): wait for it and return
+        // its captured output.
+        let output = cmd
+            .output()
+            .await
+            .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
+        return if output.status.success() {
+            Ok(String::from_utf8_lossy(&output.stdout).to_string())
+        } else {
+            Err(String::from_utf8_lossy(&output.stderr).to_string())
+        };
+    }
+
+    // `jarvis serve` is a long-running server that never exits. The old code
+    // used `.output()`, which waits for the process to exit and so hung this
+    // command forever — the "Start" button never resolved (#531). Spawn it
+    // detached instead, drain stderr (a full 4 KB Windows pipe can otherwise
+    // stall the child mid-startup, #309), and poll /health for readiness.
+    cmd.stdout(std::process::Stdio::null())
+        .stderr(std::process::Stdio::piped());
+    let mut child = cmd
+        .spawn()
+        .map_err(|e| format!("Failed to launch jarvis serve: {}", e))?;
+
+    let tail: StderrTail = Arc::new(Mutex::new(Vec::new()));
+    if let Some(stderr) = child.stderr.take() {
+        spawn_jarvis_stderr_drainer(stderr, tail.clone());
+    }
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(2))
+        .build()
+        .map_err(|e| format!("Failed to build HTTP client: {}", e))?;
+    let url = format!("http://127.0.0.1:{}/health", JARVIS_PORT);
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(120);
+
+    loop {
+        // Surface an early crash (bad venv, missing Rust ext, etc.) right away
+        // instead of waiting out the full readiness timeout.
+        if let Ok(Some(status)) = child.try_wait() {
+            let stderr = String::from_utf8_lossy(tail.lock().await.as_slice()).into_owned();
+            return Err(format!(
+                "jarvis serve exited (code {:?}) before becoming healthy:\n{}",
+                status.code(),
+                stderr.trim()
+            ));
+        }
+        if let Ok(resp) = client.get(&url).send().await {
+            if resp.status().is_success() {
+                // Leave the server running (the Child is detached on drop —
+                // kill_on_drop defaults to false); `stop` tears it down.
+                return Ok(format!(
+                    "jarvis serve is ready on http://127.0.0.1:{}",
+                    JARVIS_PORT
+                ));
+            }
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!(
+                "jarvis serve did not become healthy on port {} within 120s.",
+                JARVIS_PORT
+            ));
+        }
+        tokio::time::sleep(Duration::from_millis(500)).await;
    }
 }

@@ -1,5 +1,5 @@
 import type { ResearchEvent, SSEEvent } from '../types';
-import { getBase } from './api';
+import { getBase, authHeaders } from './api';

 export interface ChatRequest {
  model: string;
@@ -16,7 +16,7 @@ export async function* streamChat(
  const base = getBase();
  const response = await fetch(`${base}/v1/chat/completions`, {
    method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
+    headers: authHeaders({ 'Content-Type': 'application/json' }),
    body: JSON.stringify(request),
    signal,
  });
@@ -67,7 +67,7 @@ export async function* streamResearch(
  const base = getBase().replace(/\/v1\/?$/, '');
  const response = await fetch(`${base}/api/research`, {
    method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
+    headers: authHeaders({ 'Content-Type': 'application/json' }),
    body: JSON.stringify({ query }),
    signal,
  });
@@ -106,3 +106,4 @@ export async function* streamResearch(
    reader.releaseLock();
  }
 }
+
@@ -193,6 +193,8 @@ nav:
      - External MCP Servers: user-guide/mcp-external-servers.md
      - Scheduler: user-guide/scheduler.md
      - Telemetry: user-guide/telemetry.md
+      - Evaluations: user-guide/evaluations.md
+      - Benchmarks: user-guide/benchmarks.md
      - Security: user-guide/security.md
      - LLM-guided spec search: user-guide/llm-guided-spec-search.md
  - Leaderboard: leaderboard.md
@@ -152,6 +152,7 @@ Issues = "https://github.com/open-jarvis/OpenJarvis/issues"

 [project.scripts]
 jarvis = "openjarvis.cli:main"
+openjarvis-eval = "openjarvis.evals.cli:main"

 [tool.hatch.build.targets.wheel]
 packages = ["src/openjarvis"]
@@ -0,0 +1,79 @@
+"""Screen capture for vision input (``jarvis ask --screen``).
+
+Captures the primary monitor to a temporary PNG so it can be handed to a
+vision-capable model. On Windows this uses the built-in .NET
+``System.Drawing`` stack (no third-party dependency). Other platforms fall
+back to ``mss`` or ``Pillow`` if installed.
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+import tempfile
+
+# PowerShell: capture the PRIMARY monitor (more legible for a vision model
+# than a downscaled multi-monitor grab). {path} is filled in with forward
+# slashes, which .NET accepts on Windows and which avoids backslash escaping.
+_PS_CAPTURE = """
+Add-Type -AssemblyName System.Windows.Forms, System.Drawing
+$b = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds
+$bmp = New-Object System.Drawing.Bitmap($b.Width, $b.Height)
+$g = [System.Drawing.Graphics]::FromImage($bmp)
+$g.CopyFromScreen($b.X, $b.Y, 0, 0, $bmp.Size)
+$bmp.Save("{path}", [System.Drawing.Imaging.ImageFormat]::Png)
+$g.Dispose(); $bmp.Dispose()
+"""
+
+
+def capture_screen_to_temp() -> str:
+    """Capture the screen to a temp PNG and return its absolute path.
+
+    Raises ``RuntimeError`` with actionable guidance if capture fails or the
+    platform has no available backend.
+    """
+    fd, path = tempfile.mkstemp(prefix="jarvis_screen_", suffix=".png")
+    os.close(fd)
+
+    if sys.platform.startswith("win"):
+        script = _PS_CAPTURE.replace("{path}", path.replace("\\", "/"))
+        proc = subprocess.run(
+            ["powershell", "-NoProfile", "-NonInteractive", "-Command", script],
+            capture_output=True,
+            text=True,
+            timeout=30,
+        )
+        if (
+            proc.returncode != 0
+            or not os.path.exists(path)
+            or not os.path.getsize(path)
+        ):
+            raise RuntimeError(
+                "screen capture failed: "
+                + (proc.stderr.strip() or "empty image written")
+            )
+        return path
+
+    # Non-Windows: optional backends.
+    try:
+        import mss  # type: ignore
+
+        with mss.mss() as sct:
+            sct.shot(mon=-1, output=path)
+        return path
+    except ImportError:
+        pass
+    try:
+        from PIL import ImageGrab  # type: ignore
+
+        ImageGrab.grab().save(path)
+        return path
+    except Exception as exc:  # noqa: BLE001
+        raise RuntimeError(
+            "screen capture on this platform needs 'mss' or 'Pillow' "
+            "(try: pip install mss)"
+        ) from exc
+
+
+__all__ = ["capture_screen_to_temp"]
@@ -2,6 +2,7 @@

 from __future__ import annotations

+import base64
 import json as json_mod
 import logging
 import sys
@@ -619,6 +620,21 @@ def _print_profile(
        "(default: ~/.openjarvis/knowledge.db)."
    ),
 )
+@click.option(
+    "-i",
+    "--image",
+    "image_paths",
+    multiple=True,
+    type=click.Path(exists=True, dir_okay=False),
+    help="Image file for a vision model (e.g. gemma3). Repeatable.",
+)
+@click.option(
+    "-S",
+    "--screen",
+    "capture_screen",
+    is_flag=True,
+    help="Capture the current screen and send it to the vision model.",
+)
@click.option(
    "--persona",
    "persona_name",
@@ -645,6 +661,8 @@ def ask(
    research_mode: bool,
    knowledge_db: str | None,
    persona_name: str | None,
+    image_paths: tuple[str, ...] = (),
+    capture_screen: bool = False,
 ) -> None:
    """Ask Jarvis a question."""
    quiet = (ctx.obj or {}).get("quiet", False) or output_json
@@ -652,6 +670,27 @@ def ask(
    console = Console(stderr=True)
    query_text = " ".join(query)

+    # Vision: collect base64 images from --image files and/or --screen.
+    image_b64: list[str] = []
+    for _img_path in image_paths:
+        try:
+            with open(_img_path, "rb") as _fh:
+                image_b64.append(base64.b64encode(_fh.read()).decode("ascii"))
+        except OSError as exc:
+            console.print(f"[red]Could not read image {_img_path}: {exc}[/red]")
+            sys.exit(1)
+    if capture_screen:
+        try:
+            from openjarvis.cli._screen import capture_screen_to_temp
+
+            _shot = capture_screen_to_temp()
+            with open(_shot, "rb") as _fh:
+                image_b64.append(base64.b64encode(_fh.read()).decode("ascii"))
+            logger.debug("Captured screen to %s", _shot)
+        except Exception as exc:  # noqa: BLE001
+            console.print(f"[red]Screen capture failed:[/red] {exc}")
+            sys.exit(1)
+
    wall_start = time.monotonic() if enable_profile else None

    # Load config
@@ -671,11 +710,26 @@ def ask(
    # Without this fallback, `[agent].default_system_prompt` and the
    # SOUL.md / MEMORY.md / USER.md persona system are silently bypassed for
    # the most common command (`jarvis ask "..."`).
+    agent_explicitly_set = agent_name is not None
    if agent_name is None:
        configured_default = (config.agent.default_agent or "").strip()
        if configured_default:
            agent_name = configured_default

+    # Vision flows only through direct-to-engine mode. If an image/screenshot
+    # was supplied without an explicit --agent, route to direct mode so the
+    # picture reaches the model; if an agent was explicitly requested, say
+    # plainly that the image is being skipped rather than dropping it silently.
+    if image_b64:
+        if not agent_explicitly_set:
+            agent_name = ""
+        else:
+            console.print(
+                "[yellow]Note:[/yellow] --image/--screen only works in direct "
+                "mode; the image is ignored with --agent set. Re-run with "
+                '`--agent ""` to use vision.'
+            )
+
    # Track whether the user explicitly set --max-tokens
    user_set_max_tokens = max_tokens is not None

@@ -714,7 +768,13 @@ def ask(
    register_builtin_models()

    effective_engine_key = engine_key or config.intelligence.preferred_engine or None
-    resolved = get_engine(config, effective_engine_key)
+    # Pass the model we intend to run so engine selection can skip an engine
+    # that can't actually serve it (e.g. the cloud fallback when the local
+    # engine is down but only a non-OpenAI key is set — see #532). This is the
+    # -m flag or the configured default; when neither is set we leave it None
+    # and a model is chosen per-engine below.
+    selection_model = model_name or config.intelligence.default_model or None
+    resolved = get_engine(config, effective_engine_key, model=selection_model)
    if resolved is None:
        console.print(
            "[red bold]No inference engine available.[/red bold]\n\n"
@@ -865,6 +925,27 @@ def ask(
        return

    # Direct-to-engine mode (no agent)
+    # Privacy guard: a screenshot/image is sensitive, and OpenJarvis is
+    # local-first. If the active engine isn't local, warn before the image
+    # leaves the machine rather than silently uploading it to a third party.
+    _LOCAL_ENGINES = {
+        "ollama",
+        "llamacpp",
+        "vllm",
+        "sglang",
+        "exo",
+        "nexa",
+        "uzu",
+        "apple_fm",
+        "gemma_cpp",
+    }
+    if image_b64 and engine_name not in _LOCAL_ENGINES:
+        console.print(
+            f"[yellow]Privacy warning:[/yellow] sending {len(image_b64)} "
+            f"image(s) to a non-local engine ('{engine_name}'). The image will "
+            "leave this machine. Use a local engine (e.g. ollama) to keep "
+            "vision on-device."
+        )
    messages = [Message(role=Role.USER, content=query_text)]

    # Memory-augmented context injection
@@ -891,6 +972,15 @@ def ask(
        except Exception as exc:
            logger.debug("Failed to inject memory context: %s", exc)

+    # Vision: attach images to the final user message *after* any context
+    # injection (which may rebuild the list). messages_to_dicts() forwards
+    # the "images" field to Ollama's /api/chat.
+    if image_b64:
+        for _m in reversed(messages):
+            if _m.role == Role.USER:
+                _m.images = image_b64
+                break
+
    # Generate (InstrumentedEngine handles telemetry + energy recording)
    try:
        with console.status("[bold green]Generating...[/bold green]"):
@@ -332,7 +332,7 @@ def compose_bench(
        for i, rc in enumerate(run_configs, 1):
            console.print(f"\n[bold]Run {i}/{len(run_configs)}:[/bold] {rc.benchmark}")
            try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                results_table.add_row(
                    rc.benchmark,
                    f"{summary.accuracy:.4f}",
@@ -61,6 +61,12 @@ KNOWN_BENCHMARKS = {
 KNOWN_BACKENDS = {
    "jarvis-direct": "Engine-level inference (local or cloud)",
    "jarvis-agent": "Agent-level inference with tool calling",
+    "hermes": "Real Hermes Agent (Nous Research) via subprocess",
+    "openclaw": "Real OpenClaw via Node subprocess",
+    "terminalbench-native": (
+        "TerminalBench V2.1 via terminal-bench Harness "
+        "(selected with -b terminalbench-native)"
+    ),
 }


@@ -146,7 +152,9 @@ def eval_list() -> None:
    "base_url",
    default=None,
    help=(
-        "OpenAI-compat endpoint URL for hermes/openclaw backends "
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
        "(env: JARVIS_BACKEND_BASE_URL)."
    ),
 )
@@ -154,7 +162,11 @@ def eval_list() -> None:
    "--api-key",
    "api_key",
    default=None,
-    help=("API key for the hermes/openclaw endpoint (env: JARVIS_BACKEND_API_KEY)."),
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
@click.option(
    "--agent",
@@ -347,7 +359,7 @@ def eval_run(
                f"{rc.benchmark} / {rc.model}"
            )
            try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                console.print(
                    f"  [green]{summary.accuracy:.4f}[/green] "
                    f"({summary.correct}/{summary.scored_samples})"
@@ -399,8 +411,10 @@ def eval_run(
        sheets_spreadsheet_id=sheets_spreadsheet_id,
        sheets_worksheet=sheets_worksheet,
        sheets_credentials_path=sheets_credentials_path,
-        # Spec §6.2 — for hermes/openclaw external backends. Falls back to env vars
-        # so users can also set JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
+        # OpenAI-compatible endpoint for the model under eval. Required for
+        # hermes/openclaw (Spec §6.2); honored by first-party backends too on
+        # this CLI path. Falls back to env vars so users can also set
+        # JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
        base_url=base_url or os.environ.get("JARVIS_BACKEND_BASE_URL"),
        api_key=api_key or os.environ.get("JARVIS_BACKEND_API_KEY"),
    )
@@ -146,7 +146,13 @@ def serve(
        except Exception as exc:
            logger.debug("Telemetry store init failed: %s", exc)

-    resolved = get_engine(config, engine_key)
+    # Select with the model we'll actually serve so an engine that can't
+    # serve it (e.g. the cloud fallback without the matching provider key) is
+    # skipped rather than chosen and failing per-request later (see #532).
+    selection_model = (
+        model_name or config.server.model or config.intelligence.default_model or None
+    )
+    resolved = get_engine(config, engine_key, model=selection_model)
    if resolved is None:
        console.print(
            "[red bold]No inference engine available.[/red bold]\n\n"
@@ -68,6 +68,10 @@ class Message:
    tool_calls: Optional[List[ToolCall]] = None
    tool_call_id: Optional[str] = None
    metadata: Dict[str, Any] = field(default_factory=dict)
+    # Base64-encoded image data for vision-capable models (e.g. gemma3,
+    # qwen2.5-vl). Forwarded to Ollama's /api/chat "images" field; None or
+    # empty for text-only messages (the common case).
+    images: Optional[List[str]] = None


@dataclass(slots=True)
@@ -34,6 +34,10 @@ def messages_to_dicts(messages: Sequence[Message]) -> List[Dict[str, Any]]:
            ]
        if m.tool_call_id:
            d["tool_call_id"] = m.tool_call_id
+        # Vision: forward base64 images to the engine. Ollama's /api/chat
+        # accepts an "images" array on a message; text messages skip this.
+        if getattr(m, "images", None):
+            d["images"] = list(m.images)
        out.append(d)
    return out

@@ -156,12 +156,26 @@ def discover_models(


 def get_engine(
-    config: JarvisConfig, engine_key: str | None = None
+    config: JarvisConfig,
+    engine_key: str | None = None,
+    model: str | None = None,
 ) -> Tuple[str, InferenceEngine] | None:
    """Get a specific engine by key, or the default with fallback.

+    When *model* is given, an engine is selected only if it can actually
+    serve that model (``engine.can_serve(model)``). This stops the cloud
+    fallback from being chosen — when the local engine is down — for a model
+    whose provider client is missing, which otherwise surfaces as a confusing
+    "OpenAI client not available" instead of a helpful "start your local
+    engine" message (see #532). When *model* is ``None`` selection stays
+    model-agnostic (unchanged behaviour).
+
    Returns ``(key, engine_instance)`` or ``None`` if no engine is available.
    """
+
+    def _usable(engine: InferenceEngine) -> bool:
+        return engine.health() and (model is None or engine.can_serve(model))
+
    # Build an ordered list of keys to try, then fall back to full discovery.
    keys_to_try: list[str] = []
    if engine_key:
@@ -176,14 +190,16 @@ def get_engine(
            continue
        try:
            engine = _make_engine(key, config)
-            if engine.health():
+            if _usable(engine):
                return (key, engine)
        except Exception as exc:
            logger.debug("Engine %r health check failed: %s", key, exc)

-    # Fallback to any healthy engine
-    healthy = discover_engines(config)
-    return healthy[0] if healthy else None
+    # Fallback to the first healthy engine that can serve the model.
+    for key, engine in discover_engines(config):
+        if model is None or engine.can_serve(model):
+            return (key, engine)
+    return None


 __all__ = ["discover_engines", "discover_models", "get_engine"]
@@ -28,12 +28,31 @@ class _OpenAICompatibleEngine(InferenceEngine):
    _default_host: str = "http://localhost:8000"
    _api_prefix: str = "/v1"

-    def __init__(self, host: str | None = None, *, timeout: float = 600.0) -> None:
+    def __init__(
+        self,
+        host: str | None = None,
+        *,
+        api_key: str | None = None,
+        timeout: float = 600.0,
+    ) -> None:
        import os

-        env_key = f"{self.engine_id.upper()}_HOST"
-        self._host = (host or os.environ.get(env_key) or self._default_host).rstrip("/")
-        self._client = httpx.Client(base_url=self._host, timeout=timeout)
+        # Sanitize the engine id for env-var lookup ("openai-compat" ->
+        # "OPENAI_COMPAT_..."); shells cannot set hyphenated variable names.
+        env_prefix = self.engine_id.upper().replace("-", "_")
+        self._host = (
+            host or os.environ.get(f"{env_prefix}_HOST") or self._default_host
+        ).rstrip("/")
+        # Bearer auth for endpoints started with e.g. ``vllm serve --api-key``.
+        # Setting it on the client covers generate/stream/stream_full/
+        # list_models/health alike; ``None`` keeps requests header-free.
+        self._api_key = api_key or os.environ.get(f"{env_prefix}_API_KEY") or None
+        headers = (
+            {"Authorization": f"Bearer {self._api_key}"} if self._api_key else None
+        )
+        self._client = httpx.Client(
+            base_url=self._host, timeout=timeout, headers=headers
+        )

    # -- InferenceEngine interface ------------------------------------------

@@ -119,6 +119,17 @@ class InferenceEngine(ABC):
    def health(self) -> bool:
        """Return ``True`` when the engine is reachable and healthy."""

+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` if this engine can serve *model*.
+
+        Defaults to ``True``: local engines accept any model id (whether a
+        specific model is *installed* is a separate concern from engine
+        selection). Engines that multiplex provider-specific clients (e.g.
+        the cloud engine) override this so selection can skip an engine whose
+        client for the model's provider isn't configured (see #532).
+        """
+        return True
+
    def close(self) -> None:
        """Release resources (HTTP clients, connections, threads, etc.)."""

@@ -1477,6 +1477,34 @@ class CloudEngine(InferenceEngine):
            models.extend(_CODEX_MODELS)
        return models

+    def _client_for_model(self, model: str) -> Any:
+        """Return the provider client ``generate``/``stream`` will dispatch to
+        for *model* (mirrors the routing in those methods)."""
+        if _is_codex_model(model):
+            return self._codex_client
+        if _is_openrouter_model(model):
+            return self._openrouter_client
+        if _is_minimax_model(model):
+            return self._minimax_client
+        if _is_anthropic_model(model):
+            return self._anthropic_client
+        if _is_google_model(model):
+            return self._google_client
+        return self._openai_client
+
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` only if the provider client for *model* exists.
+
+        ``health()`` is ``True`` whenever *any* provider client is configured,
+        but a request for, say, a ``gpt-*`` model still needs the OpenAI
+        client specifically. Without this check the cloud engine gets picked
+        as a fallback (when the local engine is down) for a model it can't
+        serve, then dies at call time with "<provider> client not available"
+        instead of the user getting a helpful "start your local engine"
+        message (see #532).
+        """
+        return self._client_for_model(model) is not None
+
    def health(self) -> bool:
        return (
            self._openai_client is not None
@@ -23,6 +23,19 @@ from openjarvis.engine._stubs import StreamChunk
 logger = logging.getLogger(__name__)


+def _default_num_ctx() -> int:
+    """Default context window (tokens). Override with ``JARVIS_NUM_CTX``.
+
+    Raised above Ollama's 4k default so an image (which costs many tokens)
+    plus a real conversation fit. 16k is comfortable for small models on a
+    typical consumer GPU.
+    """
+    try:
+        return int(os.environ.get("JARVIS_NUM_CTX", "16384"))
+    except ValueError:
+        return 16384
+
+
@EngineRegistry.register("ollama")
 class OllamaEngine(InferenceEngine):
    """Ollama backend via its native HTTP API."""
@@ -73,7 +86,7 @@ class OllamaEngine(InferenceEngine):
            "options": {
                "temperature": temperature,
                "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
            },
        }
        # Disable extended thinking by default (Qwen3.5 etc.).
@@ -189,7 +202,7 @@ class OllamaEngine(InferenceEngine):
            "options": {
                "temperature": temperature,
                "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
            },
        }
        # Mirror generate()'s default: disable extended thinking unless the
@@ -268,7 +281,7 @@ class OllamaEngine(InferenceEngine):
            "options": {
                "temperature": temperature,
                "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
            },
        }
        if "think" not in kwargs:
@@ -1,5 +1,7 @@
 """Data-driven registration of OpenAI-compatible inference engines."""

+from __future__ import annotations
+
 from openjarvis.core.registry import EngineRegistry
 from openjarvis.engine._openai_compat import _OpenAICompatibleEngine

@@ -25,4 +27,35 @@ for _key, (_cls_name, _default_host, _api_prefix) in _ENGINES.items():
    EngineRegistry.register(_key)(_cls)
    globals()[_cls_name] = _cls

-__all__ = [name for name, _, _ in _ENGINES.values()]
+
+def normalize_openai_base_url(url: str) -> str:
+    """Strip a single trailing ``/v1`` segment from a user-supplied base URL.
+
+    Users habitually pass ``http://host:8000/v1`` (the full OpenAI-compatible
+    prefix); the engine's ``_api_prefix`` re-appends ``/v1`` to every request
+    path, so a trailing copy would double up as ``/v1/v1``. Only a literal
+    trailing ``/v1`` is stripped — proxy/gateway path prefixes are preserved.
+    """
+    base = url.rstrip("/")
+    if base.endswith("/v1"):
+        base = base[: -len("/v1")]
+    return base
+
+
+class OpenAICompatEngine(_OpenAICompatibleEngine):
+    """Generic engine for an explicitly-provided OpenAI-compatible endpoint.
+
+    Deliberately NOT registered in ``EngineRegistry``: it is only ever
+    constructed with an explicit host (e.g. ``jarvis eval --base-url``), so
+    registering it would just add a useless localhost discovery probe and
+    interact with the per-test registry wipe.
+    """
+
+    engine_id = "openai-compat"
+    _api_prefix = "/v1"
+
+
+__all__ = [name for name, _, _ in _ENGINES.values()] + [
+    "OpenAICompatEngine",
+    "normalize_openai_base_url",
+]
@@ -0,0 +1,52 @@
+"""Shared helper for targeting an explicit OpenAI-compatible endpoint.
+
+Used by the first-party eval backends (jarvis-direct, jarvis-agent) when
+``--base-url`` is given: the eval must use exactly that endpoint, with no
+silent fallback to whatever other engine discovery happens to find.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+
+def build_endpoint_engine(
+    base_url: str,
+    api_key: Optional[str] = None,
+    engine_key: Optional[str] = None,
+):
+    """Construct an :class:`OpenAICompatEngine` pinned to ``base_url``.
+
+    Pre-flight health-checks the endpoint and raises a loud, actionable
+    error when it is unreachable — engine discovery is never consulted.
+    """
+    from openjarvis.engine.openai_compat_engines import (
+        OpenAICompatEngine,
+        normalize_openai_base_url,
+    )
+
+    if engine_key:
+        logger.warning(
+            "Both an engine key (%r) and base_url (%r) were given; "
+            "base_url wins — targeting the endpoint directly.",
+            engine_key,
+            base_url,
+        )
+    host = normalize_openai_base_url(base_url)
+    engine = OpenAICompatEngine(host=host, api_key=api_key)
+    if not engine.health():
+        engine.close()
+        raise RuntimeError(
+            f"--base-url endpoint not reachable: {base_url} "
+            f"(GET {host}/v1/models failed). Is an OpenAI-compatible server "
+            "(e.g. `vllm serve`) running at that address? If it requires "
+            "authentication (HTTP 401), pass --api-key or set "
+            "JARVIS_BACKEND_API_KEY."
+        )
+    return engine
+
+
+__all__ = ["build_endpoint_engine"]
@@ -31,6 +31,8 @@ class JarvisAgentBackend(InferenceBackend):
        max_turns: Optional[int] = None,
        skills_enabled: bool = True,
        overlay_dir: Optional[Path] = None,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
    ) -> None:
        from openjarvis.system import SystemBuilder

@@ -40,7 +42,17 @@ class JarvisAgentBackend(InferenceBackend):
        self._gpu_metrics = gpu_metrics

        builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
            builder.engine(engine_key)
        if model:
            builder.model(model)
@@ -24,6 +24,8 @@ class JarvisDirectBackend(InferenceBackend):
        engine_key: Optional[str] = None,
        telemetry: bool = False,
        gpu_metrics: bool = False,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
    ) -> None:
        from openjarvis.system import SystemBuilder

@@ -31,7 +33,17 @@ class JarvisDirectBackend(InferenceBackend):
        self._gpu_metrics = gpu_metrics

        builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
            builder.engine(engine_key)
        # Propagate gpu_metrics to the runtime config so SystemBuilder
        # creates an EnergyMonitor / GpuMonitor for the InstrumentedEngine.
@@ -5,11 +5,13 @@ Uses Harness for Docker-based execution and scoring.

 from __future__ import annotations

+import inspect
 import logging
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Tuple

 from openjarvis.evals.core.backend import InferenceBackend
+from openjarvis.evals.core.types import RunSummary

 LOGGER = logging.getLogger(__name__)

@@ -20,6 +22,97 @@ try:
 except ImportError:
    _HAS_TB = False

+# terminal-bench FailureMode values that are definitionally infrastructure
+# failures (the harness broke before/while driving the agent), never a
+# judgment on the model's answer. NOTE: clean trials leave failure_mode
+# "unset" in terminal-bench 0.2.18 — both on success AND on genuine
+# unresolved misses — so failure_mode alone can NOT be used to detect
+# harness errors (it would misflag every real model miss).
+_INFRA_FAILURE_MODES = frozenset({"agent_installation_failed", "unknown_agent_error"})
+
+# Harness kwargs that older terminal-bench versions may not support.
+_TIMEOUT_KWARGS = ("global_agent_timeout_sec", "global_timeout_multiplier")
+
+
+def summarize_benchmark_results(
+    results: Any,
+    *,
+    model: str,
+    benchmark: str = "terminalbench-native",
+) -> Tuple[RunSummary, List[Dict[str, str]]]:
+    """Convert terminal-bench ``BenchmarkResults`` into a ``RunSummary``.
+
+    Trials are classified into three buckets:
+
+    - resolved: ``is_resolved`` is True -> counted correct.
+    - model miss: unresolved, but the model was actually contacted ->
+      counted in the accuracy denominator.
+    - harness/infra failure: excluded from the accuracy denominator and
+      reported in ``RunSummary.errors`` plus the returned failure list.
+
+    Zero-model-contact signal choice: terminal-bench 0.2.18 leaves
+    ``failure_mode`` UNSET both on clean success and on genuine unresolved
+    misses, so failure_mode cannot distinguish "the model tried and failed"
+    from "the agent never called the model". Token usage can: this backend
+    always runs terminus-2, which reports real LiteLLM usage, so an
+    unresolved trial with zero/missing input+output tokens means no model
+    request ever completed — an infrastructure failure (in-container setup
+    hang/death, tmux failure), not a model miss. CAVEAT: terminal-bench
+    "installed agents" (openhands, claude-code, ...) hardcode 0 tokens even
+    on success; if this backend ever honors ``agent_name`` for installed
+    agents, this heuristic must be gated on the agent type.
+    """
+    trials = list(getattr(results, "results", None) or [])
+
+    harness_failures: List[Dict[str, str]] = []
+    scored = 0
+    correct = 0
+
+    for tr in trials:
+        task_id = getattr(tr, "task_id", None) or getattr(tr, "trial_name", "unknown")
+        is_resolved = getattr(tr, "is_resolved", None) is True
+        fm = getattr(tr, "failure_mode", None)
+        fm_value = str(getattr(fm, "value", fm) or "unset").lower()
+        tokens = (getattr(tr, "total_input_tokens", None) or 0) + (
+            getattr(tr, "total_output_tokens", None) or 0
+        )
+
+        zero_model_contact = not is_resolved and tokens == 0
+        infra_failure_mode = fm_value in _INFRA_FAILURE_MODES
+
+        if zero_model_contact or infra_failure_mode:
+            harness_failures.append(
+                {
+                    "task_id": str(task_id),
+                    "failure_mode": fm_value,
+                    "reason": (
+                        "zero_model_requests" if zero_model_contact else fm_value
+                    ),
+                }
+            )
+            continue
+
+        scored += 1
+        if is_resolved:
+            correct += 1
+
+    return (
+        RunSummary(
+            benchmark=benchmark,
+            category="agentic",
+            backend="terminalbench-native",
+            model=model,
+            total_samples=len(trials),
+            scored_samples=scored,
+            correct=correct,
+            accuracy=correct / scored if scored else 0.0,
+            errors=len(harness_failures),
+            mean_latency_seconds=0.0,
+            total_cost_usd=0.0,
+        ),
+        harness_failures,
+    )
+

 class TerminalBenchNativeBackend(InferenceBackend):
    """Runs terminal-bench tasks natively via Harness with Docker execution.
@@ -44,7 +137,22 @@ class TerminalBenchNativeBackend(InferenceBackend):
        system_prompt: str = "",
        max_tokens: int = 16384,
        n_concurrent: int = 4,
+        global_agent_timeout_sec: Optional[float] = 1800.0,
+        global_timeout_multiplier: Optional[float] = None,
    ) -> None:
+        """Args of note:
+
+        global_agent_timeout_sec: Hard wall-clock bound for each trial's
+            agent phase. terminal-bench runs installed-agent SETUP inside
+            this same budget with an infinite tmux timeout, so this bounds
+            SETUP+RUN together (a setup-only timeout needs an upstream
+            terminal-bench change). When set, it REPLACES each task's own
+            ``max_agent_timeout_sec``. Set ``None`` or ``0`` to fall back
+            to per-task budgets.
+        global_timeout_multiplier: Scales per-task budgets when
+            ``global_agent_timeout_sec`` is not set. ``None`` keeps
+            terminal-bench's default (1.0).
+        """
        if not _HAS_TB:
            raise ImportError("terminal-bench is required: pip install terminal-bench")

@@ -59,6 +167,8 @@ class TerminalBenchNativeBackend(InferenceBackend):
        self._system_prompt = system_prompt
        self._max_tokens = max_tokens
        self._n_concurrent = n_concurrent
+        self._global_agent_timeout_sec = global_agent_timeout_sec
+        self._global_timeout_multiplier = global_timeout_multiplier
        self._results: Optional[BenchmarkResults] = None

    def run_harness(self, run_id: str) -> BenchmarkResults:
@@ -91,10 +201,53 @@ class TerminalBenchNativeBackend(InferenceBackend):
        if self._max_samples is not None:
            harness_kwargs["n_tasks"] = self._max_samples

+        # Bound each trial's agent phase. Without this, an in-container
+        # installed-agent SETUP hang runs with an infinite tmux timeout,
+        # bounded only by whatever budget the task happens to declare.
+        if self._global_agent_timeout_sec:
+            harness_kwargs["global_agent_timeout_sec"] = float(
+                self._global_agent_timeout_sec
+            )
+        if self._global_timeout_multiplier is not None:
+            harness_kwargs["global_timeout_multiplier"] = float(
+                self._global_timeout_multiplier
+            )
+
+        self._check_timeout_kwargs_supported(harness_kwargs)
+
        harness = Harness(**harness_kwargs)
        self._results = harness.run()
        return self._results

+    @staticmethod
+    def _check_timeout_kwargs_supported(harness_kwargs: Dict[str, Any]) -> None:
+        """Fail loudly if this terminal-bench build lacks the timeout kwargs.
+
+        terminal-bench is an undeclared, unpinned dependency, so installs may
+        predate the global timeout kwargs (added by 0.2.x). Passing an
+        unknown kwarg raises an opaque TypeError; dropping it silently would
+        re-create the unbounded-setup hang. Detect and explain instead.
+        """
+        try:
+            params = inspect.signature(Harness.__init__).parameters
+        except (TypeError, ValueError):
+            return
+        if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()):
+            return
+        unsupported = [
+            key
+            for key in _TIMEOUT_KWARGS
+            if key in harness_kwargs and key not in params
+        ]
+        if unsupported:
+            raise RuntimeError(
+                "The installed terminal-bench does not support "
+                f"{', '.join(unsupported)} (requires terminal-bench >= "
+                "0.2.18). Upgrade terminal-bench, or disable the bound by "
+                "setting global_agent_timeout_sec = 0 in the eval config "
+                "[run] section."
+            )
+
    def generate(
        self,
        prompt: str,
@@ -121,4 +274,4 @@ class TerminalBenchNativeBackend(InferenceBackend):
        pass


-__all__ = ["TerminalBenchNativeBackend"]
+__all__ = ["TerminalBenchNativeBackend", "summarize_benchmark_results"]
@@ -183,14 +183,28 @@ def _build_backend(
    max_turns: Optional[int] = None,
    base_url: Optional[str] = None,
    api_key: Optional[str] = None,
+    first_party_endpoint: bool = True,
 ):
    """Construct the appropriate backend.

-    For "hermes" and "openclaw" backends, ``base_url`` and ``api_key`` are
-    REQUIRED — these foreign frameworks need an OpenAI-compatible endpoint
-    to send model calls to. Pass them via the eval config's
-    ``[backend.external]`` section or env vars.
+    ``base_url``/``api_key`` point at the OpenAI-compatible endpoint serving
+    the model under eval:
+
+    - For "hermes" and "openclaw" they are REQUIRED — these foreign
+      frameworks always call out to an external endpoint.
+    - "jarvis-direct" and "jarvis-agent" honor them when
+      ``first_party_endpoint`` is True (the CLI ``--base-url`` path): the
+      eval targets exactly that endpoint — no engine-discovery fallback —
+      and fails fast if it is unreachable. Suite mode passes
+      ``first_party_endpoint=False`` so the suite TOML's
+      ``[backend.external]`` section stays scoped to hermes/openclaw
+      (extending it to first-party backends is explicitly deferred).
    """
+    if not first_party_endpoint:
+        fp_base_url = fp_api_key = None
+    else:
+        fp_base_url, fp_api_key = base_url, api_key
+
    if backend_name == "jarvis-agent":
        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend

@@ -202,6 +216,8 @@ def _build_backend(
            gpu_metrics=gpu_metrics,
            model=model,
            max_turns=max_turns,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
        )
    elif backend_name == "jarvis-direct":
        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
@@ -210,6 +226,8 @@ def _build_backend(
            engine_key=engine_key,
            telemetry=telemetry,
            gpu_metrics=gpu_metrics,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
        )
    elif backend_name == "hermes":
        from openjarvis.evals.backends.external import HermesBackend
@@ -656,25 +674,53 @@ def _build_trackers(config) -> list:
    return trackers


-def _run_terminalbench_native(config, console: Console) -> object:
-    """Run TerminalBench V2.1 natively via terminal-bench Harness."""
+def _run_terminalbench_native(
+    config,
+    console: Console,
+    *,
+    base_url: Optional[str] = None,
+    api_key: Optional[str] = None,
+) -> object:
+    """Run TerminalBench V2.1 natively via terminal-bench Harness.
+
+    ``base_url`` (from ``--base-url`` / JARVIS_BACKEND_BASE_URL) targets an
+    already-running OpenAI-compatible endpoint; when unset, the legacy local
+    vLLM default (http://localhost:8000/v1) is used.
+    """
+    from openjarvis.engine.openai_compat_engines import normalize_openai_base_url
    from openjarvis.evals.backends.terminalbench_native import (
        TerminalBenchNativeBackend,
+        summarize_benchmark_results,
    )
-    from openjarvis.evals.core.types import RunSummary

    model = config.model
    # LiteLLM expects "openai/<model>" for OpenAI-compatible servers
    litellm_model = f"openai/{model}"
    output_dir = getattr(config, "output_path", None) or "results/terminalbench-native/"

+    # Harness budgets: only forward explicit config values so the backend
+    # defaults (global_agent_timeout_sec=1800) apply otherwise.
+    timeout_kwargs = {}
+    if getattr(config, "global_agent_timeout_sec", None) is not None:
+        timeout_kwargs["global_agent_timeout_sec"] = config.global_agent_timeout_sec
+    if getattr(config, "global_timeout_multiplier", None) is not None:
+        timeout_kwargs["global_timeout_multiplier"] = config.global_timeout_multiplier
+
+    # Normalize to exactly one trailing "/v1" — LiteLLM's api_base wants the
+    # full OpenAI-compatible prefix, and users pass both forms of the URL.
+    if base_url:
+        api_base = normalize_openai_base_url(base_url) + "/v1"
+    else:
+        api_base = "http://localhost:8000/v1"
+
    backend = TerminalBenchNativeBackend(
        model=litellm_model,
-        api_base="http://localhost:8000/v1",
+        api_base=api_base,
        temperature=config.temperature,
        max_samples=config.max_samples,
        output_dir=output_dir,
        n_concurrent=config.max_workers or 4,
+        **timeout_kwargs,
    )

    import re
@@ -683,46 +729,83 @@ def _run_terminalbench_native(config, console: Console) -> object:
    model_slug = re.sub(r"[^a-z0-9_-]", "-", model.lower().replace("/", "-"))
    run_id = f"tb21-{model_slug}"
    console.print(f"  Running TerminalBench V2.1 natively: {model}")
+    console.print(f"  API base: {api_base}")
    console.print(f"  Harness run_id: {run_id}")

-    results = backend.run_harness(run_id)
+    if api_key:
+        # terminus-2 routes model calls through LiteLLM with the "openai/"
+        # prefix, which reads OPENAI_API_KEY from the environment. The
+        # harness runs in-process, so set the var for the duration of the
+        # run and restore the previous value afterwards.
+        prev_key = os.environ.get("OPENAI_API_KEY")
+        os.environ["OPENAI_API_KEY"] = api_key
+        try:
+            results = backend.run_harness(run_id)
+        finally:
+            if prev_key is None:
+                os.environ.pop("OPENAI_API_KEY", None)
+            else:
+                os.environ["OPENAI_API_KEY"] = prev_key
+    else:
+        results = backend.run_harness(run_id)

-    # Convert BenchmarkResults to RunSummary
-    total = len(results.trial_results) if hasattr(results, "trial_results") else 0
-    correct = 0
-    if hasattr(results, "trial_results"):
-        for tr in results.trial_results:
-            if getattr(tr, "is_resolved", False):
-                correct += 1
-
-    accuracy = correct / total if total > 0 else 0.0
-    return RunSummary(
-        benchmark="terminalbench-native",
-        category="agentic",
-        backend="terminalbench-native",
-        model=model,
-        total_samples=total,
-        scored_samples=total,
-        correct=correct,
-        accuracy=accuracy,
-        errors=0,
-        mean_latency_seconds=0.0,
-        total_cost_usd=0.0,
-    )
+    # Convert BenchmarkResults to RunSummary, classifying harness/infra
+    # failures (e.g. zero-model-contact setup hangs) out of the resolve-rate.
+    summary, harness_failures = summarize_benchmark_results(results, model=model)
+    if harness_failures:
+        console.print(
+            f"  [red bold]{len(harness_failures)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for failure in harness_failures:
+            console.print(
+                f"  [red]- {failure['task_id']}: {failure['reason']} "
+                f"(failure_mode={failure['failure_mode']})[/red]"
+            )
+    return summary


-def _run_single(config, console: Optional[Console] = None) -> object:
-    """Run a single eval from a RunConfig and return the summary."""
+def _run_single(
+    config,
+    console: Optional[Console] = None,
+    *,
+    suite_mode: bool = False,
+) -> object:
+    """Run a single eval from a RunConfig and return the summary.
+
+    ``suite_mode=True`` (TOML-suite drivers) scopes ``config.base_url`` /
+    ``config.api_key`` — stamped from the suite's ``[backend.external]``
+    section onto every RunConfig — to the hermes/openclaw backends only;
+    extending suite-level endpoint targeting to first-party backends is
+    explicitly deferred. The CLI single-run path (``suite_mode=False``)
+    honors ``--base-url``/``--api-key`` for every backend.
+    """
    from openjarvis.evals.core.runner import EvalRunner

    if console is None:
        console = Console()

+    _metadata = getattr(config, "metadata", None) or {}
+    base_url = (
+        getattr(config, "base_url", None)
+        or _metadata.get("base_url")
+        or os.environ.get("JARVIS_BACKEND_BASE_URL")
+    )
+    api_key = (
+        getattr(config, "api_key", None)
+        or _metadata.get("api_key")
+        or os.environ.get("JARVIS_BACKEND_API_KEY")
+    )
+
    # TerminalBench V2.1 native: use terminal-bench Harness directly
    if config.benchmark == "terminalbench-native":
-        return _run_terminalbench_native(config, console)
+        return _run_terminalbench_native(
+            config,
+            console,
+            base_url=None if suite_mode else base_url,
+            api_key=None if suite_mode else api_key,
+        )

-    _metadata = getattr(config, "metadata", None) or {}
    eval_backend = _build_backend(
        config.backend,
        config.engine_key,
@@ -732,16 +815,9 @@ def _run_single(config, console: Optional[Console] = None) -> object:
        gpu_metrics=getattr(config, "gpu_metrics", False),
        model=config.model,
        max_turns=getattr(config, "max_turns", None),
-        base_url=(
-            getattr(config, "base_url", None)
-            or _metadata.get("base_url")
-            or os.environ.get("JARVIS_BACKEND_BASE_URL")
-        ),
-        api_key=(
-            getattr(config, "api_key", None)
-            or _metadata.get("api_key")
-            or os.environ.get("JARVIS_BACKEND_API_KEY")
-        ),
+        base_url=base_url,
+        api_key=api_key,
+        first_party_endpoint=not suite_mode,
    )
    dataset = _build_dataset(config.benchmark)
    # Inject engine config for benchmarks that run their own simulation
@@ -964,7 +1040,9 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
    from rich.table import Table

    completed = sum(1 for t in traces if t.completed)
-    resolved = sum(1 for t in traces if t.is_resolved is True)
+    harness_errors = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = [t for t in traces if t.error_kind != "harness_error"]
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
    timed_out = sum(1 for t in traces if t.timed_out)
    total_turns = sum(t.num_turns for t in traces)
    total_tool_calls = sum(t.total_tool_calls for t in traces)
@@ -992,8 +1070,11 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
    table.add_row("Queries", str(len(traces)))
    table.add_row("Completed", f"{completed}/{len(traces)}")
    if any(t.is_resolved is not None for t in traces):
-        table.add_row("Resolved", f"{resolved}/{len(traces)}")
+        # Harness errors are excluded from the resolve-rate denominator:
+        # they are infra failures, not model misses.
+        table.add_row("Resolved", f"{resolved}/{len(model_traces)}")
    table.add_row("Timed out", str(timed_out))
+    table.add_row("Harness errors", str(len(harness_errors)))
    table.add_row("Total turns", str(total_turns))
    avg_t = f"{total_turns / len(traces):.1f}" if traces else "0"
    table.add_row("Avg turns/query", avg_t)
@@ -1020,6 +1101,19 @@ def _print_agentic_summary(console: Console, traces, config) -> None:

    console.print(table)

+    if harness_errors:
+        console.print(
+            f"[red bold]{len(harness_errors)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for t in harness_errors[:5]:
+            console.print(f"[red]  {t.query_id}: {(t.error or '')[:300]}[/red]")
+        if len(harness_errors) > 5:
+            console.print(
+                f"[red]  ... and {len(harness_errors) - 5} more "
+                "(see traces.jsonl)[/red]"
+            )
+

 def _run_from_config(
    config_path: str,
@@ -1070,7 +1164,7 @@ def _run_from_config(
            f"Run {i}/{len(run_configs)}: {rc.benchmark} / {rc.model}",
        )
        try:
-            summary = _run_single(rc, console=console)
+            summary = _run_single(rc, console=console, suite_mode=True)
            summaries.append(summary)
            console.print(
                f"  [green]{summary.accuracy:.4f}[/green] "
@@ -1115,12 +1209,21 @@ def main():
@click.option(
    "--base-url",
    default=None,
-    help="OpenAI-compat endpoint for hermes/openclaw",
+    help=(
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
+        "(env: JARVIS_BACKEND_BASE_URL)."
+    ),
 )
@click.option(
    "--api-key",
    default=None,
-    help="API key for hermes/openclaw endpoint",
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
@click.option("-m", "--model", default=None, help="Model identifier")
@click.option(
@@ -1526,8 +1629,7 @@ def summarize(jsonl_path):
    default=None,
    type=click.Path(),
    help=(
-        "Output JSONL path. Defaults to <jsonl>.reparsed when "
-        "--in-place is not set."
+        "Output JSONL path. Defaults to <jsonl>.reparsed when --in-place is not set."
    ),
 )
@click.option(
@@ -1642,9 +1744,7 @@ def reparse_judge(jsonl_path, out_path, in_place, summary_out):
        _json.dump(summary, f, indent=2)

    old_cont = [float(s) for s in old_scores if s is not None]
-    old_acc = (
-        sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
-    )
+    old_acc = sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
    old_mean = sum(old_cont) / len(old_cont) if old_cont else 0.0
    new_mean = sum(cont) / len(cont) if cont else 0.0
    mean_shift = new_mean - old_mean
@@ -20,6 +20,7 @@ from contextlib import nullcontext
 from pathlib import Path
 from typing import Any, Callable, Optional

+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.core.event_recorder import AgentEvent, EventRecorder, EventType
 from openjarvis.evals.core.trace import QueryTrace, TurnTrace

@@ -176,9 +177,12 @@ class AgenticRunner:
                )
            self._traces.append(trace)

-            status = (
-                "TIMEOUT" if trace.timed_out else ("OK" if trace.completed else "FAIL")
-            )
+            if trace.timed_out:
+                status = "TIMEOUT"
+            elif trace.error_kind == "harness_error":
+                status = "HARNESS_ERROR"
+            else:
+                status = "OK" if trace.completed else "FAIL"
            LOGGER.info(
                "Task %s: %s in %.1fs",
                query_id,
@@ -260,11 +264,12 @@ class AgenticRunner:
                        is_resolved=record.metadata.get("is_resolved"),
                    )

-                status = (
-                    "TIMEOUT"
-                    if trace.timed_out
-                    else ("OK" if trace.completed else "FAIL")
-                )
+                if trace.timed_out:
+                    status = "TIMEOUT"
+                elif trace.error_kind == "harness_error":
+                    status = "HARNESS_ERROR"
+                else:
+                    status = "OK" if trace.completed else "FAIL"
                LOGGER.info(
                    "Task %s: %s in %.1fs",
                    query_id,
@@ -444,7 +449,23 @@ class AgenticRunner:
                _run_body()

        except Exception as exc:
-            LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
+            # Distinguish infrastructure breakage (task env failed to start,
+            # Docker/tmux death) from agent failures: harness errors must be
+            # recorded distinctly so scoring excludes them from resolve-rate
+            # instead of silently counting a model miss. Either way the run
+            # continues with the next record (fail THIS task, not the run).
+            is_harness_error = isinstance(exc, TaskEnvironmentError) or bool(
+                record.metadata.get("harness_error")
+            )
+            if is_harness_error:
+                LOGGER.error(
+                    "Harness/environment failure on query %s (record %s): %s",
+                    query_id,
+                    getattr(record, "record_id", "?"),
+                    exc,
+                )
+            else:
+                LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
            end_time = time.time()
            # Unsubscribe EventBus relays
            if agent_bus is not None:
@@ -461,6 +482,8 @@ class AgenticRunner:
                total_wall_clock_s=end_time - start_time,
                completed=False,
                is_resolved=record.metadata.get("is_resolved"),
+                error=str(exc),
+                error_kind="harness_error" if is_harness_error else "agent_error",
            )

        # Unsubscribe EventBus relays
@@ -534,6 +557,48 @@ class AgenticRunner:
                    model, turn.input_tokens, turn.output_tokens
                )

+        # --- Zero-model-contact sanity check ----------------------------
+        # Failed in-container setups (e.g. an installed agent's SETUP phase
+        # hanging or dying) historically produced traces that looked like
+        # model results: completed=True, a synthetic 1-event turn,
+        # is_resolved=False from run_tests, and zero tokens — silently
+        # dragging resolve-rate down as a fake model miss. Signal choice:
+        # token usage plus LM inference events is the reliable discriminator
+        # here — a genuine model miss has token usage (and/or LM events),
+        # while "the agent never called the model" has neither. We do NOT
+        # key off completion status or is_resolved, which are identical in
+        # both cases. run_agent_loop envs drive the model directly
+        # (bypassing usage reporting), so their turn_wall_clocks count as
+        # model contact.
+        had_lm_events = any(
+            e.event_type in (EventType.LM_INFERENCE_START, EventType.LM_INFERENCE_END)
+            for e in events
+        )
+        had_loop_turns = bool(
+            task_env is not None and getattr(task_env, "turn_wall_clocks", None)
+        )
+        turn_tokens = sum(t.input_tokens + t.output_tokens for t in turns)
+        error: Optional[str] = None
+        error_kind: Optional[str] = None
+        if (
+            not had_lm_events
+            and not had_loop_turns
+            and in_tok + out_tok == 0
+            and turn_tokens == 0
+        ):
+            error = (
+                "zero_model_requests: the agent produced no LM inference "
+                "events and reported zero token usage — the model was never "
+                "contacted. This is a harness/infrastructure failure (e.g. "
+                "in-container agent setup hang/death), not a model miss, and "
+                "is excluded from resolve-rate. If your agent genuinely "
+                "contacted the model, make it report token usage or emit "
+                "LM_INFERENCE events. Response tail: "
+                f"{(response_text or '')[-500:]!r}"
+            )
+            error_kind = "harness_error"
+            LOGGER.error("Query %s: %s", query_id, error)
+
        # Query-level energy from telemetry window
        query_gpu_energy = _compute_energy_delta(readings, "gpu_energy_j")
        query_cpu_energy = _compute_energy_delta(readings, "cpu_energy_j")
@@ -563,6 +628,8 @@ class AgenticRunner:
            is_resolved=record.metadata.get("is_resolved"),
            query_mbu_avg_pct=query_mbu_avg,
            query_mbu_max_pct=query_mbu_max,
+            error=error,
+            error_kind=error_kind,
        )

        # Correlate energy with trace
@@ -143,6 +143,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
        sheets_worksheet=run_raw.get("sheets_worksheet", "Results"),
        sheets_credentials_path=run_raw.get("sheets_credentials_path", ""),
        max_turns=(int(run_raw["max_turns"]) if "max_turns" in run_raw else None),
+        global_agent_timeout_sec=(
+            float(run_raw["global_agent_timeout_sec"])
+            if "global_agent_timeout_sec" in run_raw
+            else None
+        ),
+        global_timeout_multiplier=(
+            float(run_raw["global_timeout_multiplier"])
+            if "global_timeout_multiplier" in run_raw
+            else None
+        ),
    )

    # Parse [[models]]
@@ -212,6 +222,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
                max_tokens=int(b["max_tokens"]) if "max_tokens" in b else None,
                subset=b.get("subset"),
                record_ids=record_ids,
+                global_agent_timeout_sec=(
+                    float(b["global_agent_timeout_sec"])
+                    if "global_agent_timeout_sec" in b
+                    else None
+                ),
+                global_timeout_multiplier=(
+                    float(b["global_timeout_multiplier"])
+                    if "global_timeout_multiplier" in b
+                    else None
+                ),
            )
        )

@@ -275,6 +295,14 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
            if bench.judge_model is not None:
                judge_model = bench.judge_model

+            # terminal-bench harness budgets: benchmark > [run]
+            global_agent_timeout_sec = suite.run.global_agent_timeout_sec
+            if bench.global_agent_timeout_sec is not None:
+                global_agent_timeout_sec = bench.global_agent_timeout_sec
+            global_timeout_multiplier = suite.run.global_timeout_multiplier
+            if bench.global_timeout_multiplier is not None:
+                global_timeout_multiplier = bench.global_timeout_multiplier
+
            # Auto-generate output path
            model_slug = model.name.replace("/", "-").replace(":", "-")
            output_path = f"{output_dir}/{bench.name}_{model_slug}.jsonl"
@@ -328,6 +356,8 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
                    base_url=suite.backend_external_base_url,
                    api_key=suite.backend_external_api_key,
                    record_ids=bench.record_ids,
+                    global_agent_timeout_sec=global_agent_timeout_sec,
+                    global_timeout_multiplier=global_timeout_multiplier,
                )
            )

@@ -8,6 +8,18 @@ from typing import Any, Dict, Tuple
 from openjarvis.evals.core.types import EvalRecord


+class TaskEnvironmentError(RuntimeError):
+    """A task execution environment failed to start or operate.
+
+    Raised when infrastructure backing a task (Docker container, docker
+    compose project, tmux session, recording binaries, ...) breaks. This is
+    a harness/environment failure, **not** a model failure: runners record
+    it distinctly (``QueryTrace.error_kind == "harness_error"``) so scoring
+    can exclude the sample from resolve-rate instead of silently counting
+    it as a model miss.
+    """
+
+
 class EnvironmentProvider(ABC):
    """Manages an external environment for evaluation benchmarks.

@@ -50,3 +62,6 @@ class EnvironmentProvider(ABC):
    @abstractmethod
    def teardown(self) -> None:
        """Stop the environment and release resources."""
+
+
+__all__ = ["EnvironmentProvider", "TaskEnvironmentError"]
@@ -26,13 +26,22 @@ def _agg_stats(values: Sequence[Optional[float]]) -> dict[str, Optional[float]]:
    }


+def _model_attributable(traces: list[QueryTrace]) -> list[QueryTrace]:
+    """Traces whose outcome is attributable to the model.
+
+    Harness errors (infra/setup failures, zero-model-contact runs) are
+    excluded so they never count as model misses in resolve-rate.
+    """
+    return [t for t in traces if t.error_kind != "harness_error"]
+
+
 def _compute_efficiency(
    traces: list[QueryTrace],
    total_gpu_energy: Optional[float],
    total_cpu_energy: Optional[float],
 ) -> dict[str, Optional[float]]:
    """Compute efficiency metrics from traces and aggregate energy."""
-    scored = [t for t in traces if t.is_resolved is not None]
+    scored = [t for t in _model_attributable(traces) if t.is_resolved is not None]
    resolved = sum(1 for t in scored if t.is_resolved is True)
    accuracy = resolved / len(scored) if scored else None
    gpu_powers = [
@@ -247,8 +256,12 @@ def export_summary_json(
            cpu_energy_values.append(sum(cpu_vals))
    total_cpu_energy = sum(cpu_energy_values) if cpu_energy_values else None

-    resolved = sum(1 for t in traces if t.is_resolved is True)
-    unresolved = sum(1 for t in traces if t.is_resolved is False)
+    # Harness errors (infra failures, zero-model-contact runs) are excluded
+    # from the resolve-rate denominator: they are not model misses.
+    harness_error_traces = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = _model_attributable(traces)
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
+    unresolved = sum(1 for t in model_traces if t.is_resolved is False)

    cost_values = [t.total_cost_usd for t in traces if t.total_cost_usd is not None]
    total_cost = sum(cost_values) if cost_values else None
@@ -345,6 +358,7 @@ def export_summary_json(
            "completed": completed,
            "resolved": resolved,
            "unresolved": unresolved,
+            "harness_errors": len(harness_error_traces),
            "accuracy": accuracy,
            "turns": total_turns,
            "tool_calls": total_tool_calls,
@@ -365,6 +379,12 @@ def export_summary_json(
        "efficiency": efficiency,
    }

+    if harness_error_traces:
+        summary["harness_error_details"] = [
+            {"query_id": t.query_id, "error": (t.error or "")[:500]}
+            for t in harness_error_traces
+        ]
+
    if action_totals:
        summary["action_energy_summary"] = action_totals

@@ -399,7 +419,7 @@ def export_summary_json(

    accuracy_vals: list[float] = [
        1.0 if t.is_resolved is True else 0.0
-        for t in traces
+        for t in _model_attributable(traces)
        if t.is_resolved is not None
    ]
    latency_vals = [t.total_wall_clock_s for t in traces if t.total_wall_clock_s > 0]
@@ -91,6 +91,13 @@ class QueryTrace:
    is_resolved: Optional[bool] = None
    query_mbu_avg_pct: Optional[float] = None
    query_mbu_max_pct: Optional[float] = None
+    # Error taxonomy. ``error_kind`` distinguishes infrastructure failures
+    # ("harness_error": task env / Docker / tmux broke, or the agent never
+    # contacted the model) from agent failures ("agent_error"). Harness
+    # errors are excluded from resolve-rate by export/summary code so they
+    # are never silently counted as model misses.
+    error: Optional[str] = None
+    error_kind: Optional[str] = None

    @property
    def num_turns(self) -> int:
@@ -196,6 +203,8 @@ class QueryTrace:
            "is_resolved": self.is_resolved,
            "query_mbu_avg_pct": self.query_mbu_avg_pct,
            "query_mbu_max_pct": self.query_mbu_max_pct,
+            "error": self.error,
+            "error_kind": self.error_kind,
        }

    @classmethod
@@ -216,6 +225,8 @@ class QueryTrace:
            is_resolved=d.get("is_resolved"),
            query_mbu_avg_pct=d.get("query_mbu_avg_pct"),
            query_mbu_max_pct=d.get("query_mbu_max_pct"),
+            error=d.get("error"),
+            error_kind=d.get("error_kind"),
        )

    def save_jsonl(self, path: Path) -> None:
@@ -102,6 +102,15 @@ class RunConfig:
    # specific records (e.g. recovering silent-fake records without
    # re-running the entire benchmark).
    record_ids: Optional[List[str]] = None
+    # terminal-bench harness budgets (terminalbench-native backend).
+    # global_agent_timeout_sec bounds each trial's agent phase — SETUP+RUN
+    # together, since terminal-bench runs installed-agent setup inside the
+    # agent budget with an infinite tmux timeout. When set it REPLACES the
+    # per-task max_agent_timeout_sec; 0 disables the bound (per-task budgets
+    # apply); None uses the backend default (1800 s).
+    global_agent_timeout_sec: Optional[float] = None
+    # Scales per-task budgets when global_agent_timeout_sec is not set.
+    global_timeout_multiplier: Optional[float] = None


@dataclass(slots=True)
@@ -232,6 +241,9 @@ class ExecutionConfig:
    # to JarvisConfig.agent.max_turns (default 10). Bump to 30-50 for
    # thinking/reasoning models on agentic benchmarks (GAIA, LiveResearch).
    max_turns: Optional[int] = None
+    # terminal-bench harness budgets (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None


@dataclass(slots=True)
@@ -265,6 +277,10 @@ class BenchmarkConfig:
    max_tokens: Optional[int] = None
    subset: Optional[str] = None
    record_ids: Optional[List[str]] = None
+    # Per-benchmark override of the terminal-bench harness budgets
+    # (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None


@dataclass(slots=True)
@@ -8,6 +8,8 @@ from pathlib import Path
 from types import TracebackType
 from typing import Any, MutableMapping, Optional, Type

+from openjarvis.evals.core.environment import TaskEnvironmentError
+
 LOGGER = logging.getLogger(__name__)


@@ -29,8 +31,6 @@ class TerminalBenchTaskEnv:
    # ------------------------------------------------------------------

    def __enter__(self) -> TerminalBenchTaskEnv:
-        from terminal_bench.terminal.terminal import spin_up_terminal
-
        task = self._metadata.get("task")
        task_paths = self._metadata.get("task_paths")
        task_id = self._metadata.get("task_id", "unknown")
@@ -41,6 +41,8 @@ class TerminalBenchTaskEnv:
                "Use the 'terminalbench-native' dataset."
            )

+        from terminal_bench.terminal.terminal import spin_up_terminal
+
        docker_image_prefix = f"tb__{task_id}".replace(".", "-")
        client_image_name = f"{docker_image_prefix}__client"
        client_container_name = f"oj-{task_id}".replace(".", "-")
@@ -48,19 +50,59 @@ class TerminalBenchTaskEnv:
        self._logs_tmpdir = tempfile.TemporaryDirectory(prefix="oj_tb_logs_")
        logs_path = Path(self._logs_tmpdir.name)

-        self._terminal_cm = spin_up_terminal(
-            client_container_name=client_container_name,
-            client_image_name=client_image_name,
-            docker_compose_path=task_paths.docker_compose_path,
-            docker_image_name_prefix=docker_image_prefix,
-            sessions_logs_path=logs_path,
-            disable_recording=task.disable_asciinema,
-        )
-        self._terminal = self._terminal_cm.__enter__()
+        # Everything below is exception-safe: a failure mid-startup (docker
+        # compose, tmux, asciinema) tears the spun-up terminal back down
+        # immediately instead of leaking the docker compose project until GC
+        # (or forever, when the env object is retained), and re-raises as a
+        # loud TaskEnvironmentError naming the task image so the runner can
+        # record a harness error for THIS task and continue with the rest.
+        try:
+            self._terminal_cm = spin_up_terminal(
+                client_container_name=client_container_name,
+                client_image_name=client_image_name,
+                docker_compose_path=task_paths.docker_compose_path,
+                docker_image_name_prefix=docker_image_prefix,
+                sessions_logs_path=logs_path,
+                disable_recording=task.disable_asciinema,
+            )
+            self._terminal = self._terminal_cm.__enter__()

-        session = self._terminal.create_session(
-            "agent", is_active_stream=False, as_configured_user=True
-        )
+            # Preflight BEFORE the agent loop: terminal-bench drives the
+            # agent through tmux (and records via asciinema unless the task
+            # disables it). A missing binary otherwise surfaces mid-run as
+            # an opaque RuntimeError or a fake TimeoutError.
+            self._preflight_container_binaries(
+                task, task_id, client_image_name, client_container_name
+            )
+
+            session = self._terminal.create_session(
+                "agent", is_active_stream=False, as_configured_user=True
+            )
+        except BaseException as exc:
+            self._teardown(type(exc), exc, exc.__traceback__)
+            if not isinstance(exc, Exception):
+                # KeyboardInterrupt / SystemExit: clean up but never mask.
+                raise
+            if isinstance(exc, TaskEnvironmentError):
+                self._metadata["harness_error"] = str(exc)
+                raise
+            message = (
+                f"Task '{task_id}': failed to start the task environment "
+                f"(image '{client_image_name}', container "
+                f"'{client_container_name}'): {exc}. This is a harness/"
+                "environment failure, not a model failure. Check that the "
+                "Docker daemon is healthy and that tmux is installed in the "
+                "task image."
+            )
+            # docker compose stderr is only logged at DEBUG by
+            # terminal-bench; surface it here so the failure is actionable.
+            stderr = getattr(exc, "stderr", None)
+            if stderr:
+                if isinstance(stderr, bytes):
+                    stderr = stderr.decode("utf-8", errors="replace")
+                message += f"\ndocker compose stderr (tail):\n{stderr[-2000:]}"
+            self._metadata["harness_error"] = message
+            raise TaskEnvironmentError(message) from exc

        self._metadata["terminal"] = self._terminal
        self._metadata["session"] = session
@@ -68,24 +110,99 @@ class TerminalBenchTaskEnv:

        return self

+    def _preflight_container_binaries(
+        self,
+        task: Any,
+        task_id: str,
+        client_image_name: str,
+        client_container_name: str,
+    ) -> None:
+        """Verify tmux (and asciinema if recording) exist in the container.
+
+        Raises:
+            TaskEnvironmentError: naming the task image and the missing
+                binary, with the remedy, before any agent work starts.
+        """
+        container = getattr(self._terminal, "container", None)
+        if container is None:
+            # Terminal implementation without a container handle (e.g. a
+            # future terminal-bench version); fall through to terminal-bench's
+            # own checks rather than guessing.
+            return
+
+        checks: list[tuple[str, list[str], str]] = [
+            (
+                "tmux",
+                ["tmux", "-V"],
+                f"install tmux in the task image '{client_image_name}'",
+            ),
+        ]
+        if not getattr(task, "disable_asciinema", False):
+            checks.append(
+                (
+                    "asciinema",
+                    ["asciinema", "--version"],
+                    f"install asciinema in the task image '{client_image_name}' "
+                    "or set disable_asciinema in task.yaml",
+                )
+            )
+
+        for binary, cmd, remedy in checks:
+            result = container.exec_run(cmd)
+            exit_code = getattr(result, "exit_code", 0)
+            if exit_code == 0:
+                continue
+            output = getattr(result, "output", b"")
+            if isinstance(output, bytes):
+                output = output.decode("utf-8", errors="replace")
+            raise TaskEnvironmentError(
+                f"Task '{task_id}': required binary '{binary}' is not usable "
+                f"in task image '{client_image_name}' (container "
+                f"'{client_container_name}'): exec exit code {exit_code}, "
+                f"output {str(output).strip()!r}. Remedy: {remedy}. This is "
+                "a harness/environment failure, not a model failure."
+            )
+
+    def _teardown(
+        self,
+        exc_type: Optional[Type[BaseException]] = None,
+        exc_val: Optional[BaseException] = None,
+        exc_tb: Optional[TracebackType] = None,
+    ) -> None:
+        """Idempotent cleanup shared by ``__exit__`` and failed ``__enter__``.
+
+        Secondary cleanup errors are logged, never raised, so they cannot
+        mask the original failure.
+        """
+        self._metadata.pop("terminal", None)
+        self._metadata.pop("session", None)
+        self._metadata.pop("container", None)
+
+        terminal_cm, self._terminal_cm, self._terminal = self._terminal_cm, None, None
+        if terminal_cm is not None:
+            try:
+                terminal_cm.__exit__(exc_type, exc_val, exc_tb)
+            except Exception:
+                LOGGER.exception(
+                    "Secondary error while tearing down the terminal for "
+                    "task %s (original error, if any, is re-raised)",
+                    self._metadata.get("task_id", "unknown"),
+                )
+
+        logs_tmpdir, self._logs_tmpdir = self._logs_tmpdir, None
+        if logs_tmpdir is not None:
+            try:
+                logs_tmpdir.cleanup()
+            except Exception:
+                LOGGER.exception("Failed to clean up session logs tmpdir")
+
    def __exit__(
        self,
        exc_type: Optional[Type[BaseException]],
        exc_val: Optional[BaseException],
        exc_tb: Optional[TracebackType],
    ) -> None:
-        self._metadata.pop("terminal", None)
-        self._metadata.pop("session", None)
-        self._metadata.pop("container", None)
-
-        if self._terminal_cm is not None:
-            self._terminal_cm.__exit__(exc_type, exc_val, exc_tb)
-            self._terminal_cm = None
-            self._terminal = None
-
-        if self._logs_tmpdir is not None:
-            self._logs_tmpdir.cleanup()
-            self._logs_tmpdir = None
+        self._teardown(exc_type, exc_val, exc_tb)

    # ------------------------------------------------------------------
    # Test execution
@@ -93,12 +210,6 @@ class TerminalBenchTaskEnv:

    def run_tests(self) -> tuple[bool, dict[str, Any]]:
        """Copy test scripts into container, execute, parse results."""
-        from terminal_bench.parsers.base_parser import UnitTestStatus
-        from terminal_bench.parsers.parser_factory import ParserFactory
-        from terminal_bench.terminal.docker_compose_manager import (
-            DockerComposeManager,
-        )
-
        task = self._metadata["task"]
        task_paths = self._metadata["task_paths"]
        terminal = self._terminal
@@ -110,6 +221,12 @@ class TerminalBenchTaskEnv:
            self._metadata["test_results"] = results
            return False, results

+        from terminal_bench.parsers.base_parser import UnitTestStatus
+        from terminal_bench.parsers.parser_factory import ParserFactory
+        from terminal_bench.terminal.docker_compose_manager import (
+            DockerComposeManager,
+        )
+
        try:
            paths_to_copy = [task_paths.run_tests_path]
            if task_paths.test_dir.exists():
@@ -4,6 +4,27 @@ from __future__ import annotations

 from unittest.mock import MagicMock, patch

+import httpx
+import pytest
+import respx
+
+
+def _mock_builder() -> MagicMock:
+    """A SystemBuilder mock whose fluent methods chain like the real one."""
+    builder = MagicMock()
+    for method in (
+        "engine",
+        "engine_instance",
+        "model",
+        "agent",
+        "tools",
+        "telemetry",
+        "traces",
+    ):
+        getattr(builder, method).return_value = builder
+    builder.build.return_value = MagicMock()
+    return builder
+

 class TestJarvisDirectBackend:
    @patch("openjarvis.system.SystemBuilder")
@@ -137,3 +158,130 @@ class TestJarvisAgentBackend:
        assert result["content"] == "The answer is 4."
        assert result["turns"] == 2
        assert len(result["tool_results"]) == 1
+
+
+class TestJarvisDirectBackendBaseUrl:
+    """--base-url targeting for the jarvis-direct backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        # Trailing /v1 is normalized away so request paths don't double up.
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        # The discovery path must not be engaged at all.
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisDirectBackend(base_url="http://127.0.0.1:18998")
+
+        # No silent engine substitution: the system is never built.
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisDirectBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_wins_over_engine_key(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(engine_key="vllm", base_url="http://127.0.0.1:18999")
+
+        mock_builder.engine.assert_not_called()
+        mock_builder.engine_instance.assert_called_once()
+        # The engine key is kept as the label for the injected engine.
+        assert mock_builder.engine_instance.call_args.kwargs["key"] == "vllm"
+
+
+class TestJarvisAgentBackendBaseUrl:
+    """--base-url targeting for the jarvis-agent backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisAgentBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisAgentBackend(base_url="http://127.0.0.1:18998")
+
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisAgentBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
@@ -0,0 +1,190 @@
+"""--base-url/--api-key forwarding through the eval CLI plumbing.
+
+Covers the fix for the eval-CLI endpoint gap: the flags used to be silently
+dropped for jarvis-direct/jarvis-agent and ignored by terminalbench-native
+(which hardcoded api_base="http://localhost:8000/v1").
+"""
+
+from __future__ import annotations
+
+import io
+import os
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import click
+import pytest
+from rich.console import Console
+
+from openjarvis.evals.cli import _build_backend, _run_terminalbench_native
+from openjarvis.evals.core.types import RunConfig
+
+
+def _quiet_console() -> Console:
+    return Console(file=io.StringIO())
+
+
+def _tb_config(**overrides) -> RunConfig:
+    defaults = dict(
+        benchmark="terminalbench-native",
+        backend="jarvis-direct",
+        model="my-model",
+        max_samples=1,
+        max_workers=1,
+        temperature=0.2,
+    )
+    defaults.update(overrides)
+    return RunConfig(**defaults)
+
+
+class TestBuildBackendForwardsEndpoint:
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_jarvis_direct_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_agent.JarvisAgentBackend")
+    def test_jarvis_agent_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-agent",
+            "vllm",
+            "orchestrator",
+            ["calculator"],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_suite_mode_scopes_endpoint_to_external_backends(self, mock_cls):
+        """[backend.external] suite semantics stay hermes/openclaw-only:
+        first_party_endpoint=False must not forward to first-party."""
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+            first_party_endpoint=False,
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] is None
+        assert kwargs["api_key"] is None
+
+    def test_hermes_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="hermes"):
+            _build_backend("hermes", None, "orchestrator", [])
+
+    def test_openclaw_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="openclaw"):
+            _build_backend("openclaw", None, "orchestrator", [])
+
+
+class TestTerminalBenchNativeApiBase:
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_passed_through_as_api_base(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_without_v1_gets_single_v1_suffix(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_default_api_base_unchanged_without_base_url(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(_tb_config(), _quiet_console())
+        assert mock_cls.call_args.kwargs["api_base"] == "http://localhost:8000/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_api_key_exported_as_openai_api_key_during_run(self, mock_cls, monkeypatch):
+        """terminus-2 reads OPENAI_API_KEY via LiteLLM; the var must be set
+        during harness.run() and restored afterwards."""
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+        seen: dict = {}
+
+        def fake_run_harness(run_id):
+            seen["openai_api_key"] = os.environ.get("OPENAI_API_KEY")
+            return SimpleNamespace(trial_results=[])
+
+        mock_backend = MagicMock()
+        mock_backend.run_harness.side_effect = fake_run_harness
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert seen["openai_api_key"] == "sk-tb"
+        assert "OPENAI_API_KEY" not in os.environ  # restored
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_preexisting_openai_api_key_restored(self, mock_cls, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "sk-original")
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert os.environ["OPENAI_API_KEY"] == "sk-original"
+
+
+class TestRunSingleSuiteModeGating:
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_suite_mode_drops_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console(), suite_mode=True)
+        assert mock_tb.call_args.kwargs["base_url"] is None
+        assert mock_tb.call_args.kwargs["api_key"] is None
+
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_cli_mode_forwards_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console())
+        assert mock_tb.call_args.kwargs["base_url"] == "http://node7:8123/v1"
+        assert mock_tb.call_args.kwargs["api_key"] == "sk-k"
@@ -192,6 +192,7 @@ class GuardrailsEngine(InferenceEngine):
                            tool_calls=msg.tool_calls,
                            tool_call_id=msg.tool_call_id,
                            metadata=msg.metadata,
+                            images=msg.images,
                        )
            messages = processed

@@ -33,6 +33,8 @@ class SystemBuilder:
            self._config = load_config()

        self._engine_key: Optional[str] = None
+        self._engine_instance: Optional[InferenceEngine] = None
+        self._engine_instance_key: Optional[str] = None
        self._model: Optional[str] = None
        self._agent_name: Optional[str] = None
        self._tool_names: Optional[List[str]] = None
@@ -50,6 +52,20 @@ class SystemBuilder:
        self._engine_key = key
        return self

+    def engine_instance(
+        self, engine: InferenceEngine, key: str = "openai-compat"
+    ) -> SystemBuilder:
+        """Inject a pre-built engine instance, bypassing engine discovery.
+
+        Used by callers that must target one exact endpoint (e.g.
+        ``jarvis eval --base-url``). ``build()`` health-checks the instance
+        and raises a loud error if it is unreachable — it never silently
+        substitutes a different discovered engine.
+        """
+        self._engine_instance = engine
+        self._engine_instance_key = key
+        return self
+
    def model(self, name: str) -> SystemBuilder:
        self._model = name
        return self
@@ -303,6 +319,23 @@ class SystemBuilder:
        return system

    def _resolve_engine(self, config: JarvisConfig):
+        # An explicitly injected engine instance always wins and is never
+        # silently replaced: when the caller pinned an endpoint (e.g.
+        # ``jarvis eval --base-url``) and it is down, substituting whatever
+        # other engine discovery finds would silently run against the wrong
+        # model server. Fail loudly instead.
+        if self._engine_instance is not None:
+            engine = self._engine_instance
+            key = self._engine_instance_key or "openai-compat"
+            if not engine.health():
+                host = getattr(engine, "_host", "<unknown host>")
+                raise RuntimeError(
+                    f"Injected engine {key!r} is not reachable at {host} — "
+                    "is the endpoint running and serving GET /v1/models? "
+                    "Refusing to fall back to engine discovery."
+                )
+            return engine, key
+
        from openjarvis.engine._discovery import get_engine

        pref = config.intelligence.preferred_engine
@@ -313,7 +346,18 @@ class SystemBuilder:
                "No inference engine available. "
                "Make sure an engine is running (e.g. ollama serve)."
            )
-        return resolved[1], resolved[0]
+        resolved_key, engine = resolved
+        if self._engine_key and resolved_key != self._engine_key:
+            # get_engine() falls back to any healthy discovered engine; make
+            # the substitution visible when the caller asked for a specific
+            # engine (observed: requested vllm, silently got ollama@11434).
+            logger.warning(
+                "Requested engine %r is unavailable; using %r at %s instead",
+                self._engine_key,
+                resolved_key,
+                getattr(engine, "_host", "<unknown host>"),
+            )
+        return engine, resolved_key

    def _resolve_model(self, config: JarvisConfig, engine: InferenceEngine) -> str:
        if self._model:
@@ -0,0 +1,128 @@
+"""CLI-level regression tests for ``jarvis ask`` vision input.
+
+The unit tests in ``tests/test_vision.py`` cover the ``Message.images`` ->
+``messages_to_dicts`` serialization contract in isolation. These tests lock
+the *end-to-end CLI wiring*: that ``--image`` reads a file, base64-encodes it,
+attaches it to the final user ``Message``, and that the bytes actually reach
+``engine.generate()`` -- and that the local-first privacy guard fires only for
+non-local engines.
+"""
+
+from __future__ import annotations
+
+import base64
+import importlib
+from pathlib import Path
+from typing import Any
+
+from click.testing import CliRunner
+
+from openjarvis.cli import cli
+from openjarvis.core.config import JarvisConfig
+from openjarvis.core.types import Role
+
+# Import the module (not the Click command attribute) so we can monkeypatch
+# the names it looks up at call time.
+_ask_mod = importlib.import_module("openjarvis.cli.ask")
+
+# A minimal but valid 1x1 PNG so ``click.Path(exists=True)`` is satisfied and
+# the bytes are deterministic.
+_PNG_BYTES = base64.b64decode(
+    "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk"
+    "+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
+)
+
+
+class _RecordingEngine:
+    """A fake engine that records the messages handed to ``generate()``."""
+
+    def __init__(self) -> None:
+        self.engine_id = "mock"
+        self.received: list[Any] = []
+
+    def health(self) -> bool:
+        return True
+
+    def list_models(self) -> list[str]:
+        return ["test-model"]
+
+    def generate(self, messages, *, model=None, **kwargs):
+        # Capture the exact Message objects the CLI built so the test can
+        # assert the image bytes reached the engine boundary.
+        self.received = list(messages)
+        return {
+            "content": "a 1x1 pixel",
+            "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+            "model": "test-model",
+            "finish_reason": "stop",
+        }
+
+
+def _patch_ask(monkeypatch, tmp_path: Path, *, engine_name: str) -> _RecordingEngine:
+    """Wire ``jarvis ask`` to a recording engine reported under ``engine_name``."""
+    cfg = JarvisConfig()
+    cfg.telemetry.db_path = str(tmp_path / "telemetry.db")
+    # Keep memory context out of the picture so the user message we inspect is
+    # the one the CLI built directly from the query + image.
+    cfg.agent.context_from_memory = False
+    monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg)
+
+    engine = _RecordingEngine()
+    monkeypatch.setattr(_ask_mod, "get_engine", lambda *a, **kw: (engine_name, engine))
+    monkeypatch.setattr(_ask_mod, "discover_engines", lambda c: [(engine_name, engine)])
+    monkeypatch.setattr(
+        _ask_mod, "discover_models", lambda e: {engine_name: ["test-model"]}
+    )
+    return engine
+
+
+def _write_png(tmp_path: Path) -> tuple[Path, str]:
+    img = tmp_path / "pixel.png"
+    img.write_bytes(_PNG_BYTES)
+    return img, base64.b64encode(_PNG_BYTES).decode("ascii")
+
+
+def test_image_reaches_engine_payload(monkeypatch, tmp_path: Path) -> None:
+    engine = _patch_ask(monkeypatch, tmp_path, engine_name="ollama")
+    img, expected_b64 = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    # The CLI must have routed to direct mode and called the engine.
+    assert engine.received, "engine.generate() was never called"
+    user_msgs = [m for m in engine.received if m.role == Role.USER]
+    assert user_msgs, "no USER message reached the engine"
+    assert user_msgs[-1].images == [expected_b64]
+
+
+def test_privacy_warning_for_non_local_engine(monkeypatch, tmp_path: Path) -> None:
+    engine = _patch_ask(monkeypatch, tmp_path, engine_name="openai")
+    img, expected_b64 = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    assert "Privacy warning" in result.output
+    # The warning is informational; the image must still be delivered.
+    user_msgs = [m for m in engine.received if m.role == Role.USER]
+    assert user_msgs and user_msgs[-1].images == [expected_b64]
+
+
+def test_no_privacy_warning_for_local_engine(monkeypatch, tmp_path: Path) -> None:
+    _patch_ask(monkeypatch, tmp_path, engine_name="ollama")
+    img, _ = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    assert "Privacy warning" not in result.output
@@ -437,3 +437,39 @@ class TestOpenRouterToolForwarding:
        assert result["tool_calls"][0]["id"] == "call_1"
        assert result["tool_calls"][0]["function"]["name"] == "get_weather"
        assert result["tool_calls"][0]["function"]["arguments"] == '{"city": "NYC"}'
+
+
+class TestCloudEngineCanServe:
+    """#532: can_serve gates on the per-provider client, not just health().
+
+    health() is True whenever *any* provider client is configured, but a
+    request for a gpt-* model still needs the OpenAI client specifically — so
+    engine selection must not pick the cloud engine for a model whose provider
+    client is missing.
+    """
+
+    @staticmethod
+    def _engine(**clients: object) -> CloudEngine:
+        eng = CloudEngine.__new__(CloudEngine)  # bypass real client init
+        for name in (
+            "_openai_client",
+            "_anthropic_client",
+            "_google_client",
+            "_openrouter_client",
+            "_minimax_client",
+            "_codex_client",
+        ):
+            setattr(eng, name, clients.get(name))
+        return eng
+
+    def test_openai_only_serves_openai_models(self) -> None:
+        eng = self._engine(_openai_client=object())
+        assert eng.can_serve("gpt-4o") is True
+        assert eng.can_serve("claude-sonnet-4") is False
+        assert eng.can_serve("gemini-2.5-pro") is False
+        assert eng.can_serve("openrouter/openai/gpt-4o") is False
+
+    def test_anthropic_only_serves_anthropic_models(self) -> None:
+        eng = self._engine(_anthropic_client=object())
+        assert eng.can_serve("claude-sonnet-4") is True
+        assert eng.can_serve("gpt-4o") is False
@@ -174,6 +174,51 @@ class TestGetEngine:
        assert result is not None
        assert result[0] == "running"

+    def test_skips_engine_that_cannot_serve_model(self) -> None:
+        """#532: a healthy engine that can't serve the requested model is
+        skipped for one that can — this is what stops the cloud fallback being
+        chosen (when the local engine is down) for a model whose provider
+        client is missing.
+        """
+        _reg("picky", "picky")
+        _reg("local", "local")
+
+        class _Picky(_FakeEngine):
+            def can_serve(self, model: str) -> bool:
+                return model == "servable"
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "picky"
+
+        def _make(k, c):  # noqa: ANN001
+            if k == "picky":
+                return _Picky(healthy=True)
+            return _FakeEngine(healthy=(k == "local"))
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=_make,
+        ):
+            # "picky" is healthy but cannot serve "other" -> fall back to "local"
+            result = get_engine(cfg, model="other")
+        assert result is not None
+        assert result[0] == "local"
+
+    def test_model_none_preserves_model_agnostic_selection(self) -> None:
+        """model=None keeps the legacy behaviour: first healthy engine wins."""
+        _reg("primary", "primary")
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "primary"
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=lambda k, c: _FakeEngine(healthy=True),  # noqa: ANN001
+        ):
+            result = get_engine(cfg, model=None)
+        assert result is not None
+        assert result[0] == "primary"
+

 class TestMiningSidecarEngineHandoff:
    """Engine discovery picks up (or ignores) a mining sidecar at runtime."""
@@ -0,0 +1,109 @@
+"""API-key (Authorization header) support in the OpenAI-compat engine base."""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+import respx
+
+from openjarvis.core.types import Message, Role
+from openjarvis.engine.openai_compat_engines import (
+    OpenAICompatEngine,
+    VLLMEngine,
+    normalize_openai_base_url,
+)
+
+_CHAT_RESPONSE = {
+    "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}],
+    "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+    "model": "m",
+}
+
+
+class TestAuthorizationHeader:
+    def test_bearer_header_sent_when_api_key_set(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_no_authorization_header_without_api_key(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert "authorization" not in route.calls.last.request.headers
+
+    def test_health_check_sends_bearer_header(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_env_var_fallback_sanitizes_hyphen(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        # engine_id "openai-compat" must map to OPENAI_COMPAT_API_KEY —
+        # shells cannot set hyphenated env-var names.
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-env"
+
+    def test_vllm_env_var_fallback(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        monkeypatch.setenv("VLLM_API_KEY", "sk-vllm")
+        engine = VLLMEngine(host="http://testhost:8000")
+        with respx.mock:
+            route = respx.get("http://testhost:8000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-vllm"
+
+    def test_explicit_api_key_beats_env_var(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
+        assert engine._api_key == "sk-explicit"
+
+
+class TestNormalizeOpenAIBaseUrl:
+    @pytest.mark.parametrize(
+        ("url", "expected"),
+        [
+            ("http://h:8000", "http://h:8000"),
+            ("http://h:8000/", "http://h:8000"),
+            ("http://h:8000/v1", "http://h:8000"),
+            ("http://h:8000/v1/", "http://h:8000"),
+            ("http://h:8000/gateway/v1", "http://h:8000/gateway"),
+            # Only a literal trailing "/v1" is stripped — never other paths.
+            ("http://h:8000/v1x", "http://h:8000/v1x"),
+            ("http://h:8000/v2", "http://h:8000/v2"),
+        ],
+    )
+    def test_normalization(self, url: str, expected: str) -> None:
+        assert normalize_openai_base_url(url) == expected
+
+    def test_engine_requests_have_single_v1_prefix(self) -> None:
+        """End to end: a user-supplied .../v1 URL must not produce /v1/v1."""
+        host = normalize_openai_base_url("http://testhost:9000/v1")
+        engine = OpenAICompatEngine(host=host)
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.url.path == "/v1/models"
@@ -0,0 +1,72 @@
+"""Tests for terminal-bench harness timeout plumbing through TOML configs."""
+
+from __future__ import annotations
+
+import textwrap
+
+from openjarvis.evals.core.config import expand_suite, load_eval_config
+
+
+def _write(tmp_path, body: str):
+    path = tmp_path / "suite.toml"
+    path.write_text(textwrap.dedent(body))
+    return path
+
+
+BASE = """
+    [meta]
+    name = "timeouts"
+
+    [run]
+    output_dir = "results/"
+    {run_extra}
+
+    [[models]]
+    name = "test-model"
+
+    [[benchmarks]]
+    name = "terminalbench-native"
+    backend = "terminalbench-native"
+    {bench_extra}
+"""
+
+
+class TestTimeoutConfigPlumbing:
+    def test_run_level_timeouts_parse_and_expand(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra=(
+                    "global_agent_timeout_sec = 1200\n"
+                    "    global_timeout_multiplier = 1.5"
+                ),
+                bench_extra="",
+            ),
+        )
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec == 1200.0
+        assert suite.run.global_timeout_multiplier == 1.5
+
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec == 1200.0
+        assert rc.global_timeout_multiplier == 1.5
+
+    def test_benchmark_override_wins(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra="global_agent_timeout_sec = 1200",
+                bench_extra="global_agent_timeout_sec = 300",
+            ),
+        )
+        (rc,) = expand_suite(load_eval_config(path))
+        assert rc.global_agent_timeout_sec == 300.0
+
+    def test_defaults_are_none(self, tmp_path):
+        path = _write(tmp_path, BASE.format(run_extra="", bench_extra=""))
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec is None
+        assert suite.run.global_timeout_multiplier is None
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec is None
+        assert rc.global_timeout_multiplier is None
@@ -9,6 +9,7 @@ from typing import Any, Dict, List
 import pytest

 from openjarvis.evals.core.agentic_runner import AgenticRunner, _extract_patch
+from openjarvis.evals.core.environment import TaskEnvironmentError

 # ---------------------------------------------------------------------------
 # Mock objects
@@ -50,6 +51,66 @@ class MockFailingAgent:
        raise RuntimeError("Agent error")


+class MockZeroContactAgent:
+    """Agent that returns without ever contacting the model.
+
+    Mirrors the downstream failure signature: a hung/dead in-container
+    setup yields zero LM events and zero token usage, while run_tests
+    still stamps is_resolved=False.
+    """
+
+    def ask(self, query: str) -> dict:
+        return {"content": "setup log tail ...", "usage": {}}
+
+
+class FailingTaskEnv:
+    """Task env whose __enter__ fails like a tmux/compose breakage."""
+
+    def __init__(self, metadata: Dict[str, Any]) -> None:
+        self._metadata = metadata
+
+    def __enter__(self) -> "FailingTaskEnv":
+        message = (
+            "Task 't1': required binary 'tmux' is not usable in task image "
+            "'tb__t1__client'"
+        )
+        self._metadata["harness_error"] = message
+        raise TaskEnvironmentError(message)
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+
+class ResolvingTaskEnv:
+    """Task env that stamps is_resolved into metadata like run_tests does."""
+
+    def __init__(self, metadata: Dict[str, Any], is_resolved: bool) -> None:
+        self._metadata = metadata
+        self._is_resolved = is_resolved
+
+    def __enter__(self) -> "ResolvingTaskEnv":
+        return self
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+    def run_tests(self):
+        self._metadata["is_resolved"] = self._is_resolved
+        return self._is_resolved, {}
+
+
+class EnvDataset(MockDataset):
+    """Dataset whose create_task_env is configurable per record id."""
+
+    def __init__(self, records: List[MockRecord], env_factories: Dict[str, Any]):
+        super().__init__(records)
+        self._env_factories = env_factories
+
+    def create_task_env(self, record: MockRecord):
+        factory = self._env_factories.get(record.record_id)
+        return factory(record.metadata) if factory is not None else None
+
+
 # ---------------------------------------------------------------------------
 # Tests
 # ---------------------------------------------------------------------------
@@ -99,6 +160,59 @@ class TestAgenticRunner:
        assert len(traces) == 1
        assert not traces[0].completed
        assert "Agent error" in traces[0].response_text
+        assert traces[0].error_kind == "agent_error"
+        assert "Agent error" in (traces[0].error or "")
+
+    def test_harness_failure_recorded_and_run_continues(self):
+        """(c) one task's env breakage doesn't kill the run."""
+        records = [
+            MockRecord(record_id="r1", problem="broken env"),
+            MockRecord(record_id="r2", problem="healthy"),
+        ]
+        dataset = EnvDataset(records, {"r1": FailingTaskEnv})
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert len(traces) == 2
+        # Failed task: recorded distinctly as a harness error, not a miss.
+        assert traces[0].error_kind == "harness_error"
+        assert not traces[0].completed
+        assert "tmux" in (traces[0].error or "")
+        assert traces[0].is_resolved is None
+        # Healthy task still ran to completion.
+        assert traces[1].completed
+        assert traces[1].error_kind is None
+        assert "Response to: healthy" in traces[1].response_text
+
+    def test_zero_model_contact_flagged_as_harness_error(self):
+        """(e) zero model requests -> harness_error, not a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockZeroContactAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].error_kind == "harness_error"
+        assert "zero_model_requests" in (traces[0].error or "")
+        assert traces[0].total_input_tokens == 0
+        assert traces[0].total_output_tokens == 0
+
+    def test_genuine_model_miss_not_flagged(self):
+        """(e) control: tokens>0 + is_resolved=False is a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].is_resolved is False
+        assert traces[0].error_kind is None
+        assert traces[0].error is None
+        assert traces[0].total_input_tokens > 0

    def test_synthetic_turn_created(self):
        records = [MockRecord(record_id="r1", problem="test")]
@@ -127,6 +127,84 @@ class TestExportSummaryJson:
        assert summary["totals"]["queries"] == 0


+class TestHarnessErrorExclusion:
+    """Harness errors must never count as model misses in resolve-rate."""
+
+    def _traces(self):
+        return [
+            QueryTrace(
+                query_id="q0000",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=True,
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0001",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=False,  # genuine model miss: stays in denominator
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0002",
+                workload_type="agentic",
+                completed=False,
+                is_resolved=False,  # stamped by run_tests despite zero contact
+                error="zero_model_requests: the agent never contacted the model",
+                error_kind="harness_error",
+            ),
+        ]
+
+    def test_summary_excludes_harness_errors_from_accuracy(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(self._traces(), {"model": "m"}, path)
+        summary = json.loads(path.read_text())
+
+        totals = summary["totals"]
+        assert totals["harness_errors"] == 1
+        assert totals["resolved"] == 1
+        assert totals["unresolved"] == 1  # NOT 2: harness error excluded
+        assert totals["accuracy"] == 0.5  # NOT 1/3
+        # Flat table_gen metrics exclude the harness error too.
+        assert summary["metrics"]["accuracy"]["n"] == 2
+        assert summary["metrics"]["accuracy"]["mean"] == 0.5
+        # Details surfaced for diagnosis.
+        details = summary["harness_error_details"]
+        assert details[0]["query_id"] == "q0002"
+        assert "zero_model_requests" in details[0]["error"]
+
+    def test_efficiency_excludes_harness_errors(self):
+        result = _compute_efficiency(self._traces(), None, None)
+        assert result["accuracy"] == 0.5
+
+    def test_no_harness_errors_key_absent(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(_make_traces(), {}, path)
+        summary = json.loads(path.read_text())
+        assert summary["totals"]["harness_errors"] == 0
+        assert "harness_error_details" not in summary
+
+
+class TestTraceErrorFieldRoundTrip:
+    def test_round_trip_preserves_error_fields(self):
+        trace = QueryTrace(
+            query_id="q0",
+            workload_type="agentic",
+            error="zero_model_requests: ...",
+            error_kind="harness_error",
+        )
+        restored = QueryTrace.from_dict(trace.to_dict())
+        assert restored.error == trace.error
+        assert restored.error_kind == "harness_error"
+
+    def test_old_trace_dicts_still_load(self):
+        """Backward compat: traces.jsonl written before the schema change."""
+        restored = QueryTrace.from_dict({"query_id": "q0", "workload_type": "agentic"})
+        assert restored.error is None
+        assert restored.error_kind is None
+
+
 class TestExportArtifactsManifest:
    def test_no_artifacts_dir(self, tmp_path):
        result = export_artifacts_manifest(tmp_path)
@@ -1,15 +1,104 @@
-"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency)."""
+"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency).
+
+These tests install a fake ``terminal_bench`` module tree into
+``sys.modules`` so they run without the real package or a Docker daemon
+(terminal-bench is an undeclared optional dep that CI never installs).
+"""

 from __future__ import annotations

+import sys
+import types
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from types import SimpleNamespace
+from typing import Any, Dict, List
+
 import pytest

+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.execution.terminalbench_env import TerminalBenchTaskEnv

-# terminal_bench is an optional dep — skip all tests if unavailable
-terminal_bench = pytest.importorskip(
-    "terminal_bench", reason="terminal_bench not installed"
-)
+# ---------------------------------------------------------------------------
+# Fake terminal_bench seam
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class FakeExecResult:
+    exit_code: int = 0
+    output: bytes = b""
+
+
+@dataclass
+class FakeContainer:
+    """Container stub whose exec_run results are configurable per binary."""
+
+    exec_results: Dict[str, FakeExecResult] = field(default_factory=dict)
+    exec_calls: List[List[str]] = field(default_factory=list)
+
+    def exec_run(self, cmd: List[str]) -> FakeExecResult:
+        self.exec_calls.append(list(cmd))
+        return self.exec_results.get(cmd[0], FakeExecResult())
+
+
+class FakeTerminal:
+    def __init__(self, events: List[str], container: FakeContainer) -> None:
+        self._events = events
+        self.container = container
+        self.create_session_error: Exception | None = None
+
+    def create_session(self, name: str, **_kwargs: Any) -> str:
+        self._events.append(f"create_session({name})")
+        if self.create_session_error is not None:
+            raise self.create_session_error
+        return f"session-{name}"
+
+
+@pytest.fixture()
+def fake_tb(monkeypatch, tmp_path):
+    """Install a fake terminal_bench tree; return the shared test state."""
+    events: List[str] = []
+    container = FakeContainer()
+    terminal = FakeTerminal(events, container)
+    state = SimpleNamespace(
+        events=events,
+        container=container,
+        terminal=terminal,
+        spin_up_error=None,
+    )
+
+    @contextmanager
+    def spin_up_terminal(**kwargs: Any):
+        events.append("compose_up")
+        try:
+            if state.spin_up_error is not None:
+                raise state.spin_up_error
+            yield terminal
+        finally:
+            events.append("compose_down")
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_terminal_pkg = types.ModuleType("terminal_bench.terminal")
+    mod_terminal = types.ModuleType("terminal_bench.terminal.terminal")
+    mod_terminal.spin_up_terminal = spin_up_terminal
+    mod_tb.terminal = mod_terminal_pkg
+    mod_terminal_pkg.terminal = mod_terminal
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal", mod_terminal_pkg)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal.terminal", mod_terminal)
+
+    state.metadata = {
+        "task_id": "hello.world",
+        "task": SimpleNamespace(disable_asciinema=True),
+        "task_paths": SimpleNamespace(docker_compose_path=tmp_path / "compose.yaml"),
+    }
+    return state
+
+
+# ---------------------------------------------------------------------------
+# Existing behavior (now running without the real terminal_bench package)
+# ---------------------------------------------------------------------------


 class TestTerminalBenchTaskEnv:
@@ -46,3 +135,113 @@ class TestTerminalBenchTaskEnv:
        assert is_resolved is False
        assert results["error"] == "terminal_not_running"
        assert metadata["is_resolved"] is False
+
+
+# ---------------------------------------------------------------------------
+# Exception-safe __enter__ / preflight / teardown
+# ---------------------------------------------------------------------------
+
+
+class TestEnterExceptionSafety:
+    def test_success_path(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            assert fake_tb.metadata["terminal"] is fake_tb.terminal
+            assert fake_tb.metadata["session"] == "session-agent"
+            assert fake_tb.metadata["container"] == "oj-hello-world"
+            assert "compose_down" not in fake_tb.events
+        assert fake_tb.events.count("compose_down") == 1
+        assert "terminal" not in fake_tb.metadata
+
+    def test_create_session_failure_tears_down_terminal(self, fake_tb):
+        """(a) tmux failure in __enter__ -> terminal torn down, loud error."""
+        fake_tb.terminal.create_session_error = RuntimeError(
+            "tmux is not installed in the container."
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        # No leak: compose project downed BEFORE the exception escaped.
+        assert "compose_down" in fake_tb.events
+        # Actionable: names the task, the image, and the failure.
+        message = str(excinfo.value)
+        assert "hello.world" in message
+        assert "tb__hello-world__client" in message
+        assert "tmux is not installed" in message
+        # Recorded for the runner / scorers; handles cleared.
+        assert fake_tb.metadata["harness_error"] == message
+        assert "terminal" not in fake_tb.metadata
+        assert "session" not in fake_tb.metadata
+        assert "container" not in fake_tb.metadata
+        assert env._terminal is None
+        assert env._terminal_cm is None
+        assert env._logs_tmpdir is None
+
+    def test_preflight_catches_missing_tmux(self, fake_tb):
+        """(b) preflight catches missing tmux, naming the task image."""
+        fake_tb.container.exec_results["tmux"] = FakeExecResult(
+            exit_code=127,
+            output=b'exec: "tmux": executable file not found in $PATH',
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        message = str(excinfo.value)
+        assert "tmux" in message
+        assert "tb__hello-world__client" in message  # task image named
+        assert "127" in message
+        # Fired BEFORE the agent session was created.
+        assert not any(e.startswith("create_session") for e in fake_tb.events)
+        assert "compose_down" in fake_tb.events
+        assert fake_tb.metadata["harness_error"] == message
+
+    def test_preflight_checks_asciinema_when_recording(self, fake_tb):
+        fake_tb.metadata["task"] = SimpleNamespace(disable_asciinema=False)
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError, match="asciinema"):
+            env.__enter__()
+        assert "disable_asciinema" in fake_tb.metadata["harness_error"]
+        assert "compose_down" in fake_tb.events
+
+    def test_preflight_skips_asciinema_when_disabled(self, fake_tb):
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            pass
+        assert ["asciinema", "--version"] not in fake_tb.container.exec_calls
+
+    def test_compose_up_failure_includes_stderr(self, fake_tb):
+        import subprocess
+
+        fake_tb.spin_up_error = subprocess.CalledProcessError(
+            returncode=1,
+            cmd=["docker", "compose", "up"],
+            stderr="no space left on device",
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+        assert "no space left on device" in str(excinfo.value)
+
+    def test_keyboard_interrupt_not_masked(self, fake_tb):
+        fake_tb.terminal.create_session_error = KeyboardInterrupt()
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(KeyboardInterrupt):
+            env.__enter__()
+        # Still cleaned up, but the interrupt is not wrapped.
+        assert "compose_down" in fake_tb.events
+
+    def test_teardown_is_idempotent(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        env.__enter__()
+        env.__exit__(None, None, None)
+        env.__exit__(None, None, None)
+        assert fake_tb.events.count("compose_down") == 1
@@ -0,0 +1,302 @@
+"""Tests for the TerminalBench native backend (mocked terminal_bench).
+
+Covers: timeout kwargs threading (config -> backend -> Harness kwargs),
+loud failure on terminal-bench builds without the timeout kwargs, and the
+harness-error classification in ``summarize_benchmark_results`` —
+including the zero-model-contact vs genuine-model-miss distinction.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+from types import SimpleNamespace
+from typing import Any, Dict, List, Optional
+
+import pytest
+
+import openjarvis.evals.backends.terminalbench_native as tbn
+from openjarvis.evals.backends.terminalbench_native import (
+    summarize_benchmark_results,
+)
+
+# ---------------------------------------------------------------------------
+# Helpers / fakes
+# ---------------------------------------------------------------------------
+
+
+def make_trial(
+    task_id: str,
+    *,
+    is_resolved: bool,
+    failure_mode: str = "unset",
+    input_tokens: Optional[int] = None,
+    output_tokens: Optional[int] = None,
+) -> SimpleNamespace:
+    """Build a duck-typed terminal-bench 0.2.18 TrialResults."""
+    return SimpleNamespace(
+        task_id=task_id,
+        trial_name=f"{task_id}.1-of-1",
+        is_resolved=is_resolved,
+        failure_mode=SimpleNamespace(value=failure_mode),
+        total_input_tokens=input_tokens,
+        total_output_tokens=output_tokens,
+    )
+
+
+class FakeHarness:
+    """Records constructor kwargs; run() returns the configured results."""
+
+    captured_kwargs: Dict[str, Any] = {}
+    results: Any = SimpleNamespace(results=[])
+
+    def __init__(self, **kwargs: Any) -> None:
+        type(self).captured_kwargs = kwargs
+
+    def run(self) -> Any:
+        return type(self).results
+
+
+class OldFakeHarness:
+    """A pre-timeout-kwargs Harness signature (no **kwargs)."""
+
+    def __init__(
+        self,
+        output_path: Any = None,
+        run_id: Any = None,
+        dataset_name: Any = None,
+        dataset_version: Any = None,
+        model_name: Any = None,
+        n_concurrent_trials: Any = None,
+        cleanup: Any = None,
+        agent_name: Any = None,
+        agent_kwargs: Any = None,
+        n_tasks: Any = None,
+    ) -> None:
+        pass
+
+    def run(self) -> Any:
+        return SimpleNamespace(results=[])
+
+
+@pytest.fixture()
+def fake_tb_backend(monkeypatch):
+    """Enable the backend without the real terminal_bench package."""
+    FakeHarness.captured_kwargs = {}
+    FakeHarness.results = SimpleNamespace(results=[])
+    monkeypatch.setattr(tbn, "_HAS_TB", True)
+    monkeypatch.setattr(tbn, "Harness", FakeHarness, raising=False)
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_agents = types.ModuleType("terminal_bench.agents")
+    mod_agent_name = types.ModuleType("terminal_bench.agents.agent_name")
+    mod_agent_name.AgentName = lambda name: name
+    mod_tb.agents = mod_agents
+    mod_agents.agent_name = mod_agent_name
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents", mod_agents)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents.agent_name", mod_agent_name)
+    return FakeHarness
+
+
+# ---------------------------------------------------------------------------
+# Timeout kwargs threading
+# ---------------------------------------------------------------------------
+
+
+class TestTimeoutKwargs:
+    def test_default_bound_reaches_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1800.0
+        assert "global_timeout_multiplier" not in kwargs
+
+    def test_explicit_values_reach_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path),
+            global_agent_timeout_sec=1234.0,
+            global_timeout_multiplier=2.0,
+        )
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1234.0
+        assert kwargs["global_timeout_multiplier"] == 2.0
+
+    def test_zero_disables_bound(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path), global_agent_timeout_sec=0
+        )
+        backend.run_harness("run-1")
+        assert "global_agent_timeout_sec" not in fake_tb_backend.captured_kwargs
+
+    def test_old_terminal_bench_fails_loud(
+        self, fake_tb_backend, monkeypatch, tmp_path
+    ):
+        """An old Harness without the kwargs must not hang silently."""
+        monkeypatch.setattr(tbn, "Harness", OldFakeHarness, raising=False)
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        with pytest.raises(RuntimeError, match="global_agent_timeout_sec"):
+            backend.run_harness("run-1")
+
+
+# ---------------------------------------------------------------------------
+# Harness-error classification (zero-model-contact detection)
+# ---------------------------------------------------------------------------
+
+
+class TestSummarizeBenchmarkResults:
+    def test_genuine_model_miss_is_not_flagged(self):
+        """MANDATORY: a real miss (tokens>0, failure_mode unset) stays a miss.
+
+        terminal-bench 0.2.18 leaves failure_mode UNSET on success and on
+        genuine unresolved misses — classifying on failure_mode would flag
+        every real miss as a harness error and inflate accuracy.
+        """
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-ok", is_resolved=True, input_tokens=900, output_tokens=100
+                ),
+                make_trial(
+                    "t-miss", is_resolved=False, input_tokens=800, output_tokens=50
+                ),
+                make_trial(
+                    "t-setup-dead",
+                    is_resolved=False,
+                    failure_mode="agent_installation_failed",
+                    input_tokens=0,
+                    output_tokens=0,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2  # genuine miss stays in denominator
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5  # not 1.0 (miss kept), not 1/3 (infra out)
+        assert summary.errors == 1
+        assert [f["task_id"] for f in failures] == ["t-setup-dead"]
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_zero_contact_flagged_even_with_unset_failure_mode(self):
+        """Setup hang signature: unresolved, zero requests, failure_mode unset."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-hang", is_resolved=False, input_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert summary.scored_samples == 0
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_missing_token_fields_treated_as_zero_contact(self):
+        results = SimpleNamespace(results=[make_trial("t-none", is_resolved=False)])
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+
+    def test_infra_failure_mode_flagged_despite_tokens(self):
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-crash",
+                    is_resolved=False,
+                    failure_mode="unknown_agent_error",
+                    input_tokens=500,
+                    output_tokens=20,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert failures[0]["reason"] == "unknown_agent_error"
+
+    def test_resolved_with_zero_tokens_not_flagged(self):
+        """Installed agents report 0 tokens on success — never flag resolved."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-ok", is_resolved=True, input_tokens=0, output_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 0
+        assert summary.correct == 1
+        assert summary.accuracy == 1.0
+
+    def test_empty_results(self):
+        summary, failures = summarize_benchmark_results(
+            SimpleNamespace(results=[]), model="m"
+        )
+        assert summary.total_samples == 0
+        assert summary.accuracy == 0.0
+        assert failures == []
+
+
+# ---------------------------------------------------------------------------
+# CLI wiring: config -> backend -> harness kwargs -> RunSummary
+# ---------------------------------------------------------------------------
+
+
+class TestRunTerminalbenchNativeWiring:
+    def _run(self, fake_tb_backend, tmp_path, trials: List[Any], **config_kwargs):
+        from rich.console import Console
+
+        from openjarvis.evals.cli import _run_terminalbench_native
+        from openjarvis.evals.core.types import RunConfig
+
+        fake_tb_backend.results = SimpleNamespace(results=trials)
+        config = RunConfig(
+            benchmark="terminalbench-native",
+            backend="terminalbench-native",
+            model="test-model",
+            output_path=str(tmp_path / "out"),
+            **config_kwargs,
+        )
+        console = Console(record=True, width=120)
+        summary = _run_terminalbench_native(config, console)
+        return summary, console.export_text()
+
+    def test_config_timeouts_reach_harness_kwargs(self, fake_tb_backend, tmp_path):
+        """(d) timeout kwargs travel config -> backend -> harness_kwargs."""
+        self._run(
+            fake_tb_backend,
+            tmp_path,
+            [],
+            global_agent_timeout_sec=901.0,
+            global_timeout_multiplier=1.5,
+        )
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 901.0
+        assert kwargs["global_timeout_multiplier"] == 1.5
+
+    def test_config_defaults_use_backend_bound(self, fake_tb_backend, tmp_path):
+        self._run(fake_tb_backend, tmp_path, [])
+        assert fake_tb_backend.captured_kwargs["global_agent_timeout_sec"] == 1800.0
+
+    def test_summary_counts_real_trials(self, fake_tb_backend, tmp_path):
+        """Regression: results field is ``results``, not ``trial_results``.
+
+        The old conversion read the nonexistent ``trial_results`` attribute
+        and hardcoded errors=0, rendering every run as 0 samples / 0.0.
+        """
+        trials = [
+            make_trial("t-ok", is_resolved=True, input_tokens=10, output_tokens=10),
+            make_trial("t-miss", is_resolved=False, input_tokens=10, output_tokens=2),
+            make_trial(
+                "t-hang",
+                is_resolved=False,
+                failure_mode="agent_timeout",
+                input_tokens=0,
+                output_tokens=0,
+            ),
+        ]
+        summary, output = self._run(fake_tb_backend, tmp_path, trials)
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5
+        assert summary.errors == 1
+        # The harness failure is reported loudly with its task id.
+        assert "t-hang" in output
+        assert "zero_model_requests" in output
@@ -2,7 +2,7 @@

 from __future__ import annotations

-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch

 import pytest

@@ -327,6 +327,85 @@ class TestSystemBuilder:
        assert builder._engine_key == "ollama"


+class TestSystemBuilderEngineInstance:
+    """Explicit engine injection (jarvis eval --base-url path)."""
+
+    @staticmethod
+    def _fake_engine(healthy: bool = True) -> MagicMock:
+        engine = MagicMock(
+            spec=["health", "can_serve", "generate", "list_models", "close"]
+        )
+        engine.health.return_value = healthy
+        engine._host = "http://127.0.0.1:18999"
+        return engine
+
+    def test_engine_instance_is_fluent(self):
+        builder = SystemBuilder(JarvisConfig())
+        engine = self._fake_engine()
+        result = builder.engine_instance(engine, key="my-endpoint")
+        assert result is builder
+        assert builder._engine_instance is engine
+        assert builder._engine_instance_key == "my-endpoint"
+
+    def test_resolve_engine_returns_injected_instance(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        resolved_engine, resolved_key = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        assert resolved_key == "endpoint"
+
+    def test_unhealthy_injected_instance_raises_naming_host(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18999"):
+            builder._resolve_engine(config)
+
+    def test_unhealthy_injected_instance_never_consults_discovery(self):
+        """The observed failure mode: an explicit endpoint must NOT be
+        silently replaced by whatever other engine discovery finds."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine)
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            with pytest.raises(RuntimeError, match="Refusing to fall back"):
+                builder._resolve_engine(config)
+        mock_get_engine.assert_not_called()
+
+    def test_healthy_injected_instance_never_consults_discovery(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            resolved_engine, _ = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        mock_get_engine.assert_not_called()
+
+    def test_build_wires_injected_engine(self):
+        """build() must use the injected engine (possibly behind security
+        wrappers) instead of running discovery."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        engine.list_models.return_value = ["stub-model"]
+        builder = (
+            SystemBuilder(config)
+            .engine_instance(engine, key="endpoint")
+            .model("stub-model")
+            .telemetry(False)
+            .traces(False)
+        )
+        system = builder.build()
+        try:
+            inner = system.engine
+            while hasattr(inner, "_engine"):
+                inner = inner._engine
+            assert inner is engine
+            assert system.engine_key == "endpoint"
+        finally:
+            system.close()
+
+
 class TestJarvisSystemClose:
    def test_close_with_scheduler_store(self):
        engine = MagicMock()
@@ -0,0 +1,94 @@
+"""Tests for vision input support: ``Message.images`` -> Ollama payload.
+
+These cover the data-flow contract that makes vision work end to end:
+a ``Message`` can carry base64 images, the engine serializer forwards them
+to Ollama's ``/api/chat`` ``images`` field, and text-only messages are
+completely unaffected. The security guardrail must preserve images when it
+rewrites a flagged message.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import openjarvis.engine.ollama as ollama_mod
+from openjarvis.core.types import Message, Role
+from openjarvis.engine._base import messages_to_dicts
+
+
+def test_message_defaults_to_no_images() -> None:
+    assert Message(role=Role.USER, content="hi").images is None
+
+
+def test_messages_to_dicts_omits_images_for_text() -> None:
+    dicts = messages_to_dicts([Message(role=Role.USER, content="hi")])
+    assert "images" not in dicts[0]
+
+
+def test_messages_to_dicts_forwards_images() -> None:
+    b64 = "aGVsbG8="  # "hello"
+    dicts = messages_to_dicts(
+        [Message(role=Role.USER, content="what is this?", images=[b64])]
+    )
+    assert dicts[0]["role"] == "user"
+    assert dicts[0]["content"] == "what is this?"
+    assert dicts[0]["images"] == [b64]
+
+
+def test_messages_to_dicts_empty_images_treated_as_text() -> None:
+    dicts = messages_to_dicts([Message(role=Role.USER, content="hi", images=[])])
+    assert "images" not in dicts[0]
+
+
+def test_default_num_ctx_default_and_override(monkeypatch) -> None:
+    monkeypatch.delenv("JARVIS_NUM_CTX", raising=False)
+    assert ollama_mod._default_num_ctx() == 16384
+
+    monkeypatch.setenv("JARVIS_NUM_CTX", "8000")
+    assert ollama_mod._default_num_ctx() == 8000
+
+    # A non-integer override must fall back to the safe default, not crash.
+    monkeypatch.setenv("JARVIS_NUM_CTX", "not-an-int")
+    assert ollama_mod._default_num_ctx() == 16384
+
+
+def test_guardrails_preserves_images_when_sanitizing() -> None:
+    """A flagged message gets rewritten; its image must survive the rewrite."""
+    from openjarvis.security.guardrails import GuardrailsEngine
+
+    class _RecordingEngine:
+        """Captures the messages the guardrail forwards to the real engine."""
+
+        def __init__(self) -> None:
+            self.received: list[Message] = []
+
+        def generate(self, messages, *, model, **kwargs):
+            self.received = list(messages)
+            return {"content": "ok"}
+
+    class _AlwaysFlag:
+        """A scanner that flags everything, forcing the sanitize rewrite path."""
+
+        def scan(self, text: str):
+            finding = SimpleNamespace(
+                pattern_name="test",
+                threat_level=SimpleNamespace(value="low"),
+                description="always flags",
+            )
+            return SimpleNamespace(findings=[finding])
+
+        def redact(self, text: str) -> str:
+            return text
+
+    engine = _RecordingEngine()
+    guarded = GuardrailsEngine(
+        engine,
+        scanners=[_AlwaysFlag()],
+        scan_input=True,
+        scan_output=False,
+    )
+    msg = Message(role=Role.USER, content="suspicious", images=["aGVsbG8="])
+
+    guarded.generate([msg], model="x")
+
+    assert engine.received[0].images == ["aGVsbG8="]
Author	SHA1	Message	Date
Joshua Azmy Claude Opus 4.8 Jon Saad-Falcon	79e23719d4	feat(vision): add image + screen capture input for vision models (#486 ) OpenJarvis can run vision-capable local models (gemma3, qwen2.5-vl), but the CLI had no way to send them a picture -- the Ollama engine only serialized text. This adds end-to-end image input. What's new - `jarvis ask -i/--image <file>` attaches one or more images to the query. - `jarvis ask -S/--screen` captures the primary monitor (dependency-free on Windows via .NET; mss/Pillow fallback elsewhere). - Vision auto-routes to direct-to-engine mode; with an explicit --agent it warns rather than silently dropping the image. - Privacy guard: warns before sending an image to a non-local engine, keeping OpenJarvis local-first by default. - Context-window default raised 8k -> 16k (JARVIS_NUM_CTX) so an image plus a conversation fit. Implementation - Message.images carries base64 data; messages_to_dicts() forwards it to Ollama's /api/chat "images" field. Text-only messages are unchanged. - GuardrailsEngine preserves images when it rewrites a flagged message. Tests (tests/test_vision.py, 6/6 pass, ruff-clean) - payload forwarding, text path untouched, num_ctx override, guardrail image preservation. Verified on AMD RX 9070 XT (Ollama/Vulkan, 100% GPU) with gemma3:4b: solid-color image, file image, and live screen capture all described. Co-authored-by: Claude Opus 4.8 <noreply@anthropic.com> Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>	2026-06-14 18:19:00 -07:00
SANJAYandsanjayravit	7ba334b5f0	fix(gui): inject bearer token into streaming chat/research routes (#499 ) Ensures that the desktop GUI correctly sends the Authorization header when an API key is configured. This resolves the 'Failed to get response' bug on Windows systems with enabled authentication. Ref: #266 Co-authored-by: sanjayravit <sanjay@example.com>	2026-06-14 17:32:43 -07:00
github-actions[bot]	48a2627c9a	chore: update clone traffic data [skip ci]	2026-06-14 07:38:47 +00:00
github-actions[bot]	8625f4f95f	chore: update clone traffic data [skip ci]	2026-06-13 07:21:38 +00:00
github-actions[bot]	cf08f164c0	chore: update clone traffic data [skip ci]	2026-06-12 07:41:03 +00:00
Jon Saad-FalconandClaude Opus 4.8	b21463aab6	fix(evals): harden terminalbench-native harness against tmux death and setup hangs (#536 ) Two failure classes hit by a downstream team: 1) tmux/task-env death (TerminalBenchTaskEnv.__enter__, terminalbench_env.py): create_session ran inside the spin_up_terminal generator-CM with no exception safety — a tmux failure leaked the docker compose project (down deferred to GC, never if the env was retained) and surfaced as an opaque mid-run death. Now: exception-safe __enter__ with an idempotent _teardown(), a tmux/asciinema preflight in the container BEFORE the agent loop (TaskEnvironmentError naming the task image + remedy), and fail-that-task-cleanly semantics — the failure is recorded as a harness error (QueryTrace.error_kind="harness_error"), the container is downed, and the run continues. 2) OpenHands in-container SETUP hang misattributed as a model result: harness.run() had no bound (terminal-bench runs installed-agent setup with max_timeout_sec=inf inside the per-trial agent budget), and the summary conversion read the nonexistent results.trial_results attr and hardcoded errors=0, folding zero-model-request trials into resolve-rate as model misses. Now: global_agent_timeout_sec / global_timeout_multiplier are threaded config -> backend -> Harness kwargs (default 1800 s bound on SETUP+RUN; configurable per [run]/[[benchmarks]] TOML), and summarize_benchmark_results() classifies harness/infra failures out of the accuracy denominator keyed on token usage (zero/missing tokens + unresolved = the agent never contacted the model), NOT failure_mode — terminal-bench 0.2.18 leaves failure_mode UNSET on success AND on genuine misses, so a failure_mode-based check would misflag every real model miss. Genuine misses (tokens>0, is_resolved=false) stay in the denominator. QueryTrace gains error/error_kind (wired through to_dict/from_dict so the fields actually reach traces.jsonl; backward-compatible loads), the AgenticRunner flags zero-model-contact traces and TaskEnvironmentError as harness errors, and export/summary/console output exclude harness errors from resolve-rate while reporting them loudly. terminal-bench stays an undeclared dep on purpose: it requires Python >=3.12 while this project supports >=3.10,<3.14, so an unmarked extra would break uv lock. pyproject/uv.lock untouched. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 14:30:10 -07:00
Jon Saad-FalconandClaude Opus 4.8	0cac61d3bb	docs(evals): rewrite evaluations guide to match the real CLI surface; add openjarvis-eval alias (#534 ) docs/user-guide/evaluations.md documented a standalone "openjarvis-evals" package, a "uv sync --extra eval" install, and an "openjarvis-eval" console script — a layout from commit `bd493832` that was never an ancestor of main. Rewrite the page against the real surface (jarvis eval / python -m openjarvis.evals), document all 40 registered benchmark keys and 4 backends, fix the judge-model default, correct run-all semantics, and split the option reference into the jarvis-eval subset and the module CLI's research-only options. Add the one-line [project.scripts] alias openjarvis-eval = "openjarvis.evals.cli:main" (the click group the module CLI already dispatches to) so the long-documented command name works again. The page was a complete orphan: add it (and the equally orphaned benchmarks.md) to the mkdocs nav and link it from docs/index.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 13:50:17 -07:00
Jon Saad-FalconandClaude Opus 4.8	50993dfa4d	fix(evals): honor --base-url/--api-key for first-party eval backends (#535 ) `jarvis eval run --base-url ... --api-key ...` was silently dropped for jarvis-direct/jarvis-agent (_build_backend only forwarded the flags to hermes/openclaw) and ignored by terminalbench-native, which hardcoded api_base="http://localhost:8000/v1". Worse, with --base-url set the engine-discovery fallback silently substituted ANY healthy local engine (observed: requested vllm + healthy endpoint at --base-url, got OllamaEngine@localhost:11434 — the requested URL was never contacted). Changes: - _OpenAICompatibleEngine gains an api_key param (Bearer Authorization header on the httpx client; {ENGINE_ID}_API_KEY env fallback with hyphen-sanitized names; no header when unset). - New non-registered OpenAICompatEngine + normalize_openai_base_url() (strips a single literal trailing "/v1" so request paths don't double). - SystemBuilder.engine_instance() injects a pre-built engine; build() health-checks it and fails loudly naming the host instead of falling back to discovery. Discovery substitution after an explicit -e key now logs a warning. - JarvisDirectBackend/JarvisAgentBackend accept base_url/api_key; on base_url they pin an OpenAICompatEngine to that endpoint with a fail-fast pre-flight (actionable error naming the URL and probe). - _build_backend forwards base_url/api_key to first-party backends on the CLI path; _run_terminalbench_native receives --base-url as api_base (single /v1 suffix) and exports OPENAI_API_KEY around the in-process harness run (terminus-2 routes via LiteLLM). - Suite TOML [backend.external] stays scoped to hermes/openclaw (suite_mode=True in the suite drivers) — first-party suite semantics are explicitly deferred. The config-host path is untouched. - Help text updated on both CLI surfaces; KNOWN_BACKENDS now lists hermes/openclaw/terminalbench-native. Fixes the eval-CLI endpoint gap reported by the downstream team. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 13:49:37 -07:00
github-actions[bot]	527f84f960	chore: update clone traffic data [skip ci]	2026-06-11 07:44:18 +00:00
Robby Manihani Claude Opus 4.8 Jon Saad-Falcon	8eaeb3a754	fix(windows): desktop backend spawn + model-aware engine selection (#533 ) * fix(windows): desktop backend spawn (#531) + model-aware engine selection (#532) Two runtime bugs found during end-to-end testing on a clean Windows 11 24H2 Azure VM. #531 - Desktop "Failed to get response": run_jarvis_command spawned the backend with .output(), which waits for the process to exit. `jarvis serve` never exits, so the Tauri command hung forever (the Start button never resolved); and it ran `uv run jarvis` with no cwd, so in a packaged install -- where the cwd isn't the checkout -- `jarvis` wasn't found and the server never started. Now: run from find_project_root(), and for `serve` spawn detached (.spawn()), drain stderr, and poll /health for readiness (mirrors start_backend); short commands keep .output(). The server layer itself was verified healthy on Windows (/health and /v1/chat/completions both 200, localhost included) -- the fault was the Tauri spawn path. #532 - "OpenAI client not available" after reboot: when the local engine is down, get_engine's fallback selected CloudEngine because health() is True if ANY provider client exists -- without checking the resolved model's provider has a client. A user with e.g. OPENROUTER_API_KEY and a gpt-* model then hit the OpenAI path with no client. Add CloudEngine.can_serve(model) (checks the specific provider client via the same routing generate()/stream() use) + a default can_serve->True on the base engine, and make get_engine model-aware so it skips an engine that can't serve the model -- the user falls through to the helpful "no engine available / start ollama" message instead. Tests: engine discovery/cloud/model-matrix + cli serve/ask suites pass (the one ask_e2e failure is a pre-existing version-banner flake, fails identically on main). The Tauri crate couldn't be compiled locally (no GTK/webkit sys-libs in this env); relies on CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(engine): cover model-aware engine selection + CloudEngine.can_serve (#532) #533 added a `model` arg to get_engine and a can_serve() gate but shipped no tests. Add them: - get_engine skips a healthy engine that can't serve the requested model (the cloud-fallback-for-unservable-model case behind #532), - model=None preserves the legacy model-agnostic selection, - CloudEngine.can_serve gates on the per-provider client (gpt->OpenAI, claude->Anthropic, ...), verified empirically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>	2026-06-10 19:52:36 -07:00