docs(evals): rewrite evaluations guide to match the real CLI surface; add openjarvis-eval alias (#534 )

docs/user-guide/evaluations.md documented a standalone "openjarvis-evals" package, a "uv sync --extra eval" install, and an "openjarvis-eval" console script — a layout from commit bd493832 that was never an ancestor of main. Rewrite the page against the real surface (jarvis eval / python -m openjarvis.evals), document all 40 registered benchmark keys and 4 backends, fix the judge-model default, correct run-all semantics, and split the option reference into the jarvis-eval subset and the module CLI's research-only options. Add the one-line [project.scripts] alias openjarvis-eval = "openjarvis.evals.cli:main" (the click group the module CLI already dispatches to) so the long-documented command name works again. The page was a complete orphan: add it (and the equally orphaned benchmarks.md) to the mkdocs nav and link it from docs/index.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
fix(evals): honor --base-url/--api-key for first-party eval backends (#535 )
2026-08-16 18:02:02 +00:00 · 2026-06-11 13:50:17 -07:00 · 2026-06-11 13:49:37 -07:00 · 2026-06-11 07:44:18 +00:00 · 2026-06-10 19:52:36 -07:00
27 changed files with 1275 additions and 123 deletions
@@ -1,7 +1,7 @@
 {
  "schemaVersion": 1,
  "label": "Git Clones",
-  "message": "107,695",
+  "message": "110,366",
  "color": "green",
  "namedLogo": "git"
 }
@@ -1,6 +1,6 @@
 {
-  "total_clones": 107695,
-  "last_updated": "2026-06-10T07:32:26Z",
+  "total_clones": 110366,
+  "last_updated": "2026-06-11T07:44:18Z",
  "daily": {
    "2026-03-27": 2189,
    "2026-03-28": 1874,
@@ -75,6 +75,8 @@
    "2026-06-05": 2127,
    "2026-06-06": 2204,
    "2026-06-07": 1174,
-    "2026-06-08": 2369
+    "2026-06-08": 2369,
+    "2026-06-09": 1361,
+    "2026-06-10": 1310
  }
 }
@@ -183,7 +183,7 @@ OpenJarvis is built around five composable layers. Each has a clean interface an

    ---

-    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), agents, memory, tools, and telemetry.
+    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), [Evaluations](user-guide/evaluations.md), agents, memory, tools, and telemetry.

 -   **[Architecture](architecture/overview.md)**

@@ -1,14 +1,14 @@
 # Evaluations

-The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correctness and accuracy** on academic datasets. It is a separate package from the main OpenJarvis library and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
+The OpenJarvis evaluation framework (`openjarvis.evals`) measures model **correctness and accuracy** on academic datasets. It ships inside the main `openjarvis` package (at `src/openjarvis/evals/`) and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.

 !!! info "Evals vs. Benchmarks"
    OpenJarvis has two distinct measurement systems that complement each other:

-    | System | Package | Measures | Entry Point |
-    |--------|---------|----------|-------------|
-    | **Evaluations** | `openjarvis-evals` | Correctness on academic datasets (accuracy, pass rate) | `openjarvis-eval` |
-    | **Benchmarks** | `openjarvis` | Engine performance (latency, throughput) | `jarvis bench` |
+    | System | Module | Measures | Entry Point |
+    |--------|--------|----------|-------------|
+    | **Evaluations** | `openjarvis.evals` | Correctness on academic datasets (accuracy, pass rate) | `jarvis eval` |
+    | **Benchmarks** | `openjarvis.bench` | Engine performance (latency, throughput) | `jarvis bench` |

    Use evaluations to answer "does this model get the right answer?" and benchmarks to answer "how fast does this model respond?". See the [Benchmarks guide](benchmarks.md) for the performance measurement system.

@@ -18,22 +18,38 @@ The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correc

 ## Installation

-The evaluation framework is a standalone package in the `evals/` directory. Install it alongside OpenJarvis:
+The evaluation framework is part of the main `openjarvis` package — no separate install or extra is required. The standard dev setup is enough:

 ```bash
-uv sync --extra eval
+uv sync --extra dev
 ```

-This installs the `openjarvis-eval` CLI entry point and all required dependencies (`datasets`, `huggingface-hub`, `tqdm`, `rich`).
+The framework's core dependencies (`click`, `datasets`, `rich`) are base dependencies of `openjarvis`. Two optional extras enable experiment tracking integrations:
+
+```bash
+uv sync --extra dev --extra eval-wandb     # Weights & Biases run tracking
+uv sync --extra dev --extra eval-sheets    # Google Sheets results export
+```

 !!! note "Python version requirement"
-    Python 3.10 requires the `tomli` package for TOML config parsing. The `evals/pyproject.toml` includes this as a conditional dependency, so it is installed automatically.
+    Python 3.10 requires the `tomli` package for TOML config parsing. `openjarvis` declares it as a conditional dependency, so it is installed automatically.
+
+## Entry Points
+
+Two equivalent entry points expose the framework:
+
+| Command | Surface |
+|---------|---------|
+| `jarvis eval {list,run,compare,report}` | Canonical CLI. `run` covers the common options; `compare` and `report` post-process result files. |
+| `python -m openjarvis.evals {list,run,run-all,summarize,reparse-judge}` | Full research surface, including judge configuration, the agentic runner, and episode mode. |
+
+The `openjarvis-eval` console script is an alias for `python -m openjarvis.evals` — same commands, same options. This guide uses `jarvis eval` wherever its option set suffices and the module form for research-only options.

 ---

 ## Datasets

-The framework ships with **30+ datasets** covering academic reasoning, agentic tasks, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below.
+The framework ships with **40 registered benchmarks** covering academic reasoning, agentic tasks, coding, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below; `uv run python -m openjarvis.evals list` prints the authoritative registry.

 ### Use-Case Benchmarks

@@ -64,6 +80,7 @@ These benchmarks measure reasoning and knowledge on established academic dataset
 | **MATH-500** | `math500` | reasoning | Competition-level math problems |
 | **NaturalReasoning** | `natural-reasoning` | reasoning | Natural language reasoning |
 | **HLE** | `hle` | reasoning | Humanity's Last Exam hard challenges |
+| **LiveResearchBench** | `liveresearchbench` | reasoning | Recent research comprehension (Salesforce) |
 | **SimpleQA** | `simpleqa` | chat | Short-form factual question answering |
 | **IPW** | `ipw` | chat | Intelligence Per Watt mixed benchmark |

@@ -79,6 +96,11 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **TerminalBench** | `terminalbench` | agentic | Terminal-based task completion |
 | **TerminalBench Native** | `terminalbench-native` | agentic | TerminalBench with native Docker execution |
 | **TerminalBench V2.1** | `terminalbench-v2.1` | agentic | TB v2.1 Harbor-style Docker tasks |
+| **PinchBench** | `pinchbench` | agentic | Real-world agent tasks |
+| **TauBench** | `taubench` | agentic | Multi-turn customer service |
+| **DeepResearchBench** | `liveresearch` | agentic | Deep research report generation |
+| **DeepResearchBench (alias)** | `deepresearch` | agentic | Same benchmark as `liveresearch` |
+| **ToolCall-15** | `toolcall15` | agentic | Tool calling benchmark |
 | **LifelongAgent** | `lifelong-agent` | agentic | Sequential task learning across sessions |
 | **PaperArena** | `paperarena` | agentic | Scientific paper analysis |
 | **DeepPlanning** | `deepplanning` | agentic | Shopping constraint planning |
@@ -87,6 +109,14 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **WebChoreArena** | `webchorearena` | agentic | Web chore tasks |
 | **WorkArena** | `workarena` | agentic | WorkArena++ enterprise workflows |

+Both `liveresearch` and `deepresearch` are registered keys for the DeepResearchBench report-generation benchmark.
+
+### Coding Benchmarks
+
+| Dataset | Key | Category | Description |
+|---------|-----|----------|-------------|
+| **LiveCodeBench** | `livecodebench` | coding | Competitive programming |
+
 ### Retrieval Benchmarks

 | Dataset | Key | Category | Description |
@@ -123,7 +153,7 @@ The framework includes two pre-built configs for evaluating models on the five c
 ### Cloud models

 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
 ```

 This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, GPT-5.4, GPT-5 Mini) against all 5 use-case benchmarks with 30 samples each, producing a 6x5 = 30-run matrix. Results are written to `results/use-cases-v2-cloud/`.
@@ -131,7 +161,7 @@ This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gem
 ### Local models

 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_local.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_local.toml
 ```

 This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS 120B, GLM4, Qwen3.5 35B-A3B, GLM-4.7-Flash) against the same 5 benchmarks, producing a 5x5 = 25-run matrix. Uses 2 workers (suitable for single-GPU setups). Results are written to `results/use-cases-v2-local/`.
@@ -143,15 +173,22 @@ This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS

 ## Inference Backends

-Every evaluation run routes model calls through one of two backends:
+Every evaluation run routes model calls through one of four backends:

 | Backend | Key | Description |
 |---------|-----|-------------|
 | **jarvis-direct** | `jarvis-direct` | Engine-level inference via `SystemBuilder`. Works for local (Ollama, vLLM, llama.cpp) and cloud models. |
 | **jarvis-agent** | `jarvis-agent` | Agent-level inference with tool calling. Uses `JarvisSystem.ask()` with the specified agent and tools. |
+| **hermes** | `hermes` | Real Hermes Agent (Nous Research) via subprocess. Requires `--base-url` and `--api-key`. |
+| **openclaw** | `openclaw` | Real OpenClaw via Node subprocess. Requires `--base-url` and `--api-key`. |

 Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark requires tool use — for example, GAIA tasks that reference files that must be read with `file_read`, or arithmetic tasks that benefit from `calculator`.

+The `hermes` and `openclaw` backends shell out to external agent frameworks and need an OpenAI-compatible endpoint for their model calls: pass `--base-url`/`--api-key`, set the `JARVIS_BACKEND_BASE_URL`/`JARVIS_BACKEND_API_KEY` environment variables, or add a `[backend.external]` section to your config (see [Config Reference](#backendexternal)).
+
+!!! note "TerminalBench Native"
+    `jarvis eval run --backend` additionally accepts `terminalbench-native`, a Docker-based execution backend used by the TerminalBench Native benchmark.
+
 ---

 ## CLI Usage
@@ -159,73 +196,106 @@ Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark
 ### List available benchmarks and backends

 ```bash
-openjarvis-eval list
+uv run python -m openjarvis.evals list
 ```

-Output:
+Abridged output (40 benchmarks, 4 backends):

 ```
-Benchmarks:
-  supergpqa    [reasoning  ] SuperGPQA multiple-choice
-  gaia         [agentic    ] GAIA agentic benchmark
-  frames       [rag        ] FRAMES multi-hop RAG
-  wildchat     [chat       ] WildChat conversation quality
-
-Backends:
-  jarvis-direct    Engine-level inference (local or cloud)
-  jarvis-agent     Agent-level inference with tool calling
+                         Available Benchmarks
+┌──────────────────────┬───────────┬───────────────────────────────────┐
+│ Name                 │ Category  │ Description                       │
+├──────────────────────┼───────────┼───────────────────────────────────┤
+│ supergpqa            │ reasoning │ SuperGPQA multiple-choice         │
+│ gpqa                 │ reasoning │ GPQA graduate-level MCQ           │
+│ ...                  │ ...       │ ...                               │
+│ livecodebench        │ coding    │ LiveCodeBench competitive progr.  │
+│ toolcall15           │ agentic   │ ToolCall-15 tool calling benchmark│
+└──────────────────────┴───────────┴───────────────────────────────────┘
+                         Available Backends
+┌───────────────┬──────────────────────────────────────────────────┐
+│ jarvis-direct │ Engine-level inference (local or cloud)          │
+│ jarvis-agent  │ Agent-level inference with tool calling          │
+│ hermes        │ Real Hermes Agent (Nous Research) via subprocess │
+│ openclaw      │ Real OpenClaw via Node subprocess                │
+└───────────────┴──────────────────────────────────────────────────┘
 ```

+`jarvis eval list` prints a similar table but currently shows a curated subset of the registry; the module form above is the authoritative listing.
+
 ### Run a single benchmark

 ```bash
-# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples default)
-openjarvis-eval run -b supergpqa -m qwen3:8b
+# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples)
+uv run jarvis eval run -b supergpqa -m qwen3:8b -n 10

-# Evaluate GPT-4o on GAIA using the agent backend with tools
-openjarvis-eval run -b gaia -m gpt-4o --backend jarvis-agent \
+# Evaluate GPT-5 Mini on GAIA using the agent backend with tools
+uv run jarvis eval run -b gaia -m gpt-5-mini --backend jarvis-agent \
    --agent orchestrator --tools calculator,file_read -n 50

-# Run FRAMES with vLLM engine, write output to a file
-openjarvis-eval run -b frames -m llama3:70b -e vllm \
+# Run FRAMES with the vLLM engine, write output to a file
+uv run jarvis eval run -b frames -m llama3:70b -e vllm \
    -o results/frames_llama70b.jsonl

 # Run WildChat with a higher temperature for chat quality
-openjarvis-eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
+uv run jarvis eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
 ```

-#### Full option reference
+#### `jarvis eval run` option reference

 | Option | Short | Type | Default | Description |
 |--------|-------|------|---------|-------------|
 | `--config` | `-c` | path | — | TOML config file; when provided, `-b` and `-m` are not required |
-| `--benchmark` | `-b` | choice | required* | `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `--backend` | | choice | `jarvis-direct` | `jarvis-direct` or `jarvis-agent` |
-| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-4o`) |
-| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
-| `--agent` | | str | `orchestrator` | Agent name for `jarvis-agent` backend |
-| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--benchmark` | `-b` | str | required* | Any registered benchmark key (see `... list`) |
+| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-5-mini`) |
 | `--max-samples` | `-n` | int | all | Limit the number of samples evaluated |
-| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
-| `--judge-model` | | str | `gpt-4o` | LLM used for judge-based scoring |
-| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--backend` | | choice | `jarvis-direct` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
+| `--base-url` | | str | — | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `--api-key` | | str | — | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
+| `--agent` | | str | — | Agent name for `jarvis-agent` backend (e.g., `orchestrator`) |
+| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
+| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--telemetry/--no-telemetry` | | flag | off | Enable telemetry collection during eval |
+| `--gpu-metrics/--no-gpu-metrics` | | flag | off | Enable GPU metric polling |
 | `--seed` | | int | `42` | Random seed for dataset shuffling |
-| `--split` | | str | dataset default | Override the dataset split |
 | `--temperature` | | float | `0.0` | Generation temperature |
 | `--max-tokens` | | int | `2048` | Maximum output tokens |
+| `--model-filter` | | str | — | Filter models by name substring (multi-model configs) |
+| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--wandb-project` / `--wandb-entity` / `--wandb-tags` / `--wandb-group` | | str | `""` | Weights & Biases tracking (requires `eval-wandb` extra) |
+| `--sheets-id` / `--sheets-worksheet` / `--sheets-creds` | | str | `""` | Google Sheets export (requires `eval-sheets` extra) |
 | `--verbose` | `-v` | flag | off | Enable debug logging |

 *Required when `--config` is not provided.

+#### Research-only options (`python -m openjarvis.evals run`)
+
+The module CLI accepts everything above plus research-grade options that `jarvis eval run` does not expose:
+
+| Option | Short | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
+| `--judge-model` | | str | `gpt-5-mini-2025-08-07` | LLM used for judge-based scoring (see `--help` for the current default) |
+| `--judge-engine` | | str | `cloud` | Engine key for the LLM judge; use `vllm` to judge locally |
+| `--split` | | str | dataset default | Override the dataset split |
+| `--compact` | | flag | off | Dense single-table output |
+| `--trace-detail` | | flag | off | Full per-step trace listing |
+| `--agentic` | | flag | off | Use `AgenticRunner` for multi-turn agent execution |
+| `--episode-mode` | | flag | off | Sequential episode processing with lifelong learning (required for `lifelong-agent` and similar benchmarks) |
+| `--concurrency` | | int | `1` | Parallel query execution (AgenticRunner only) |
+| `--query-timeout` | | float | — | Per-query wall-clock timeout in seconds (AgenticRunner only) |
+
+Note: the module CLI's `--backend` choice covers `jarvis-direct`, `jarvis-agent`, `hermes`, and `openclaw`; `terminalbench-native` as a backend is available via `jarvis eval run` and TOML configs.
+
 ### Run all benchmarks at once

-The `run-all` command evaluates a single model against all four benchmarks sequentially and writes results to an output directory:
+The `run-all` command (module CLI only) evaluates a single model against **every registered benchmark** sequentially and writes results to an output directory:

 ```bash
-openjarvis-eval run-all -m qwen3:8b
+uv run python -m openjarvis.evals run-all -m qwen3:8b

 # With options
-openjarvis-eval run-all -m gpt-4o -n 100 --output-dir results/gpt4o/
+uv run python -m openjarvis.evals run-all -m gpt-5-mini -n 100 --output-dir results/gpt5mini/
 ```

 Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The model slug replaces `/` and `:` with `-`, so `qwen3:8b` becomes `qwen3-8b`.
@@ -235,7 +305,7 @@ Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The m
 After a run, inspect a JSONL results file:

 ```bash
-openjarvis-eval summarize results/supergpqa_qwen3-8b.jsonl
+uv run python -m openjarvis.evals summarize results/supergpqa_qwen3-8b.jsonl
 ```

 Output:
@@ -251,6 +321,55 @@ Accuracy:  0.7222
 Errors:    2
 ```

+The module CLI also provides `reparse-judge`, which re-parses stored judge output in a results file and recovers records whose judge verdicts initially failed to parse — useful after improving the judge-output parser without re-running inference.
+
+### Compare and report
+
+`jarvis eval` adds two post-processing commands for result files:
+
+```bash
+# Side-by-side metric comparison across runs
+uv run jarvis eval compare results/supergpqa_qwen3-8b.jsonl results/supergpqa_gpt-5-mini.jsonl
+
+# Detailed report (accuracy, latency, cost, per-subject breakdown) for one run
+uv run jarvis eval report results/supergpqa_qwen3-8b.jsonl
+```
+
+---
+
+## Evaluating an Already-Running Endpoint
+
+If you already have an OpenAI-compatible server running — `jarvis serve`, vLLM, SGLang, llama.cpp's server, or a hosted endpoint — point an eval directly at it with `--base-url` and `--api-key`:
+
+```bash
+# A vLLM server is already serving Qwen/Qwen3-8B on a GPU node:
+#   vllm serve Qwen/Qwen3-8B --port 8000
+uv run jarvis eval run -b supergpqa -m Qwen/Qwen3-8B \
+    --base-url http://gpu-node:8000/v1 \
+    --api-key local-key \
+    -n 50
+```
+
+The `-m` value must match a model id the server reports at `GET /v1/models`. Both flags fall back to the `JARVIS_BACKEND_BASE_URL` and `JARVIS_BACKEND_API_KEY` environment variables, so CI jobs can set them once:
+
+```bash
+export JARVIS_BACKEND_BASE_URL=http://gpu-node:8000/v1
+export JARVIS_BACKEND_API_KEY=local-key
+uv run jarvis eval run -b gaia -m Qwen/Qwen3-8B --backend jarvis-agent -n 25
+```
+
+For the external `hermes` and `openclaw` backends these values are **required** (the foreign frameworks need an endpoint to send model calls to).
+
+!!! tip "Engine-level alternative for vLLM"
+    The vLLM engine also honors the `VLLM_HOST` environment variable (default `http://localhost:8000`):
+
+    ```bash
+    VLLM_HOST=http://gpu-node:8000 uv run python -m openjarvis.evals run \
+        -b supergpqa -m Qwen/Qwen3-8B -e vllm -n 50
+    ```
+
+    `VLLM_HOST` is process-global — if the candidate and the judge both use the `vllm` engine, they share the same endpoint. Prefer `--base-url` when you need them separate.
+
 ---

 ## TOML Config System
@@ -260,7 +379,7 @@ For research workflows that compare multiple models across multiple benchmarks,
 ### Running from a config

 ```bash
-openjarvis-eval run --config src/openjarvis/evals/configs/full-suite.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/full-suite.toml
 ```

 When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options are not required. All settings come from the config file. The CLI expands the matrix, prints a progress table, and writes results to the configured `output_dir`.
@@ -269,7 +388,7 @@ When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options a

 A config file has six sections: `[meta]`, `[defaults]`, `[judge]`, `[run]`, `[[models]]`, and `[[benchmarks]]`. Only `[[models]]` and `[[benchmarks]]` are required — all other sections are optional and fall back to built-in defaults.

-```toml title="evals/configs/full-suite.toml"
+```toml title="src/openjarvis/evals/configs/full-suite.toml"
 # Suite-level metadata (optional)
 [meta]
 name = "full-suite-v1"
@@ -353,7 +472,7 @@ For example, `temperature` is resolved as: use `[defaults].temperature` (0.0), t

 A config requires only one `[[models]]` and one `[[benchmarks]]` entry:

-```toml title="evals/configs/minimal.toml"
+```toml title="src/openjarvis/evals/configs/minimal.toml"
 [[models]]
 name = "qwen3:8b"

@@ -365,7 +484,7 @@ This runs SuperGPQA against qwen3:8b with all default settings. Use this as a st

 ### Single-run config with full options

-```toml title="evals/configs/single-run.toml"
+```toml title="src/openjarvis/evals/configs/single-run.toml"
 [meta]
 name = "single-run-example"
 description = "Evaluate SuperGPQA with a single model and full configuration"
@@ -425,7 +544,8 @@ Configuration for the LLM used as a judge in GAIA, FRAMES, and WildChat scoring.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `model` | str | `"gpt-4o"` | Judge model identifier |
+| `model` | str | `"gpt-5-mini-2025-08-07"` | Judge model identifier |
+| `engine` | str | `None` | Engine key for the judge (e.g., `"vllm"` to judge locally; defaults to cloud) |
 | `provider` | str | `None` | Provider override (e.g., `"openai"`) |
 | `temperature` | float | `0.0` | Judge sampling temperature |
 | `max_tokens` | int | `1024` | Maximum judge output tokens |
@@ -444,6 +564,20 @@ Execution settings that apply to the entire suite.
 | `seed` | int | `42` | Random seed for dataset shuffling |
 | `telemetry` | bool | `false` | Enable GPU telemetry capture (energy, power, utilization, throughput) |
 | `gpu_metrics` | bool | `false` | Enable GPU metric polling via `pynvml` (requires `pynvml` or `nvidia-ml-py`) |
+| `warmup_samples` | int | `0` | Untimed warmup samples before measurement |
+| `energy_vendor` | str | `""` | GPU energy vendor override |
+| `max_turns` | int | `None` | Maximum agent turns per query |
+| `wandb_project` / `wandb_entity` / `wandb_tags` / `wandb_group` | str | `""` | Weights & Biases tracking |
+| `sheets_spreadsheet_id` / `sheets_worksheet` / `sheets_credentials_path` | str | `""` / `"Results"` / `""` | Google Sheets export |
+
+### `[backend.external]`
+
+Endpoint settings for the `hermes` and `openclaw` backends. Environment variables override TOML values.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `base_url` | str | `None` | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `api_key` | str | `None` | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |

 ### `[[models]]`

@@ -451,7 +585,7 @@ One block per model. The `name` field is required.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-4o"`) |
+| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-5-mini"`) |
 | `engine` | str | `None` | Engine key to use (`"ollama"`, `"vllm"`, `"cloud"`, ...) |
 | `provider` | str | `None` | Provider override for cloud models (e.g., `"openai"`) |
 | `temperature` | float | `None` | Override `[defaults].temperature` for this model |
@@ -468,10 +602,12 @@ One block per benchmark. The `name` field is required.

 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Benchmark key: `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `backend` | str | `"jarvis-direct"` | Inference backend: `jarvis-direct` or `jarvis-agent` |
+| `name` | str | required | Any registered benchmark key (see `uv run python -m openjarvis.evals list`) |
+| `backend` | str | `"jarvis-direct"` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
 | `max_samples` | int | `None` | Limit number of samples; `None` evaluates the full dataset |
 | `split` | str | `None` | Override the default dataset split |
+| `subset` | str | `None` | Dataset subset/variant (benchmark-specific) |
+| `record_ids` | list[str] | `None` | Evaluate only these record ids |
 | `agent` | str | `None` | Agent name for `jarvis-agent` backend (e.g., `"orchestrator"`) |
 | `tools` | list[str] | `[]` | Tool names for `jarvis-agent` backend |
 | `judge_model` | str | `None` | Override `[judge].model` for this benchmark only |
@@ -647,7 +783,7 @@ The `EvalRunner` processes samples concurrently using a `ThreadPoolExecutor`. Re

 ```bash
 # Use more workers for faster evaluation (if the engine supports concurrent requests)
-openjarvis-eval run -b supergpqa -m qwen3:8b -w 8 -n 500
+uv run python -m openjarvis.evals run -b supergpqa -m qwen3:8b -w 8 -n 500
 ```

 !!! warning "Worker count and engine load"
@@ -1540,19 +1540,89 @@ async fn fetch_models(api_url: String) -> Result<serde_json::Value, String> {

 #[tauri::command]
 async fn run_jarvis_command(args: Vec<String>) -> Result<String, String> {
-    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
-    cmd_args.extend(args);
    let uv_bin = resolve_bin("uv");
-    let output = tokio::process::Command::new(&uv_bin)
-        .args(&cmd_args)
-        .output()
-        .await
-        .map_err(|e| format!("Failed to launch jarvis: {}", e))?;

-    if output.status.success() {
-        Ok(String::from_utf8_lossy(&output.stdout).to_string())
-    } else {
-        Err(String::from_utf8_lossy(&output.stderr).to_string())
+    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
+    cmd_args.extend(args.iter().cloned());
+
+    let mut cmd = tokio::process::Command::new(&uv_bin);
+    cmd.args(&cmd_args);
+    // Run from the project root so `uv run jarvis` resolves the OpenJarvis
+    // project regardless of the app's launch cwd. In a packaged install the
+    // cwd isn't the checkout, so without this `jarvis` isn't found and the
+    // backend never starts — the UI then shows "Failed to get response"
+    // (see #531).
+    if let Some(ref root) = find_project_root() {
+        cmd.current_dir(root);
+    }
+
+    let is_serve = args.first().map(|a| a.as_str() == "serve").unwrap_or(false);
+
+    if !is_serve {
+        // Short-lived command (e.g. `stop`, `status`): wait for it and return
+        // its captured output.
+        let output = cmd
+            .output()
+            .await
+            .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
+        return if output.status.success() {
+            Ok(String::from_utf8_lossy(&output.stdout).to_string())
+        } else {
+            Err(String::from_utf8_lossy(&output.stderr).to_string())
+        };
+    }
+
+    // `jarvis serve` is a long-running server that never exits. The old code
+    // used `.output()`, which waits for the process to exit and so hung this
+    // command forever — the "Start" button never resolved (#531). Spawn it
+    // detached instead, drain stderr (a full 4 KB Windows pipe can otherwise
+    // stall the child mid-startup, #309), and poll /health for readiness.
+    cmd.stdout(std::process::Stdio::null())
+        .stderr(std::process::Stdio::piped());
+    let mut child = cmd
+        .spawn()
+        .map_err(|e| format!("Failed to launch jarvis serve: {}", e))?;
+
+    let tail: StderrTail = Arc::new(Mutex::new(Vec::new()));
+    if let Some(stderr) = child.stderr.take() {
+        spawn_jarvis_stderr_drainer(stderr, tail.clone());
+    }
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(2))
+        .build()
+        .map_err(|e| format!("Failed to build HTTP client: {}", e))?;
+    let url = format!("http://127.0.0.1:{}/health", JARVIS_PORT);
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(120);
+
+    loop {
+        // Surface an early crash (bad venv, missing Rust ext, etc.) right away
+        // instead of waiting out the full readiness timeout.
+        if let Ok(Some(status)) = child.try_wait() {
+            let stderr = String::from_utf8_lossy(tail.lock().await.as_slice()).into_owned();
+            return Err(format!(
+                "jarvis serve exited (code {:?}) before becoming healthy:\n{}",
+                status.code(),
+                stderr.trim()
+            ));
+        }
+        if let Ok(resp) = client.get(&url).send().await {
+            if resp.status().is_success() {
+                // Leave the server running (the Child is detached on drop —
+                // kill_on_drop defaults to false); `stop` tears it down.
+                return Ok(format!(
+                    "jarvis serve is ready on http://127.0.0.1:{}",
+                    JARVIS_PORT
+                ));
+            }
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!(
+                "jarvis serve did not become healthy on port {} within 120s.",
+                JARVIS_PORT
+            ));
+        }
+        tokio::time::sleep(Duration::from_millis(500)).await;
    }
 }

@@ -193,6 +193,8 @@ nav:
      - External MCP Servers: user-guide/mcp-external-servers.md
      - Scheduler: user-guide/scheduler.md
      - Telemetry: user-guide/telemetry.md
+      - Evaluations: user-guide/evaluations.md
+      - Benchmarks: user-guide/benchmarks.md
      - Security: user-guide/security.md
      - LLM-guided spec search: user-guide/llm-guided-spec-search.md
  - Leaderboard: leaderboard.md
@@ -152,6 +152,7 @@ Issues = "https://github.com/open-jarvis/OpenJarvis/issues"

 [project.scripts]
 jarvis = "openjarvis.cli:main"
+openjarvis-eval = "openjarvis.evals.cli:main"

 [tool.hatch.build.targets.wheel]
 packages = ["src/openjarvis"]
@@ -714,7 +714,13 @@ def ask(
    register_builtin_models()

    effective_engine_key = engine_key or config.intelligence.preferred_engine or None
-    resolved = get_engine(config, effective_engine_key)
+    # Pass the model we intend to run so engine selection can skip an engine
+    # that can't actually serve it (e.g. the cloud fallback when the local
+    # engine is down but only a non-OpenAI key is set — see #532). This is the
+    # -m flag or the configured default; when neither is set we leave it None
+    # and a model is chosen per-engine below.
+    selection_model = model_name or config.intelligence.default_model or None
+    resolved = get_engine(config, effective_engine_key, model=selection_model)
    if resolved is None:
        console.print(
            "[red bold]No inference engine available.[/red bold]\n\n"
@@ -332,7 +332,7 @@ def compose_bench(
        for i, rc in enumerate(run_configs, 1):
            console.print(f"\n[bold]Run {i}/{len(run_configs)}:[/bold] {rc.benchmark}")
            try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                results_table.add_row(
                    rc.benchmark,
                    f"{summary.accuracy:.4f}",
@@ -61,6 +61,12 @@ KNOWN_BENCHMARKS = {
 KNOWN_BACKENDS = {
    "jarvis-direct": "Engine-level inference (local or cloud)",
    "jarvis-agent": "Agent-level inference with tool calling",
+    "hermes": "Real Hermes Agent (Nous Research) via subprocess",
+    "openclaw": "Real OpenClaw via Node subprocess",
+    "terminalbench-native": (
+        "TerminalBench V2.1 via terminal-bench Harness "
+        "(selected with -b terminalbench-native)"
+    ),
 }


@@ -146,7 +152,9 @@ def eval_list() -> None:
    "base_url",
    default=None,
    help=(
-        "OpenAI-compat endpoint URL for hermes/openclaw backends "
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
        "(env: JARVIS_BACKEND_BASE_URL)."
    ),
 )
@@ -154,7 +162,11 @@ def eval_list() -> None:
    "--api-key",
    "api_key",
    default=None,
-    help=("API key for the hermes/openclaw endpoint (env: JARVIS_BACKEND_API_KEY)."),
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
@click.option(
    "--agent",
@@ -347,7 +359,7 @@ def eval_run(
                f"{rc.benchmark} / {rc.model}"
            )
            try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                console.print(
                    f"  [green]{summary.accuracy:.4f}[/green] "
                    f"({summary.correct}/{summary.scored_samples})"
@@ -399,8 +411,10 @@ def eval_run(
        sheets_spreadsheet_id=sheets_spreadsheet_id,
        sheets_worksheet=sheets_worksheet,
        sheets_credentials_path=sheets_credentials_path,
-        # Spec §6.2 — for hermes/openclaw external backends. Falls back to env vars
-        # so users can also set JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
+        # OpenAI-compatible endpoint for the model under eval. Required for
+        # hermes/openclaw (Spec §6.2); honored by first-party backends too on
+        # this CLI path. Falls back to env vars so users can also set
+        # JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
        base_url=base_url or os.environ.get("JARVIS_BACKEND_BASE_URL"),
        api_key=api_key or os.environ.get("JARVIS_BACKEND_API_KEY"),
    )
@@ -146,7 +146,13 @@ def serve(
        except Exception as exc:
            logger.debug("Telemetry store init failed: %s", exc)

-    resolved = get_engine(config, engine_key)
+    # Select with the model we'll actually serve so an engine that can't
+    # serve it (e.g. the cloud fallback without the matching provider key) is
+    # skipped rather than chosen and failing per-request later (see #532).
+    selection_model = (
+        model_name or config.server.model or config.intelligence.default_model or None
+    )
+    resolved = get_engine(config, engine_key, model=selection_model)
    if resolved is None:
        console.print(
            "[red bold]No inference engine available.[/red bold]\n\n"
@@ -156,12 +156,26 @@ def discover_models(


 def get_engine(
-    config: JarvisConfig, engine_key: str | None = None
+    config: JarvisConfig,
+    engine_key: str | None = None,
+    model: str | None = None,
 ) -> Tuple[str, InferenceEngine] | None:
    """Get a specific engine by key, or the default with fallback.

+    When *model* is given, an engine is selected only if it can actually
+    serve that model (``engine.can_serve(model)``). This stops the cloud
+    fallback from being chosen — when the local engine is down — for a model
+    whose provider client is missing, which otherwise surfaces as a confusing
+    "OpenAI client not available" instead of a helpful "start your local
+    engine" message (see #532). When *model* is ``None`` selection stays
+    model-agnostic (unchanged behaviour).
+
    Returns ``(key, engine_instance)`` or ``None`` if no engine is available.
    """
+
+    def _usable(engine: InferenceEngine) -> bool:
+        return engine.health() and (model is None or engine.can_serve(model))
+
    # Build an ordered list of keys to try, then fall back to full discovery.
    keys_to_try: list[str] = []
    if engine_key:
@@ -176,14 +190,16 @@ def get_engine(
            continue
        try:
            engine = _make_engine(key, config)
-            if engine.health():
+            if _usable(engine):
                return (key, engine)
        except Exception as exc:
            logger.debug("Engine %r health check failed: %s", key, exc)

-    # Fallback to any healthy engine
-    healthy = discover_engines(config)
-    return healthy[0] if healthy else None
+    # Fallback to the first healthy engine that can serve the model.
+    for key, engine in discover_engines(config):
+        if model is None or engine.can_serve(model):
+            return (key, engine)
+    return None


 __all__ = ["discover_engines", "discover_models", "get_engine"]
@@ -28,12 +28,31 @@ class _OpenAICompatibleEngine(InferenceEngine):
    _default_host: str = "http://localhost:8000"
    _api_prefix: str = "/v1"

-    def __init__(self, host: str | None = None, *, timeout: float = 600.0) -> None:
+    def __init__(
+        self,
+        host: str | None = None,
+        *,
+        api_key: str | None = None,
+        timeout: float = 600.0,
+    ) -> None:
        import os

-        env_key = f"{self.engine_id.upper()}_HOST"
-        self._host = (host or os.environ.get(env_key) or self._default_host).rstrip("/")
-        self._client = httpx.Client(base_url=self._host, timeout=timeout)
+        # Sanitize the engine id for env-var lookup ("openai-compat" ->
+        # "OPENAI_COMPAT_..."); shells cannot set hyphenated variable names.
+        env_prefix = self.engine_id.upper().replace("-", "_")
+        self._host = (
+            host or os.environ.get(f"{env_prefix}_HOST") or self._default_host
+        ).rstrip("/")
+        # Bearer auth for endpoints started with e.g. ``vllm serve --api-key``.
+        # Setting it on the client covers generate/stream/stream_full/
+        # list_models/health alike; ``None`` keeps requests header-free.
+        self._api_key = api_key or os.environ.get(f"{env_prefix}_API_KEY") or None
+        headers = (
+            {"Authorization": f"Bearer {self._api_key}"} if self._api_key else None
+        )
+        self._client = httpx.Client(
+            base_url=self._host, timeout=timeout, headers=headers
+        )

    # -- InferenceEngine interface ------------------------------------------

@@ -119,6 +119,17 @@ class InferenceEngine(ABC):
    def health(self) -> bool:
        """Return ``True`` when the engine is reachable and healthy."""

+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` if this engine can serve *model*.
+
+        Defaults to ``True``: local engines accept any model id (whether a
+        specific model is *installed* is a separate concern from engine
+        selection). Engines that multiplex provider-specific clients (e.g.
+        the cloud engine) override this so selection can skip an engine whose
+        client for the model's provider isn't configured (see #532).
+        """
+        return True
+
    def close(self) -> None:
        """Release resources (HTTP clients, connections, threads, etc.)."""

@@ -1477,6 +1477,34 @@ class CloudEngine(InferenceEngine):
            models.extend(_CODEX_MODELS)
        return models

+    def _client_for_model(self, model: str) -> Any:
+        """Return the provider client ``generate``/``stream`` will dispatch to
+        for *model* (mirrors the routing in those methods)."""
+        if _is_codex_model(model):
+            return self._codex_client
+        if _is_openrouter_model(model):
+            return self._openrouter_client
+        if _is_minimax_model(model):
+            return self._minimax_client
+        if _is_anthropic_model(model):
+            return self._anthropic_client
+        if _is_google_model(model):
+            return self._google_client
+        return self._openai_client
+
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` only if the provider client for *model* exists.
+
+        ``health()`` is ``True`` whenever *any* provider client is configured,
+        but a request for, say, a ``gpt-*`` model still needs the OpenAI
+        client specifically. Without this check the cloud engine gets picked
+        as a fallback (when the local engine is down) for a model it can't
+        serve, then dies at call time with "<provider> client not available"
+        instead of the user getting a helpful "start your local engine"
+        message (see #532).
+        """
+        return self._client_for_model(model) is not None
+
    def health(self) -> bool:
        return (
            self._openai_client is not None
@@ -1,5 +1,7 @@
 """Data-driven registration of OpenAI-compatible inference engines."""

+from __future__ import annotations
+
 from openjarvis.core.registry import EngineRegistry
 from openjarvis.engine._openai_compat import _OpenAICompatibleEngine

@@ -25,4 +27,35 @@ for _key, (_cls_name, _default_host, _api_prefix) in _ENGINES.items():
    EngineRegistry.register(_key)(_cls)
    globals()[_cls_name] = _cls

-__all__ = [name for name, _, _ in _ENGINES.values()]
+
+def normalize_openai_base_url(url: str) -> str:
+    """Strip a single trailing ``/v1`` segment from a user-supplied base URL.
+
+    Users habitually pass ``http://host:8000/v1`` (the full OpenAI-compatible
+    prefix); the engine's ``_api_prefix`` re-appends ``/v1`` to every request
+    path, so a trailing copy would double up as ``/v1/v1``. Only a literal
+    trailing ``/v1`` is stripped — proxy/gateway path prefixes are preserved.
+    """
+    base = url.rstrip("/")
+    if base.endswith("/v1"):
+        base = base[: -len("/v1")]
+    return base
+
+
+class OpenAICompatEngine(_OpenAICompatibleEngine):
+    """Generic engine for an explicitly-provided OpenAI-compatible endpoint.
+
+    Deliberately NOT registered in ``EngineRegistry``: it is only ever
+    constructed with an explicit host (e.g. ``jarvis eval --base-url``), so
+    registering it would just add a useless localhost discovery probe and
+    interact with the per-test registry wipe.
+    """
+
+    engine_id = "openai-compat"
+    _api_prefix = "/v1"
+
+
+__all__ = [name for name, _, _ in _ENGINES.values()] + [
+    "OpenAICompatEngine",
+    "normalize_openai_base_url",
+]
@@ -0,0 +1,52 @@
+"""Shared helper for targeting an explicit OpenAI-compatible endpoint.
+
+Used by the first-party eval backends (jarvis-direct, jarvis-agent) when
+``--base-url`` is given: the eval must use exactly that endpoint, with no
+silent fallback to whatever other engine discovery happens to find.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+
+def build_endpoint_engine(
+    base_url: str,
+    api_key: Optional[str] = None,
+    engine_key: Optional[str] = None,
+):
+    """Construct an :class:`OpenAICompatEngine` pinned to ``base_url``.
+
+    Pre-flight health-checks the endpoint and raises a loud, actionable
+    error when it is unreachable — engine discovery is never consulted.
+    """
+    from openjarvis.engine.openai_compat_engines import (
+        OpenAICompatEngine,
+        normalize_openai_base_url,
+    )
+
+    if engine_key:
+        logger.warning(
+            "Both an engine key (%r) and base_url (%r) were given; "
+            "base_url wins — targeting the endpoint directly.",
+            engine_key,
+            base_url,
+        )
+    host = normalize_openai_base_url(base_url)
+    engine = OpenAICompatEngine(host=host, api_key=api_key)
+    if not engine.health():
+        engine.close()
+        raise RuntimeError(
+            f"--base-url endpoint not reachable: {base_url} "
+            f"(GET {host}/v1/models failed). Is an OpenAI-compatible server "
+            "(e.g. `vllm serve`) running at that address? If it requires "
+            "authentication (HTTP 401), pass --api-key or set "
+            "JARVIS_BACKEND_API_KEY."
+        )
+    return engine
+
+
+__all__ = ["build_endpoint_engine"]
@@ -31,6 +31,8 @@ class JarvisAgentBackend(InferenceBackend):
        max_turns: Optional[int] = None,
        skills_enabled: bool = True,
        overlay_dir: Optional[Path] = None,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
    ) -> None:
        from openjarvis.system import SystemBuilder

@@ -40,7 +42,17 @@ class JarvisAgentBackend(InferenceBackend):
        self._gpu_metrics = gpu_metrics

        builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
            builder.engine(engine_key)
        if model:
            builder.model(model)
@@ -24,6 +24,8 @@ class JarvisDirectBackend(InferenceBackend):
        engine_key: Optional[str] = None,
        telemetry: bool = False,
        gpu_metrics: bool = False,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
    ) -> None:
        from openjarvis.system import SystemBuilder

@@ -31,7 +33,17 @@ class JarvisDirectBackend(InferenceBackend):
        self._gpu_metrics = gpu_metrics

        builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
            builder.engine(engine_key)
        # Propagate gpu_metrics to the runtime config so SystemBuilder
        # creates an EnergyMonitor / GpuMonitor for the InstrumentedEngine.
@@ -183,14 +183,28 @@ def _build_backend(
    max_turns: Optional[int] = None,
    base_url: Optional[str] = None,
    api_key: Optional[str] = None,
+    first_party_endpoint: bool = True,
 ):
    """Construct the appropriate backend.

-    For "hermes" and "openclaw" backends, ``base_url`` and ``api_key`` are
-    REQUIRED — these foreign frameworks need an OpenAI-compatible endpoint
-    to send model calls to. Pass them via the eval config's
-    ``[backend.external]`` section or env vars.
+    ``base_url``/``api_key`` point at the OpenAI-compatible endpoint serving
+    the model under eval:
+
+    - For "hermes" and "openclaw" they are REQUIRED — these foreign
+      frameworks always call out to an external endpoint.
+    - "jarvis-direct" and "jarvis-agent" honor them when
+      ``first_party_endpoint`` is True (the CLI ``--base-url`` path): the
+      eval targets exactly that endpoint — no engine-discovery fallback —
+      and fails fast if it is unreachable. Suite mode passes
+      ``first_party_endpoint=False`` so the suite TOML's
+      ``[backend.external]`` section stays scoped to hermes/openclaw
+      (extending it to first-party backends is explicitly deferred).
    """
+    if not first_party_endpoint:
+        fp_base_url = fp_api_key = None
+    else:
+        fp_base_url, fp_api_key = base_url, api_key
+
    if backend_name == "jarvis-agent":
        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend

@@ -202,6 +216,8 @@ def _build_backend(
            gpu_metrics=gpu_metrics,
            model=model,
            max_turns=max_turns,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
        )
    elif backend_name == "jarvis-direct":
        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
@@ -210,6 +226,8 @@ def _build_backend(
            engine_key=engine_key,
            telemetry=telemetry,
            gpu_metrics=gpu_metrics,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
        )
    elif backend_name == "hermes":
        from openjarvis.evals.backends.external import HermesBackend
@@ -656,8 +674,20 @@ def _build_trackers(config) -> list:
    return trackers


-def _run_terminalbench_native(config, console: Console) -> object:
-    """Run TerminalBench V2.1 natively via terminal-bench Harness."""
+def _run_terminalbench_native(
+    config,
+    console: Console,
+    *,
+    base_url: Optional[str] = None,
+    api_key: Optional[str] = None,
+) -> object:
+    """Run TerminalBench V2.1 natively via terminal-bench Harness.
+
+    ``base_url`` (from ``--base-url`` / JARVIS_BACKEND_BASE_URL) targets an
+    already-running OpenAI-compatible endpoint; when unset, the legacy local
+    vLLM default (http://localhost:8000/v1) is used.
+    """
+    from openjarvis.engine.openai_compat_engines import normalize_openai_base_url
    from openjarvis.evals.backends.terminalbench_native import (
        TerminalBenchNativeBackend,
    )
@@ -668,9 +698,16 @@ def _run_terminalbench_native(config, console: Console) -> object:
    litellm_model = f"openai/{model}"
    output_dir = getattr(config, "output_path", None) or "results/terminalbench-native/"

+    # Normalize to exactly one trailing "/v1" — LiteLLM's api_base wants the
+    # full OpenAI-compatible prefix, and users pass both forms of the URL.
+    if base_url:
+        api_base = normalize_openai_base_url(base_url) + "/v1"
+    else:
+        api_base = "http://localhost:8000/v1"
+
    backend = TerminalBenchNativeBackend(
        model=litellm_model,
-        api_base="http://localhost:8000/v1",
+        api_base=api_base,
        temperature=config.temperature,
        max_samples=config.max_samples,
        output_dir=output_dir,
@@ -683,9 +720,25 @@ def _run_terminalbench_native(config, console: Console) -> object:
    model_slug = re.sub(r"[^a-z0-9_-]", "-", model.lower().replace("/", "-"))
    run_id = f"tb21-{model_slug}"
    console.print(f"  Running TerminalBench V2.1 natively: {model}")
+    console.print(f"  API base: {api_base}")
    console.print(f"  Harness run_id: {run_id}")

-    results = backend.run_harness(run_id)
+    if api_key:
+        # terminus-2 routes model calls through LiteLLM with the "openai/"
+        # prefix, which reads OPENAI_API_KEY from the environment. The
+        # harness runs in-process, so set the var for the duration of the
+        # run and restore the previous value afterwards.
+        prev_key = os.environ.get("OPENAI_API_KEY")
+        os.environ["OPENAI_API_KEY"] = api_key
+        try:
+            results = backend.run_harness(run_id)
+        finally:
+            if prev_key is None:
+                os.environ.pop("OPENAI_API_KEY", None)
+            else:
+                os.environ["OPENAI_API_KEY"] = prev_key
+    else:
+        results = backend.run_harness(run_id)

    # Convert BenchmarkResults to RunSummary
    total = len(results.trial_results) if hasattr(results, "trial_results") else 0
@@ -711,18 +764,47 @@ def _run_terminalbench_native(config, console: Console) -> object:
    )


-def _run_single(config, console: Optional[Console] = None) -> object:
-    """Run a single eval from a RunConfig and return the summary."""
+def _run_single(
+    config,
+    console: Optional[Console] = None,
+    *,
+    suite_mode: bool = False,
+) -> object:
+    """Run a single eval from a RunConfig and return the summary.
+
+    ``suite_mode=True`` (TOML-suite drivers) scopes ``config.base_url`` /
+    ``config.api_key`` — stamped from the suite's ``[backend.external]``
+    section onto every RunConfig — to the hermes/openclaw backends only;
+    extending suite-level endpoint targeting to first-party backends is
+    explicitly deferred. The CLI single-run path (``suite_mode=False``)
+    honors ``--base-url``/``--api-key`` for every backend.
+    """
    from openjarvis.evals.core.runner import EvalRunner

    if console is None:
        console = Console()

+    _metadata = getattr(config, "metadata", None) or {}
+    base_url = (
+        getattr(config, "base_url", None)
+        or _metadata.get("base_url")
+        or os.environ.get("JARVIS_BACKEND_BASE_URL")
+    )
+    api_key = (
+        getattr(config, "api_key", None)
+        or _metadata.get("api_key")
+        or os.environ.get("JARVIS_BACKEND_API_KEY")
+    )
+
    # TerminalBench V2.1 native: use terminal-bench Harness directly
    if config.benchmark == "terminalbench-native":
-        return _run_terminalbench_native(config, console)
+        return _run_terminalbench_native(
+            config,
+            console,
+            base_url=None if suite_mode else base_url,
+            api_key=None if suite_mode else api_key,
+        )

-    _metadata = getattr(config, "metadata", None) or {}
    eval_backend = _build_backend(
        config.backend,
        config.engine_key,
@@ -732,16 +814,9 @@ def _run_single(config, console: Optional[Console] = None) -> object:
        gpu_metrics=getattr(config, "gpu_metrics", False),
        model=config.model,
        max_turns=getattr(config, "max_turns", None),
-        base_url=(
-            getattr(config, "base_url", None)
-            or _metadata.get("base_url")
-            or os.environ.get("JARVIS_BACKEND_BASE_URL")
-        ),
-        api_key=(
-            getattr(config, "api_key", None)
-            or _metadata.get("api_key")
-            or os.environ.get("JARVIS_BACKEND_API_KEY")
-        ),
+        base_url=base_url,
+        api_key=api_key,
+        first_party_endpoint=not suite_mode,
    )
    dataset = _build_dataset(config.benchmark)
    # Inject engine config for benchmarks that run their own simulation
@@ -1070,7 +1145,7 @@ def _run_from_config(
            f"Run {i}/{len(run_configs)}: {rc.benchmark} / {rc.model}",
        )
        try:
-            summary = _run_single(rc, console=console)
+            summary = _run_single(rc, console=console, suite_mode=True)
            summaries.append(summary)
            console.print(
                f"  [green]{summary.accuracy:.4f}[/green] "
@@ -1115,12 +1190,21 @@ def main():
@click.option(
    "--base-url",
    default=None,
-    help="OpenAI-compat endpoint for hermes/openclaw",
+    help=(
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
+        "(env: JARVIS_BACKEND_BASE_URL)."
+    ),
 )
@click.option(
    "--api-key",
    default=None,
-    help="API key for hermes/openclaw endpoint",
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
@click.option("-m", "--model", default=None, help="Model identifier")
@click.option(
@@ -1526,8 +1610,7 @@ def summarize(jsonl_path):
    default=None,
    type=click.Path(),
    help=(
-        "Output JSONL path. Defaults to <jsonl>.reparsed when "
-        "--in-place is not set."
+        "Output JSONL path. Defaults to <jsonl>.reparsed when --in-place is not set."
    ),
 )
@click.option(
@@ -1642,9 +1725,7 @@ def reparse_judge(jsonl_path, out_path, in_place, summary_out):
        _json.dump(summary, f, indent=2)

    old_cont = [float(s) for s in old_scores if s is not None]
-    old_acc = (
-        sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
-    )
+    old_acc = sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
    old_mean = sum(old_cont) / len(old_cont) if old_cont else 0.0
    new_mean = sum(cont) / len(cont) if cont else 0.0
    mean_shift = new_mean - old_mean
@@ -4,6 +4,27 @@ from __future__ import annotations

 from unittest.mock import MagicMock, patch

+import httpx
+import pytest
+import respx
+
+
+def _mock_builder() -> MagicMock:
+    """A SystemBuilder mock whose fluent methods chain like the real one."""
+    builder = MagicMock()
+    for method in (
+        "engine",
+        "engine_instance",
+        "model",
+        "agent",
+        "tools",
+        "telemetry",
+        "traces",
+    ):
+        getattr(builder, method).return_value = builder
+    builder.build.return_value = MagicMock()
+    return builder
+

 class TestJarvisDirectBackend:
    @patch("openjarvis.system.SystemBuilder")
@@ -137,3 +158,130 @@ class TestJarvisAgentBackend:
        assert result["content"] == "The answer is 4."
        assert result["turns"] == 2
        assert len(result["tool_results"]) == 1
+
+
+class TestJarvisDirectBackendBaseUrl:
+    """--base-url targeting for the jarvis-direct backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        # Trailing /v1 is normalized away so request paths don't double up.
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        # The discovery path must not be engaged at all.
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisDirectBackend(base_url="http://127.0.0.1:18998")
+
+        # No silent engine substitution: the system is never built.
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisDirectBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_wins_over_engine_key(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(engine_key="vllm", base_url="http://127.0.0.1:18999")
+
+        mock_builder.engine.assert_not_called()
+        mock_builder.engine_instance.assert_called_once()
+        # The engine key is kept as the label for the injected engine.
+        assert mock_builder.engine_instance.call_args.kwargs["key"] == "vllm"
+
+
+class TestJarvisAgentBackendBaseUrl:
+    """--base-url targeting for the jarvis-agent backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisAgentBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisAgentBackend(base_url="http://127.0.0.1:18998")
+
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisAgentBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
@@ -0,0 +1,190 @@
+"""--base-url/--api-key forwarding through the eval CLI plumbing.
+
+Covers the fix for the eval-CLI endpoint gap: the flags used to be silently
+dropped for jarvis-direct/jarvis-agent and ignored by terminalbench-native
+(which hardcoded api_base="http://localhost:8000/v1").
+"""
+
+from __future__ import annotations
+
+import io
+import os
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import click
+import pytest
+from rich.console import Console
+
+from openjarvis.evals.cli import _build_backend, _run_terminalbench_native
+from openjarvis.evals.core.types import RunConfig
+
+
+def _quiet_console() -> Console:
+    return Console(file=io.StringIO())
+
+
+def _tb_config(**overrides) -> RunConfig:
+    defaults = dict(
+        benchmark="terminalbench-native",
+        backend="jarvis-direct",
+        model="my-model",
+        max_samples=1,
+        max_workers=1,
+        temperature=0.2,
+    )
+    defaults.update(overrides)
+    return RunConfig(**defaults)
+
+
+class TestBuildBackendForwardsEndpoint:
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_jarvis_direct_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_agent.JarvisAgentBackend")
+    def test_jarvis_agent_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-agent",
+            "vllm",
+            "orchestrator",
+            ["calculator"],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_suite_mode_scopes_endpoint_to_external_backends(self, mock_cls):
+        """[backend.external] suite semantics stay hermes/openclaw-only:
+        first_party_endpoint=False must not forward to first-party."""
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+            first_party_endpoint=False,
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] is None
+        assert kwargs["api_key"] is None
+
+    def test_hermes_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="hermes"):
+            _build_backend("hermes", None, "orchestrator", [])
+
+    def test_openclaw_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="openclaw"):
+            _build_backend("openclaw", None, "orchestrator", [])
+
+
+class TestTerminalBenchNativeApiBase:
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_passed_through_as_api_base(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_without_v1_gets_single_v1_suffix(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_default_api_base_unchanged_without_base_url(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(_tb_config(), _quiet_console())
+        assert mock_cls.call_args.kwargs["api_base"] == "http://localhost:8000/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_api_key_exported_as_openai_api_key_during_run(self, mock_cls, monkeypatch):
+        """terminus-2 reads OPENAI_API_KEY via LiteLLM; the var must be set
+        during harness.run() and restored afterwards."""
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+        seen: dict = {}
+
+        def fake_run_harness(run_id):
+            seen["openai_api_key"] = os.environ.get("OPENAI_API_KEY")
+            return SimpleNamespace(trial_results=[])
+
+        mock_backend = MagicMock()
+        mock_backend.run_harness.side_effect = fake_run_harness
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert seen["openai_api_key"] == "sk-tb"
+        assert "OPENAI_API_KEY" not in os.environ  # restored
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_preexisting_openai_api_key_restored(self, mock_cls, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "sk-original")
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert os.environ["OPENAI_API_KEY"] == "sk-original"
+
+
+class TestRunSingleSuiteModeGating:
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_suite_mode_drops_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console(), suite_mode=True)
+        assert mock_tb.call_args.kwargs["base_url"] is None
+        assert mock_tb.call_args.kwargs["api_key"] is None
+
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_cli_mode_forwards_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console())
+        assert mock_tb.call_args.kwargs["base_url"] == "http://node7:8123/v1"
+        assert mock_tb.call_args.kwargs["api_key"] == "sk-k"
@@ -33,6 +33,8 @@ class SystemBuilder:
            self._config = load_config()

        self._engine_key: Optional[str] = None
+        self._engine_instance: Optional[InferenceEngine] = None
+        self._engine_instance_key: Optional[str] = None
        self._model: Optional[str] = None
        self._agent_name: Optional[str] = None
        self._tool_names: Optional[List[str]] = None
@@ -50,6 +52,20 @@ class SystemBuilder:
        self._engine_key = key
        return self

+    def engine_instance(
+        self, engine: InferenceEngine, key: str = "openai-compat"
+    ) -> SystemBuilder:
+        """Inject a pre-built engine instance, bypassing engine discovery.
+
+        Used by callers that must target one exact endpoint (e.g.
+        ``jarvis eval --base-url``). ``build()`` health-checks the instance
+        and raises a loud error if it is unreachable — it never silently
+        substitutes a different discovered engine.
+        """
+        self._engine_instance = engine
+        self._engine_instance_key = key
+        return self
+
    def model(self, name: str) -> SystemBuilder:
        self._model = name
        return self
@@ -303,6 +319,23 @@ class SystemBuilder:
        return system

    def _resolve_engine(self, config: JarvisConfig):
+        # An explicitly injected engine instance always wins and is never
+        # silently replaced: when the caller pinned an endpoint (e.g.
+        # ``jarvis eval --base-url``) and it is down, substituting whatever
+        # other engine discovery finds would silently run against the wrong
+        # model server. Fail loudly instead.
+        if self._engine_instance is not None:
+            engine = self._engine_instance
+            key = self._engine_instance_key or "openai-compat"
+            if not engine.health():
+                host = getattr(engine, "_host", "<unknown host>")
+                raise RuntimeError(
+                    f"Injected engine {key!r} is not reachable at {host} — "
+                    "is the endpoint running and serving GET /v1/models? "
+                    "Refusing to fall back to engine discovery."
+                )
+            return engine, key
+
        from openjarvis.engine._discovery import get_engine

        pref = config.intelligence.preferred_engine
@@ -313,7 +346,18 @@ class SystemBuilder:
                "No inference engine available. "
                "Make sure an engine is running (e.g. ollama serve)."
            )
-        return resolved[1], resolved[0]
+        resolved_key, engine = resolved
+        if self._engine_key and resolved_key != self._engine_key:
+            # get_engine() falls back to any healthy discovered engine; make
+            # the substitution visible when the caller asked for a specific
+            # engine (observed: requested vllm, silently got ollama@11434).
+            logger.warning(
+                "Requested engine %r is unavailable; using %r at %s instead",
+                self._engine_key,
+                resolved_key,
+                getattr(engine, "_host", "<unknown host>"),
+            )
+        return engine, resolved_key

    def _resolve_model(self, config: JarvisConfig, engine: InferenceEngine) -> str:
        if self._model:
@@ -437,3 +437,39 @@ class TestOpenRouterToolForwarding:
        assert result["tool_calls"][0]["id"] == "call_1"
        assert result["tool_calls"][0]["function"]["name"] == "get_weather"
        assert result["tool_calls"][0]["function"]["arguments"] == '{"city": "NYC"}'
+
+
+class TestCloudEngineCanServe:
+    """#532: can_serve gates on the per-provider client, not just health().
+
+    health() is True whenever *any* provider client is configured, but a
+    request for a gpt-* model still needs the OpenAI client specifically — so
+    engine selection must not pick the cloud engine for a model whose provider
+    client is missing.
+    """
+
+    @staticmethod
+    def _engine(**clients: object) -> CloudEngine:
+        eng = CloudEngine.__new__(CloudEngine)  # bypass real client init
+        for name in (
+            "_openai_client",
+            "_anthropic_client",
+            "_google_client",
+            "_openrouter_client",
+            "_minimax_client",
+            "_codex_client",
+        ):
+            setattr(eng, name, clients.get(name))
+        return eng
+
+    def test_openai_only_serves_openai_models(self) -> None:
+        eng = self._engine(_openai_client=object())
+        assert eng.can_serve("gpt-4o") is True
+        assert eng.can_serve("claude-sonnet-4") is False
+        assert eng.can_serve("gemini-2.5-pro") is False
+        assert eng.can_serve("openrouter/openai/gpt-4o") is False
+
+    def test_anthropic_only_serves_anthropic_models(self) -> None:
+        eng = self._engine(_anthropic_client=object())
+        assert eng.can_serve("claude-sonnet-4") is True
+        assert eng.can_serve("gpt-4o") is False
@@ -174,6 +174,51 @@ class TestGetEngine:
        assert result is not None
        assert result[0] == "running"

+    def test_skips_engine_that_cannot_serve_model(self) -> None:
+        """#532: a healthy engine that can't serve the requested model is
+        skipped for one that can — this is what stops the cloud fallback being
+        chosen (when the local engine is down) for a model whose provider
+        client is missing.
+        """
+        _reg("picky", "picky")
+        _reg("local", "local")
+
+        class _Picky(_FakeEngine):
+            def can_serve(self, model: str) -> bool:
+                return model == "servable"
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "picky"
+
+        def _make(k, c):  # noqa: ANN001
+            if k == "picky":
+                return _Picky(healthy=True)
+            return _FakeEngine(healthy=(k == "local"))
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=_make,
+        ):
+            # "picky" is healthy but cannot serve "other" -> fall back to "local"
+            result = get_engine(cfg, model="other")
+        assert result is not None
+        assert result[0] == "local"
+
+    def test_model_none_preserves_model_agnostic_selection(self) -> None:
+        """model=None keeps the legacy behaviour: first healthy engine wins."""
+        _reg("primary", "primary")
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "primary"
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=lambda k, c: _FakeEngine(healthy=True),  # noqa: ANN001
+        ):
+            result = get_engine(cfg, model=None)
+        assert result is not None
+        assert result[0] == "primary"
+

 class TestMiningSidecarEngineHandoff:
    """Engine discovery picks up (or ignores) a mining sidecar at runtime."""
@@ -0,0 +1,109 @@
+"""API-key (Authorization header) support in the OpenAI-compat engine base."""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+import respx
+
+from openjarvis.core.types import Message, Role
+from openjarvis.engine.openai_compat_engines import (
+    OpenAICompatEngine,
+    VLLMEngine,
+    normalize_openai_base_url,
+)
+
+_CHAT_RESPONSE = {
+    "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}],
+    "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+    "model": "m",
+}
+
+
+class TestAuthorizationHeader:
+    def test_bearer_header_sent_when_api_key_set(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_no_authorization_header_without_api_key(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert "authorization" not in route.calls.last.request.headers
+
+    def test_health_check_sends_bearer_header(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_env_var_fallback_sanitizes_hyphen(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        # engine_id "openai-compat" must map to OPENAI_COMPAT_API_KEY —
+        # shells cannot set hyphenated env-var names.
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-env"
+
+    def test_vllm_env_var_fallback(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        monkeypatch.setenv("VLLM_API_KEY", "sk-vllm")
+        engine = VLLMEngine(host="http://testhost:8000")
+        with respx.mock:
+            route = respx.get("http://testhost:8000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-vllm"
+
+    def test_explicit_api_key_beats_env_var(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
+        assert engine._api_key == "sk-explicit"
+
+
+class TestNormalizeOpenAIBaseUrl:
+    @pytest.mark.parametrize(
+        ("url", "expected"),
+        [
+            ("http://h:8000", "http://h:8000"),
+            ("http://h:8000/", "http://h:8000"),
+            ("http://h:8000/v1", "http://h:8000"),
+            ("http://h:8000/v1/", "http://h:8000"),
+            ("http://h:8000/gateway/v1", "http://h:8000/gateway"),
+            # Only a literal trailing "/v1" is stripped — never other paths.
+            ("http://h:8000/v1x", "http://h:8000/v1x"),
+            ("http://h:8000/v2", "http://h:8000/v2"),
+        ],
+    )
+    def test_normalization(self, url: str, expected: str) -> None:
+        assert normalize_openai_base_url(url) == expected
+
+    def test_engine_requests_have_single_v1_prefix(self) -> None:
+        """End to end: a user-supplied .../v1 URL must not produce /v1/v1."""
+        host = normalize_openai_base_url("http://testhost:9000/v1")
+        engine = OpenAICompatEngine(host=host)
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.url.path == "/v1/models"
@@ -2,7 +2,7 @@

 from __future__ import annotations

-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch

 import pytest

@@ -327,6 +327,85 @@ class TestSystemBuilder:
        assert builder._engine_key == "ollama"


+class TestSystemBuilderEngineInstance:
+    """Explicit engine injection (jarvis eval --base-url path)."""
+
+    @staticmethod
+    def _fake_engine(healthy: bool = True) -> MagicMock:
+        engine = MagicMock(
+            spec=["health", "can_serve", "generate", "list_models", "close"]
+        )
+        engine.health.return_value = healthy
+        engine._host = "http://127.0.0.1:18999"
+        return engine
+
+    def test_engine_instance_is_fluent(self):
+        builder = SystemBuilder(JarvisConfig())
+        engine = self._fake_engine()
+        result = builder.engine_instance(engine, key="my-endpoint")
+        assert result is builder
+        assert builder._engine_instance is engine
+        assert builder._engine_instance_key == "my-endpoint"
+
+    def test_resolve_engine_returns_injected_instance(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        resolved_engine, resolved_key = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        assert resolved_key == "endpoint"
+
+    def test_unhealthy_injected_instance_raises_naming_host(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18999"):
+            builder._resolve_engine(config)
+
+    def test_unhealthy_injected_instance_never_consults_discovery(self):
+        """The observed failure mode: an explicit endpoint must NOT be
+        silently replaced by whatever other engine discovery finds."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine)
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            with pytest.raises(RuntimeError, match="Refusing to fall back"):
+                builder._resolve_engine(config)
+        mock_get_engine.assert_not_called()
+
+    def test_healthy_injected_instance_never_consults_discovery(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            resolved_engine, _ = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        mock_get_engine.assert_not_called()
+
+    def test_build_wires_injected_engine(self):
+        """build() must use the injected engine (possibly behind security
+        wrappers) instead of running discovery."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        engine.list_models.return_value = ["stub-model"]
+        builder = (
+            SystemBuilder(config)
+            .engine_instance(engine, key="endpoint")
+            .model("stub-model")
+            .telemetry(False)
+            .traces(False)
+        )
+        system = builder.build()
+        try:
+            inner = system.engine
+            while hasattr(inner, "_engine"):
+                inner = inner._engine
+            assert inner is engine
+            assert system.engine_key == "endpoint"
+        finally:
+            system.close()
+
+
 class TestJarvisSystemClose:
    def test_close_with_scheduler_store(self):
        engine = MagicMock()
Author	SHA1	Message	Date
Jon Saad-FalconandClaude Opus 4.8	0cac61d3bb	docs(evals): rewrite evaluations guide to match the real CLI surface; add openjarvis-eval alias (#534 ) docs/user-guide/evaluations.md documented a standalone "openjarvis-evals" package, a "uv sync --extra eval" install, and an "openjarvis-eval" console script — a layout from commit `bd493832` that was never an ancestor of main. Rewrite the page against the real surface (jarvis eval / python -m openjarvis.evals), document all 40 registered benchmark keys and 4 backends, fix the judge-model default, correct run-all semantics, and split the option reference into the jarvis-eval subset and the module CLI's research-only options. Add the one-line [project.scripts] alias openjarvis-eval = "openjarvis.evals.cli:main" (the click group the module CLI already dispatches to) so the long-documented command name works again. The page was a complete orphan: add it (and the equally orphaned benchmarks.md) to the mkdocs nav and link it from docs/index.md. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 13:50:17 -07:00
Jon Saad-FalconandClaude Opus 4.8	50993dfa4d	fix(evals): honor --base-url/--api-key for first-party eval backends (#535 ) `jarvis eval run --base-url ... --api-key ...` was silently dropped for jarvis-direct/jarvis-agent (_build_backend only forwarded the flags to hermes/openclaw) and ignored by terminalbench-native, which hardcoded api_base="http://localhost:8000/v1". Worse, with --base-url set the engine-discovery fallback silently substituted ANY healthy local engine (observed: requested vllm + healthy endpoint at --base-url, got OllamaEngine@localhost:11434 — the requested URL was never contacted). Changes: - _OpenAICompatibleEngine gains an api_key param (Bearer Authorization header on the httpx client; {ENGINE_ID}_API_KEY env fallback with hyphen-sanitized names; no header when unset). - New non-registered OpenAICompatEngine + normalize_openai_base_url() (strips a single literal trailing "/v1" so request paths don't double). - SystemBuilder.engine_instance() injects a pre-built engine; build() health-checks it and fails loudly naming the host instead of falling back to discovery. Discovery substitution after an explicit -e key now logs a warning. - JarvisDirectBackend/JarvisAgentBackend accept base_url/api_key; on base_url they pin an OpenAICompatEngine to that endpoint with a fail-fast pre-flight (actionable error naming the URL and probe). - _build_backend forwards base_url/api_key to first-party backends on the CLI path; _run_terminalbench_native receives --base-url as api_base (single /v1 suffix) and exports OPENAI_API_KEY around the in-process harness run (terminus-2 routes via LiteLLM). - Suite TOML [backend.external] stays scoped to hermes/openclaw (suite_mode=True in the suite drivers) — first-party suite semantics are explicitly deferred. The config-host path is untouched. - Help text updated on both CLI surfaces; KNOWN_BACKENDS now lists hermes/openclaw/terminalbench-native. Fixes the eval-CLI endpoint gap reported by the downstream team. Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>	2026-06-11 13:49:37 -07:00
github-actions[bot]	527f84f960	chore: update clone traffic data [skip ci]	2026-06-11 07:44:18 +00:00
Robby Manihani Claude Opus 4.8 Jon Saad-Falcon	8eaeb3a754	fix(windows): desktop backend spawn + model-aware engine selection (#533 ) * fix(windows): desktop backend spawn (#531) + model-aware engine selection (#532) Two runtime bugs found during end-to-end testing on a clean Windows 11 24H2 Azure VM. #531 - Desktop "Failed to get response": run_jarvis_command spawned the backend with .output(), which waits for the process to exit. `jarvis serve` never exits, so the Tauri command hung forever (the Start button never resolved); and it ran `uv run jarvis` with no cwd, so in a packaged install -- where the cwd isn't the checkout -- `jarvis` wasn't found and the server never started. Now: run from find_project_root(), and for `serve` spawn detached (.spawn()), drain stderr, and poll /health for readiness (mirrors start_backend); short commands keep .output(). The server layer itself was verified healthy on Windows (/health and /v1/chat/completions both 200, localhost included) -- the fault was the Tauri spawn path. #532 - "OpenAI client not available" after reboot: when the local engine is down, get_engine's fallback selected CloudEngine because health() is True if ANY provider client exists -- without checking the resolved model's provider has a client. A user with e.g. OPENROUTER_API_KEY and a gpt-* model then hit the OpenAI path with no client. Add CloudEngine.can_serve(model) (checks the specific provider client via the same routing generate()/stream() use) + a default can_serve->True on the base engine, and make get_engine model-aware so it skips an engine that can't serve the model -- the user falls through to the helpful "no engine available / start ollama" message instead. Tests: engine discovery/cloud/model-matrix + cli serve/ask suites pass (the one ask_e2e failure is a pre-existing version-banner flake, fails identically on main). The Tauri crate couldn't be compiled locally (no GTK/webkit sys-libs in this env); relies on CI. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> * test(engine): cover model-aware engine selection + CloudEngine.can_serve (#532) #533 added a `model` arg to get_engine and a can_serve() gate but shipped no tests. Add them: - get_engine skips a healthy engine that can't serve the requested model (the cloud-fallback-for-unservable-model case behind #532), - model=None preserves the legacy model-agnostic selection, - CloudEngine.can_serve gates on the per-provider client (gpt->OpenAI, claude->Anthropic, ...), verified empirically. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com> --------- Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com> Co-authored-by: Jon Saad-Falcon <jonsaadfalcon@gmail.com>	2026-06-10 19:52:36 -07:00