fix(connectors): complete Google OAuth and register Drive in Data Sources (fixes #512) (#548)

Pasting a Google Client ID / Secret never completed OAuth: Drive (and its
Google siblings) accepted the credentials, showed no error, opened no browser,
and never appeared in Data Sources. Root cause is three coupled defects, all
reproduced at the unit level against main with a FastAPI TestClient (no Google
creds, network-free):

(A/B) POST /connect routed a `client_id:client_secret` pair into the
  connector's handle_callback, which spawned a daemon thread that popped a
  browser and ran its own localhost:8789 callback server. That thread fails
  silently in the bundled desktop context (`except Exception: pass`), so the
  connector never gained an access_token; /connect returned status "pending"
  and the UI's 20x2s poll timed out with no error.
  Fix: in POST /connect, an OAuth `client_id:client_secret` pair now persists
  the client credentials to every Google credential file and returns an
  `oauth_required` directive pointing at the in-process server flow, instead of
  the silent background thread. The Google connectors' handle_callback no longer
  spawns the browser thread for the pair case — it only persists the creds; the
  server's /oauth/start -> /oauth/callback owns the consent round-trip.

(C) The would-be-correct server flow was itself broken: under
  `from __future__ import annotations` plus a `Request` import local to the
  router factory, FastAPI could not resolve the stringized `request: Request`
  annotation. /oauth/start returned HTTP 422 (request mis-bound as a query
  param) and /oauth/callback injected None -> AttributeError on
  `request.base_url`. Fix: import `Request` at module scope and make the
  callback's `request` a required injected dependency.

A malformed/blank client pair now raises HTTP 400 with the provider setup URL
instead of a perpetual silent "pending" (REVIEW.md silent-failure discipline).

Frontend: DataSourcesPage now opens the server OAuth window when /connect
returns `oauth_required`, then polls until connected; connect errors surface the
backend detail; the Drive setup steps document the "Web application" OAuth
client + server-callback redirect URI the in-process flow requires.

Tests (run on the main venv, hermetic — no ~/.openjarvis pollution):
- test_oauth_flow.py: the three handle_callback tests now assert NO browser is
  opened and only client creds are persisted (was: assert background flow ran).
- test_connectors_router_oauth.py (new): reproduces + fixes all three defects via
  TestClient with mocked token exchange; parametrized over gdrive/gcalendar/
  gcontacts/gmail/google_tasks to prove the shared OAuth path is fixed for every
  sibling and that a single consent writes the access_token to all six Google
  credential files and flips is_connected() to True.
Full tests/connectors suite: 355 passed.

Relationship to PR #510: #510 rewrites all of these files (account-scoped
retrieval) but still carries all three defects. This fix is intentionally scoped
to the OAuth path and does not modify oauth.py, to minimize collision. A
maintainer can either merge this and rebase #510 on top, or port these changes
into #510. See PR body for details.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/clone-stats/badge.json

@@ -1,7 +1,7 @@
 {
   "schemaVersion": 1,
   "label": "Git Clones",
-  "message": "107,695",
+  "message": "117,047",
   "color": "green",
   "namedLogo": "git"
 }

.github/clone-stats/clone-data.json

@@ -1,6 +1,6 @@
 {
-  "total_clones": 107695,
-  "last_updated": "2026-06-10T07:32:26Z",
+  "total_clones": 117047,
+  "last_updated": "2026-06-14T07:38:47Z",
   "daily": {
     "2026-03-27": 2189,
     "2026-03-28": 1874,
@@ -75,6 +75,11 @@
     "2026-06-05": 2127,
     "2026-06-06": 2204,
     "2026-06-07": 1174,
-    "2026-06-08": 2369
+    "2026-06-08": 2369,
+    "2026-06-09": 1361,
+    "2026-06-10": 1310,
+    "2026-06-11": 2564,
+    "2026-06-12": 1313,
+    "2026-06-13": 2804
   }
 }

.github/workflows/autotag.yml

@@ -21,14 +21,20 @@ jobs:
         id: version
         run: |
           set -euo pipefail
-          # Base version is the next patch above whatever is in pyproject.toml.
+          # Base version is the next patch above the latest plain release tag
+          # (vX.Y.Z) reachable from HEAD. pyproject.toml no longer carries a
+          # static version (#526 switched it to hatch-vcs), so the release tag
+          # is the source of truth. `.devN`/`.rcN`/`desktop-*` tags are excluded
+          # so they can't be mistaken for the release base.
           # Any future manual `X.Y.Z` release will outrank every `X.Y.Z.devN`
           # autotag — PEP 440 sorts dev releases strictly below the final.
-          BASE=$(grep -E '^version = "' pyproject.toml | head -1 | sed -E 's/^version = "([^"]+)"/\1/')
-          if [[ -z "$BASE" ]]; then
-            echo "::error::Could not parse version from pyproject.toml"
+          LATEST_RELEASE=$(git tag --list 'v[0-9]*' --merged HEAD \
+            | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1)
+          if [[ -z "$LATEST_RELEASE" ]]; then
+            echo "::error::No release tag (vX.Y.Z) reachable from HEAD"
             exit 1
           fi
+          BASE="${LATEST_RELEASE#v}"
           MAJOR=$(echo "$BASE" | cut -d. -f1)
           MINOR=$(echo "$BASE" | cut -d. -f2)
           PATCH=$(echo "$BASE" | cut -d. -f3 | sed -E 's/[^0-9].*$//')

.github/workflows/claude-issues.yml

@@ -11,22 +11,33 @@ concurrency:
   group: claude-issues-${{ github.event.issue.number || github.run_id }}
   cancel-in-progress: true
 
+# Least-privilege: only what the issue-fixer job actually needs.
+# id-token (OIDC) is intentionally omitted — claude-code-action@v1 is passed
+# github_token directly, so OIDC is unused here.
 permissions:
   contents: write
   pull-requests: write
   issues: write
-  id-token: write
 
 jobs:
   fix:
     runs-on: ubuntu-latest
-    timeout-minutes: 60
+    timeout-minutes: 15
+    # Security gate: this job reaches secrets.ANTHROPIC_API_KEY and holds a
+    # write-scoped GITHUB_TOKEN. `issues` / `issue_comment` are public,
+    # attacker-controllable events that run in the base-repo context with full
+    # secret access, so the human-triggered paths are restricted to actors with
+    # write-level association (OWNER / MEMBER / COLLABORATOR). This blocks
+    # external / first-time contributors from draining the API budget or
+    # creating branches/PRs, while leaving maintainer use unaffected.
     if: |
       github.event_name == 'workflow_dispatch' ||
       (github.event_name == 'issues' &&
+       contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.issue.author_association) &&
        (contains(github.event.issue.labels.*.name, 'bug') ||
         contains(github.event.issue.labels.*.name, 'autofix'))) ||
       (github.event_name == 'issue_comment' &&
+       contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) &&
        !github.event.issue.pull_request &&
        contains(github.event.comment.body, '@claude') &&
        github.actor != 'claude[bot]')

.github/workflows/claude-review.yml

@@ -11,23 +11,33 @@ concurrency:
   group: claude-review-${{ github.event.pull_request.number || github.event.issue.number || github.run_id }}
   cancel-in-progress: true
 
+# Least-privilege: PR review only needs to post comments on the PR.
+# id-token (OIDC) is omitted — claude-code-action@v1 is passed github_token
+# directly, so OIDC is unused here.
 permissions:
   contents: read
   pull-requests: write
   issues: write
-  id-token: write
 
 jobs:
   review:
     runs-on: ubuntu-latest
     timeout-minutes: 30
+    # Security gate: this job reaches secrets.ANTHROPIC_API_KEY. Both
+    # issue_comment and pull_request_review_comment are public,
+    # attacker-controllable events that run in the base-repo context with full
+    # secret access, so the @claude paths are restricted to actors with
+    # write-level association (OWNER / MEMBER / COLLABORATOR). External /
+    # first-time contributors cannot trigger the key; maintainers are unaffected.
     if: |
       github.event_name == 'workflow_dispatch' ||
       (github.event_name == 'issue_comment' &&
+       contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) &&
        github.event.issue.pull_request &&
        contains(github.event.comment.body, '@claude') &&
        github.actor != 'claude[bot]') ||
       (github.event_name == 'pull_request_review_comment' &&
+       contains(fromJSON('["OWNER", "MEMBER", "COLLABORATOR"]'), github.event.comment.author_association) &&
        contains(github.event.comment.body, '@claude') &&
        github.actor != 'claude[bot]')
     steps:

.github/workflows/desktop.yml

@@ -114,6 +114,11 @@ jobs:
 
     steps:
       - uses: actions/checkout@v6
+        with:
+          # Full history + tags so the workflow_dispatch fallback in
+          # "Determine release info" can derive the dev version from the
+          # latest release tag (#526).
+          fetch-depth: 0
 
       - name: Install system dependencies (Linux)
         if: matrix.platform == 'ubuntu-22.04'
@@ -183,7 +188,16 @@ jobs:
             # workflow_dispatch fallback (manual UI dispatch without --ref).
             # Derive a PEP 440 dev version aligned with autotag.yml so we
             # don't burn the X.Y.Z release-version namespace.
-            BASE=$(grep -E '^version = "' pyproject.toml | head -1 | sed -E 's/^version = "([^"]+)"/\1/')
+            # pyproject.toml no longer carries a static version (#526), so the
+            # base comes from the latest plain release tag (vX.Y.Z), matching
+            # autotag.yml. .dev/.rc/desktop-* tags are excluded.
+            LATEST_RELEASE=$(git tag --list 'v[0-9]*' --merged HEAD \
+              | grep -E '^v[0-9]+\.[0-9]+\.[0-9]+$' | sort -V | tail -1)
+            if [[ -z "$LATEST_RELEASE" ]]; then
+              echo "::error::No release tag (vX.Y.Z) reachable from HEAD"
+              exit 1
+            fi
+            BASE="${LATEST_RELEASE#v}"
             MAJOR=$(echo "$BASE" | cut -d. -f1)
             MINOR=$(echo "$BASE" | cut -d. -f2)
             PATCH=$(echo "$BASE" | cut -d. -f3 | sed -E 's/[^0-9].*$//')

.github/workflows/pypi-publish.yml

@@ -12,6 +12,11 @@ on:
         description: 'Tag to publish (e.g. v1.0.2.dev500). Overrides github.ref.'
         required: false
         type: string
+      dry_run:
+        description: 'Dry run: build + validate, then publish to TestPyPI instead of PyPI (no production upload).'
+        required: false
+        default: false
+        type: boolean
 
 permissions:
   contents: read
@@ -67,27 +72,41 @@ jobs:
             exit 1
           }
 
-      - name: Set version from tag
+      - name: Resolve build version from tag
         env:
           REF: ${{ steps.ref.outputs.ref }}
         run: |
           set -euo pipefail
-          # Strip leading "v" if present (e.g. v1.0.2.dev500 -> 1.0.2.dev500)
+          # Strip leading "v" (e.g. v1.0.3.dev825 -> 1.0.3.dev825).
           VERSION="${REF#v}"
-          if [[ -z "$VERSION" ]]; then
-            echo "::error::Could not resolve version from ref '$REF'"
+          if ! [[ "$VERSION" =~ ^[0-9]+\.[0-9]+\.[0-9]+ ]]; then
+            echo "::error::ref '$REF' is not a version tag (expected vX.Y.Z[.devN]); pass -f tag=vX.Y.Z"
             exit 1
           fi
-          sed -i "s/^version = \".*\"/version = \"${VERSION}\"/" pyproject.toml
-          # Sanity check the substitution actually took
-          grep -q "^version = \"${VERSION}\"" pyproject.toml || {
-            echo "::error::sed failed to update pyproject.toml version"
-            exit 1
-          }
-          echo "Building version $VERSION"
+          # pyproject.toml is now dynamic = ["version"] via hatch-vcs (#526), so
+          # there is no static line to sed. setuptools_scm cannot bump custom
+          # `.devN` tags, so we pin the exact build version explicitly — the
+          # published version always equals the pushed tag.
+          echo "SETUPTOOLS_SCM_PRETEND_VERSION=${VERSION}" >> "$GITHUB_ENV"
+          echo "Building version ${VERSION}"
 
       - name: Build package
         run: uv build
 
+      - name: Publish to TestPyPI (dry run)
+        if: ${{ inputs.dry_run }}
+        env:
+          UV_PUBLISH_TOKEN: ${{ secrets.TEST_PYPI_API_TOKEN }}
+        run: |
+          set -euo pipefail
+          if [[ -z "${UV_PUBLISH_TOKEN:-}" ]]; then
+            echo "::warning::TEST_PYPI_API_TOKEN is not set — skipping the TestPyPI upload."
+            echo "Build + twine check passed, which validated version derivation and packaging end to end."
+            echo "To exercise a real upload, add a TEST_PYPI_API_TOKEN secret (or a TestPyPI trusted publisher)."
+            exit 0
+          fi
+          uv publish --publish-url https://test.pypi.org/legacy/
+
       - name: Publish to PyPI
+        if: ${{ !inputs.dry_run }}
         run: uv publish

CHANGELOG.md

@@ -8,6 +8,19 @@ The format is based on [Keep a Changelog](https://keepachangelog.com/en/1.1.0/).
 
 ## [Unreleased]
 
+### Added
+
+**Vision input for `jarvis ask`** — attach images to a query with
+`-i`/`--image` (repeatable) or capture the current screen with
+`-S`/`--screen`, for vision-capable models such as `gemma3:4b`. Images flow
+through `Message.images` into Ollama's `/api/chat` `images` field; text-only
+requests are unaffected. A privacy guard warns before any image is sent to a
+non-local engine, and the security guardrail now preserves images when it
+sanitizes a flagged prompt. Screen capture uses the built-in Windows .NET
+stack with `mss`/`Pillow` fallbacks on other platforms. Adds the
+`JARVIS_NUM_CTX` environment variable to tune the Ollama context window
+(default `16384`).
+
 ## [1.0.2] - 2026-05-24
 
 A patch release that fixes a packaging bug which broke the v1.0.1

docs/index.md

@@ -183,7 +183,7 @@ OpenJarvis is built around five composable layers. Each has a clean interface an
 
     ---
 
-    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), agents, memory, tools, and telemetry.
+    CLI, Python SDK, and guides for [Morning Digest](user-guide/morning-digest.md), [Deep Research](user-guide/deep-research.md), [Code Assistant](user-guide/code-assistant.md), [Scheduled Monitor](user-guide/scheduled-monitor.md), [Simple Chat](user-guide/chat-simple.md), [Evaluations](user-guide/evaluations.md), agents, memory, tools, and telemetry.
 
 -   **[Architecture](architecture/overview.md)**
 

docs/user-guide/cli.md

@@ -66,6 +66,8 @@ jarvis ask "What is the capital of France?"
 | `--no-context`                | flag    | off        | Disable memory context injection                       |
 | `-a`, `--agent AGENT`         | string  | none       | Agent to use (`simple`, `orchestrator`)                |
 | `--tools TOOLS`               | string  | none       | Comma-separated tool names to enable                   |
+| `-i`, `--image PATH`          | path    | none       | Image file for a vision model (e.g. `gemma3:4b`); repeatable |
+| `-S`, `--screen`              | flag    | off        | Capture the current screen and send it to the vision model  |
 
 ### Direct Mode vs Agent Mode
 
@@ -105,6 +107,39 @@ jarvis ask --no-context "Tell me about Python"
 jarvis ask --max-tokens 2048 "Write a detailed essay about AI"
 ```
 
+### Vision Input
+
+Vision-capable models (such as `gemma3:4b`) can read images alongside your
+text prompt. Attach one or more image files with `-i`/`--image`, or capture
+the current screen with `-S`/`--screen`:
+
+```bash
+# Ask about a local image
+jarvis ask -i screenshot.png "What is shown in this image?"
+
+# Send multiple images (the flag is repeatable)
+jarvis ask -i chart-a.png -i chart-b.png "Compare these two charts"
+
+# Capture the current screen and ask about it
+jarvis ask --screen "Summarize what's on my screen"
+```
+
+Vision runs in **direct mode** only. If you also pass `--agent`, the image is
+ignored and a note is printed — re-run with `--agent ""` to force direct mode.
+
+The Ollama context window can be tuned for large images or long prompts with
+the `JARVIS_NUM_CTX` environment variable (default `16384`):
+
+```bash
+JARVIS_NUM_CTX=8192 jarvis ask --screen "What's on my screen?"
+```
+
+!!! note "Keep vision on-device"
+    Images are sensitive. OpenJarvis prints a privacy warning before sending
+    an image to a non-local engine, so a screenshot never leaves your machine
+    unnoticed. Use a local engine (e.g. `ollama` with `gemma3:4b`) to keep
+    vision fully local.
+
 ### JSON Output Format
 
 When using `--json` in **direct mode**, the output includes:

docs/user-guide/evaluations.md

@@ -1,14 +1,14 @@
 # Evaluations
 
-The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correctness and accuracy** on academic datasets. It is a separate package from the main OpenJarvis library and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
+The OpenJarvis evaluation framework (`openjarvis.evals`) measures model **correctness and accuracy** on academic datasets. It ships inside the main `openjarvis` package (at `src/openjarvis/evals/`) and is designed specifically for research workflows where you need reproducible, dataset-driven quality assessments.
 
 !!! info "Evals vs. Benchmarks"
     OpenJarvis has two distinct measurement systems that complement each other:
 
-    | System | Package | Measures | Entry Point |
-    |--------|---------|----------|-------------|
-    | **Evaluations** | `openjarvis-evals` | Correctness on academic datasets (accuracy, pass rate) | `openjarvis-eval` |
-    | **Benchmarks** | `openjarvis` | Engine performance (latency, throughput) | `jarvis bench` |
+    | System | Module | Measures | Entry Point |
+    |--------|--------|----------|-------------|
+    | **Evaluations** | `openjarvis.evals` | Correctness on academic datasets (accuracy, pass rate) | `jarvis eval` |
+    | **Benchmarks** | `openjarvis.bench` | Engine performance (latency, throughput) | `jarvis bench` |
 
     Use evaluations to answer "does this model get the right answer?" and benchmarks to answer "how fast does this model respond?". See the [Benchmarks guide](benchmarks.md) for the performance measurement system.
 
@@ -18,22 +18,38 @@ The OpenJarvis evaluation framework (`openjarvis-evals`) measures model **correc
 
 ## Installation
 
-The evaluation framework is a standalone package in the `evals/` directory. Install it alongside OpenJarvis:
+The evaluation framework is part of the main `openjarvis` package — no separate install or extra is required. The standard dev setup is enough:
 
 ```bash
-uv sync --extra eval
+uv sync --extra dev
 ```
 
-This installs the `openjarvis-eval` CLI entry point and all required dependencies (`datasets`, `huggingface-hub`, `tqdm`, `rich`).
+The framework's core dependencies (`click`, `datasets`, `rich`) are base dependencies of `openjarvis`. Two optional extras enable experiment tracking integrations:
+
+```bash
+uv sync --extra dev --extra eval-wandb     # Weights & Biases run tracking
+uv sync --extra dev --extra eval-sheets    # Google Sheets results export
+```
 
 !!! note "Python version requirement"
-    Python 3.10 requires the `tomli` package for TOML config parsing. The `evals/pyproject.toml` includes this as a conditional dependency, so it is installed automatically.
+    Python 3.10 requires the `tomli` package for TOML config parsing. `openjarvis` declares it as a conditional dependency, so it is installed automatically.
+
+## Entry Points
+
+Two equivalent entry points expose the framework:
+
+| Command | Surface |
+|---------|---------|
+| `jarvis eval {list,run,compare,report}` | Canonical CLI. `run` covers the common options; `compare` and `report` post-process result files. |
+| `python -m openjarvis.evals {list,run,run-all,summarize,reparse-judge}` | Full research surface, including judge configuration, the agentic runner, and episode mode. |
+
+The `openjarvis-eval` console script is an alias for `python -m openjarvis.evals` — same commands, same options. This guide uses `jarvis eval` wherever its option set suffices and the module form for research-only options.
 
 ---
 
 ## Datasets
 
-The framework ships with **30+ datasets** covering academic reasoning, agentic tasks, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below.
+The framework ships with **40 registered benchmarks** covering academic reasoning, agentic tasks, coding, retrieval, conversation quality, and practical use-case benchmarks. Datasets are grouped by category below; `uv run python -m openjarvis.evals list` prints the authoritative registry.
 
 ### Use-Case Benchmarks
 
@@ -64,6 +80,7 @@ These benchmarks measure reasoning and knowledge on established academic dataset
 | **MATH-500** | `math500` | reasoning | Competition-level math problems |
 | **NaturalReasoning** | `natural-reasoning` | reasoning | Natural language reasoning |
 | **HLE** | `hle` | reasoning | Humanity's Last Exam hard challenges |
+| **LiveResearchBench** | `liveresearchbench` | reasoning | Recent research comprehension (Salesforce) |
 | **SimpleQA** | `simpleqa` | chat | Short-form factual question answering |
 | **IPW** | `ipw` | chat | Intelligence Per Watt mixed benchmark |
 
@@ -79,6 +96,11 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **TerminalBench** | `terminalbench` | agentic | Terminal-based task completion |
 | **TerminalBench Native** | `terminalbench-native` | agentic | TerminalBench with native Docker execution |
 | **TerminalBench V2.1** | `terminalbench-v2.1` | agentic | TB v2.1 Harbor-style Docker tasks |
+| **PinchBench** | `pinchbench` | agentic | Real-world agent tasks |
+| **TauBench** | `taubench` | agentic | Multi-turn customer service |
+| **DeepResearchBench** | `liveresearch` | agentic | Deep research report generation |
+| **DeepResearchBench (alias)** | `deepresearch` | agentic | Same benchmark as `liveresearch` |
+| **ToolCall-15** | `toolcall15` | agentic | Tool calling benchmark |
 | **LifelongAgent** | `lifelong-agent` | agentic | Sequential task learning across sessions |
 | **PaperArena** | `paperarena` | agentic | Scientific paper analysis |
 | **DeepPlanning** | `deepplanning` | agentic | Shopping constraint planning |
@@ -87,6 +109,14 @@ These benchmarks test multi-step agent capabilities including tool use, code gen
 | **WebChoreArena** | `webchorearena` | agentic | Web chore tasks |
 | **WorkArena** | `workarena` | agentic | WorkArena++ enterprise workflows |
 
+Both `liveresearch` and `deepresearch` are registered keys for the DeepResearchBench report-generation benchmark.
+
+### Coding Benchmarks
+
+| Dataset | Key | Category | Description |
+|---------|-----|----------|-------------|
+| **LiveCodeBench** | `livecodebench` | coding | Competitive programming |
+
 ### Retrieval Benchmarks
 
 | Dataset | Key | Category | Description |
@@ -123,7 +153,7 @@ The framework includes two pre-built configs for evaluating models on the five c
 ### Cloud models
 
 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_cloud.toml
 ```
 
 This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gemini 3.1 Pro, Gemini 3.1 Flash Lite, GPT-5.4, GPT-5 Mini) against all 5 use-case benchmarks with 30 samples each, producing a 6x5 = 30-run matrix. Results are written to `results/use-cases-v2-cloud/`.
@@ -131,7 +161,7 @@ This config evaluates **6 cloud models** (Claude Opus 4.6, Claude Haiku 4.5, Gem
 ### Local models
 
 ```bash
-uv run python -m openjarvis.evals --config src/openjarvis/evals/configs/use_case_v2_local.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/use_case_v2_local.toml
 ```
 
 This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS 120B, GLM4, Qwen3.5 35B-A3B, GLM-4.7-Flash) against the same 5 benchmarks, producing a 5x5 = 25-run matrix. Uses 2 workers (suitable for single-GPU setups). Results are written to `results/use-cases-v2-local/`.
@@ -143,15 +173,22 @@ This config evaluates **5 local models** via Ollama (Qwen3.5 122B-A10B, GPT-OSS
 
 ## Inference Backends
 
-Every evaluation run routes model calls through one of two backends:
+Every evaluation run routes model calls through one of four backends:
 
 | Backend | Key | Description |
 |---------|-----|-------------|
 | **jarvis-direct** | `jarvis-direct` | Engine-level inference via `SystemBuilder`. Works for local (Ollama, vLLM, llama.cpp) and cloud models. |
 | **jarvis-agent** | `jarvis-agent` | Agent-level inference with tool calling. Uses `JarvisSystem.ask()` with the specified agent and tools. |
+| **hermes** | `hermes` | Real Hermes Agent (Nous Research) via subprocess. Requires `--base-url` and `--api-key`. |
+| **openclaw** | `openclaw` | Real OpenClaw via Node subprocess. Requires `--base-url` and `--api-key`. |
 
 Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark requires tool use — for example, GAIA tasks that reference files that must be read with `file_read`, or arithmetic tasks that benefit from `calculator`.
 
+The `hermes` and `openclaw` backends shell out to external agent frameworks and need an OpenAI-compatible endpoint for their model calls: pass `--base-url`/`--api-key`, set the `JARVIS_BACKEND_BASE_URL`/`JARVIS_BACKEND_API_KEY` environment variables, or add a `[backend.external]` section to your config (see [Config Reference](#backendexternal)).
+
+!!! note "TerminalBench Native"
+    `jarvis eval run --backend` additionally accepts `terminalbench-native`, a Docker-based execution backend used by the TerminalBench Native benchmark.
+
 ---
 
 ## CLI Usage
@@ -159,73 +196,106 @@ Use `jarvis-direct` for most evaluations. Use `jarvis-agent` when the benchmark
 ### List available benchmarks and backends
 
 ```bash
-openjarvis-eval list
+uv run python -m openjarvis.evals list
 ```
 
-Output:
+Abridged output (40 benchmarks, 4 backends):
 
 ```
-Benchmarks:
-  supergpqa    [reasoning  ] SuperGPQA multiple-choice
-  gaia         [agentic    ] GAIA agentic benchmark
-  frames       [rag        ] FRAMES multi-hop RAG
-  wildchat     [chat       ] WildChat conversation quality
-
-Backends:
-  jarvis-direct    Engine-level inference (local or cloud)
-  jarvis-agent     Agent-level inference with tool calling
+                         Available Benchmarks
+┌──────────────────────┬───────────┬───────────────────────────────────┐
+│ Name                 │ Category  │ Description                       │
+├──────────────────────┼───────────┼───────────────────────────────────┤
+│ supergpqa            │ reasoning │ SuperGPQA multiple-choice         │
+│ gpqa                 │ reasoning │ GPQA graduate-level MCQ           │
+│ ...                  │ ...       │ ...                               │
+│ livecodebench        │ coding    │ LiveCodeBench competitive progr.  │
+│ toolcall15           │ agentic   │ ToolCall-15 tool calling benchmark│
+└──────────────────────┴───────────┴───────────────────────────────────┘
+                         Available Backends
+┌───────────────┬──────────────────────────────────────────────────┐
+│ jarvis-direct │ Engine-level inference (local or cloud)          │
+│ jarvis-agent  │ Agent-level inference with tool calling          │
+│ hermes        │ Real Hermes Agent (Nous Research) via subprocess │
+│ openclaw      │ Real OpenClaw via Node subprocess                │
+└───────────────┴──────────────────────────────────────────────────┘
 ```
 
+`jarvis eval list` prints a similar table but currently shows a curated subset of the registry; the module form above is the authoritative listing.
+
 ### Run a single benchmark
 
 ```bash
-# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples default)
-openjarvis-eval run -b supergpqa -m qwen3:8b
+# Evaluate qwen3:8b on SuperGPQA (engine-level, 10 samples)
+uv run jarvis eval run -b supergpqa -m qwen3:8b -n 10
 
-# Evaluate GPT-4o on GAIA using the agent backend with tools
-openjarvis-eval run -b gaia -m gpt-4o --backend jarvis-agent \
+# Evaluate GPT-5 Mini on GAIA using the agent backend with tools
+uv run jarvis eval run -b gaia -m gpt-5-mini --backend jarvis-agent \
     --agent orchestrator --tools calculator,file_read -n 50
 
-# Run FRAMES with vLLM engine, write output to a file
-openjarvis-eval run -b frames -m llama3:70b -e vllm \
+# Run FRAMES with the vLLM engine, write output to a file
+uv run jarvis eval run -b frames -m llama3:70b -e vllm \
     -o results/frames_llama70b.jsonl
 
 # Run WildChat with a higher temperature for chat quality
-openjarvis-eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
+uv run jarvis eval run -b wildchat -m qwen3:8b --temperature 0.7 -n 100
 ```
 
-#### Full option reference
+#### `jarvis eval run` option reference
 
 | Option | Short | Type | Default | Description |
 |--------|-------|------|---------|-------------|
 | `--config` | `-c` | path | — | TOML config file; when provided, `-b` and `-m` are not required |
-| `--benchmark` | `-b` | choice | required* | `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `--backend` | | choice | `jarvis-direct` | `jarvis-direct` or `jarvis-agent` |
-| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-4o`) |
-| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
-| `--agent` | | str | `orchestrator` | Agent name for `jarvis-agent` backend |
-| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--benchmark` | `-b` | str | required* | Any registered benchmark key (see `... list`) |
+| `--model` | `-m` | str | required* | Model identifier (e.g., `qwen3:8b`, `gpt-5-mini`) |
 | `--max-samples` | `-n` | int | all | Limit the number of samples evaluated |
-| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
-| `--judge-model` | | str | `gpt-4o` | LLM used for judge-based scoring |
-| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--backend` | | choice | `jarvis-direct` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
+| `--base-url` | | str | — | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `--api-key` | | str | — | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
+| `--agent` | | str | — | Agent name for `jarvis-agent` backend (e.g., `orchestrator`) |
+| `--engine` | `-e` | str | auto | Engine key (`ollama`, `vllm`, `cloud`, ...) |
+| `--tools` | | str | `""` | Comma-separated tool names (e.g., `calculator,file_read`) |
+| `--telemetry/--no-telemetry` | | flag | off | Enable telemetry collection during eval |
+| `--gpu-metrics/--no-gpu-metrics` | | flag | off | Enable GPU metric polling |
 | `--seed` | | int | `42` | Random seed for dataset shuffling |
-| `--split` | | str | dataset default | Override the dataset split |
 | `--temperature` | | float | `0.0` | Generation temperature |
 | `--max-tokens` | | int | `2048` | Maximum output tokens |
+| `--model-filter` | | str | — | Filter models by name substring (multi-model configs) |
+| `--output` | `-o` | path | auto-generated | Output JSONL file path |
+| `--wandb-project` / `--wandb-entity` / `--wandb-tags` / `--wandb-group` | | str | `""` | Weights & Biases tracking (requires `eval-wandb` extra) |
+| `--sheets-id` / `--sheets-worksheet` / `--sheets-creds` | | str | `""` | Google Sheets export (requires `eval-sheets` extra) |
 | `--verbose` | `-v` | flag | off | Enable debug logging |
 
 *Required when `--config` is not provided.
 
+#### Research-only options (`python -m openjarvis.evals run`)
+
+The module CLI accepts everything above plus research-grade options that `jarvis eval run` does not expose:
+
+| Option | Short | Type | Default | Description |
+|--------|-------|------|---------|-------------|
+| `--max-workers` | `-w` | int | `4` | Parallel evaluation workers |
+| `--judge-model` | | str | `gpt-5-mini-2025-08-07` | LLM used for judge-based scoring (see `--help` for the current default) |
+| `--judge-engine` | | str | `cloud` | Engine key for the LLM judge; use `vllm` to judge locally |
+| `--split` | | str | dataset default | Override the dataset split |
+| `--compact` | | flag | off | Dense single-table output |
+| `--trace-detail` | | flag | off | Full per-step trace listing |
+| `--agentic` | | flag | off | Use `AgenticRunner` for multi-turn agent execution |
+| `--episode-mode` | | flag | off | Sequential episode processing with lifelong learning (required for `lifelong-agent` and similar benchmarks) |
+| `--concurrency` | | int | `1` | Parallel query execution (AgenticRunner only) |
+| `--query-timeout` | | float | — | Per-query wall-clock timeout in seconds (AgenticRunner only) |
+
+Note: the module CLI's `--backend` choice covers `jarvis-direct`, `jarvis-agent`, `hermes`, and `openclaw`; `terminalbench-native` as a backend is available via `jarvis eval run` and TOML configs.
+
 ### Run all benchmarks at once
 
-The `run-all` command evaluates a single model against all four benchmarks sequentially and writes results to an output directory:
+The `run-all` command (module CLI only) evaluates a single model against **every registered benchmark** sequentially and writes results to an output directory:
 
 ```bash
-openjarvis-eval run-all -m qwen3:8b
+uv run python -m openjarvis.evals run-all -m qwen3:8b
 
 # With options
-openjarvis-eval run-all -m gpt-4o -n 100 --output-dir results/gpt4o/
+uv run python -m openjarvis.evals run-all -m gpt-5-mini -n 100 --output-dir results/gpt5mini/
 ```
 
 Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The model slug replaces `/` and `:` with `-`, so `qwen3:8b` becomes `qwen3-8b`.
@@ -235,7 +305,7 @@ Output files are written as `{output_dir}/{benchmark}_{model-slug}.jsonl`. The m
 After a run, inspect a JSONL results file:
 
 ```bash
-openjarvis-eval summarize results/supergpqa_qwen3-8b.jsonl
+uv run python -m openjarvis.evals summarize results/supergpqa_qwen3-8b.jsonl
 ```
 
 Output:
@@ -251,6 +321,55 @@ Accuracy:  0.7222
 Errors:    2
 ```
 
+The module CLI also provides `reparse-judge`, which re-parses stored judge output in a results file and recovers records whose judge verdicts initially failed to parse — useful after improving the judge-output parser without re-running inference.
+
+### Compare and report
+
+`jarvis eval` adds two post-processing commands for result files:
+
+```bash
+# Side-by-side metric comparison across runs
+uv run jarvis eval compare results/supergpqa_qwen3-8b.jsonl results/supergpqa_gpt-5-mini.jsonl
+
+# Detailed report (accuracy, latency, cost, per-subject breakdown) for one run
+uv run jarvis eval report results/supergpqa_qwen3-8b.jsonl
+```
+
+---
+
+## Evaluating an Already-Running Endpoint
+
+If you already have an OpenAI-compatible server running — `jarvis serve`, vLLM, SGLang, llama.cpp's server, or a hosted endpoint — point an eval directly at it with `--base-url` and `--api-key`:
+
+```bash
+# A vLLM server is already serving Qwen/Qwen3-8B on a GPU node:
+#   vllm serve Qwen/Qwen3-8B --port 8000
+uv run jarvis eval run -b supergpqa -m Qwen/Qwen3-8B \
+    --base-url http://gpu-node:8000/v1 \
+    --api-key local-key \
+    -n 50
+```
+
+The `-m` value must match a model id the server reports at `GET /v1/models`. Both flags fall back to the `JARVIS_BACKEND_BASE_URL` and `JARVIS_BACKEND_API_KEY` environment variables, so CI jobs can set them once:
+
+```bash
+export JARVIS_BACKEND_BASE_URL=http://gpu-node:8000/v1
+export JARVIS_BACKEND_API_KEY=local-key
+uv run jarvis eval run -b gaia -m Qwen/Qwen3-8B --backend jarvis-agent -n 25
+```
+
+For the external `hermes` and `openclaw` backends these values are **required** (the foreign frameworks need an endpoint to send model calls to).
+
+!!! tip "Engine-level alternative for vLLM"
+    The vLLM engine also honors the `VLLM_HOST` environment variable (default `http://localhost:8000`):
+
+    ```bash
+    VLLM_HOST=http://gpu-node:8000 uv run python -m openjarvis.evals run \
+        -b supergpqa -m Qwen/Qwen3-8B -e vllm -n 50
+    ```
+
+    `VLLM_HOST` is process-global — if the candidate and the judge both use the `vllm` engine, they share the same endpoint. Prefer `--base-url` when you need them separate.
+
 ---
 
 ## TOML Config System
@@ -260,7 +379,7 @@ For research workflows that compare multiple models across multiple benchmarks,
 ### Running from a config
 
 ```bash
-openjarvis-eval run --config src/openjarvis/evals/configs/full-suite.toml
+uv run jarvis eval run --config src/openjarvis/evals/configs/full-suite.toml
 ```
 
 When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options are not required. All settings come from the config file. The CLI expands the matrix, prints a progress table, and writes results to the configured `output_dir`.
@@ -269,7 +388,7 @@ When `--config` is provided, the `-b`/`--benchmark` and `-m`/`--model` options a
 
 A config file has six sections: `[meta]`, `[defaults]`, `[judge]`, `[run]`, `[[models]]`, and `[[benchmarks]]`. Only `[[models]]` and `[[benchmarks]]` are required — all other sections are optional and fall back to built-in defaults.
 
-```toml title="evals/configs/full-suite.toml"
+```toml title="src/openjarvis/evals/configs/full-suite.toml"
 # Suite-level metadata (optional)
 [meta]
 name = "full-suite-v1"
@@ -353,7 +472,7 @@ For example, `temperature` is resolved as: use `[defaults].temperature` (0.0), t
 
 A config requires only one `[[models]]` and one `[[benchmarks]]` entry:
 
-```toml title="evals/configs/minimal.toml"
+```toml title="src/openjarvis/evals/configs/minimal.toml"
 [[models]]
 name = "qwen3:8b"
 
@@ -365,7 +484,7 @@ This runs SuperGPQA against qwen3:8b with all default settings. Use this as a st
 
 ### Single-run config with full options
 
-```toml title="evals/configs/single-run.toml"
+```toml title="src/openjarvis/evals/configs/single-run.toml"
 [meta]
 name = "single-run-example"
 description = "Evaluate SuperGPQA with a single model and full configuration"
@@ -425,7 +544,8 @@ Configuration for the LLM used as a judge in GAIA, FRAMES, and WildChat scoring.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `model` | str | `"gpt-4o"` | Judge model identifier |
+| `model` | str | `"gpt-5-mini-2025-08-07"` | Judge model identifier |
+| `engine` | str | `None` | Engine key for the judge (e.g., `"vllm"` to judge locally; defaults to cloud) |
 | `provider` | str | `None` | Provider override (e.g., `"openai"`) |
 | `temperature` | float | `0.0` | Judge sampling temperature |
 | `max_tokens` | int | `1024` | Maximum judge output tokens |
@@ -444,6 +564,20 @@ Execution settings that apply to the entire suite.
 | `seed` | int | `42` | Random seed for dataset shuffling |
 | `telemetry` | bool | `false` | Enable GPU telemetry capture (energy, power, utilization, throughput) |
 | `gpu_metrics` | bool | `false` | Enable GPU metric polling via `pynvml` (requires `pynvml` or `nvidia-ml-py`) |
+| `warmup_samples` | int | `0` | Untimed warmup samples before measurement |
+| `energy_vendor` | str | `""` | GPU energy vendor override |
+| `max_turns` | int | `None` | Maximum agent turns per query |
+| `wandb_project` / `wandb_entity` / `wandb_tags` / `wandb_group` | str | `""` | Weights & Biases tracking |
+| `sheets_spreadsheet_id` / `sheets_worksheet` / `sheets_credentials_path` | str | `""` / `"Results"` / `""` | Google Sheets export |
+
+### `[backend.external]`
+
+Endpoint settings for the `hermes` and `openclaw` backends. Environment variables override TOML values.
+
+| Field | Type | Default | Description |
+|-------|------|---------|-------------|
+| `base_url` | str | `None` | OpenAI-compatible endpoint URL (env: `JARVIS_BACKEND_BASE_URL`) |
+| `api_key` | str | `None` | API key for the endpoint (env: `JARVIS_BACKEND_API_KEY`) |
 
 ### `[[models]]`
 
@@ -451,7 +585,7 @@ One block per model. The `name` field is required.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-4o"`) |
+| `name` | str | required | Model identifier (e.g., `"qwen3:8b"`, `"gpt-5-mini"`) |
 | `engine` | str | `None` | Engine key to use (`"ollama"`, `"vllm"`, `"cloud"`, ...) |
 | `provider` | str | `None` | Provider override for cloud models (e.g., `"openai"`) |
 | `temperature` | float | `None` | Override `[defaults].temperature` for this model |
@@ -468,10 +602,12 @@ One block per benchmark. The `name` field is required.
 
 | Field | Type | Default | Description |
 |-------|------|---------|-------------|
-| `name` | str | required | Benchmark key: `supergpqa`, `gaia`, `frames`, or `wildchat` |
-| `backend` | str | `"jarvis-direct"` | Inference backend: `jarvis-direct` or `jarvis-agent` |
+| `name` | str | required | Any registered benchmark key (see `uv run python -m openjarvis.evals list`) |
+| `backend` | str | `"jarvis-direct"` | `jarvis-direct`, `jarvis-agent`, `hermes`, `openclaw`, or `terminalbench-native` |
 | `max_samples` | int | `None` | Limit number of samples; `None` evaluates the full dataset |
 | `split` | str | `None` | Override the default dataset split |
+| `subset` | str | `None` | Dataset subset/variant (benchmark-specific) |
+| `record_ids` | list[str] | `None` | Evaluate only these record ids |
 | `agent` | str | `None` | Agent name for `jarvis-agent` backend (e.g., `"orchestrator"`) |
 | `tools` | list[str] | `[]` | Tool names for `jarvis-agent` backend |
 | `judge_model` | str | `None` | Override `[judge].model` for this benchmark only |
@@ -647,7 +783,7 @@ The `EvalRunner` processes samples concurrently using a `ThreadPoolExecutor`. Re
 
 ```bash
 # Use more workers for faster evaluation (if the engine supports concurrent requests)
-openjarvis-eval run -b supergpqa -m qwen3:8b -w 8 -n 500
+uv run python -m openjarvis.evals run -b supergpqa -m qwen3:8b -w 8 -n 500
 ```
 
 !!! warning "Worker count and engine load"

frontend/src-tauri/src/lib.rs

@@ -1540,19 +1540,89 @@ async fn fetch_models(api_url: String) -> Result<serde_json::Value, String> {
 
 #[tauri::command]
 async fn run_jarvis_command(args: Vec<String>) -> Result<String, String> {
-    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
-    cmd_args.extend(args);
     let uv_bin = resolve_bin("uv");
-    let output = tokio::process::Command::new(&uv_bin)
-        .args(&cmd_args)
-        .output()
-        .await
-        .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
 
-    if output.status.success() {
-        Ok(String::from_utf8_lossy(&output.stdout).to_string())
-    } else {
-        Err(String::from_utf8_lossy(&output.stderr).to_string())
+    let mut cmd_args = vec!["run".to_string(), "jarvis".to_string()];
+    cmd_args.extend(args.iter().cloned());
+
+    let mut cmd = tokio::process::Command::new(&uv_bin);
+    cmd.args(&cmd_args);
+    // Run from the project root so `uv run jarvis` resolves the OpenJarvis
+    // project regardless of the app's launch cwd. In a packaged install the
+    // cwd isn't the checkout, so without this `jarvis` isn't found and the
+    // backend never starts — the UI then shows "Failed to get response"
+    // (see #531).
+    if let Some(ref root) = find_project_root() {
+        cmd.current_dir(root);
+    }
+
+    let is_serve = args.first().map(|a| a.as_str() == "serve").unwrap_or(false);
+
+    if !is_serve {
+        // Short-lived command (e.g. `stop`, `status`): wait for it and return
+        // its captured output.
+        let output = cmd
+            .output()
+            .await
+            .map_err(|e| format!("Failed to launch jarvis: {}", e))?;
+        return if output.status.success() {
+            Ok(String::from_utf8_lossy(&output.stdout).to_string())
+        } else {
+            Err(String::from_utf8_lossy(&output.stderr).to_string())
+        };
+    }
+
+    // `jarvis serve` is a long-running server that never exits. The old code
+    // used `.output()`, which waits for the process to exit and so hung this
+    // command forever — the "Start" button never resolved (#531). Spawn it
+    // detached instead, drain stderr (a full 4 KB Windows pipe can otherwise
+    // stall the child mid-startup, #309), and poll /health for readiness.
+    cmd.stdout(std::process::Stdio::null())
+        .stderr(std::process::Stdio::piped());
+    let mut child = cmd
+        .spawn()
+        .map_err(|e| format!("Failed to launch jarvis serve: {}", e))?;
+
+    let tail: StderrTail = Arc::new(Mutex::new(Vec::new()));
+    if let Some(stderr) = child.stderr.take() {
+        spawn_jarvis_stderr_drainer(stderr, tail.clone());
+    }
+
+    let client = reqwest::Client::builder()
+        .timeout(Duration::from_secs(2))
+        .build()
+        .map_err(|e| format!("Failed to build HTTP client: {}", e))?;
+    let url = format!("http://127.0.0.1:{}/health", JARVIS_PORT);
+    let deadline = tokio::time::Instant::now() + Duration::from_secs(120);
+
+    loop {
+        // Surface an early crash (bad venv, missing Rust ext, etc.) right away
+        // instead of waiting out the full readiness timeout.
+        if let Ok(Some(status)) = child.try_wait() {
+            let stderr = String::from_utf8_lossy(tail.lock().await.as_slice()).into_owned();
+            return Err(format!(
+                "jarvis serve exited (code {:?}) before becoming healthy:\n{}",
+                status.code(),
+                stderr.trim()
+            ));
+        }
+        if let Ok(resp) = client.get(&url).send().await {
+            if resp.status().is_success() {
+                // Leave the server running (the Child is detached on drop —
+                // kill_on_drop defaults to false); `stop` tears it down.
+                return Ok(format!(
+                    "jarvis serve is ready on http://127.0.0.1:{}",
+                    JARVIS_PORT
+                ));
+            }
+        }
+        if tokio::time::Instant::now() >= deadline {
+            return Err(format!(
+                "jarvis serve did not become healthy on port {} within 120s.",
+                JARVIS_PORT
+            ));
+        }
+        tokio::time::sleep(Duration::from_millis(500)).await;
     }
 }
 

frontend/src/lib/connectors-api.ts

@@ -1,5 +1,5 @@
 import { getBase } from './api';
-import type { ConnectorInfo, SyncStatus, ConnectRequest } from '../types/connectors';
+import type { ConnectorInfo, SyncStatus, ConnectRequest, ConnectResponse } from '../types/connectors';
 
 // ---------------------------------------------------------------------------
 // Connectors API
@@ -18,16 +18,47 @@ export async function getConnector(id: string): Promise<ConnectorInfo> {
   return res.json();
 }
 
-export async function connectSource(id: string, req: ConnectRequest): Promise<ConnectorInfo> {
+export async function connectSource(id: string, req: ConnectRequest): Promise<ConnectResponse> {
   const res = await fetch(`${getBase()}/v1/connectors/${encodeURIComponent(id)}/connect`, {
     method: 'POST',
     headers: { 'Content-Type': 'application/json' },
     body: JSON.stringify(req),
   });
-  if (!res.ok) throw new Error(`Failed to connect ${id}: ${res.status}`);
+  if (!res.ok) {
+    // Surface the backend's actionable detail (e.g. malformed Client ID /
+    // Secret) instead of a bare status code so the UI can render it.
+    const err = await res.json().catch(() => ({ detail: res.statusText }));
+    throw new Error(err.detail || `Failed to connect ${id}: ${res.status}`);
+  }
   return res.json();
 }
 
+/** Open the server-side OAuth consent flow in a popup and resolve once the
+ *  connector reports connected (or reject on timeout). Reused for any OAuth
+ *  connector whose /connect returned `oauth_required` (issue #512). */
+export function startServerOAuth(id: string, oauthStartPath?: string): Promise<void> {
+  const path = oauthStartPath || `/v1/connectors/${encodeURIComponent(id)}/oauth/start`;
+  window.open(`${getBase()}${path}`, '_blank', 'width=600,height=700');
+  return new Promise((resolve, reject) => {
+    const interval = setInterval(async () => {
+      try {
+        const info = await getConnector(id);
+        if (info.connected) {
+          clearInterval(interval);
+          clearTimeout(timer);
+          resolve();
+        }
+      } catch {
+        // ignore transient polling errors
+      }
+    }, 2000);
+    const timer = setTimeout(() => {
+      clearInterval(interval);
+      reject(new Error('Authorization timed out — please try again.'));
+    }, 180000);
+  });
+}
+
 export async function disconnectSource(id: string): Promise<void> {
   const res = await fetch(`${getBase()}/v1/connectors/${encodeURIComponent(id)}/disconnect`, {
     method: 'POST',

frontend/src/lib/sse.ts

@@ -1,5 +1,5 @@
 import type { ResearchEvent, SSEEvent } from '../types';
-import { getBase } from './api';
+import { getBase, authHeaders } from './api';
 
 export interface ChatRequest {
   model: string;
@@ -16,7 +16,7 @@ export async function* streamChat(
   const base = getBase();
   const response = await fetch(`${base}/v1/chat/completions`, {
     method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
+    headers: authHeaders({ 'Content-Type': 'application/json' }),
     body: JSON.stringify(request),
     signal,
   });
@@ -67,7 +67,7 @@ export async function* streamResearch(
   const base = getBase().replace(/\/v1\/?$/, '');
   const response = await fetch(`${base}/api/research`, {
     method: 'POST',
-    headers: { 'Content-Type': 'application/json' },
+    headers: authHeaders({ 'Content-Type': 'application/json' }),
     body: JSON.stringify({ query }),
     signal,
   });
@@ -106,3 +106,4 @@ export async function* streamResearch(
     reader.releaseLock();
   }
 }
+

frontend/src/pages/DataSourcesPage.tsx

@@ -24,7 +24,7 @@ import {
 import type { LucideIcon } from 'lucide-react';
 import { SOURCE_CATALOG } from '../types/connectors';
 import type { ConnectRequest } from '../types/connectors';
-import { listConnectors, connectSource, disconnectSource, getSyncStatus, triggerSync } from '../lib/connectors-api';
+import { listConnectors, connectSource, disconnectSource, getSyncStatus, triggerSync, startServerOAuth } from '../lib/connectors-api';
 import type { SyncStatus } from '../types/connectors';
 
 // ---------------------------------------------------------------------------
@@ -673,7 +673,19 @@ function DataSourcesSection() {
     setConnectStage('Connecting...');
     setConnectError('');
     try {
-      await connectSource(id, req);
+      const resp = await connectSource(id, req);
+
+      // OAuth connectors (Google Drive/Calendar/Contacts/Gmail/Tasks): pasting
+      // a Client ID / Secret only registers the app credentials. The backend
+      // returns `oauth_required` with the path to the in-process consent flow,
+      // which is the only path that actually mints an access token. Open it now
+      // and wait for the callback to flip the connector to connected. Without
+      // this the connector would stay "pending" forever — the exact #512 bug.
+      if (resp.status === 'oauth_required') {
+        setConnectStage('Opening Google sign-in...');
+        await startServerOAuth(id, resp.oauth_start);
+      }
+
       setConnectStage('Connected! Starting sync...');
 
       // Wait for connector to show as connected

frontend/src/types/connectors.ts

@@ -55,6 +55,19 @@ export interface ConnectRequest {
   password?: string;
 }
 
+/** Response from POST /v1/connectors/{id}/connect.
+ *  For OAuth connectors, pasting a Client ID / Secret pair only registers the
+ *  app credentials; the backend returns `status: "oauth_required"` plus an
+ *  `oauth_start` path the UI must open to run the browser consent flow that
+ *  actually mints an access token (see issue #512). */
+export interface ConnectResponse {
+  connector_id: string;
+  connected: boolean;
+  status: "connected" | "pending" | "oauth_required" | "disconnected";
+  oauth_start?: string;
+  sync_status?: string | null;
+}
+
 export type WizardStep = "pick" | "connect" | "ingest" | "ready";
 
 // Backward-compatible alias
@@ -257,12 +270,12 @@ export const SOURCE_CATALOG: ConnectorMeta[] = [
         urlLabel: 'Enable Drive API',
       },
       {
-        label: 'Create OAuth credentials: go to Credentials (link below) → click "+ Create Credentials" → choose "OAuth client ID" → Application type: "Desktop app" → click "Create"',
+        label: 'Create OAuth credentials: go to Credentials (link below) → click "+ Create Credentials" → choose "OAuth client ID" → Application type: "Web application". Under "Authorized redirect URIs" add this server\'s callback (e.g. http://localhost:1313/v1/connectors/gdrive/oauth/callback — match the host/port your OpenJarvis server is bound to) → click "Create".',
         url: 'https://console.cloud.google.com/apis/credentials',
         urlLabel: 'Open Credentials',
       },
       {
-        label: 'A dialog will show your Client ID and Client Secret. Copy both and paste them below. (If you miss it, click the download icon next to your OAuth client to see them again)',
+        label: 'A dialog will show your Client ID and Client Secret. Copy both and paste them below, then click Connect — a Google sign-in window opens to finish authorization. (If you miss the dialog, click the download icon next to your OAuth client to see them again.)',
       },
     ],
     inputFields: [

mkdocs.yml

@@ -193,6 +193,8 @@ nav:
       - External MCP Servers: user-guide/mcp-external-servers.md
       - Scheduler: user-guide/scheduler.md
       - Telemetry: user-guide/telemetry.md
+      - Evaluations: user-guide/evaluations.md
+      - Benchmarks: user-guide/benchmarks.md
       - Security: user-guide/security.md
       - LLM-guided spec search: user-guide/llm-guided-spec-search.md
   - Leaderboard: leaderboard.md

pyproject.toml

@@ -1,10 +1,10 @@
 [build-system]
-requires = ["hatchling"]
+requires = ["hatchling", "hatch-vcs"]
 build-backend = "hatchling.build"
 
 [project]
 name = "OpenJarvis"
-version = "1.0.2"
+dynamic = ["version"]
 description = "OpenJarvis — modular AI assistant backend with composable intelligence primitives"
 readme = "README.md"
 # Upper bound: numpy 2.2.x (pinned transitively via datasets/pandas) ships no
@@ -152,6 +152,31 @@ Issues = "https://github.com/open-jarvis/OpenJarvis/issues"
 
 [project.scripts]
 jarvis = "openjarvis.cli:main"
+openjarvis-eval = "openjarvis.evals.cli:main"
+
+# Version is derived from git tags by hatch-vcs (see #526). For source/editable
+# checkouts this yields the true `git describe` version (e.g. 1.0.3.dev109+g<sha>)
+# rather than a stale static string. CI release builds override this with
+# SETUPTOOLS_SCM_PRETEND_VERSION so the published version equals the pushed tag.
+#
+# setuptools_scm cannot bump custom `.devN` tags (only `.dev0`), so the autotag
+# `vX.Y.Z.devN` tags are deliberately EXCLUDED from version derivation here; the
+# base is taken from the latest plain release tag (vX.Y.Z) and the dev distance
+# is computed from commit count since that release.
+[tool.hatch.version]
+source = "vcs"
+
+[tool.hatch.version.raw-options]
+tag_regex = '^v(?P<version>[0-9]+\.[0-9]+\.[0-9]+)$'
+git_describe_command = [
+    "git", "describe", "--dirty", "--tags", "--long",
+    "--match", "v[0-9]*", "--exclude", "*dev*", "--exclude", "*rc*", "--exclude", "desktop-*",
+]
+# Builds without a git checkout (e.g. the `COPY src/ src/` Docker stages, which
+# never include .git) can't run `git describe`. Without a fallback that would
+# hard-fail the build. Mirror the runtime sentinel in src/openjarvis/__init__.py.
+# Such builds can inject the real version via SETUPTOOLS_SCM_PRETEND_VERSION.
+fallback_version = "0.0.0+unknown"
 
 [tool.hatch.build.targets.wheel]
 packages = ["src/openjarvis"]

src/openjarvis/cli/_screen.py

@@ -0,0 +1,79 @@
+"""Screen capture for vision input (``jarvis ask --screen``).
+
+Captures the primary monitor to a temporary PNG so it can be handed to a
+vision-capable model. On Windows this uses the built-in .NET
+``System.Drawing`` stack (no third-party dependency). Other platforms fall
+back to ``mss`` or ``Pillow`` if installed.
+"""
+
+from __future__ import annotations
+
+import os
+import subprocess
+import sys
+import tempfile
+
+# PowerShell: capture the PRIMARY monitor (more legible for a vision model
+# than a downscaled multi-monitor grab). {path} is filled in with forward
+# slashes, which .NET accepts on Windows and which avoids backslash escaping.
+_PS_CAPTURE = """
+Add-Type -AssemblyName System.Windows.Forms, System.Drawing
+$b = [System.Windows.Forms.Screen]::PrimaryScreen.Bounds
+$bmp = New-Object System.Drawing.Bitmap($b.Width, $b.Height)
+$g = [System.Drawing.Graphics]::FromImage($bmp)
+$g.CopyFromScreen($b.X, $b.Y, 0, 0, $bmp.Size)
+$bmp.Save("{path}", [System.Drawing.Imaging.ImageFormat]::Png)
+$g.Dispose(); $bmp.Dispose()
+"""
+
+
+def capture_screen_to_temp() -> str:
+    """Capture the screen to a temp PNG and return its absolute path.
+
+    Raises ``RuntimeError`` with actionable guidance if capture fails or the
+    platform has no available backend.
+    """
+    fd, path = tempfile.mkstemp(prefix="jarvis_screen_", suffix=".png")
+    os.close(fd)
+
+    if sys.platform.startswith("win"):
+        script = _PS_CAPTURE.replace("{path}", path.replace("\\", "/"))
+        proc = subprocess.run(
+            ["powershell", "-NoProfile", "-NonInteractive", "-Command", script],
+            capture_output=True,
+            text=True,
+            timeout=30,
+        )
+        if (
+            proc.returncode != 0
+            or not os.path.exists(path)
+            or not os.path.getsize(path)
+        ):
+            raise RuntimeError(
+                "screen capture failed: "
+                + (proc.stderr.strip() or "empty image written")
+            )
+        return path
+
+    # Non-Windows: optional backends.
+    try:
+        import mss  # type: ignore
+
+        with mss.mss() as sct:
+            sct.shot(mon=-1, output=path)
+        return path
+    except ImportError:
+        pass
+    try:
+        from PIL import ImageGrab  # type: ignore
+
+        ImageGrab.grab().save(path)
+        return path
+    except Exception as exc:  # noqa: BLE001
+        raise RuntimeError(
+            "screen capture on this platform needs 'mss' or 'Pillow' "
+            "(try: pip install mss)"
+        ) from exc
+
+
+__all__ = ["capture_screen_to_temp"]

src/openjarvis/cli/ask.py

@@ -2,6 +2,7 @@
 
 from __future__ import annotations
 
+import base64
 import json as json_mod
 import logging
 import sys
@@ -619,6 +620,21 @@ def _print_profile(
         "(default: ~/.openjarvis/knowledge.db)."
     ),
 )
+@click.option(
+    "-i",
+    "--image",
+    "image_paths",
+    multiple=True,
+    type=click.Path(exists=True, dir_okay=False),
+    help="Image file for a vision model (e.g. gemma3). Repeatable.",
+)
+@click.option(
+    "-S",
+    "--screen",
+    "capture_screen",
+    is_flag=True,
+    help="Capture the current screen and send it to the vision model.",
+)
 @click.option(
     "--persona",
     "persona_name",
@@ -645,6 +661,8 @@ def ask(
     research_mode: bool,
     knowledge_db: str | None,
     persona_name: str | None,
+    image_paths: tuple[str, ...] = (),
+    capture_screen: bool = False,
 ) -> None:
     """Ask Jarvis a question."""
     quiet = (ctx.obj or {}).get("quiet", False) or output_json
@@ -652,6 +670,27 @@ def ask(
     console = Console(stderr=True)
     query_text = " ".join(query)
 
+    # Vision: collect base64 images from --image files and/or --screen.
+    image_b64: list[str] = []
+    for _img_path in image_paths:
+        try:
+            with open(_img_path, "rb") as _fh:
+                image_b64.append(base64.b64encode(_fh.read()).decode("ascii"))
+        except OSError as exc:
+            console.print(f"[red]Could not read image {_img_path}: {exc}[/red]")
+            sys.exit(1)
+    if capture_screen:
+        try:
+            from openjarvis.cli._screen import capture_screen_to_temp
+
+            _shot = capture_screen_to_temp()
+            with open(_shot, "rb") as _fh:
+                image_b64.append(base64.b64encode(_fh.read()).decode("ascii"))
+            logger.debug("Captured screen to %s", _shot)
+        except Exception as exc:  # noqa: BLE001
+            console.print(f"[red]Screen capture failed:[/red] {exc}")
+            sys.exit(1)
+
     wall_start = time.monotonic() if enable_profile else None
 
     # Load config
@@ -671,11 +710,26 @@ def ask(
     # Without this fallback, `[agent].default_system_prompt` and the
     # SOUL.md / MEMORY.md / USER.md persona system are silently bypassed for
     # the most common command (`jarvis ask "..."`).
+    agent_explicitly_set = agent_name is not None
     if agent_name is None:
         configured_default = (config.agent.default_agent or "").strip()
         if configured_default:
             agent_name = configured_default
 
+    # Vision flows only through direct-to-engine mode. If an image/screenshot
+    # was supplied without an explicit --agent, route to direct mode so the
+    # picture reaches the model; if an agent was explicitly requested, say
+    # plainly that the image is being skipped rather than dropping it silently.
+    if image_b64:
+        if not agent_explicitly_set:
+            agent_name = ""
+        else:
+            console.print(
+                "[yellow]Note:[/yellow] --image/--screen only works in direct "
+                "mode; the image is ignored with --agent set. Re-run with "
+                '`--agent ""` to use vision.'
+            )
+
     # Track whether the user explicitly set --max-tokens
     user_set_max_tokens = max_tokens is not None
 
@@ -714,7 +768,13 @@ def ask(
     register_builtin_models()
 
     effective_engine_key = engine_key or config.intelligence.preferred_engine or None
-    resolved = get_engine(config, effective_engine_key)
+    # Pass the model we intend to run so engine selection can skip an engine
+    # that can't actually serve it (e.g. the cloud fallback when the local
+    # engine is down but only a non-OpenAI key is set — see #532). This is the
+    # -m flag or the configured default; when neither is set we leave it None
+    # and a model is chosen per-engine below.
+    selection_model = model_name or config.intelligence.default_model or None
+    resolved = get_engine(config, effective_engine_key, model=selection_model)
     if resolved is None:
         console.print(
             "[red bold]No inference engine available.[/red bold]\n\n"
@@ -865,6 +925,27 @@ def ask(
         return
 
     # Direct-to-engine mode (no agent)
+    # Privacy guard: a screenshot/image is sensitive, and OpenJarvis is
+    # local-first. If the active engine isn't local, warn before the image
+    # leaves the machine rather than silently uploading it to a third party.
+    _LOCAL_ENGINES = {
+        "ollama",
+        "llamacpp",
+        "vllm",
+        "sglang",
+        "exo",
+        "nexa",
+        "uzu",
+        "apple_fm",
+        "gemma_cpp",
+    }
+    if image_b64 and engine_name not in _LOCAL_ENGINES:
+        console.print(
+            f"[yellow]Privacy warning:[/yellow] sending {len(image_b64)} "
+            f"image(s) to a non-local engine ('{engine_name}'). The image will "
+            "leave this machine. Use a local engine (e.g. ollama) to keep "
+            "vision on-device."
+        )
     messages = [Message(role=Role.USER, content=query_text)]
 
     # Memory-augmented context injection
@@ -891,6 +972,15 @@ def ask(
         except Exception as exc:
             logger.debug("Failed to inject memory context: %s", exc)
 
+    # Vision: attach images to the final user message *after* any context
+    # injection (which may rebuild the list). messages_to_dicts() forwards
+    # the "images" field to Ollama's /api/chat.
+    if image_b64:
+        for _m in reversed(messages):
+            if _m.role == Role.USER:
+                _m.images = image_b64
+                break
+
     # Generate (InstrumentedEngine handles telemetry + energy recording)
     try:
         with console.status("[bold green]Generating...[/bold green]"):

src/openjarvis/cli/compose_cmd.py

@@ -332,7 +332,7 @@ def compose_bench(
         for i, rc in enumerate(run_configs, 1):
             console.print(f"\n[bold]Run {i}/{len(run_configs)}:[/bold] {rc.benchmark}")
             try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                 results_table.add_row(
                     rc.benchmark,
                     f"{summary.accuracy:.4f}",

src/openjarvis/cli/eval_cmd.py

@@ -61,6 +61,12 @@ KNOWN_BENCHMARKS = {
 KNOWN_BACKENDS = {
     "jarvis-direct": "Engine-level inference (local or cloud)",
     "jarvis-agent": "Agent-level inference with tool calling",
+    "hermes": "Real Hermes Agent (Nous Research) via subprocess",
+    "openclaw": "Real OpenClaw via Node subprocess",
+    "terminalbench-native": (
+        "TerminalBench V2.1 via terminal-bench Harness "
+        "(selected with -b terminalbench-native)"
+    ),
 }
 
 
@@ -146,7 +152,9 @@ def eval_list() -> None:
     "base_url",
     default=None,
     help=(
-        "OpenAI-compat endpoint URL for hermes/openclaw backends "
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
         "(env: JARVIS_BACKEND_BASE_URL)."
     ),
 )
@@ -154,7 +162,11 @@ def eval_list() -> None:
     "--api-key",
     "api_key",
     default=None,
-    help=("API key for the hermes/openclaw endpoint (env: JARVIS_BACKEND_API_KEY)."),
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
 @click.option(
     "--agent",
@@ -347,7 +359,7 @@ def eval_run(
                 f"{rc.benchmark} / {rc.model}"
             )
             try:
-                summary = _run_single(rc, console=console)
+                summary = _run_single(rc, console=console, suite_mode=True)
                 console.print(
                     f"  [green]{summary.accuracy:.4f}[/green] "
                     f"({summary.correct}/{summary.scored_samples})"
@@ -399,8 +411,10 @@ def eval_run(
         sheets_spreadsheet_id=sheets_spreadsheet_id,
         sheets_worksheet=sheets_worksheet,
         sheets_credentials_path=sheets_credentials_path,
-        # Spec §6.2 — for hermes/openclaw external backends. Falls back to env vars
-        # so users can also set JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
+        # OpenAI-compatible endpoint for the model under eval. Required for
+        # hermes/openclaw (Spec §6.2); honored by first-party backends too on
+        # this CLI path. Falls back to env vars so users can also set
+        # JARVIS_BACKEND_BASE_URL/JARVIS_BACKEND_API_KEY.
         base_url=base_url or os.environ.get("JARVIS_BACKEND_BASE_URL"),
         api_key=api_key or os.environ.get("JARVIS_BACKEND_API_KEY"),
     )

src/openjarvis/cli/serve.py

@@ -146,7 +146,13 @@ def serve(
         except Exception as exc:
             logger.debug("Telemetry store init failed: %s", exc)
 
-    resolved = get_engine(config, engine_key)
+    # Select with the model we'll actually serve so an engine that can't
+    # serve it (e.g. the cloud fallback without the matching provider key) is
+    # skipped rather than chosen and failing per-request later (see #532).
+    selection_model = (
+        model_name or config.server.model or config.intelligence.default_model or None
+    )
+    resolved = get_engine(config, engine_key, model=selection_model)
     if resolved is None:
         console.print(
             "[red bold]No inference engine available.[/red bold]\n\n"

src/openjarvis/connectors/gcalendar.py

@@ -20,7 +20,6 @@ from openjarvis.connectors.oauth import (
     delete_tokens,
     load_tokens,
     resolve_google_credentials,
-    run_oauth_flow,
     save_tokens,
 )
 from openjarvis.core.config import DEFAULT_CONFIG_DIR
@@ -290,12 +289,18 @@ class GCalendarConnector(BaseConnector):
         """Handle the OAuth callback.
 
         If *code* looks like a ``client_id:client_secret`` pair (containing
-        ``.apps.googleusercontent.com``), store the credentials and trigger
-        the full browser-based OAuth flow.  Otherwise treat it as a raw
-        token / auth code.
+        ``.apps.googleusercontent.com``), persist the client credentials only.
+        The browser consent + code→token exchange is owned by the in-process
+        server flow (``/v1/connectors/{id}/oauth/start`` → ``/oauth/callback``),
+        which writes the real ``access_token`` to every Google credential file.
+
+        The previous daemon-thread browser flow (its own ``localhost:8789``
+        callback server) failed silently in the bundled desktop context and is
+        intentionally removed here (issue #512).
+
+        Any other *code* is treated as a raw token / auth code.
         """
         code = code.strip()
-        # If user pastes client_id:client_secret, store and run OAuth flow
         if ":" in code and ".apps.googleusercontent.com" in code:
             client_id, client_secret = code.split(":", 1)
             save_tokens(
@@ -305,20 +310,6 @@ class GCalendarConnector(BaseConnector):
                     "client_secret": client_secret.strip(),
                 },
             )
-            import threading
-
-            def _run() -> None:
-                try:
-                    run_oauth_flow(
-                        client_id=client_id.strip(),
-                        client_secret=client_secret.strip(),
-                        scopes=GOOGLE_ALL_SCOPES,
-                        credentials_path=self._credentials_path,
-                    )
-                except Exception:  # noqa: BLE001
-                    pass
-
-            threading.Thread(target=_run, daemon=True).start()
         else:
             # Raw token or auth code
             save_tokens(self._credentials_path, {"token": code})

src/openjarvis/connectors/gcontacts.py

@@ -20,7 +20,6 @@ from openjarvis.connectors.oauth import (
     delete_tokens,
     load_tokens,
     resolve_google_credentials,
-    run_oauth_flow,
     save_tokens,
 )
 from openjarvis.core.config import DEFAULT_CONFIG_DIR
@@ -195,12 +194,18 @@ class GContactsConnector(BaseConnector):
         """Handle the OAuth callback.
 
         If *code* looks like a ``client_id:client_secret`` pair (containing
-        ``.apps.googleusercontent.com``), store the credentials and trigger
-        the full browser-based OAuth flow.  Otherwise treat it as a raw
-        token / auth code.
+        ``.apps.googleusercontent.com``), persist the client credentials only.
+        The browser consent + code→token exchange is owned by the in-process
+        server flow (``/v1/connectors/{id}/oauth/start`` → ``/oauth/callback``),
+        which writes the real ``access_token`` to every Google credential file.
+
+        The previous daemon-thread browser flow (its own ``localhost:8789``
+        callback server) failed silently in the bundled desktop context and is
+        intentionally removed here (issue #512).
+
+        Any other *code* is treated as a raw token / auth code.
         """
         code = code.strip()
-        # If user pastes client_id:client_secret, store and run OAuth flow
         if ":" in code and ".apps.googleusercontent.com" in code:
             client_id, client_secret = code.split(":", 1)
             save_tokens(
@@ -210,20 +215,6 @@ class GContactsConnector(BaseConnector):
                     "client_secret": client_secret.strip(),
                 },
             )
-            import threading
-
-            def _run() -> None:
-                try:
-                    run_oauth_flow(
-                        client_id=client_id.strip(),
-                        client_secret=client_secret.strip(),
-                        scopes=GOOGLE_ALL_SCOPES,
-                        credentials_path=self._credentials_path,
-                    )
-                except Exception:  # noqa: BLE001
-                    pass
-
-            threading.Thread(target=_run, daemon=True).start()
         else:
             # Raw token or auth code
             save_tokens(self._credentials_path, {"token": code})

src/openjarvis/connectors/gdrive.py

@@ -20,7 +20,6 @@ from openjarvis.connectors.oauth import (
     delete_tokens,
     load_tokens,
     resolve_google_credentials,
-    run_oauth_flow,
     save_tokens,
 )
 from openjarvis.core.config import DEFAULT_CONFIG_DIR
@@ -178,15 +177,25 @@ class GDriveConnector(BaseConnector):
         """Handle the OAuth callback.
 
         If *code* looks like a ``client_id:client_secret`` pair (containing
-        ``.apps.googleusercontent.com``), store the credentials and trigger
-        the full browser-based OAuth flow.  Otherwise treat it as a raw
-        token / auth code.
+        ``.apps.googleusercontent.com``), persist the client credentials only.
+        The actual browser consent + code→token exchange is owned by the
+        in-process server flow (``/v1/connectors/{id}/oauth/start`` →
+        ``/oauth/callback``), which writes the real ``access_token`` to every
+        Google credential file.
+
+        Previously this spawned a daemon thread that popped a browser and ran
+        its own ``localhost:8789`` callback server; that thread failed silently
+        in the bundled desktop context, so the connector never gained an access
+        token and never appeared in Data Sources (issue #512). The background
+        flow is intentionally removed here.
+
+        Any other *code* is treated as a raw token / auth code.
         """
         code = code.strip()
-        # If user pastes client_id:client_secret, store and run OAuth flow
+        # A pasted client_id:client_secret pair is the app registration, not a
+        # completed credential — persist it and let the server flow finish auth.
         if ":" in code and ".apps.googleusercontent.com" in code:
             client_id, client_secret = code.split(":", 1)
-            # Save credentials immediately
             save_tokens(
                 self._credentials_path,
                 {
@@ -194,21 +203,6 @@ class GDriveConnector(BaseConnector):
                     "client_secret": client_secret.strip(),
                 },
             )
-            # Run OAuth flow in background thread to avoid blocking
-            import threading
-
-            def _run() -> None:
-                try:
-                    run_oauth_flow(
-                        client_id=client_id.strip(),
-                        client_secret=client_secret.strip(),
-                        scopes=GOOGLE_ALL_SCOPES,
-                        credentials_path=self._credentials_path,
-                    )
-                except Exception:  # noqa: BLE001
-                    pass
-
-            threading.Thread(target=_run, daemon=True).start()
         else:
             # Raw token or auth code
             save_tokens(self._credentials_path, {"token": code})

src/openjarvis/core/config.py

@@ -946,8 +946,10 @@ class AgentConfig:
     system_prompt_path: str = ""  # path to system prompt file (.txt, .md)
     context_from_memory: bool = True  # inject relevant memory context into prompts
     default_system_prompt: str = (
-        "You are a helpful AI assistant running locally on the user's own "
-        "hardware through OpenJarvis. You are not a cloud service. Respond "
+        "You are OpenJarvis, a helpful AI assistant running locally on the "
+        "user's own hardware. You are not a cloud service, and you are not "
+        "Claude, ChatGPT, Gemini, or any other branded assistant. If asked "
+        "who or what you are, identify yourself as OpenJarvis. Respond "
         "helpfully, concisely, and accurately."
     )
 

src/openjarvis/core/types.py

@@ -68,6 +68,10 @@ class Message:
     tool_calls: Optional[List[ToolCall]] = None
     tool_call_id: Optional[str] = None
     metadata: Dict[str, Any] = field(default_factory=dict)
+    # Base64-encoded image data for vision-capable models (e.g. gemma3,
+    # qwen2.5-vl). Forwarded to Ollama's /api/chat "images" field; None or
+    # empty for text-only messages (the common case).
+    images: Optional[List[str]] = None
 
 
 @dataclass(slots=True)

src/openjarvis/engine/_base.py

@@ -34,6 +34,10 @@ def messages_to_dicts(messages: Sequence[Message]) -> List[Dict[str, Any]]:
             ]
         if m.tool_call_id:
             d["tool_call_id"] = m.tool_call_id
+        # Vision: forward base64 images to the engine. Ollama's /api/chat
+        # accepts an "images" array on a message; text messages skip this.
+        if getattr(m, "images", None):
+            d["images"] = list(m.images)
         out.append(d)
     return out
 

src/openjarvis/engine/_discovery.py

@@ -156,12 +156,26 @@ def discover_models(
 
 
 def get_engine(
-    config: JarvisConfig, engine_key: str | None = None
+    config: JarvisConfig,
+    engine_key: str | None = None,
+    model: str | None = None,
 ) -> Tuple[str, InferenceEngine] | None:
     """Get a specific engine by key, or the default with fallback.
 
+    When *model* is given, an engine is selected only if it can actually
+    serve that model (``engine.can_serve(model)``). This stops the cloud
+    fallback from being chosen — when the local engine is down — for a model
+    whose provider client is missing, which otherwise surfaces as a confusing
+    "OpenAI client not available" instead of a helpful "start your local
+    engine" message (see #532). When *model* is ``None`` selection stays
+    model-agnostic (unchanged behaviour).
+
     Returns ``(key, engine_instance)`` or ``None`` if no engine is available.
     """
+
+    def _usable(engine: InferenceEngine) -> bool:
+        return engine.health() and (model is None or engine.can_serve(model))
+
     # Build an ordered list of keys to try, then fall back to full discovery.
     keys_to_try: list[str] = []
     if engine_key:
@@ -176,14 +190,16 @@ def get_engine(
             continue
         try:
             engine = _make_engine(key, config)
-            if engine.health():
+            if _usable(engine):
                 return (key, engine)
         except Exception as exc:
             logger.debug("Engine %r health check failed: %s", key, exc)
 
-    # Fallback to any healthy engine
-    healthy = discover_engines(config)
-    return healthy[0] if healthy else None
+    # Fallback to the first healthy engine that can serve the model.
+    for key, engine in discover_engines(config):
+        if model is None or engine.can_serve(model):
+            return (key, engine)
+    return None
 
 
 __all__ = ["discover_engines", "discover_models", "get_engine"]

src/openjarvis/engine/_openai_compat.py

@@ -28,12 +28,31 @@ class _OpenAICompatibleEngine(InferenceEngine):
     _default_host: str = "http://localhost:8000"
     _api_prefix: str = "/v1"
 
-    def __init__(self, host: str | None = None, *, timeout: float = 600.0) -> None:
+    def __init__(
+        self,
+        host: str | None = None,
+        *,
+        api_key: str | None = None,
+        timeout: float = 600.0,
+    ) -> None:
         import os
 
-        env_key = f"{self.engine_id.upper()}_HOST"
-        self._host = (host or os.environ.get(env_key) or self._default_host).rstrip("/")
-        self._client = httpx.Client(base_url=self._host, timeout=timeout)
+        # Sanitize the engine id for env-var lookup ("openai-compat" ->
+        # "OPENAI_COMPAT_..."); shells cannot set hyphenated variable names.
+        env_prefix = self.engine_id.upper().replace("-", "_")
+        self._host = (
+            host or os.environ.get(f"{env_prefix}_HOST") or self._default_host
+        ).rstrip("/")
+        # Bearer auth for endpoints started with e.g. ``vllm serve --api-key``.
+        # Setting it on the client covers generate/stream/stream_full/
+        # list_models/health alike; ``None`` keeps requests header-free.
+        self._api_key = api_key or os.environ.get(f"{env_prefix}_API_KEY") or None
+        headers = (
+            {"Authorization": f"Bearer {self._api_key}"} if self._api_key else None
+        )
+        self._client = httpx.Client(
+            base_url=self._host, timeout=timeout, headers=headers
+        )
 
     # -- InferenceEngine interface ------------------------------------------
 

src/openjarvis/engine/_stubs.py

@@ -119,6 +119,17 @@ class InferenceEngine(ABC):
     def health(self) -> bool:
         """Return ``True`` when the engine is reachable and healthy."""
 
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` if this engine can serve *model*.
+
+        Defaults to ``True``: local engines accept any model id (whether a
+        specific model is *installed* is a separate concern from engine
+        selection). Engines that multiplex provider-specific clients (e.g.
+        the cloud engine) override this so selection can skip an engine whose
+        client for the model's provider isn't configured (see #532).
+        """
+        return True
+
     def close(self) -> None:
         """Release resources (HTTP clients, connections, threads, etc.)."""
 

src/openjarvis/engine/cloud.py

@@ -1,4 +1,7 @@
-"""Cloud inference engine — OpenAI, Anthropic, Google, and MiniMax API backends."""
+"""Cloud inference engine.
+
+OpenAI, Anthropic, Google, MiniMax, and DeepSeek API backends.
+"""
 
 from __future__ import annotations
 
@@ -48,6 +51,8 @@ PRICING: Dict[str, tuple[float, float]] = {
     "MiniMax-M2.7-highspeed": (0.60, 2.40),
     "MiniMax-M2.5": (0.30, 1.20),
     "MiniMax-M2.5-highspeed": (0.60, 2.40),
+    "deepseek-v4-flash": (0.27, 1.10),
+    "deepseek-v4-pro": (0.55, 2.19),
 }
 
 # Well-known model IDs per provider
@@ -83,6 +88,10 @@ _MINIMAX_MODELS = [
     "MiniMax-M2.5",
     "MiniMax-M2.5-highspeed",
 ]
+_DEEPSEEK_MODELS = [
+    "deepseek-v4-flash",
+    "deepseek-v4-pro",
+]
 
 # OpenRouter models — prefixed with "openrouter/" so they can be identified
 _OPENROUTER_POPULAR = [
@@ -111,6 +120,10 @@ def _is_minimax_model(model: str) -> bool:
     return model.lower().startswith("minimax")
 
 
+def _is_deepseek_model(model: str) -> bool:
+    return model.lower().startswith("deepseek")
+
+
 def _is_openrouter_model(model: str) -> bool:
     return model.startswith("openrouter/")
 
@@ -127,6 +140,35 @@ def _is_google_model(model: str) -> bool:
     return "gemini" in model.lower() and not _is_openrouter_model(model)
 
 
+# Positive prefix predicate for genuine OpenAI models. Kept in sync with
+# ``server/cloud_router.py:_OPENAI_PREFIXES`` so local-vs-cloud classification
+# agrees across the codebase. Used by ``_client_for_model``/``can_serve`` so the
+# cloud engine never claims it can serve an unrecognized (e.g. local Ollama)
+# model name just because an OpenAI key happens to be present (see #335).
+_OPENAI_PREFIXES = ("gpt-", "chatgpt-", "o1", "o3", "o4")
+
+
+def _is_openai_model(model: str) -> bool:
+    """True only for genuine OpenAI models (gpt-*, chatgpt-*, o1/o3/o4 series).
+
+    Defined positively so that an unrecognized model name (a local Ollama model
+    like ``qwen3.5:0.8b``, or a typo) is NOT treated as an OpenAI model. This is
+    the routing surface ``can_serve`` relies on; ``generate``/``stream`` keep
+    their OpenAI fall-through so an explicitly-requested unknown cloud model
+    still errors loudly at call time.
+
+    Caveat: a user may repoint the OpenAI client at an OpenAI-compatible server
+    (vLLM/LM Studio) via ``OPENAI_BASE_URL`` and legitimately serve non-gpt
+    names. That path is undocumented/untested in this engine; if it is added,
+    this predicate (or ``_client_for_model``) should treat a configured custom
+    base_url as "serves anything".
+    """
+    m = model.lower()
+    if m in (name.lower() for name in _OPENAI_MODELS):
+        return True
+    return m.startswith(_OPENAI_PREFIXES)
+
+
 def _is_openai_reasoning_model(model: str) -> bool:
     """Check if model is an OpenAI reasoning model that restricts temperature."""
     m = model.lower()
@@ -269,7 +311,7 @@ def _convert_tools_to_google(
 
 @EngineRegistry.register("cloud")
 class CloudEngine(InferenceEngine):
-    """Cloud inference via OpenAI, Anthropic, Google, and MiniMax SDKs."""
+    """Cloud inference via OpenAI, Anthropic, Google, MiniMax, and DeepSeek SDKs."""
 
     engine_id = "cloud"
     is_cloud = True
@@ -280,6 +322,7 @@ class CloudEngine(InferenceEngine):
         self._google_client: Any = None
         self._openrouter_client: Any = None
         self._minimax_client: Any = None
+        self._deepseek_client: Any = None
         self._codex_client: Any = None
         # Gemini thought_signatures: tool_call_id -> signature bytes
         self._thought_sigs: Dict[str, bytes] = {}
@@ -332,6 +375,17 @@ class CloudEngine(InferenceEngine):
                 )
             except ImportError:
                 pass
+        deepseek_key = os.environ.get("DEEPSEEK_API_KEY")
+        if deepseek_key:
+            try:
+                import openai
+
+                self._deepseek_client = openai.OpenAI(
+                    base_url="https://api.deepseek.com/v1",
+                    api_key=deepseek_key,
+                )
+            except ImportError:
+                pass
         # Codex — uses the OpenAI Responses API.
         # Supports both standard API keys (api.openai.com) and ChatGPT
         # OAuth tokens (chatgpt.com) via OPENAI_CODEX_BASE_URL override.
@@ -985,6 +1039,56 @@ class CloudEngine(InferenceEngine):
             ]
         return result
 
+    def _generate_deepseek(
+        self,
+        messages: Sequence[Message],
+        *,
+        model: str,
+        temperature: float,
+        max_tokens: int,
+        **kwargs: Any,
+    ) -> Dict[str, Any]:
+        if self._deepseek_client is None:
+            raise EngineConnectionError(
+                "DeepSeek client not available — set DEEPSEEK_API_KEY"
+            )
+        kwargs.pop("response_format", None)
+        create_kwargs: Dict[str, Any] = {
+            "model": model,
+            "messages": messages_to_dicts(messages),
+            "max_tokens": max_tokens,
+            "temperature": temperature,
+        }
+        t0 = time.monotonic()
+        resp = self._deepseek_client.chat.completions.create(**create_kwargs)
+        elapsed = time.monotonic() - t0
+        choice = resp.choices[0]
+        usage = resp.usage
+        prompt_tokens = usage.prompt_tokens if usage else 0
+        completion_tokens = usage.completion_tokens if usage else 0
+        result: Dict[str, Any] = {
+            "content": choice.message.content or "",
+            "usage": {
+                "prompt_tokens": prompt_tokens,
+                "completion_tokens": completion_tokens,
+                "total_tokens": (usage.total_tokens if usage else 0),
+            },
+            "model": resp.model,
+            "finish_reason": choice.finish_reason or "stop",
+            "cost_usd": estimate_cost(model, prompt_tokens, completion_tokens),
+            "ttft": elapsed,
+        }
+        if hasattr(choice.message, "tool_calls") and choice.message.tool_calls:
+            result["tool_calls"] = [
+                {
+                    "id": tc.id,
+                    "name": tc.function.name,
+                    "arguments": tc.function.arguments,
+                }
+                for tc in choice.message.tool_calls
+            ]
+        return result
+
     def generate(
         self,
         messages: Sequence[Message],
@@ -1006,6 +1110,8 @@ class CloudEngine(InferenceEngine):
             return self._generate_openrouter(messages, **kw)
         if _is_minimax_model(model):
             return self._generate_minimax(messages, **kw)
+        if _is_deepseek_model(model):
+            return self._generate_deepseek(messages, **kw)
         if _is_anthropic_model(model):
             return self._generate_anthropic(messages, **kw)
         if _is_google_model(model):
@@ -1036,6 +1142,9 @@ class CloudEngine(InferenceEngine):
         elif _is_minimax_model(model):
             async for token in self._stream_minimax(messages, **kw):
                 yield token
+        elif _is_deepseek_model(model):
+            async for token in self._stream_deepseek(messages, **kw):
+                yield token
         elif _is_anthropic_model(model):
             async for token in self._stream_anthropic(messages, **kw):
                 yield token
@@ -1254,6 +1363,30 @@ class CloudEngine(InferenceEngine):
             if delta and delta.content:
                 yield delta.content
 
+    async def _stream_deepseek(
+        self,
+        messages: Sequence[Message],
+        *,
+        model: str,
+        temperature: float,
+        max_tokens: int,
+        **kwargs: Any,
+    ) -> AsyncIterator[str]:
+        if self._deepseek_client is None:
+            raise EngineConnectionError("DeepSeek client not available")
+        create_kwargs: Dict[str, Any] = {
+            "model": model,
+            "messages": messages_to_dicts(messages),
+            "max_tokens": max_tokens,
+            "temperature": temperature,
+            "stream": True,
+        }
+        resp = self._deepseek_client.chat.completions.create(**create_kwargs)
+        for chunk in resp:
+            delta = chunk.choices[0].delta if chunk.choices else None
+            if delta and delta.content:
+                yield delta.content
+
     # -- stream_full: rich streaming with tool_calls support ----------------
 
     async def _stream_full_openai(
@@ -1307,6 +1440,18 @@ class CloudEngine(InferenceEngine):
                 "stream": True,
                 **kwargs,
             }
+        elif _is_deepseek_model(model):
+            client = self._deepseek_client
+            if client is None:
+                raise EngineConnectionError("DeepSeek client not available")
+            create_kwargs = {
+                "model": model,
+                "messages": messages_to_dicts(messages),
+                "max_tokens": max_tokens,
+                "temperature": temperature,
+                "stream": True,
+                **kwargs,
+            }
         else:
             client = self._openai_client
             if client is None:
@@ -1473,10 +1618,54 @@ class CloudEngine(InferenceEngine):
             models.extend(_OPENROUTER_POPULAR)
         if self._minimax_client is not None:
             models.extend(_MINIMAX_MODELS)
+        if self._deepseek_client is not None:
+            models.extend(_DEEPSEEK_MODELS)
         if self._codex_client is not None:
             models.extend(_CODEX_MODELS)
         return models
 
+    def _client_for_model(self, model: str) -> Any:
+        """Return the provider client ``generate``/``stream`` will dispatch to
+        for *model*, or ``None`` for a model this engine cannot route.
+
+        Mirrors the routing in ``generate``/``stream``, but is intentionally
+        *stricter* on the OpenAI fall-through: only genuine OpenAI models map to
+        the OpenAI client. Unrecognized names (e.g. a local Ollama model like
+        ``qwen3.5:0.8b``) return ``None`` so ``can_serve`` declines them and the
+        cloud engine is not mis-selected as a fallback when the local engine is
+        transiently down and any (even dummy) ``OPENAI_API_KEY`` is set (#335).
+        ``generate``/``stream`` keep their OpenAI fall-through, so an
+        explicitly-requested unknown cloud model still fails loudly at call time.
+        """
+        if _is_codex_model(model):
+            return self._codex_client
+        if _is_openrouter_model(model):
+            return self._openrouter_client
+        if _is_minimax_model(model):
+            return self._minimax_client
+        if _is_deepseek_model(model):
+            return self._deepseek_client
+        if _is_anthropic_model(model):
+            return self._anthropic_client
+        if _is_google_model(model):
+            return self._google_client
+        if _is_openai_model(model):
+            return self._openai_client
+        return None
+
+    def can_serve(self, model: str) -> bool:
+        """Return ``True`` only if the provider client for *model* exists.
+
+        ``health()`` is ``True`` whenever *any* provider client is configured,
+        but a request for, say, a ``gpt-*`` model still needs the OpenAI
+        client specifically. Without this check the cloud engine gets picked
+        as a fallback (when the local engine is down) for a model it can't
+        serve, then dies at call time with "<provider> client not available"
+        instead of the user getting a helpful "start your local engine"
+        message (see #532).
+        """
+        return self._client_for_model(model) is not None
+
     def health(self) -> bool:
         return (
             self._openai_client is not None
@@ -1484,6 +1673,7 @@ class CloudEngine(InferenceEngine):
             or self._google_client is not None
             or self._openrouter_client is not None
             or self._minimax_client is not None
+            or self._deepseek_client is not None
             or self._codex_client is not None
         )
 

src/openjarvis/engine/ollama.py

@@ -23,6 +23,19 @@ from openjarvis.engine._stubs import StreamChunk
 logger = logging.getLogger(__name__)
 
 
+def _default_num_ctx() -> int:
+    """Default context window (tokens). Override with ``JARVIS_NUM_CTX``.
+
+    Raised above Ollama's 4k default so an image (which costs many tokens)
+    plus a real conversation fit. 16k is comfortable for small models on a
+    typical consumer GPU.
+    """
+    try:
+        return int(os.environ.get("JARVIS_NUM_CTX", "16384"))
+    except ValueError:
+        return 16384
+
+
 @EngineRegistry.register("ollama")
 class OllamaEngine(InferenceEngine):
     """Ollama backend via its native HTTP API."""
@@ -73,7 +86,7 @@ class OllamaEngine(InferenceEngine):
             "options": {
                 "temperature": temperature,
                 "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
             },
         }
         # Disable extended thinking by default (Qwen3.5 etc.).
@@ -189,7 +202,7 @@ class OllamaEngine(InferenceEngine):
             "options": {
                 "temperature": temperature,
                 "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
             },
         }
         # Mirror generate()'s default: disable extended thinking unless the
@@ -268,7 +281,7 @@ class OllamaEngine(InferenceEngine):
             "options": {
                 "temperature": temperature,
                 "num_predict": max_tokens,
-                "num_ctx": kwargs.get("num_ctx", 8192),
+                "num_ctx": kwargs.get("num_ctx", _default_num_ctx()),
             },
         }
         if "think" not in kwargs:

src/openjarvis/engine/openai_compat_engines.py

@@ -1,5 +1,7 @@
 """Data-driven registration of OpenAI-compatible inference engines."""
 
+from __future__ import annotations
+
 from openjarvis.core.registry import EngineRegistry
 from openjarvis.engine._openai_compat import _OpenAICompatibleEngine
 
@@ -25,4 +27,35 @@ for _key, (_cls_name, _default_host, _api_prefix) in _ENGINES.items():
     EngineRegistry.register(_key)(_cls)
     globals()[_cls_name] = _cls
 
-__all__ = [name for name, _, _ in _ENGINES.values()]
+
+def normalize_openai_base_url(url: str) -> str:
+    """Strip a single trailing ``/v1`` segment from a user-supplied base URL.
+
+    Users habitually pass ``http://host:8000/v1`` (the full OpenAI-compatible
+    prefix); the engine's ``_api_prefix`` re-appends ``/v1`` to every request
+    path, so a trailing copy would double up as ``/v1/v1``. Only a literal
+    trailing ``/v1`` is stripped — proxy/gateway path prefixes are preserved.
+    """
+    base = url.rstrip("/")
+    if base.endswith("/v1"):
+        base = base[: -len("/v1")]
+    return base
+
+
+class OpenAICompatEngine(_OpenAICompatibleEngine):
+    """Generic engine for an explicitly-provided OpenAI-compatible endpoint.
+
+    Deliberately NOT registered in ``EngineRegistry``: it is only ever
+    constructed with an explicit host (e.g. ``jarvis eval --base-url``), so
+    registering it would just add a useless localhost discovery probe and
+    interact with the per-test registry wipe.
+    """
+
+    engine_id = "openai-compat"
+    _api_prefix = "/v1"
+
+
+__all__ = [name for name, _, _ in _ENGINES.values()] + [
+    "OpenAICompatEngine",
+    "normalize_openai_base_url",
+]

src/openjarvis/evals/backends/_endpoint_util.py

@@ -0,0 +1,52 @@
+"""Shared helper for targeting an explicit OpenAI-compatible endpoint.
+
+Used by the first-party eval backends (jarvis-direct, jarvis-agent) when
+``--base-url`` is given: the eval must use exactly that endpoint, with no
+silent fallback to whatever other engine discovery happens to find.
+"""
+
+from __future__ import annotations
+
+import logging
+from typing import Optional
+
+logger = logging.getLogger(__name__)
+
+
+def build_endpoint_engine(
+    base_url: str,
+    api_key: Optional[str] = None,
+    engine_key: Optional[str] = None,
+):
+    """Construct an :class:`OpenAICompatEngine` pinned to ``base_url``.
+
+    Pre-flight health-checks the endpoint and raises a loud, actionable
+    error when it is unreachable — engine discovery is never consulted.
+    """
+    from openjarvis.engine.openai_compat_engines import (
+        OpenAICompatEngine,
+        normalize_openai_base_url,
+    )
+
+    if engine_key:
+        logger.warning(
+            "Both an engine key (%r) and base_url (%r) were given; "
+            "base_url wins — targeting the endpoint directly.",
+            engine_key,
+            base_url,
+        )
+    host = normalize_openai_base_url(base_url)
+    engine = OpenAICompatEngine(host=host, api_key=api_key)
+    if not engine.health():
+        engine.close()
+        raise RuntimeError(
+            f"--base-url endpoint not reachable: {base_url} "
+            f"(GET {host}/v1/models failed). Is an OpenAI-compatible server "
+            "(e.g. `vllm serve`) running at that address? If it requires "
+            "authentication (HTTP 401), pass --api-key or set "
+            "JARVIS_BACKEND_API_KEY."
+        )
+    return engine
+
+
+__all__ = ["build_endpoint_engine"]

src/openjarvis/evals/backends/jarvis_agent.py

@@ -31,6 +31,8 @@ class JarvisAgentBackend(InferenceBackend):
         max_turns: Optional[int] = None,
         skills_enabled: bool = True,
         overlay_dir: Optional[Path] = None,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
     ) -> None:
         from openjarvis.system import SystemBuilder
 
@@ -40,7 +42,17 @@ class JarvisAgentBackend(InferenceBackend):
         self._gpu_metrics = gpu_metrics
 
         builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
             builder.engine(engine_key)
         if model:
             builder.model(model)

src/openjarvis/evals/backends/jarvis_direct.py

@@ -24,6 +24,8 @@ class JarvisDirectBackend(InferenceBackend):
         engine_key: Optional[str] = None,
         telemetry: bool = False,
         gpu_metrics: bool = False,
+        base_url: Optional[str] = None,
+        api_key: Optional[str] = None,
     ) -> None:
         from openjarvis.system import SystemBuilder
 
@@ -31,7 +33,17 @@ class JarvisDirectBackend(InferenceBackend):
         self._gpu_metrics = gpu_metrics
 
         builder = SystemBuilder()
-        if engine_key:
+        if base_url:
+            # Explicit endpoint targeting (--base-url): pin the eval to
+            # exactly this OpenAI-compatible endpoint. Fails fast if it is
+            # unreachable; never falls back to a discovered engine.
+            from openjarvis.evals.backends._endpoint_util import (
+                build_endpoint_engine,
+            )
+
+            engine = build_endpoint_engine(base_url, api_key, engine_key)
+            builder.engine_instance(engine, key=engine_key or "openai-compat")
+        elif engine_key:
             builder.engine(engine_key)
         # Propagate gpu_metrics to the runtime config so SystemBuilder
         # creates an EnergyMonitor / GpuMonitor for the InstrumentedEngine.

src/openjarvis/evals/backends/terminalbench_native.py

@@ -5,11 +5,13 @@ Uses Harness for Docker-based execution and scoring.
 
 from __future__ import annotations
 
+import inspect
 import logging
 from pathlib import Path
-from typing import Any, Dict, Optional
+from typing import Any, Dict, List, Optional, Tuple
 
 from openjarvis.evals.core.backend import InferenceBackend
+from openjarvis.evals.core.types import RunSummary
 
 LOGGER = logging.getLogger(__name__)
 
@@ -20,6 +22,97 @@ try:
 except ImportError:
     _HAS_TB = False
 
+# terminal-bench FailureMode values that are definitionally infrastructure
+# failures (the harness broke before/while driving the agent), never a
+# judgment on the model's answer. NOTE: clean trials leave failure_mode
+# "unset" in terminal-bench 0.2.18 — both on success AND on genuine
+# unresolved misses — so failure_mode alone can NOT be used to detect
+# harness errors (it would misflag every real model miss).
+_INFRA_FAILURE_MODES = frozenset({"agent_installation_failed", "unknown_agent_error"})
+
+# Harness kwargs that older terminal-bench versions may not support.
+_TIMEOUT_KWARGS = ("global_agent_timeout_sec", "global_timeout_multiplier")
+
+
+def summarize_benchmark_results(
+    results: Any,
+    *,
+    model: str,
+    benchmark: str = "terminalbench-native",
+) -> Tuple[RunSummary, List[Dict[str, str]]]:
+    """Convert terminal-bench ``BenchmarkResults`` into a ``RunSummary``.
+
+    Trials are classified into three buckets:
+
+    - resolved: ``is_resolved`` is True -> counted correct.
+    - model miss: unresolved, but the model was actually contacted ->
+      counted in the accuracy denominator.
+    - harness/infra failure: excluded from the accuracy denominator and
+      reported in ``RunSummary.errors`` plus the returned failure list.
+
+    Zero-model-contact signal choice: terminal-bench 0.2.18 leaves
+    ``failure_mode`` UNSET both on clean success and on genuine unresolved
+    misses, so failure_mode cannot distinguish "the model tried and failed"
+    from "the agent never called the model". Token usage can: this backend
+    always runs terminus-2, which reports real LiteLLM usage, so an
+    unresolved trial with zero/missing input+output tokens means no model
+    request ever completed — an infrastructure failure (in-container setup
+    hang/death, tmux failure), not a model miss. CAVEAT: terminal-bench
+    "installed agents" (openhands, claude-code, ...) hardcode 0 tokens even
+    on success; if this backend ever honors ``agent_name`` for installed
+    agents, this heuristic must be gated on the agent type.
+    """
+    trials = list(getattr(results, "results", None) or [])
+
+    harness_failures: List[Dict[str, str]] = []
+    scored = 0
+    correct = 0
+
+    for tr in trials:
+        task_id = getattr(tr, "task_id", None) or getattr(tr, "trial_name", "unknown")
+        is_resolved = getattr(tr, "is_resolved", None) is True
+        fm = getattr(tr, "failure_mode", None)
+        fm_value = str(getattr(fm, "value", fm) or "unset").lower()
+        tokens = (getattr(tr, "total_input_tokens", None) or 0) + (
+            getattr(tr, "total_output_tokens", None) or 0
+        )
+
+        zero_model_contact = not is_resolved and tokens == 0
+        infra_failure_mode = fm_value in _INFRA_FAILURE_MODES
+
+        if zero_model_contact or infra_failure_mode:
+            harness_failures.append(
+                {
+                    "task_id": str(task_id),
+                    "failure_mode": fm_value,
+                    "reason": (
+                        "zero_model_requests" if zero_model_contact else fm_value
+                    ),
+                }
+            )
+            continue
+
+        scored += 1
+        if is_resolved:
+            correct += 1
+
+    return (
+        RunSummary(
+            benchmark=benchmark,
+            category="agentic",
+            backend="terminalbench-native",
+            model=model,
+            total_samples=len(trials),
+            scored_samples=scored,
+            correct=correct,
+            accuracy=correct / scored if scored else 0.0,
+            errors=len(harness_failures),
+            mean_latency_seconds=0.0,
+            total_cost_usd=0.0,
+        ),
+        harness_failures,
+    )
+
 
 class TerminalBenchNativeBackend(InferenceBackend):
     """Runs terminal-bench tasks natively via Harness with Docker execution.
@@ -44,7 +137,22 @@ class TerminalBenchNativeBackend(InferenceBackend):
         system_prompt: str = "",
         max_tokens: int = 16384,
         n_concurrent: int = 4,
+        global_agent_timeout_sec: Optional[float] = 1800.0,
+        global_timeout_multiplier: Optional[float] = None,
     ) -> None:
+        """Args of note:
+
+        global_agent_timeout_sec: Hard wall-clock bound for each trial's
+            agent phase. terminal-bench runs installed-agent SETUP inside
+            this same budget with an infinite tmux timeout, so this bounds
+            SETUP+RUN together (a setup-only timeout needs an upstream
+            terminal-bench change). When set, it REPLACES each task's own
+            ``max_agent_timeout_sec``. Set ``None`` or ``0`` to fall back
+            to per-task budgets.
+        global_timeout_multiplier: Scales per-task budgets when
+            ``global_agent_timeout_sec`` is not set. ``None`` keeps
+            terminal-bench's default (1.0).
+        """
         if not _HAS_TB:
             raise ImportError("terminal-bench is required: pip install terminal-bench")
 
@@ -59,6 +167,8 @@ class TerminalBenchNativeBackend(InferenceBackend):
         self._system_prompt = system_prompt
         self._max_tokens = max_tokens
         self._n_concurrent = n_concurrent
+        self._global_agent_timeout_sec = global_agent_timeout_sec
+        self._global_timeout_multiplier = global_timeout_multiplier
         self._results: Optional[BenchmarkResults] = None
 
     def run_harness(self, run_id: str) -> BenchmarkResults:
@@ -91,10 +201,53 @@ class TerminalBenchNativeBackend(InferenceBackend):
         if self._max_samples is not None:
             harness_kwargs["n_tasks"] = self._max_samples
 
+        # Bound each trial's agent phase. Without this, an in-container
+        # installed-agent SETUP hang runs with an infinite tmux timeout,
+        # bounded only by whatever budget the task happens to declare.
+        if self._global_agent_timeout_sec:
+            harness_kwargs["global_agent_timeout_sec"] = float(
+                self._global_agent_timeout_sec
+            )
+        if self._global_timeout_multiplier is not None:
+            harness_kwargs["global_timeout_multiplier"] = float(
+                self._global_timeout_multiplier
+            )
+
+        self._check_timeout_kwargs_supported(harness_kwargs)
+
         harness = Harness(**harness_kwargs)
         self._results = harness.run()
         return self._results
 
+    @staticmethod
+    def _check_timeout_kwargs_supported(harness_kwargs: Dict[str, Any]) -> None:
+        """Fail loudly if this terminal-bench build lacks the timeout kwargs.
+
+        terminal-bench is an undeclared, unpinned dependency, so installs may
+        predate the global timeout kwargs (added by 0.2.x). Passing an
+        unknown kwarg raises an opaque TypeError; dropping it silently would
+        re-create the unbounded-setup hang. Detect and explain instead.
+        """
+        try:
+            params = inspect.signature(Harness.__init__).parameters
+        except (TypeError, ValueError):
+            return
+        if any(p.kind is inspect.Parameter.VAR_KEYWORD for p in params.values()):
+            return
+        unsupported = [
+            key
+            for key in _TIMEOUT_KWARGS
+            if key in harness_kwargs and key not in params
+        ]
+        if unsupported:
+            raise RuntimeError(
+                "The installed terminal-bench does not support "
+                f"{', '.join(unsupported)} (requires terminal-bench >= "
+                "0.2.18). Upgrade terminal-bench, or disable the bound by "
+                "setting global_agent_timeout_sec = 0 in the eval config "
+                "[run] section."
+            )
+
     def generate(
         self,
         prompt: str,
@@ -121,4 +274,4 @@ class TerminalBenchNativeBackend(InferenceBackend):
         pass
 
 
-__all__ = ["TerminalBenchNativeBackend"]
+__all__ = ["TerminalBenchNativeBackend", "summarize_benchmark_results"]

src/openjarvis/evals/cli.py

@@ -183,14 +183,28 @@ def _build_backend(
     max_turns: Optional[int] = None,
     base_url: Optional[str] = None,
     api_key: Optional[str] = None,
+    first_party_endpoint: bool = True,
 ):
     """Construct the appropriate backend.
 
-    For "hermes" and "openclaw" backends, ``base_url`` and ``api_key`` are
-    REQUIRED — these foreign frameworks need an OpenAI-compatible endpoint
-    to send model calls to. Pass them via the eval config's
-    ``[backend.external]`` section or env vars.
+    ``base_url``/``api_key`` point at the OpenAI-compatible endpoint serving
+    the model under eval:
+
+    - For "hermes" and "openclaw" they are REQUIRED — these foreign
+      frameworks always call out to an external endpoint.
+    - "jarvis-direct" and "jarvis-agent" honor them when
+      ``first_party_endpoint`` is True (the CLI ``--base-url`` path): the
+      eval targets exactly that endpoint — no engine-discovery fallback —
+      and fails fast if it is unreachable. Suite mode passes
+      ``first_party_endpoint=False`` so the suite TOML's
+      ``[backend.external]`` section stays scoped to hermes/openclaw
+      (extending it to first-party backends is explicitly deferred).
     """
+    if not first_party_endpoint:
+        fp_base_url = fp_api_key = None
+    else:
+        fp_base_url, fp_api_key = base_url, api_key
+
     if backend_name == "jarvis-agent":
         from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
 
@@ -202,6 +216,8 @@ def _build_backend(
             gpu_metrics=gpu_metrics,
             model=model,
             max_turns=max_turns,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
         )
     elif backend_name == "jarvis-direct":
         from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
@@ -210,6 +226,8 @@ def _build_backend(
             engine_key=engine_key,
             telemetry=telemetry,
             gpu_metrics=gpu_metrics,
+            base_url=fp_base_url,
+            api_key=fp_api_key,
         )
     elif backend_name == "hermes":
         from openjarvis.evals.backends.external import HermesBackend
@@ -656,25 +674,53 @@ def _build_trackers(config) -> list:
     return trackers
 
 
-def _run_terminalbench_native(config, console: Console) -> object:
-    """Run TerminalBench V2.1 natively via terminal-bench Harness."""
+def _run_terminalbench_native(
+    config,
+    console: Console,
+    *,
+    base_url: Optional[str] = None,
+    api_key: Optional[str] = None,
+) -> object:
+    """Run TerminalBench V2.1 natively via terminal-bench Harness.
+
+    ``base_url`` (from ``--base-url`` / JARVIS_BACKEND_BASE_URL) targets an
+    already-running OpenAI-compatible endpoint; when unset, the legacy local
+    vLLM default (http://localhost:8000/v1) is used.
+    """
+    from openjarvis.engine.openai_compat_engines import normalize_openai_base_url
     from openjarvis.evals.backends.terminalbench_native import (
         TerminalBenchNativeBackend,
+        summarize_benchmark_results,
     )
-    from openjarvis.evals.core.types import RunSummary
 
     model = config.model
     # LiteLLM expects "openai/<model>" for OpenAI-compatible servers
     litellm_model = f"openai/{model}"
     output_dir = getattr(config, "output_path", None) or "results/terminalbench-native/"
 
+    # Harness budgets: only forward explicit config values so the backend
+    # defaults (global_agent_timeout_sec=1800) apply otherwise.
+    timeout_kwargs = {}
+    if getattr(config, "global_agent_timeout_sec", None) is not None:
+        timeout_kwargs["global_agent_timeout_sec"] = config.global_agent_timeout_sec
+    if getattr(config, "global_timeout_multiplier", None) is not None:
+        timeout_kwargs["global_timeout_multiplier"] = config.global_timeout_multiplier
+
+    # Normalize to exactly one trailing "/v1" — LiteLLM's api_base wants the
+    # full OpenAI-compatible prefix, and users pass both forms of the URL.
+    if base_url:
+        api_base = normalize_openai_base_url(base_url) + "/v1"
+    else:
+        api_base = "http://localhost:8000/v1"
+
     backend = TerminalBenchNativeBackend(
         model=litellm_model,
-        api_base="http://localhost:8000/v1",
+        api_base=api_base,
         temperature=config.temperature,
         max_samples=config.max_samples,
         output_dir=output_dir,
         n_concurrent=config.max_workers or 4,
+        **timeout_kwargs,
     )
 
     import re
@@ -683,46 +729,83 @@ def _run_terminalbench_native(config, console: Console) -> object:
     model_slug = re.sub(r"[^a-z0-9_-]", "-", model.lower().replace("/", "-"))
     run_id = f"tb21-{model_slug}"
     console.print(f"  Running TerminalBench V2.1 natively: {model}")
+    console.print(f"  API base: {api_base}")
     console.print(f"  Harness run_id: {run_id}")
 
-    results = backend.run_harness(run_id)
+    if api_key:
+        # terminus-2 routes model calls through LiteLLM with the "openai/"
+        # prefix, which reads OPENAI_API_KEY from the environment. The
+        # harness runs in-process, so set the var for the duration of the
+        # run and restore the previous value afterwards.
+        prev_key = os.environ.get("OPENAI_API_KEY")
+        os.environ["OPENAI_API_KEY"] = api_key
+        try:
+            results = backend.run_harness(run_id)
+        finally:
+            if prev_key is None:
+                os.environ.pop("OPENAI_API_KEY", None)
+            else:
+                os.environ["OPENAI_API_KEY"] = prev_key
+    else:
+        results = backend.run_harness(run_id)
 
-    # Convert BenchmarkResults to RunSummary
-    total = len(results.trial_results) if hasattr(results, "trial_results") else 0
-    correct = 0
-    if hasattr(results, "trial_results"):
-        for tr in results.trial_results:
-            if getattr(tr, "is_resolved", False):
-                correct += 1
-
-    accuracy = correct / total if total > 0 else 0.0
-    return RunSummary(
-        benchmark="terminalbench-native",
-        category="agentic",
-        backend="terminalbench-native",
-        model=model,
-        total_samples=total,
-        scored_samples=total,
-        correct=correct,
-        accuracy=accuracy,
-        errors=0,
-        mean_latency_seconds=0.0,
-        total_cost_usd=0.0,
-    )
+    # Convert BenchmarkResults to RunSummary, classifying harness/infra
+    # failures (e.g. zero-model-contact setup hangs) out of the resolve-rate.
+    summary, harness_failures = summarize_benchmark_results(results, model=model)
+    if harness_failures:
+        console.print(
+            f"  [red bold]{len(harness_failures)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for failure in harness_failures:
+            console.print(
+                f"  [red]- {failure['task_id']}: {failure['reason']} "
+                f"(failure_mode={failure['failure_mode']})[/red]"
+            )
+    return summary
 
 
-def _run_single(config, console: Optional[Console] = None) -> object:
-    """Run a single eval from a RunConfig and return the summary."""
+def _run_single(
+    config,
+    console: Optional[Console] = None,
+    *,
+    suite_mode: bool = False,
+) -> object:
+    """Run a single eval from a RunConfig and return the summary.
+
+    ``suite_mode=True`` (TOML-suite drivers) scopes ``config.base_url`` /
+    ``config.api_key`` — stamped from the suite's ``[backend.external]``
+    section onto every RunConfig — to the hermes/openclaw backends only;
+    extending suite-level endpoint targeting to first-party backends is
+    explicitly deferred. The CLI single-run path (``suite_mode=False``)
+    honors ``--base-url``/``--api-key`` for every backend.
+    """
     from openjarvis.evals.core.runner import EvalRunner
 
     if console is None:
         console = Console()
 
+    _metadata = getattr(config, "metadata", None) or {}
+    base_url = (
+        getattr(config, "base_url", None)
+        or _metadata.get("base_url")
+        or os.environ.get("JARVIS_BACKEND_BASE_URL")
+    )
+    api_key = (
+        getattr(config, "api_key", None)
+        or _metadata.get("api_key")
+        or os.environ.get("JARVIS_BACKEND_API_KEY")
+    )
+
     # TerminalBench V2.1 native: use terminal-bench Harness directly
     if config.benchmark == "terminalbench-native":
-        return _run_terminalbench_native(config, console)
+        return _run_terminalbench_native(
+            config,
+            console,
+            base_url=None if suite_mode else base_url,
+            api_key=None if suite_mode else api_key,
+        )
 
-    _metadata = getattr(config, "metadata", None) or {}
     eval_backend = _build_backend(
         config.backend,
         config.engine_key,
@@ -732,16 +815,9 @@ def _run_single(config, console: Optional[Console] = None) -> object:
         gpu_metrics=getattr(config, "gpu_metrics", False),
         model=config.model,
         max_turns=getattr(config, "max_turns", None),
-        base_url=(
-            getattr(config, "base_url", None)
-            or _metadata.get("base_url")
-            or os.environ.get("JARVIS_BACKEND_BASE_URL")
-        ),
-        api_key=(
-            getattr(config, "api_key", None)
-            or _metadata.get("api_key")
-            or os.environ.get("JARVIS_BACKEND_API_KEY")
-        ),
+        base_url=base_url,
+        api_key=api_key,
+        first_party_endpoint=not suite_mode,
     )
     dataset = _build_dataset(config.benchmark)
     # Inject engine config for benchmarks that run their own simulation
@@ -964,7 +1040,9 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
     from rich.table import Table
 
     completed = sum(1 for t in traces if t.completed)
-    resolved = sum(1 for t in traces if t.is_resolved is True)
+    harness_errors = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = [t for t in traces if t.error_kind != "harness_error"]
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
     timed_out = sum(1 for t in traces if t.timed_out)
     total_turns = sum(t.num_turns for t in traces)
     total_tool_calls = sum(t.total_tool_calls for t in traces)
@@ -992,8 +1070,11 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
     table.add_row("Queries", str(len(traces)))
     table.add_row("Completed", f"{completed}/{len(traces)}")
     if any(t.is_resolved is not None for t in traces):
-        table.add_row("Resolved", f"{resolved}/{len(traces)}")
+        # Harness errors are excluded from the resolve-rate denominator:
+        # they are infra failures, not model misses.
+        table.add_row("Resolved", f"{resolved}/{len(model_traces)}")
     table.add_row("Timed out", str(timed_out))
+    table.add_row("Harness errors", str(len(harness_errors)))
     table.add_row("Total turns", str(total_turns))
     avg_t = f"{total_turns / len(traces):.1f}" if traces else "0"
     table.add_row("Avg turns/query", avg_t)
@@ -1020,6 +1101,19 @@ def _print_agentic_summary(console: Console, traces, config) -> None:
 
     console.print(table)
 
+    if harness_errors:
+        console.print(
+            f"[red bold]{len(harness_errors)} harness/infra failure(s) "
+            "excluded from resolve-rate:[/red bold]"
+        )
+        for t in harness_errors[:5]:
+            console.print(f"[red]  {t.query_id}: {(t.error or '')[:300]}[/red]")
+        if len(harness_errors) > 5:
+            console.print(
+                f"[red]  ... and {len(harness_errors) - 5} more "
+                "(see traces.jsonl)[/red]"
+            )
+
 
 def _run_from_config(
     config_path: str,
@@ -1070,7 +1164,7 @@ def _run_from_config(
             f"Run {i}/{len(run_configs)}: {rc.benchmark} / {rc.model}",
         )
         try:
-            summary = _run_single(rc, console=console)
+            summary = _run_single(rc, console=console, suite_mode=True)
             summaries.append(summary)
             console.print(
                 f"  [green]{summary.accuracy:.4f}[/green] "
@@ -1115,12 +1209,21 @@ def main():
 @click.option(
     "--base-url",
     default=None,
-    help="OpenAI-compat endpoint for hermes/openclaw",
+    help=(
+        "OpenAI-compatible endpoint for the model under eval. Required for "
+        "hermes/openclaw; for jarvis-direct/jarvis-agent/terminalbench-native "
+        "it bypasses engine discovery and targets this URL directly "
+        "(env: JARVIS_BACKEND_BASE_URL)."
+    ),
 )
 @click.option(
     "--api-key",
     default=None,
-    help="API key for hermes/openclaw endpoint",
+    help=(
+        "API key for the --base-url endpoint, sent as a Bearer token. "
+        "Required for hermes/openclaw; optional for first-party backends "
+        "(env: JARVIS_BACKEND_API_KEY)."
+    ),
 )
 @click.option("-m", "--model", default=None, help="Model identifier")
 @click.option(
@@ -1526,8 +1629,7 @@ def summarize(jsonl_path):
     default=None,
     type=click.Path(),
     help=(
-        "Output JSONL path. Defaults to <jsonl>.reparsed when "
-        "--in-place is not set."
+        "Output JSONL path. Defaults to <jsonl>.reparsed when --in-place is not set."
     ),
 )
 @click.option(
@@ -1642,9 +1744,7 @@ def reparse_judge(jsonl_path, out_path, in_place, summary_out):
         _json.dump(summary, f, indent=2)
 
     old_cont = [float(s) for s in old_scores if s is not None]
-    old_acc = (
-        sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
-    )
+    old_acc = sum(1 for s in old_cont if s >= 0.5) / len(old_cont) if old_cont else 0.0
     old_mean = sum(old_cont) / len(old_cont) if old_cont else 0.0
     new_mean = sum(cont) / len(cont) if cont else 0.0
     mean_shift = new_mean - old_mean

src/openjarvis/evals/core/agentic_runner.py

@@ -20,6 +20,7 @@ from contextlib import nullcontext
 from pathlib import Path
 from typing import Any, Callable, Optional
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.core.event_recorder import AgentEvent, EventRecorder, EventType
 from openjarvis.evals.core.trace import QueryTrace, TurnTrace
 
@@ -176,9 +177,12 @@ class AgenticRunner:
                 )
             self._traces.append(trace)
 
-            status = (
-                "TIMEOUT" if trace.timed_out else ("OK" if trace.completed else "FAIL")
-            )
+            if trace.timed_out:
+                status = "TIMEOUT"
+            elif trace.error_kind == "harness_error":
+                status = "HARNESS_ERROR"
+            else:
+                status = "OK" if trace.completed else "FAIL"
             LOGGER.info(
                 "Task %s: %s in %.1fs",
                 query_id,
@@ -260,11 +264,12 @@ class AgenticRunner:
                         is_resolved=record.metadata.get("is_resolved"),
                     )
 
-                status = (
-                    "TIMEOUT"
-                    if trace.timed_out
-                    else ("OK" if trace.completed else "FAIL")
-                )
+                if trace.timed_out:
+                    status = "TIMEOUT"
+                elif trace.error_kind == "harness_error":
+                    status = "HARNESS_ERROR"
+                else:
+                    status = "OK" if trace.completed else "FAIL"
                 LOGGER.info(
                     "Task %s: %s in %.1fs",
                     query_id,
@@ -444,7 +449,23 @@ class AgenticRunner:
                 _run_body()
 
         except Exception as exc:
-            LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
+            # Distinguish infrastructure breakage (task env failed to start,
+            # Docker/tmux death) from agent failures: harness errors must be
+            # recorded distinctly so scoring excludes them from resolve-rate
+            # instead of silently counting a model miss. Either way the run
+            # continues with the next record (fail THIS task, not the run).
+            is_harness_error = isinstance(exc, TaskEnvironmentError) or bool(
+                record.metadata.get("harness_error")
+            )
+            if is_harness_error:
+                LOGGER.error(
+                    "Harness/environment failure on query %s (record %s): %s",
+                    query_id,
+                    getattr(record, "record_id", "?"),
+                    exc,
+                )
+            else:
+                LOGGER.warning("Agent failed on query %s: %s", query_id, exc)
             end_time = time.time()
             # Unsubscribe EventBus relays
             if agent_bus is not None:
@@ -461,6 +482,8 @@ class AgenticRunner:
                 total_wall_clock_s=end_time - start_time,
                 completed=False,
                 is_resolved=record.metadata.get("is_resolved"),
+                error=str(exc),
+                error_kind="harness_error" if is_harness_error else "agent_error",
             )
 
         # Unsubscribe EventBus relays
@@ -534,6 +557,48 @@ class AgenticRunner:
                     model, turn.input_tokens, turn.output_tokens
                 )
 
+        # --- Zero-model-contact sanity check ----------------------------
+        # Failed in-container setups (e.g. an installed agent's SETUP phase
+        # hanging or dying) historically produced traces that looked like
+        # model results: completed=True, a synthetic 1-event turn,
+        # is_resolved=False from run_tests, and zero tokens — silently
+        # dragging resolve-rate down as a fake model miss. Signal choice:
+        # token usage plus LM inference events is the reliable discriminator
+        # here — a genuine model miss has token usage (and/or LM events),
+        # while "the agent never called the model" has neither. We do NOT
+        # key off completion status or is_resolved, which are identical in
+        # both cases. run_agent_loop envs drive the model directly
+        # (bypassing usage reporting), so their turn_wall_clocks count as
+        # model contact.
+        had_lm_events = any(
+            e.event_type in (EventType.LM_INFERENCE_START, EventType.LM_INFERENCE_END)
+            for e in events
+        )
+        had_loop_turns = bool(
+            task_env is not None and getattr(task_env, "turn_wall_clocks", None)
+        )
+        turn_tokens = sum(t.input_tokens + t.output_tokens for t in turns)
+        error: Optional[str] = None
+        error_kind: Optional[str] = None
+        if (
+            not had_lm_events
+            and not had_loop_turns
+            and in_tok + out_tok == 0
+            and turn_tokens == 0
+        ):
+            error = (
+                "zero_model_requests: the agent produced no LM inference "
+                "events and reported zero token usage — the model was never "
+                "contacted. This is a harness/infrastructure failure (e.g. "
+                "in-container agent setup hang/death), not a model miss, and "
+                "is excluded from resolve-rate. If your agent genuinely "
+                "contacted the model, make it report token usage or emit "
+                "LM_INFERENCE events. Response tail: "
+                f"{(response_text or '')[-500:]!r}"
+            )
+            error_kind = "harness_error"
+            LOGGER.error("Query %s: %s", query_id, error)
+
         # Query-level energy from telemetry window
         query_gpu_energy = _compute_energy_delta(readings, "gpu_energy_j")
         query_cpu_energy = _compute_energy_delta(readings, "cpu_energy_j")
@@ -563,6 +628,8 @@ class AgenticRunner:
             is_resolved=record.metadata.get("is_resolved"),
             query_mbu_avg_pct=query_mbu_avg,
             query_mbu_max_pct=query_mbu_max,
+            error=error,
+            error_kind=error_kind,
         )
 
         # Correlate energy with trace

src/openjarvis/evals/core/config.py

@@ -143,6 +143,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
         sheets_worksheet=run_raw.get("sheets_worksheet", "Results"),
         sheets_credentials_path=run_raw.get("sheets_credentials_path", ""),
         max_turns=(int(run_raw["max_turns"]) if "max_turns" in run_raw else None),
+        global_agent_timeout_sec=(
+            float(run_raw["global_agent_timeout_sec"])
+            if "global_agent_timeout_sec" in run_raw
+            else None
+        ),
+        global_timeout_multiplier=(
+            float(run_raw["global_timeout_multiplier"])
+            if "global_timeout_multiplier" in run_raw
+            else None
+        ),
     )
 
     # Parse [[models]]
@@ -212,6 +222,16 @@ def load_eval_config(path: str | Path) -> EvalSuiteConfig:
                 max_tokens=int(b["max_tokens"]) if "max_tokens" in b else None,
                 subset=b.get("subset"),
                 record_ids=record_ids,
+                global_agent_timeout_sec=(
+                    float(b["global_agent_timeout_sec"])
+                    if "global_agent_timeout_sec" in b
+                    else None
+                ),
+                global_timeout_multiplier=(
+                    float(b["global_timeout_multiplier"])
+                    if "global_timeout_multiplier" in b
+                    else None
+                ),
             )
         )
 
@@ -275,6 +295,14 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
             if bench.judge_model is not None:
                 judge_model = bench.judge_model
 
+            # terminal-bench harness budgets: benchmark > [run]
+            global_agent_timeout_sec = suite.run.global_agent_timeout_sec
+            if bench.global_agent_timeout_sec is not None:
+                global_agent_timeout_sec = bench.global_agent_timeout_sec
+            global_timeout_multiplier = suite.run.global_timeout_multiplier
+            if bench.global_timeout_multiplier is not None:
+                global_timeout_multiplier = bench.global_timeout_multiplier
+
             # Auto-generate output path
             model_slug = model.name.replace("/", "-").replace(":", "-")
             output_path = f"{output_dir}/{bench.name}_{model_slug}.jsonl"
@@ -328,6 +356,8 @@ def expand_suite(suite: EvalSuiteConfig) -> List[RunConfig]:
                     base_url=suite.backend_external_base_url,
                     api_key=suite.backend_external_api_key,
                     record_ids=bench.record_ids,
+                    global_agent_timeout_sec=global_agent_timeout_sec,
+                    global_timeout_multiplier=global_timeout_multiplier,
                 )
             )
 

src/openjarvis/evals/core/environment.py

@@ -8,6 +8,18 @@ from typing import Any, Dict, Tuple
 from openjarvis.evals.core.types import EvalRecord
 
 
+class TaskEnvironmentError(RuntimeError):
+    """A task execution environment failed to start or operate.
+
+    Raised when infrastructure backing a task (Docker container, docker
+    compose project, tmux session, recording binaries, ...) breaks. This is
+    a harness/environment failure, **not** a model failure: runners record
+    it distinctly (``QueryTrace.error_kind == "harness_error"``) so scoring
+    can exclude the sample from resolve-rate instead of silently counting
+    it as a model miss.
+    """
+
+
 class EnvironmentProvider(ABC):
     """Manages an external environment for evaluation benchmarks.
 
@@ -50,3 +62,6 @@ class EnvironmentProvider(ABC):
     @abstractmethod
     def teardown(self) -> None:
         """Stop the environment and release resources."""
+
+
+__all__ = ["EnvironmentProvider", "TaskEnvironmentError"]

src/openjarvis/evals/core/export.py

@@ -26,13 +26,22 @@ def _agg_stats(values: Sequence[Optional[float]]) -> dict[str, Optional[float]]:
     }
 
 
+def _model_attributable(traces: list[QueryTrace]) -> list[QueryTrace]:
+    """Traces whose outcome is attributable to the model.
+
+    Harness errors (infra/setup failures, zero-model-contact runs) are
+    excluded so they never count as model misses in resolve-rate.
+    """
+    return [t for t in traces if t.error_kind != "harness_error"]
+
+
 def _compute_efficiency(
     traces: list[QueryTrace],
     total_gpu_energy: Optional[float],
     total_cpu_energy: Optional[float],
 ) -> dict[str, Optional[float]]:
     """Compute efficiency metrics from traces and aggregate energy."""
-    scored = [t for t in traces if t.is_resolved is not None]
+    scored = [t for t in _model_attributable(traces) if t.is_resolved is not None]
     resolved = sum(1 for t in scored if t.is_resolved is True)
     accuracy = resolved / len(scored) if scored else None
     gpu_powers = [
@@ -247,8 +256,12 @@ def export_summary_json(
             cpu_energy_values.append(sum(cpu_vals))
     total_cpu_energy = sum(cpu_energy_values) if cpu_energy_values else None
 
-    resolved = sum(1 for t in traces if t.is_resolved is True)
-    unresolved = sum(1 for t in traces if t.is_resolved is False)
+    # Harness errors (infra failures, zero-model-contact runs) are excluded
+    # from the resolve-rate denominator: they are not model misses.
+    harness_error_traces = [t for t in traces if t.error_kind == "harness_error"]
+    model_traces = _model_attributable(traces)
+    resolved = sum(1 for t in model_traces if t.is_resolved is True)
+    unresolved = sum(1 for t in model_traces if t.is_resolved is False)
 
     cost_values = [t.total_cost_usd for t in traces if t.total_cost_usd is not None]
     total_cost = sum(cost_values) if cost_values else None
@@ -345,6 +358,7 @@ def export_summary_json(
             "completed": completed,
             "resolved": resolved,
             "unresolved": unresolved,
+            "harness_errors": len(harness_error_traces),
             "accuracy": accuracy,
             "turns": total_turns,
             "tool_calls": total_tool_calls,
@@ -365,6 +379,12 @@ def export_summary_json(
         "efficiency": efficiency,
     }
 
+    if harness_error_traces:
+        summary["harness_error_details"] = [
+            {"query_id": t.query_id, "error": (t.error or "")[:500]}
+            for t in harness_error_traces
+        ]
+
     if action_totals:
         summary["action_energy_summary"] = action_totals
 
@@ -399,7 +419,7 @@ def export_summary_json(
 
     accuracy_vals: list[float] = [
         1.0 if t.is_resolved is True else 0.0
-        for t in traces
+        for t in _model_attributable(traces)
         if t.is_resolved is not None
     ]
     latency_vals = [t.total_wall_clock_s for t in traces if t.total_wall_clock_s > 0]

src/openjarvis/evals/core/trace.py

@@ -91,6 +91,13 @@ class QueryTrace:
     is_resolved: Optional[bool] = None
     query_mbu_avg_pct: Optional[float] = None
     query_mbu_max_pct: Optional[float] = None
+    # Error taxonomy. ``error_kind`` distinguishes infrastructure failures
+    # ("harness_error": task env / Docker / tmux broke, or the agent never
+    # contacted the model) from agent failures ("agent_error"). Harness
+    # errors are excluded from resolve-rate by export/summary code so they
+    # are never silently counted as model misses.
+    error: Optional[str] = None
+    error_kind: Optional[str] = None
 
     @property
     def num_turns(self) -> int:
@@ -196,6 +203,8 @@ class QueryTrace:
             "is_resolved": self.is_resolved,
             "query_mbu_avg_pct": self.query_mbu_avg_pct,
             "query_mbu_max_pct": self.query_mbu_max_pct,
+            "error": self.error,
+            "error_kind": self.error_kind,
         }
 
     @classmethod
@@ -216,6 +225,8 @@ class QueryTrace:
             is_resolved=d.get("is_resolved"),
             query_mbu_avg_pct=d.get("query_mbu_avg_pct"),
             query_mbu_max_pct=d.get("query_mbu_max_pct"),
+            error=d.get("error"),
+            error_kind=d.get("error_kind"),
         )
 
     def save_jsonl(self, path: Path) -> None:

src/openjarvis/evals/core/types.py

@@ -102,6 +102,15 @@ class RunConfig:
     # specific records (e.g. recovering silent-fake records without
     # re-running the entire benchmark).
     record_ids: Optional[List[str]] = None
+    # terminal-bench harness budgets (terminalbench-native backend).
+    # global_agent_timeout_sec bounds each trial's agent phase — SETUP+RUN
+    # together, since terminal-bench runs installed-agent setup inside the
+    # agent budget with an infinite tmux timeout. When set it REPLACES the
+    # per-task max_agent_timeout_sec; 0 disables the bound (per-task budgets
+    # apply); None uses the backend default (1800 s).
+    global_agent_timeout_sec: Optional[float] = None
+    # Scales per-task budgets when global_agent_timeout_sec is not set.
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)
@@ -232,6 +241,9 @@ class ExecutionConfig:
     # to JarvisConfig.agent.max_turns (default 10). Bump to 30-50 for
     # thinking/reasoning models on agentic benchmarks (GAIA, LiveResearch).
     max_turns: Optional[int] = None
+    # terminal-bench harness budgets (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)
@@ -265,6 +277,10 @@ class BenchmarkConfig:
     max_tokens: Optional[int] = None
     subset: Optional[str] = None
     record_ids: Optional[List[str]] = None
+    # Per-benchmark override of the terminal-bench harness budgets
+    # (see RunConfig for semantics).
+    global_agent_timeout_sec: Optional[float] = None
+    global_timeout_multiplier: Optional[float] = None
 
 
 @dataclass(slots=True)

src/openjarvis/evals/execution/terminalbench_env.py

@@ -8,6 +8,8 @@ from pathlib import Path
 from types import TracebackType
 from typing import Any, MutableMapping, Optional, Type
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
+
 LOGGER = logging.getLogger(__name__)
 
 
@@ -29,8 +31,6 @@ class TerminalBenchTaskEnv:
     # ------------------------------------------------------------------
 
     def __enter__(self) -> TerminalBenchTaskEnv:
-        from terminal_bench.terminal.terminal import spin_up_terminal
-
         task = self._metadata.get("task")
         task_paths = self._metadata.get("task_paths")
         task_id = self._metadata.get("task_id", "unknown")
@@ -41,6 +41,8 @@ class TerminalBenchTaskEnv:
                 "Use the 'terminalbench-native' dataset."
             )
 
+        from terminal_bench.terminal.terminal import spin_up_terminal
+
         docker_image_prefix = f"tb__{task_id}".replace(".", "-")
         client_image_name = f"{docker_image_prefix}__client"
         client_container_name = f"oj-{task_id}".replace(".", "-")
@@ -48,19 +50,59 @@ class TerminalBenchTaskEnv:
         self._logs_tmpdir = tempfile.TemporaryDirectory(prefix="oj_tb_logs_")
         logs_path = Path(self._logs_tmpdir.name)
 
-        self._terminal_cm = spin_up_terminal(
-            client_container_name=client_container_name,
-            client_image_name=client_image_name,
-            docker_compose_path=task_paths.docker_compose_path,
-            docker_image_name_prefix=docker_image_prefix,
-            sessions_logs_path=logs_path,
-            disable_recording=task.disable_asciinema,
-        )
-        self._terminal = self._terminal_cm.__enter__()
+        # Everything below is exception-safe: a failure mid-startup (docker
+        # compose, tmux, asciinema) tears the spun-up terminal back down
+        # immediately instead of leaking the docker compose project until GC
+        # (or forever, when the env object is retained), and re-raises as a
+        # loud TaskEnvironmentError naming the task image so the runner can
+        # record a harness error for THIS task and continue with the rest.
+        try:
+            self._terminal_cm = spin_up_terminal(
+                client_container_name=client_container_name,
+                client_image_name=client_image_name,
+                docker_compose_path=task_paths.docker_compose_path,
+                docker_image_name_prefix=docker_image_prefix,
+                sessions_logs_path=logs_path,
+                disable_recording=task.disable_asciinema,
+            )
+            self._terminal = self._terminal_cm.__enter__()
 
-        session = self._terminal.create_session(
-            "agent", is_active_stream=False, as_configured_user=True
-        )
+            # Preflight BEFORE the agent loop: terminal-bench drives the
+            # agent through tmux (and records via asciinema unless the task
+            # disables it). A missing binary otherwise surfaces mid-run as
+            # an opaque RuntimeError or a fake TimeoutError.
+            self._preflight_container_binaries(
+                task, task_id, client_image_name, client_container_name
+            )
+
+            session = self._terminal.create_session(
+                "agent", is_active_stream=False, as_configured_user=True
+            )
+        except BaseException as exc:
+            self._teardown(type(exc), exc, exc.__traceback__)
+            if not isinstance(exc, Exception):
+                # KeyboardInterrupt / SystemExit: clean up but never mask.
+                raise
+            if isinstance(exc, TaskEnvironmentError):
+                self._metadata["harness_error"] = str(exc)
+                raise
+            message = (
+                f"Task '{task_id}': failed to start the task environment "
+                f"(image '{client_image_name}', container "
+                f"'{client_container_name}'): {exc}. This is a harness/"
+                "environment failure, not a model failure. Check that the "
+                "Docker daemon is healthy and that tmux is installed in the "
+                "task image."
+            )
+            # docker compose stderr is only logged at DEBUG by
+            # terminal-bench; surface it here so the failure is actionable.
+            stderr = getattr(exc, "stderr", None)
+            if stderr:
+                if isinstance(stderr, bytes):
+                    stderr = stderr.decode("utf-8", errors="replace")
+                message += f"\ndocker compose stderr (tail):\n{stderr[-2000:]}"
+            self._metadata["harness_error"] = message
+            raise TaskEnvironmentError(message) from exc
 
         self._metadata["terminal"] = self._terminal
         self._metadata["session"] = session
@@ -68,24 +110,99 @@ class TerminalBenchTaskEnv:
 
         return self
 
+    def _preflight_container_binaries(
+        self,
+        task: Any,
+        task_id: str,
+        client_image_name: str,
+        client_container_name: str,
+    ) -> None:
+        """Verify tmux (and asciinema if recording) exist in the container.
+
+        Raises:
+            TaskEnvironmentError: naming the task image and the missing
+                binary, with the remedy, before any agent work starts.
+        """
+        container = getattr(self._terminal, "container", None)
+        if container is None:
+            # Terminal implementation without a container handle (e.g. a
+            # future terminal-bench version); fall through to terminal-bench's
+            # own checks rather than guessing.
+            return
+
+        checks: list[tuple[str, list[str], str]] = [
+            (
+                "tmux",
+                ["tmux", "-V"],
+                f"install tmux in the task image '{client_image_name}'",
+            ),
+        ]
+        if not getattr(task, "disable_asciinema", False):
+            checks.append(
+                (
+                    "asciinema",
+                    ["asciinema", "--version"],
+                    f"install asciinema in the task image '{client_image_name}' "
+                    "or set disable_asciinema in task.yaml",
+                )
+            )
+
+        for binary, cmd, remedy in checks:
+            result = container.exec_run(cmd)
+            exit_code = getattr(result, "exit_code", 0)
+            if exit_code == 0:
+                continue
+            output = getattr(result, "output", b"")
+            if isinstance(output, bytes):
+                output = output.decode("utf-8", errors="replace")
+            raise TaskEnvironmentError(
+                f"Task '{task_id}': required binary '{binary}' is not usable "
+                f"in task image '{client_image_name}' (container "
+                f"'{client_container_name}'): exec exit code {exit_code}, "
+                f"output {str(output).strip()!r}. Remedy: {remedy}. This is "
+                "a harness/environment failure, not a model failure."
+            )
+
+    def _teardown(
+        self,
+        exc_type: Optional[Type[BaseException]] = None,
+        exc_val: Optional[BaseException] = None,
+        exc_tb: Optional[TracebackType] = None,
+    ) -> None:
+        """Idempotent cleanup shared by ``__exit__`` and failed ``__enter__``.
+
+        Secondary cleanup errors are logged, never raised, so they cannot
+        mask the original failure.
+        """
+        self._metadata.pop("terminal", None)
+        self._metadata.pop("session", None)
+        self._metadata.pop("container", None)
+
+        terminal_cm, self._terminal_cm, self._terminal = self._terminal_cm, None, None
+        if terminal_cm is not None:
+            try:
+                terminal_cm.__exit__(exc_type, exc_val, exc_tb)
+            except Exception:
+                LOGGER.exception(
+                    "Secondary error while tearing down the terminal for "
+                    "task %s (original error, if any, is re-raised)",
+                    self._metadata.get("task_id", "unknown"),
+                )
+
+        logs_tmpdir, self._logs_tmpdir = self._logs_tmpdir, None
+        if logs_tmpdir is not None:
+            try:
+                logs_tmpdir.cleanup()
+            except Exception:
+                LOGGER.exception("Failed to clean up session logs tmpdir")
+
     def __exit__(
         self,
         exc_type: Optional[Type[BaseException]],
         exc_val: Optional[BaseException],
         exc_tb: Optional[TracebackType],
     ) -> None:
-        self._metadata.pop("terminal", None)
-        self._metadata.pop("session", None)
-        self._metadata.pop("container", None)
-
-        if self._terminal_cm is not None:
-            self._terminal_cm.__exit__(exc_type, exc_val, exc_tb)
-            self._terminal_cm = None
-            self._terminal = None
-
-        if self._logs_tmpdir is not None:
-            self._logs_tmpdir.cleanup()
-            self._logs_tmpdir = None
+        self._teardown(exc_type, exc_val, exc_tb)
 
     # ------------------------------------------------------------------
     # Test execution
@@ -93,12 +210,6 @@ class TerminalBenchTaskEnv:
 
     def run_tests(self) -> tuple[bool, dict[str, Any]]:
         """Copy test scripts into container, execute, parse results."""
-        from terminal_bench.parsers.base_parser import UnitTestStatus
-        from terminal_bench.parsers.parser_factory import ParserFactory
-        from terminal_bench.terminal.docker_compose_manager import (
-            DockerComposeManager,
-        )
-
         task = self._metadata["task"]
         task_paths = self._metadata["task_paths"]
         terminal = self._terminal
@@ -110,6 +221,12 @@ class TerminalBenchTaskEnv:
             self._metadata["test_results"] = results
             return False, results
 
+        from terminal_bench.parsers.base_parser import UnitTestStatus
+        from terminal_bench.parsers.parser_factory import ParserFactory
+        from terminal_bench.terminal.docker_compose_manager import (
+            DockerComposeManager,
+        )
+
         try:
             paths_to_copy = [task_paths.run_tests_path]
             if task_paths.test_dir.exists():

src/openjarvis/evals/tests/test_backends.py

@@ -4,6 +4,27 @@ from __future__ import annotations
 
 from unittest.mock import MagicMock, patch
 
+import httpx
+import pytest
+import respx
+
+
+def _mock_builder() -> MagicMock:
+    """A SystemBuilder mock whose fluent methods chain like the real one."""
+    builder = MagicMock()
+    for method in (
+        "engine",
+        "engine_instance",
+        "model",
+        "agent",
+        "tools",
+        "telemetry",
+        "traces",
+    ):
+        getattr(builder, method).return_value = builder
+    builder.build.return_value = MagicMock()
+    return builder
+
 
 class TestJarvisDirectBackend:
     @patch("openjarvis.system.SystemBuilder")
@@ -137,3 +158,130 @@ class TestJarvisAgentBackend:
         assert result["content"] == "The answer is 4."
         assert result["turns"] == 2
         assert len(result["tool_results"]) == 1
+
+
+class TestJarvisDirectBackendBaseUrl:
+    """--base-url targeting for the jarvis-direct backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        # Trailing /v1 is normalized away so request paths don't double up.
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        # The discovery path must not be engaged at all.
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisDirectBackend(base_url="http://127.0.0.1:18998")
+
+        # No silent engine substitution: the system is never built.
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisDirectBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_wins_over_engine_key(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_direct import JarvisDirectBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisDirectBackend(engine_key="vllm", base_url="http://127.0.0.1:18999")
+
+        mock_builder.engine.assert_not_called()
+        mock_builder.engine_instance.assert_called_once()
+        # The engine key is kept as the label for the injected engine.
+        assert mock_builder.engine_instance.call_args.kwargs["key"] == "vllm"
+
+
+class TestJarvisAgentBackendBaseUrl:
+    """--base-url targeting for the jarvis-agent backend."""
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_base_url_injects_pinned_openai_compat_engine(self, mock_builder_cls):
+        from openjarvis.engine.openai_compat_engines import OpenAICompatEngine
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18999/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            JarvisAgentBackend(base_url="http://127.0.0.1:18999/v1", api_key="sk-x")
+
+        mock_builder.engine_instance.assert_called_once()
+        injected = mock_builder.engine_instance.call_args[0][0]
+        assert isinstance(injected, OpenAICompatEngine)
+        assert injected._host == "http://127.0.0.1:18999"
+        assert injected._api_key == "sk-x"
+        mock_builder.engine.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_unreachable_base_url_fails_fast_naming_url(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        with respx.mock:
+            respx.get("http://127.0.0.1:18998/v1/models").mock(
+                side_effect=httpx.ConnectError("connection refused")
+            )
+            with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18998"):
+                JarvisAgentBackend(base_url="http://127.0.0.1:18998")
+
+        mock_builder.engine_instance.assert_not_called()
+        mock_builder.build.assert_not_called()
+
+    @patch("openjarvis.system.SystemBuilder")
+    def test_no_base_url_keeps_engine_key_path(self, mock_builder_cls):
+        from openjarvis.evals.backends.jarvis_agent import JarvisAgentBackend
+
+        mock_builder = _mock_builder()
+        mock_builder_cls.return_value = mock_builder
+
+        JarvisAgentBackend(engine_key="vllm")
+        mock_builder.engine.assert_called_with("vllm")
+        mock_builder.engine_instance.assert_not_called()

src/openjarvis/evals/tests/test_cli_endpoint.py

@@ -0,0 +1,190 @@
+"""--base-url/--api-key forwarding through the eval CLI plumbing.
+
+Covers the fix for the eval-CLI endpoint gap: the flags used to be silently
+dropped for jarvis-direct/jarvis-agent and ignored by terminalbench-native
+(which hardcoded api_base="http://localhost:8000/v1").
+"""
+
+from __future__ import annotations
+
+import io
+import os
+from types import SimpleNamespace
+from unittest.mock import MagicMock, patch
+
+import click
+import pytest
+from rich.console import Console
+
+from openjarvis.evals.cli import _build_backend, _run_terminalbench_native
+from openjarvis.evals.core.types import RunConfig
+
+
+def _quiet_console() -> Console:
+    return Console(file=io.StringIO())
+
+
+def _tb_config(**overrides) -> RunConfig:
+    defaults = dict(
+        benchmark="terminalbench-native",
+        backend="jarvis-direct",
+        model="my-model",
+        max_samples=1,
+        max_workers=1,
+        temperature=0.2,
+    )
+    defaults.update(overrides)
+    return RunConfig(**defaults)
+
+
+class TestBuildBackendForwardsEndpoint:
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_jarvis_direct_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_agent.JarvisAgentBackend")
+    def test_jarvis_agent_receives_base_url_and_api_key(self, mock_cls):
+        _build_backend(
+            "jarvis-agent",
+            "vllm",
+            "orchestrator",
+            ["calculator"],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] == "http://node7:8123/v1"
+        assert kwargs["api_key"] == "sk-k"
+
+    @patch("openjarvis.evals.backends.jarvis_direct.JarvisDirectBackend")
+    def test_suite_mode_scopes_endpoint_to_external_backends(self, mock_cls):
+        """[backend.external] suite semantics stay hermes/openclaw-only:
+        first_party_endpoint=False must not forward to first-party."""
+        _build_backend(
+            "jarvis-direct",
+            "vllm",
+            "orchestrator",
+            [],
+            base_url="http://node7:8123/v1",
+            api_key="sk-k",
+            first_party_endpoint=False,
+        )
+        kwargs = mock_cls.call_args.kwargs
+        assert kwargs["base_url"] is None
+        assert kwargs["api_key"] is None
+
+    def test_hermes_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="hermes"):
+            _build_backend("hermes", None, "orchestrator", [])
+
+    def test_openclaw_still_requires_base_url_and_api_key(self):
+        with pytest.raises(click.UsageError, match="openclaw"):
+            _build_backend("openclaw", None, "orchestrator", [])
+
+
+class TestTerminalBenchNativeApiBase:
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_passed_through_as_api_base(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_base_url_without_v1_gets_single_v1_suffix(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123",
+        )
+        assert mock_cls.call_args.kwargs["api_base"] == "http://node7:8123/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_default_api_base_unchanged_without_base_url(self, mock_cls):
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(_tb_config(), _quiet_console())
+        assert mock_cls.call_args.kwargs["api_base"] == "http://localhost:8000/v1"
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_api_key_exported_as_openai_api_key_during_run(self, mock_cls, monkeypatch):
+        """terminus-2 reads OPENAI_API_KEY via LiteLLM; the var must be set
+        during harness.run() and restored afterwards."""
+        monkeypatch.delenv("OPENAI_API_KEY", raising=False)
+        seen: dict = {}
+
+        def fake_run_harness(run_id):
+            seen["openai_api_key"] = os.environ.get("OPENAI_API_KEY")
+            return SimpleNamespace(trial_results=[])
+
+        mock_backend = MagicMock()
+        mock_backend.run_harness.side_effect = fake_run_harness
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert seen["openai_api_key"] == "sk-tb"
+        assert "OPENAI_API_KEY" not in os.environ  # restored
+
+    @patch("openjarvis.evals.backends.terminalbench_native.TerminalBenchNativeBackend")
+    def test_preexisting_openai_api_key_restored(self, mock_cls, monkeypatch):
+        monkeypatch.setenv("OPENAI_API_KEY", "sk-original")
+        mock_backend = MagicMock()
+        mock_backend.run_harness.return_value = SimpleNamespace(trial_results=[])
+        mock_cls.return_value = mock_backend
+
+        _run_terminalbench_native(
+            _tb_config(),
+            _quiet_console(),
+            base_url="http://node7:8123/v1",
+            api_key="sk-tb",
+        )
+        assert os.environ["OPENAI_API_KEY"] == "sk-original"
+
+
+class TestRunSingleSuiteModeGating:
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_suite_mode_drops_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console(), suite_mode=True)
+        assert mock_tb.call_args.kwargs["base_url"] is None
+        assert mock_tb.call_args.kwargs["api_key"] is None
+
+    @patch("openjarvis.evals.cli._run_terminalbench_native")
+    def test_cli_mode_forwards_endpoint_for_terminalbench(self, mock_tb):
+        from openjarvis.evals.cli import _run_single
+
+        mock_tb.return_value = SimpleNamespace(accuracy=0.0)
+        config = _tb_config(base_url="http://node7:8123/v1", api_key="sk-k")
+        _run_single(config, console=_quiet_console())
+        assert mock_tb.call_args.kwargs["base_url"] == "http://node7:8123/v1"
+        assert mock_tb.call_args.kwargs["api_key"] == "sk-k"

src/openjarvis/security/guardrails.py

@@ -192,6 +192,7 @@ class GuardrailsEngine(InferenceEngine):
                             tool_calls=msg.tool_calls,
                             tool_call_id=msg.tool_call_id,
                             metadata=msg.metadata,
+                            images=msg.images,
                         )
             messages = processed
 

src/openjarvis/server/connectors_router.py

@@ -3,7 +3,23 @@
 from __future__ import annotations
 
 import logging
-from typing import Any, Dict, Optional
+from typing import TYPE_CHECKING, Any, Dict, Optional
+
+# ``Request`` must be importable at *module* scope so that FastAPI can resolve
+# the stringized ``request: Request`` annotations on the OAuth endpoints below.
+# Because this module uses ``from __future__ import annotations``, every
+# annotation is a string that FastAPI evaluates against the module globals; a
+# ``Request`` imported only inside ``create_connectors_router()`` is invisible
+# there, which makes FastAPI mistake ``request`` for a required *query* param
+# (HTTP 422 on /oauth/start) or inject ``None`` (AttributeError on
+# /oauth/callback). Keep this import at top level. See issue #512.
+if TYPE_CHECKING:
+    from starlette.requests import Request
+else:
+    try:
+        from starlette.requests import Request
+    except ImportError:  # starlette ships with fastapi; absent only without it
+        Request = Any  # type: ignore[assignment,misc]
 
 logger = logging.getLogger(__name__)
 
@@ -78,7 +94,7 @@ def create_connectors_router():
     this package.
     """
     try:
-        from fastapi import APIRouter, HTTPException, Request
+        from fastapi import APIRouter, HTTPException
     except ImportError as exc:
         raise ImportError(
             "fastapi and pydantic are required for the connectors router"
@@ -125,6 +141,69 @@ def create_connectors_router():
             "chunks": chunks,
         }
 
+    def _maybe_oauth_client_pair(
+        connector_id: str, req: ConnectRequest
+    ) -> Optional[Dict[str, Any]]:
+        """Handle a pasted ``client_id:client_secret`` for an OAuth connector.
+
+        Returns an ``oauth_required`` directive (and persists the client
+        credentials to every credential file for the provider) when *req*
+        carries a Google ``client_id:client_secret`` pair, so the caller can
+        return early instead of triggering the silent background OAuth flow.
+        Returns ``None`` when there is no such pair (the caller then falls
+        through to the normal ``handle_callback`` / token path).
+
+        Raises ``HTTPException(400)`` when the pair is present but malformed or
+        the connector has no OAuth provider — per the silent-failure discipline
+        in REVIEW.md, a bad credential surfaces an actionable error rather than
+        a perpetual ``pending`` state.
+        """
+        from openjarvis.connectors.oauth import (
+            get_provider_for_connector,
+            save_client_credentials,
+        )
+
+        raw = (req.code or req.token or "").strip()
+        # Only the client-registration pair routes through the server flow.
+        # A raw access token (no ".apps.googleusercontent.com") is handled by
+        # the connector's handle_callback unchanged.
+        if ".apps.googleusercontent.com" not in raw or ":" not in raw:
+            return None
+
+        provider = get_provider_for_connector(connector_id)
+        if provider is None:
+            raise HTTPException(
+                status_code=400,
+                detail=f"No OAuth provider configured for '{connector_id}'",
+            )
+
+        client_id, client_secret = raw.split(":", 1)
+        client_id = client_id.strip()
+        client_secret = client_secret.strip()
+        if not client_id or not client_secret:
+            raise HTTPException(
+                status_code=400,
+                detail=(
+                    "Malformed credentials — expected 'CLIENT_ID:CLIENT_SECRET'. "
+                    f"Create an OAuth client at: {provider.setup_url}"
+                ),
+            )
+
+        save_client_credentials(provider, client_id, client_secret)
+        # Cached instances may have resolved a stale credentials path before
+        # these client creds existed; drop them so /oauth/callback rebuilds
+        # them against the freshly written files.
+        for cid in provider.connector_ids:
+            _instances.pop(cid, None)
+
+        return {
+            "connector_id": connector_id,
+            "connected": False,
+            "status": "oauth_required",
+            "oauth_start": f"/v1/connectors/{connector_id}/oauth/start",
+            "sync_status": None,
+        }
+
     # ------------------------------------------------------------------
     # Background-sync state tracking. Defined here (before the endpoints)
     # so that POST /connect can fire-and-forget into the same machinery
@@ -317,6 +396,19 @@ def create_connectors_router():
                     instance._connected = Path(req.path).is_dir()
 
             elif auth_type == "oauth":
+                # A pasted ``client_id:client_secret`` pair is NOT a completed
+                # OAuth credential — it is the app registration. Persist it and
+                # hand the UI a directive to run the in-process browser consent
+                # flow (/oauth/start → /oauth/callback), which is the only path
+                # that actually exchanges a code for an access_token. Previously
+                # this routed into the connector's handle_callback, which spawned
+                # a daemon thread that popped a browser + ran its own
+                # localhost:8789 callback server; that thread fails silently in
+                # the bundled desktop context, so the connector never became
+                # connected and never appeared in Data Sources (issue #512).
+                directive = _maybe_oauth_client_pair(connector_id, req)
+                if directive is not None:
+                    return directive
                 if req.code:
                     instance.handle_callback(req.code)
                 elif req.token:
@@ -433,9 +525,9 @@ def create_connectors_router():
     @router.get("/{connector_id}/oauth/callback")
     async def oauth_callback(
         connector_id: str,
+        request: Request,
         code: str = "",
         error: str = "",
-        request: Request = None,
     ):
         """Handle OAuth callback from the provider."""
         from fastapi.responses import HTMLResponse

src/openjarvis/server/routes.py

@@ -43,6 +43,51 @@ def _to_messages(chat_messages) -> list[Message]:
     return messages
 
 
+def _ensure_identity_prompt(messages: list[Message], app_config) -> list[Message]:
+    """Prepend OpenJarvis's identity system prompt when the client omits one.
+
+    The desktop UI's chat backend posts only user/assistant turns to
+    ``/v1/chat/completions`` (see ``frontend/.../Chat/InputArea.tsx``), so
+    nothing grounds the model's identity. Without a system prompt the model
+    answers from its training identity (e.g. "I'm Claude", "I am Qwen"),
+    which is what #540 reported. The CLI paths inject this via
+    ``SystemPromptBuilder`` / ``BaseAgent``; the engine-direct server paths
+    did not. This mirrors the agent fallback in ``agents/_stubs.py``.
+
+    If any message already carries a system role, the caller has supplied
+    their own grounding and we leave the list untouched (no double-prompting).
+
+    Resolution of the identity text: ``app_config.agent.default_system_prompt``
+    when a config is wired onto ``app.state``; otherwise fall back to
+    ``load_config()``. Config resolution is wrapped so a broken/missing
+    config degrades to "no injection" rather than crashing the endpoint, but
+    the failure is logged (per REVIEW.md — never silently swallow).
+    """
+    if any(m.role == Role.SYSTEM for m in messages):
+        return messages
+
+    prompt = ""
+    try:
+        if app_config is not None:
+            prompt = app_config.agent.default_system_prompt or ""
+        else:
+            from openjarvis.core.config import load_config
+
+            prompt = load_config().agent.default_system_prompt or ""
+    except Exception:
+        logging.getLogger("openjarvis.server").debug(
+            "Identity system prompt resolution failed; "
+            "serving request without identity grounding",
+            exc_info=True,
+        )
+        return messages
+
+    if not prompt:
+        return messages
+
+    return [Message(role=Role.SYSTEM, content=prompt), *messages]
+
+
 @router.post("/v1/chat/completions")
 async def chat_completions(request_body: ChatCompletionRequest, request: Request):
     """Handle chat completion requests (streaming and non-streaming)."""
@@ -149,7 +194,7 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request
         # from the engine for true real-time output.
         if request_body.tools:
             return await _handle_stream_tools(
-                engine, model, request_body, complexity_info
+                engine, model, request_body, complexity_info, app_config=config
             )
         return await _handle_stream(
             engine,
@@ -157,6 +202,7 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request
             request_body,
             complexity_info,
             trace_store=getattr(request.app.state, "trace_store", None),
+            app_config=config,
         )
 
     # Non-streaming: use agent if available, otherwise direct engine call.
@@ -192,6 +238,7 @@ async def chat_completions(request_body: ChatCompletionRequest, request: Request
         request_body,
         bus=bus,
         complexity_info=complexity_info,
+        app_config=config,
     )
 
 
@@ -201,9 +248,11 @@ def _handle_direct(
     req: ChatCompletionRequest,
     bus=None,
     complexity_info=None,
+    app_config=None,
 ) -> ChatCompletionResponse:
     """Direct engine call without agent."""
     messages = _to_messages(req.messages)
+    messages = _ensure_identity_prompt(messages, app_config)
     kwargs: dict[str, Any] = {}
     if req.tools:
         kwargs["tools"] = req.tools
@@ -380,6 +429,8 @@ async def _handle_stream_tools(
     model: str,
     req: ChatCompletionRequest,
     complexity_info=None,
+    *,
+    app_config=None,
 ):
     """Stream a raw OpenAI-compat function-calling response via SSE.
 
@@ -397,6 +448,7 @@ async def _handle_stream_tools(
     from openjarvis.server.cloud_router import is_cloud_model
 
     messages = _to_messages(req.messages)
+    messages = _ensure_identity_prompt(messages, app_config)
     chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
     use_cloud = is_cloud_model(model)
 
@@ -491,6 +543,7 @@ async def _handle_stream(
     complexity_info=None,
     *,
     trace_store=None,
+    app_config=None,
 ):
     """Stream response using SSE format.
 
@@ -509,6 +562,7 @@ async def _handle_stream(
     )
 
     messages = _to_messages(req.messages)
+    messages = _ensure_identity_prompt(messages, app_config)
     chunk_id = f"chatcmpl-{uuid.uuid4().hex[:12]}"
 
     # Last user message — recorded as the trace query.

src/openjarvis/system/builder.py

@@ -33,6 +33,8 @@ class SystemBuilder:
             self._config = load_config()
 
         self._engine_key: Optional[str] = None
+        self._engine_instance: Optional[InferenceEngine] = None
+        self._engine_instance_key: Optional[str] = None
         self._model: Optional[str] = None
         self._agent_name: Optional[str] = None
         self._tool_names: Optional[List[str]] = None
@@ -50,6 +52,20 @@ class SystemBuilder:
         self._engine_key = key
         return self
 
+    def engine_instance(
+        self, engine: InferenceEngine, key: str = "openai-compat"
+    ) -> SystemBuilder:
+        """Inject a pre-built engine instance, bypassing engine discovery.
+
+        Used by callers that must target one exact endpoint (e.g.
+        ``jarvis eval --base-url``). ``build()`` health-checks the instance
+        and raises a loud error if it is unreachable — it never silently
+        substitutes a different discovered engine.
+        """
+        self._engine_instance = engine
+        self._engine_instance_key = key
+        return self
+
     def model(self, name: str) -> SystemBuilder:
         self._model = name
         return self
@@ -303,6 +319,23 @@ class SystemBuilder:
         return system
 
     def _resolve_engine(self, config: JarvisConfig):
+        # An explicitly injected engine instance always wins and is never
+        # silently replaced: when the caller pinned an endpoint (e.g.
+        # ``jarvis eval --base-url``) and it is down, substituting whatever
+        # other engine discovery finds would silently run against the wrong
+        # model server. Fail loudly instead.
+        if self._engine_instance is not None:
+            engine = self._engine_instance
+            key = self._engine_instance_key or "openai-compat"
+            if not engine.health():
+                host = getattr(engine, "_host", "<unknown host>")
+                raise RuntimeError(
+                    f"Injected engine {key!r} is not reachable at {host} — "
+                    "is the endpoint running and serving GET /v1/models? "
+                    "Refusing to fall back to engine discovery."
+                )
+            return engine, key
+
         from openjarvis.engine._discovery import get_engine
 
         pref = config.intelligence.preferred_engine
@@ -313,7 +346,18 @@ class SystemBuilder:
                 "No inference engine available. "
                 "Make sure an engine is running (e.g. ollama serve)."
             )
-        return resolved[1], resolved[0]
+        resolved_key, engine = resolved
+        if self._engine_key and resolved_key != self._engine_key:
+            # get_engine() falls back to any healthy discovered engine; make
+            # the substitution visible when the caller asked for a specific
+            # engine (observed: requested vllm, silently got ollama@11434).
+            logger.warning(
+                "Requested engine %r is unavailable; using %r at %s instead",
+                self._engine_key,
+                resolved_key,
+                getattr(engine, "_host", "<unknown host>"),
+            )
+        return engine, resolved_key
 
     def _resolve_model(self, config: JarvisConfig, engine: InferenceEngine) -> str:
         if self._model:

tests/cli/test_ask_vision.py

@@ -0,0 +1,128 @@
+"""CLI-level regression tests for ``jarvis ask`` vision input.
+
+The unit tests in ``tests/test_vision.py`` cover the ``Message.images`` ->
+``messages_to_dicts`` serialization contract in isolation. These tests lock
+the *end-to-end CLI wiring*: that ``--image`` reads a file, base64-encodes it,
+attaches it to the final user ``Message``, and that the bytes actually reach
+``engine.generate()`` -- and that the local-first privacy guard fires only for
+non-local engines.
+"""
+
+from __future__ import annotations
+
+import base64
+import importlib
+from pathlib import Path
+from typing import Any
+
+from click.testing import CliRunner
+
+from openjarvis.cli import cli
+from openjarvis.core.config import JarvisConfig
+from openjarvis.core.types import Role
+
+# Import the module (not the Click command attribute) so we can monkeypatch
+# the names it looks up at call time.
+_ask_mod = importlib.import_module("openjarvis.cli.ask")
+
+# A minimal but valid 1x1 PNG so ``click.Path(exists=True)`` is satisfied and
+# the bytes are deterministic.
+_PNG_BYTES = base64.b64decode(
+    "iVBORw0KGgoAAAANSUhEUgAAAAEAAAABCAQAAAC1HAwCAAAAC0lEQVR42mNk"
+    "+M9QDwADhgGAWjR9awAAAABJRU5ErkJggg=="
+)
+
+
+class _RecordingEngine:
+    """A fake engine that records the messages handed to ``generate()``."""
+
+    def __init__(self) -> None:
+        self.engine_id = "mock"
+        self.received: list[Any] = []
+
+    def health(self) -> bool:
+        return True
+
+    def list_models(self) -> list[str]:
+        return ["test-model"]
+
+    def generate(self, messages, *, model=None, **kwargs):
+        # Capture the exact Message objects the CLI built so the test can
+        # assert the image bytes reached the engine boundary.
+        self.received = list(messages)
+        return {
+            "content": "a 1x1 pixel",
+            "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+            "model": "test-model",
+            "finish_reason": "stop",
+        }
+
+
+def _patch_ask(monkeypatch, tmp_path: Path, *, engine_name: str) -> _RecordingEngine:
+    """Wire ``jarvis ask`` to a recording engine reported under ``engine_name``."""
+    cfg = JarvisConfig()
+    cfg.telemetry.db_path = str(tmp_path / "telemetry.db")
+    # Keep memory context out of the picture so the user message we inspect is
+    # the one the CLI built directly from the query + image.
+    cfg.agent.context_from_memory = False
+    monkeypatch.setattr(_ask_mod, "load_config", lambda: cfg)
+
+    engine = _RecordingEngine()
+    monkeypatch.setattr(_ask_mod, "get_engine", lambda *a, **kw: (engine_name, engine))
+    monkeypatch.setattr(_ask_mod, "discover_engines", lambda c: [(engine_name, engine)])
+    monkeypatch.setattr(
+        _ask_mod, "discover_models", lambda e: {engine_name: ["test-model"]}
+    )
+    return engine
+
+
+def _write_png(tmp_path: Path) -> tuple[Path, str]:
+    img = tmp_path / "pixel.png"
+    img.write_bytes(_PNG_BYTES)
+    return img, base64.b64encode(_PNG_BYTES).decode("ascii")
+
+
+def test_image_reaches_engine_payload(monkeypatch, tmp_path: Path) -> None:
+    engine = _patch_ask(monkeypatch, tmp_path, engine_name="ollama")
+    img, expected_b64 = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    # The CLI must have routed to direct mode and called the engine.
+    assert engine.received, "engine.generate() was never called"
+    user_msgs = [m for m in engine.received if m.role == Role.USER]
+    assert user_msgs, "no USER message reached the engine"
+    assert user_msgs[-1].images == [expected_b64]
+
+
+def test_privacy_warning_for_non_local_engine(monkeypatch, tmp_path: Path) -> None:
+    engine = _patch_ask(monkeypatch, tmp_path, engine_name="openai")
+    img, expected_b64 = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    assert "Privacy warning" in result.output
+    # The warning is informational; the image must still be delivered.
+    user_msgs = [m for m in engine.received if m.role == Role.USER]
+    assert user_msgs and user_msgs[-1].images == [expected_b64]
+
+
+def test_no_privacy_warning_for_local_engine(monkeypatch, tmp_path: Path) -> None:
+    _patch_ask(monkeypatch, tmp_path, engine_name="ollama")
+    img, _ = _write_png(tmp_path)
+
+    result = CliRunner().invoke(
+        cli,
+        ["ask", "-i", str(img), "--no-context", "--agent", "", "describe this"],
+    )
+
+    assert result.exit_code == 0, result.output
+    assert "Privacy warning" not in result.output

tests/connectors/test_connectors_router_oauth.py

@@ -0,0 +1,273 @@
+"""Regression tests for the connectors-router OAuth flow (issue #512).
+
+These tests reproduce the three coupled defects that prevented Google Drive
+(and its Google siblings) from ever completing OAuth and appearing in Data
+Sources, and assert the fixed behaviour:
+
+(A/B) ``POST /connect`` with a pasted ``client_id:client_secret`` pair must
+      persist the client credentials and return an ``oauth_required`` directive
+      pointing at ``/oauth/start`` — NOT silently spawn a background browser
+      thread and report a perpetual ``pending`` state.
+(C-1) ``GET /oauth/start`` must return a redirect to the provider's consent
+      page (regression: HTTP 422 because ``request: Request`` was mis-bound as
+      a query param under ``from __future__ import annotations`` + a local
+      ``Request`` import).
+(C-2) ``GET /oauth/callback`` must read ``request.base_url`` and exchange the
+      code for tokens without crashing (regression: ``request`` defaulted to
+      ``None`` → ``AttributeError``), persisting the access token to every
+      Google credential file and flipping ``is_connected()`` to True.
+
+All tests are hermetic: the connectors directory, the shared Google
+credentials path, and every Google connector's default credentials path are
+redirected to ``tmp_path`` so the suite neither depends on nor pollutes
+``~/.openjarvis/connectors`` (a real source of spurious failures — see the
+verifier note on ``resolve_google_credentials`` silently substituting the
+shared file when the caller-supplied path does not yet exist on disk).
+"""
+
+from __future__ import annotations
+
+import json
+from pathlib import Path
+from typing import Any, Iterator
+from unittest.mock import patch
+
+import pytest
+
+fastapi = pytest.importorskip("fastapi", reason="requires the 'server' extra")
+from fastapi import FastAPI  # noqa: E402
+from fastapi.testclient import TestClient  # noqa: E402
+
+_CLIENT_PAIR = "myid-123.apps.googleusercontent.com:GOCSPX-secret"
+_CLIENT_ID = "myid-123.apps.googleusercontent.com"
+
+_ALL_GOOGLE_FILES = (
+    "google.json",
+    "gdrive.json",
+    "gcalendar.json",
+    "gcontacts.json",
+    "gmail.json",
+    "google_tasks.json",
+)
+
+
+@pytest.fixture()
+def hermetic_connectors(tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> Path:
+    """Redirect all Google credential paths into *tmp_path*.
+
+    Ensures connector instances created by the router's ``_get_or_create``
+    resolve to the same directory the OAuth callback writes to, and that the
+    test leaves ``~/.openjarvis`` untouched.
+
+    Why this is more than a one-line monkeypatch: the autouse registry-clear
+    fixture causes ``_ensure_connectors_registered()`` to ``importlib.reload``
+    each connector module on the first router call, which re-executes the
+    module body. To survive that reload we patch ``DEFAULT_CONFIG_DIR`` at its
+    *source* (``openjarvis.core.config``) — every connector re-derives
+    ``_DEFAULT_CREDENTIALS_PATH`` from it on reload, so the tmp dir sticks.
+    We also pre-register + pre-reload the connectors inside the fixture so the
+    reload happens while the patch is live, then reset module state on
+    teardown so a later test that imports these modules fresh is unaffected.
+    """
+    import importlib
+    import sys
+
+    import openjarvis.connectors.oauth as oauth_mod
+    import openjarvis.core.config as config_mod
+    import openjarvis.server.connectors_router as router_mod
+    from openjarvis.core.registry import ConnectorRegistry
+
+    conn_dir = tmp_path / "connectors"
+    conn_dir.mkdir(parents=True, exist_ok=True)
+
+    monkeypatch.setattr(config_mod, "DEFAULT_CONFIG_DIR", tmp_path)
+    monkeypatch.setattr(oauth_mod, "_CONNECTORS_DIR", conn_dir)
+    monkeypatch.setattr(
+        oauth_mod, "_SHARED_GOOGLE_CREDENTIALS_PATH", str(conn_dir / "google.json")
+    )
+
+    # Force the connector modules to re-derive their default paths from the
+    # patched DEFAULT_CONFIG_DIR now, before any request, and register them so
+    # the router's lazy reload-on-empty-registry path is a no-op.
+    google_mods = [
+        "openjarvis.connectors.gdrive",
+        "openjarvis.connectors.gcalendar",
+        "openjarvis.connectors.gcontacts",
+        "openjarvis.connectors.gmail",
+        "openjarvis.connectors.google_tasks",
+    ]
+    for name in google_mods:
+        if name in sys.modules:
+            importlib.reload(sys.modules[name])
+
+    router_mod._instances.clear()
+    yield conn_dir
+    router_mod._instances.clear()
+    ConnectorRegistry.clear()
+    # Restore the connector modules to their real (unpatched) default paths so
+    # subsequent tests in the same process see ~/.openjarvis again.
+    for name in google_mods:
+        if name in sys.modules:
+            importlib.reload(sys.modules[name])
+
+
+@pytest.fixture()
+def client(hermetic_connectors: Path) -> Iterator[TestClient]:
+    from openjarvis.server.connectors_router import create_connectors_router
+
+    app = FastAPI()
+    app.include_router(create_connectors_router())
+    with TestClient(app) as c:
+        yield c
+
+
+# ---------------------------------------------------------------------------
+# Defect A/B — POST /connect must not silently spawn a background OAuth thread
+# ---------------------------------------------------------------------------
+
+
+@pytest.mark.parametrize(
+    "connector_id", ["gdrive", "gcalendar", "gcontacts", "gmail", "google_tasks"]
+)
+def test_connect_client_pair_returns_oauth_required_no_browser(
+    client: TestClient, hermetic_connectors: Path, connector_id: str
+) -> None:
+    """Pasting client_id:secret persists creds + asks the UI to run the flow.
+
+    Covers every Google connector that shares the OAuth provider, proving the
+    sibling connectors are fixed too (not just gdrive).
+    """
+    with patch("openjarvis.core.open_browser") as mock_browser:
+        resp = client.post(
+            f"/v1/connectors/{connector_id}/connect", json={"code": _CLIENT_PAIR}
+        )
+
+    assert resp.status_code == 200, resp.text
+    body = resp.json()
+    assert body["status"] == "oauth_required"
+    assert body["oauth_start"] == f"/v1/connectors/{connector_id}/oauth/start"
+    assert body["connected"] is False
+    # No fire-and-forget browser thread (the root cause of "nothing happens").
+    mock_browser.assert_not_called()
+
+    # Client credentials persisted to EVERY Google credential file so a single
+    # consent covers all Google connectors.
+    for filename in _ALL_GOOGLE_FILES:
+        path = hermetic_connectors / filename
+        assert path.exists(), f"{filename} not written"
+        assert json.loads(path.read_text())["client_id"] == _CLIENT_ID
+
+
+def test_connect_malformed_client_pair_raises_400(
+    client: TestClient,
+) -> None:
+    """A blank secret surfaces an actionable 400 — not a silent pending state."""
+    resp = client.post(
+        "/v1/connectors/gdrive/connect",
+        json={"code": "myid-123.apps.googleusercontent.com:"},
+    )
+    assert resp.status_code == 400
+    assert "CLIENT_ID:CLIENT_SECRET" in resp.json()["detail"]
+
+
+def test_connect_raw_token_still_handled(
+    client: TestClient, hermetic_connectors: Path
+) -> None:
+    """A raw token (not a client pair) still flows through handle_callback."""
+    resp = client.post(
+        "/v1/connectors/gdrive/connect", json={"token": "ya29.raw-access-token"}
+    )
+    assert resp.status_code == 200, resp.text
+    saved = json.loads((hermetic_connectors / "gdrive.json").read_text())
+    assert saved.get("token") == "ya29.raw-access-token"
+
+
+# ---------------------------------------------------------------------------
+# Defect C-1 — GET /oauth/start must redirect (was HTTP 422)
+# ---------------------------------------------------------------------------
+
+
+def test_oauth_start_redirects_to_consent(
+    client: TestClient,
+) -> None:
+    # First save client creds via the connect call.
+    client.post("/v1/connectors/gdrive/connect", json={"code": _CLIENT_PAIR})
+
+    resp = client.get("/v1/connectors/gdrive/oauth/start", follow_redirects=False)
+    # FastAPI's RedirectResponse defaults to 307; any 3xx is a pass (was 422).
+    assert resp.status_code in (302, 307), resp.text
+    location = resp.headers["location"]
+    assert location.startswith("https://accounts.google.com/o/oauth2/v2/auth")
+    assert _CLIENT_ID in location
+    # redirect_uri must point back at OUR in-process callback.
+    assert "oauth%2Fcallback" in location or "oauth/callback" in location
+
+
+def test_oauth_start_without_creds_returns_400(client: TestClient) -> None:
+    resp = client.get("/v1/connectors/gdrive/oauth/start", follow_redirects=False)
+    assert resp.status_code == 400
+    assert "client credentials" in resp.json()["detail"].lower()
+
+
+# ---------------------------------------------------------------------------
+# Defect C-2 — GET /oauth/callback must exchange + persist (was 500 on None)
+# ---------------------------------------------------------------------------
+
+
+def test_oauth_callback_exchanges_and_connects(
+    client: TestClient, hermetic_connectors: Path
+) -> None:
+    import openjarvis.connectors.oauth as oauth_mod
+
+    client.post("/v1/connectors/gdrive/connect", json={"code": _CLIENT_PAIR})
+
+    fake_tokens = {
+        "access_token": "ya29.REAL",
+        "refresh_token": "1//REAL",
+        "token_type": "Bearer",
+        "expires_in": 3600,
+    }
+    with patch.object(oauth_mod, "_exchange_token", return_value=fake_tokens) as ex:
+        resp = client.get("/v1/connectors/gdrive/oauth/callback?code=authcode123")
+
+    assert resp.status_code == 200, resp.text
+    assert "Connected!" in resp.text
+    ex.assert_called_once()
+
+    # Access token written to ALL Google credential files.
+    for filename in _ALL_GOOGLE_FILES:
+        saved = json.loads((hermetic_connectors / filename).read_text())
+        assert saved["access_token"] == "ya29.REAL"
+        assert saved["refresh_token"] == "1//REAL"
+
+    # The connector now reports connected, and GET /connectors agrees.
+    from openjarvis.connectors.gdrive import GDriveConnector
+
+    assert GDriveConnector().is_connected() is True
+
+    listing = client.get("/v1/connectors").json()["connectors"]
+    gdrive = next(c for c in listing if c["connector_id"] == "gdrive")
+    assert gdrive["connected"] is True
+
+
+def test_oauth_callback_error_param_renders_failure(client: TestClient) -> None:
+    resp = client.get("/v1/connectors/gdrive/oauth/callback?error=access_denied")
+    assert resp.status_code == 400
+    assert "access_denied" in resp.text
+
+
+def test_oauth_callback_exchange_failure_renders_error(
+    client: TestClient,
+) -> None:
+    import openjarvis.connectors.oauth as oauth_mod
+
+    client.post("/v1/connectors/gdrive/connect", json={"code": _CLIENT_PAIR})
+
+    def _boom(*_a: Any, **_k: Any) -> dict[str, Any]:
+        raise RuntimeError("token endpoint 400")
+
+    with patch.object(oauth_mod, "_exchange_token", side_effect=_boom):
+        resp = client.get("/v1/connectors/gdrive/oauth/callback?code=bad")
+
+    assert resp.status_code == 500
+    assert "Token Exchange Failed" in resp.text

tests/connectors/test_oauth_flow.py

@@ -30,19 +30,35 @@ def test_exchange_google_token_calls_endpoint() -> None:
     mock_post.assert_called_once()
 
 
-def test_gdrive_handle_callback_triggers_oauth(tmp_path: Path) -> None:
+def test_gdrive_handle_callback_persists_creds_no_background_flow(
+    tmp_path: Path,
+) -> None:
+    """A pasted client pair persists creds ONLY — no silent browser thread.
+
+    Regression for issue #512: the previous implementation spawned a daemon
+    thread that popped a browser and ran its own localhost:8789 callback
+    server. That thread failed silently in the bundled desktop context, so the
+    connector never gained an access token. ``handle_callback`` must now only
+    save the client_id/secret; the in-process server flow owns the consent
+    round-trip. We assert ``open_browser`` is never invoked.
+    """
     from openjarvis.connectors.gdrive import GDriveConnector
+    from openjarvis.connectors.oauth import load_tokens
 
     creds = str(tmp_path / "gdrive.json")
     conn = GDriveConnector(credentials_path=creds)
 
-    with patch("openjarvis.connectors.gdrive.run_oauth_flow") as mock_flow:
-        mock_flow.return_value = {"access_token": "ya29.test"}
+    with patch("openjarvis.core.open_browser") as mock_browser:
         conn.handle_callback("test-id.apps.googleusercontent.com:test-secret")
 
-    mock_flow.assert_called_once()
-    call_kwargs = mock_flow.call_args
-    assert "test-id.apps.googleusercontent.com" in str(call_kwargs)
+    mock_browser.assert_not_called()
+    tokens = load_tokens(creds)
+    assert tokens is not None
+    assert tokens["client_id"] == "test-id.apps.googleusercontent.com"
+    assert tokens["client_secret"] == "test-secret"
+    # No access token yet — that arrives via /oauth/callback.
+    assert not tokens.get("access_token")
+    assert conn.is_connected() is False
 
 
 def test_gdrive_is_connected_requires_access_token(tmp_path: Path) -> None:
@@ -61,48 +77,46 @@ def test_gdrive_is_connected_requires_access_token(tmp_path: Path) -> None:
     assert conn.is_connected() is True
 
 
-def test_gcalendar_handle_callback_triggers_oauth(tmp_path: Path) -> None:
+def test_gcalendar_handle_callback_persists_creds_no_background_flow(
+    tmp_path: Path,
+) -> None:
+    """Sibling connector shares the fix: creds saved, no browser thread (#512)."""
     from openjarvis.connectors.gcalendar import GCalendarConnector
+    from openjarvis.connectors.oauth import load_tokens
 
     creds = str(tmp_path / "gcalendar.json")
     conn = GCalendarConnector(credentials_path=creds)
 
-    with patch("openjarvis.connectors.gcalendar.run_oauth_flow") as mock_flow:
-        mock_flow.return_value = {"access_token": "ya29.test"}
+    with patch("openjarvis.core.open_browser") as mock_browser:
         conn.handle_callback("test-id.apps.googleusercontent.com:test-secret")
 
-    mock_flow.assert_called_once()
-
-
-def test_gcontacts_handle_callback_triggers_oauth(tmp_path: Path) -> None:
-    from openjarvis.connectors.gcontacts import GContactsConnector
-
-    creds = str(tmp_path / "gcontacts.json")
-    conn = GContactsConnector(credentials_path=creds)
-
-    with patch("openjarvis.connectors.gcontacts.run_oauth_flow") as mock_flow:
-        mock_flow.return_value = {"access_token": "ya29.test"}
-        conn.handle_callback("test-id.apps.googleusercontent.com:test-secret")
-
-    mock_flow.assert_called_once()
-
-
-def test_gdrive_handle_callback_fallback_on_failure(tmp_path: Path) -> None:
-    from openjarvis.connectors.gdrive import GDriveConnector
-    from openjarvis.connectors.oauth import load_tokens
-
-    creds = str(tmp_path / "gdrive.json")
-    conn = GDriveConnector(credentials_path=creds)
-
-    with patch("openjarvis.connectors.gdrive.run_oauth_flow") as mock_flow:
-        mock_flow.side_effect = RuntimeError("OAuth failed")
-        conn.handle_callback("test-id.apps.googleusercontent.com:test-secret")
-
-    # Should have saved client_id and client_secret as fallback
+    mock_browser.assert_not_called()
     tokens = load_tokens(creds)
     assert tokens is not None
     assert tokens["client_id"] == "test-id.apps.googleusercontent.com"
     assert tokens["client_secret"] == "test-secret"
+    assert conn.is_connected() is False
+
+
+def test_gcontacts_handle_callback_persists_creds_no_background_flow(
+    tmp_path: Path,
+) -> None:
+    """Sibling connector shares the fix: creds saved, no browser thread (#512)."""
+    from openjarvis.connectors.gcontacts import GContactsConnector
+    from openjarvis.connectors.oauth import load_tokens
+
+    creds = str(tmp_path / "gcontacts.json")
+    conn = GContactsConnector(credentials_path=creds)
+
+    with patch("openjarvis.core.open_browser") as mock_browser:
+        conn.handle_callback("test-id.apps.googleusercontent.com:test-secret")
+
+    mock_browser.assert_not_called()
+    tokens = load_tokens(creds)
+    assert tokens is not None
+    assert tokens["client_id"] == "test-id.apps.googleusercontent.com"
+    assert tokens["client_secret"] == "test-secret"
+    assert conn.is_connected() is False
 
 
 def test_gdrive_handle_callback_raw_token(tmp_path: Path) -> None:

tests/core/test_config.py

@@ -237,6 +237,14 @@ class TestAgentConfigNew:
             or isinstance(getattr(ac.__class__, "temperature", None), property) is False
         )
 
+    def test_default_system_prompt_anchors_identity(self) -> None:
+        """#540: the hardened wording must name OpenJarvis and explicitly
+        deny the model's training identity so distilled models stop
+        claiming to be Claude/ChatGPT/etc."""
+        prompt = AgentConfig().default_system_prompt
+        assert "OpenJarvis" in prompt
+        assert "not Claude" in prompt
+
 
 class TestNestedEngineConfig:
     def test_nested_access(self) -> None:
@@ -561,6 +569,7 @@ class TestWhatsAppBaileysChannelConfig:
 
 def test_mining_config_absent_means_none(tmp_path):
     from openjarvis.core.config import load_config
+
     cfg_path = tmp_path / "config.toml"
     cfg_path.write_text("")  # empty config
     cfg = load_config(cfg_path)

tests/engine/test_cloud.py

@@ -9,9 +9,13 @@ import pytest
 
 from openjarvis.core.registry import EngineRegistry
 from openjarvis.core.types import Message, Role
+from openjarvis.engine._base import EngineConnectionError
 from openjarvis.engine.cloud import (
     CloudEngine,
     _is_codex_model,
+    _is_deepseek_model,
+    _is_openai_model,
+    _is_openrouter_model,
     estimate_cost,
 )
 
@@ -437,3 +441,187 @@ class TestOpenRouterToolForwarding:
         assert result["tool_calls"][0]["id"] == "call_1"
         assert result["tool_calls"][0]["function"]["name"] == "get_weather"
         assert result["tool_calls"][0]["function"]["arguments"] == '{"city": "NYC"}'
+
+
+class TestCloudEngineCanServe:
+    """#532: can_serve gates on the per-provider client, not just health().
+
+    health() is True whenever *any* provider client is configured, but a
+    request for a gpt-* model still needs the OpenAI client specifically — so
+    engine selection must not pick the cloud engine for a model whose provider
+    client is missing.
+    """
+
+    @staticmethod
+    def _engine(**clients: object) -> CloudEngine:
+        eng = CloudEngine.__new__(CloudEngine)  # bypass real client init
+        for name in (
+            "_openai_client",
+            "_anthropic_client",
+            "_google_client",
+            "_openrouter_client",
+            "_minimax_client",
+            "_deepseek_client",
+            "_codex_client",
+        ):
+            setattr(eng, name, clients.get(name))
+        return eng
+
+    def test_openai_only_serves_openai_models(self) -> None:
+        eng = self._engine(_openai_client=object())
+        assert eng.can_serve("gpt-4o") is True
+        assert eng.can_serve("claude-sonnet-4") is False
+        assert eng.can_serve("gemini-2.5-pro") is False
+        assert eng.can_serve("openrouter/openai/gpt-4o") is False
+
+    def test_openai_key_does_not_claim_local_models(self) -> None:
+        """#335: with only the OpenAI client set (e.g. a present-but-dummy
+        OPENAI_API_KEY), the cloud engine must NOT claim it can serve a local
+        Ollama model name — otherwise it gets mis-selected as a fallback when
+        the local engine is transiently down and dies with "OpenAI client not
+        available". Only genuine OpenAI models route to the OpenAI client.
+        """
+        eng = self._engine(_openai_client=object())
+        # Local Ollama / unrecognized names are NOT served by the cloud engine.
+        assert eng.can_serve("qwen3.5:0.8b") is False
+        assert eng.can_serve("llama3.2") is False
+        assert eng.can_serve("mistral") is False
+        assert eng.can_serve("phi3:mini") is False
+        assert eng.can_serve("some-unknown-model") is False
+        # Genuine OpenAI families still served.
+        assert eng.can_serve("gpt-4o") is True
+        assert eng.can_serve("gpt-5.4") is True
+        assert eng.can_serve("o3-mini") is True
+
+    def test_unknown_model_not_served_even_with_all_clients(self) -> None:
+        """#335: an unrecognized model is declined regardless of how many
+        provider clients are configured — it never falls through to OpenAI."""
+        eng = self._engine(
+            _openai_client=object(),
+            _anthropic_client=object(),
+            _google_client=object(),
+            _minimax_client=object(),
+            _deepseek_client=object(),
+        )
+        assert eng.can_serve("qwen3.5:0.8b") is False
+        assert eng.can_serve("totally-made-up") is False
+
+    def test_anthropic_only_serves_anthropic_models(self) -> None:
+        eng = self._engine(_anthropic_client=object())
+        assert eng.can_serve("claude-sonnet-4") is True
+        assert eng.can_serve("gpt-4o") is False
+
+    def test_deepseek_only_serves_deepseek_models(self) -> None:
+        """The DeepSeek client serves deepseek-* models (and only those)."""
+        eng = self._engine(_deepseek_client=object())
+        assert eng.can_serve("deepseek-v4-flash") is True
+        assert eng.can_serve("deepseek-v4-pro") is True
+        assert eng.can_serve("DeepSeek-V4-Pro") is True  # case-insensitive
+        assert eng.can_serve("gpt-4o") is False
+        # OpenRouter-prefixed deepseek is NOT the direct DeepSeek provider.
+        assert eng.can_serve("openrouter/deepseek/deepseek-r1") is False
+
+
+class TestCloudEngineDeepSeek:
+    """PR #504: DeepSeek as a first-class cloud provider (OpenAI-compatible)."""
+
+    def test_is_deepseek_model_predicate(self) -> None:
+        assert _is_deepseek_model("deepseek-v4-flash") is True
+        assert _is_deepseek_model("deepseek-v4-pro") is True
+        assert _is_deepseek_model("DeepSeek-V4-Pro") is True  # case-insensitive
+        assert _is_deepseek_model("gpt-4o") is False
+        # No predicate collision: openrouter/deepseek/* belongs to OpenRouter.
+        assert _is_deepseek_model("openrouter/deepseek/deepseek-r1") is False
+        assert _is_openrouter_model("openrouter/deepseek/deepseek-r1") is True
+        # And a deepseek name is not mistaken for an OpenAI model.
+        assert _is_openai_model("deepseek-v4-pro") is False
+
+    def test_pricing_entries_present(self) -> None:
+        assert estimate_cost("deepseek-v4-flash", 1_000_000, 1_000_000) == (
+            pytest.approx(1.37)  # 0.27 + 1.10
+        )
+        assert estimate_cost("deepseek-v4-pro", 1_000_000, 1_000_000) == (
+            pytest.approx(2.74)  # 0.55 + 2.19
+        )
+
+    def test_init_wires_deepseek_client(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        """DEEPSEEK_API_KEY builds an openai client pointed at api.deepseek.com."""
+        for var in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY"):
+            monkeypatch.delenv(var, raising=False)
+        monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-deepseek-test")
+
+        fake_openai = mock.MagicMock()
+        with mock.patch.dict("sys.modules", {"openai": fake_openai}):
+            EngineRegistry.register_value("cloud", CloudEngine)
+            engine = CloudEngine()
+
+        fake_openai.OpenAI.assert_any_call(
+            base_url="https://api.deepseek.com/v1",
+            api_key="sk-deepseek-test",
+        )
+        assert engine._deepseek_client is not None
+
+    def test_health_and_list_models_gated_on_deepseek_key(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        for var in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY"):
+            monkeypatch.delenv(var, raising=False)
+        monkeypatch.setenv("DEEPSEEK_API_KEY", "sk-deepseek-test")
+
+        fake_openai = mock.MagicMock()
+        with mock.patch.dict("sys.modules", {"openai": fake_openai}):
+            EngineRegistry.register_value("cloud", CloudEngine)
+            engine = CloudEngine()
+
+        assert engine.health() is True
+        models = engine.list_models()
+        assert "deepseek-v4-flash" in models
+        assert "deepseek-v4-pro" in models
+        # can_serve must agree with list_models (regression for the missing
+        # _client_for_model deepseek branch flagged by the #504 verifier).
+        assert engine.can_serve("deepseek-v4-pro") is True
+        assert engine.can_serve("deepseek-v4-flash") is True
+
+    def test_generate_routes_to_deepseek_client(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        for var in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "DEEPSEEK_API_KEY"):
+            monkeypatch.delenv(var, raising=False)
+
+        fake_usage = SimpleNamespace(
+            prompt_tokens=7, completion_tokens=3, total_tokens=10
+        )
+        fake_choice = SimpleNamespace(
+            message=SimpleNamespace(content="ds-hello"),
+            finish_reason="stop",
+        )
+        fake_resp = SimpleNamespace(
+            choices=[fake_choice], usage=fake_usage, model="deepseek-v4-pro"
+        )
+        fake_client = mock.MagicMock()
+        fake_client.chat.completions.create.return_value = fake_resp
+
+        EngineRegistry.register_value("cloud", CloudEngine)
+        engine = CloudEngine()
+        engine._deepseek_client = fake_client
+
+        result = engine.generate(
+            [Message(role=Role.USER, content="Hi")], model="deepseek-v4-pro"
+        )
+        assert result["content"] == "ds-hello"
+        assert result["usage"]["prompt_tokens"] == 7
+        # Routed to the DeepSeek client, not OpenAI.
+        fake_client.chat.completions.create.assert_called_once()
+
+    def test_generate_without_client_raises(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        for var in ("OPENAI_API_KEY", "ANTHROPIC_API_KEY", "DEEPSEEK_API_KEY"):
+            monkeypatch.delenv(var, raising=False)
+        EngineRegistry.register_value("cloud", CloudEngine)
+        engine = CloudEngine()
+        assert engine._deepseek_client is None
+        with pytest.raises(EngineConnectionError):
+            engine.generate(
+                [Message(role=Role.USER, content="Hi")], model="deepseek-v4-pro"
+            )

tests/engine/test_discovery.py

@@ -174,6 +174,99 @@ class TestGetEngine:
         assert result is not None
         assert result[0] == "running"
 
+    def test_skips_engine_that_cannot_serve_model(self) -> None:
+        """#532: a healthy engine that can't serve the requested model is
+        skipped for one that can — this is what stops the cloud fallback being
+        chosen (when the local engine is down) for a model whose provider
+        client is missing.
+        """
+        _reg("picky", "picky")
+        _reg("local", "local")
+
+        class _Picky(_FakeEngine):
+            def can_serve(self, model: str) -> bool:
+                return model == "servable"
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "picky"
+
+        def _make(k, c):  # noqa: ANN001
+            if k == "picky":
+                return _Picky(healthy=True)
+            return _FakeEngine(healthy=(k == "local"))
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=_make,
+        ):
+            # "picky" is healthy but cannot serve "other" -> fall back to "local"
+            result = get_engine(cfg, model="other")
+        assert result is not None
+        assert result[0] == "local"
+
+    def test_dummy_openai_key_does_not_misroute_local_model(
+        self, monkeypatch: object
+    ) -> None:
+        """#335: a present-but-dummy OPENAI_API_KEY + a down local engine must
+        NOT cause a local Ollama model to be routed to the cloud engine.
+
+        Before the fix, CloudEngine.can_serve('qwen3.5:0.8b') returned True
+        whenever any OpenAI client existed (even a junk key), so get_engine
+        picked 'cloud' and the request later died with "OpenAI client not
+        available". With the strict _client_for_model fall-through it returns
+        None for unrecognized names, so get_engine declines cloud and (with the
+        local engine down) returns None — surfacing a "start your local engine"
+        failure instead.
+        """
+        from openjarvis.engine.cloud import CloudEngine
+
+        _reg("ollama", "ollama")
+        EngineRegistry.register_value("cloud", CloudEngine)
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "ollama"
+
+        def _make(k, c):  # noqa: ANN001
+            if k == "ollama":
+                # Local engine is down (post-restart Ollama not yet up).
+                return _FakeEngine(healthy=False, models=["qwen3.5:0.8b"])
+            # Real CloudEngine with only a (dummy) OpenAI client wired.
+            eng = CloudEngine.__new__(CloudEngine)
+            for name in (
+                "_openai_client",
+                "_anthropic_client",
+                "_google_client",
+                "_openrouter_client",
+                "_minimax_client",
+                "_deepseek_client",
+                "_codex_client",
+            ):
+                setattr(eng, name, object() if name == "_openai_client" else None)
+            return eng
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=_make,
+        ):
+            result = get_engine(cfg, model="qwen3.5:0.8b")
+        # Cloud must NOT be selected for a local model name.
+        assert result is None
+
+    def test_model_none_preserves_model_agnostic_selection(self) -> None:
+        """model=None keeps the legacy behaviour: first healthy engine wins."""
+        _reg("primary", "primary")
+
+        cfg = JarvisConfig()
+        cfg.engine.default = "primary"
+
+        with mock.patch(
+            "openjarvis.engine._discovery._make_engine",
+            side_effect=lambda k, c: _FakeEngine(healthy=True),  # noqa: ANN001
+        ):
+            result = get_engine(cfg, model=None)
+        assert result is not None
+        assert result[0] == "primary"
+
 
 class TestMiningSidecarEngineHandoff:
     """Engine discovery picks up (or ignores) a mining sidecar at runtime."""

tests/engine/test_openai_compat_api_key.py

@@ -0,0 +1,109 @@
+"""API-key (Authorization header) support in the OpenAI-compat engine base."""
+
+from __future__ import annotations
+
+import httpx
+import pytest
+import respx
+
+from openjarvis.core.types import Message, Role
+from openjarvis.engine.openai_compat_engines import (
+    OpenAICompatEngine,
+    VLLMEngine,
+    normalize_openai_base_url,
+)
+
+_CHAT_RESPONSE = {
+    "choices": [{"message": {"content": "ok"}, "finish_reason": "stop"}],
+    "usage": {"prompt_tokens": 1, "completion_tokens": 1, "total_tokens": 2},
+    "model": "m",
+}
+
+
+class TestAuthorizationHeader:
+    def test_bearer_header_sent_when_api_key_set(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_no_authorization_header_without_api_key(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.post("http://testhost:9000/v1/chat/completions").mock(
+                return_value=httpx.Response(200, json=_CHAT_RESPONSE)
+            )
+            engine.generate([Message(role=Role.USER, content="hi")], model="m")
+        assert "authorization" not in route.calls.last.request.headers
+
+    def test_health_check_sends_bearer_header(self) -> None:
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-test")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-test"
+
+    def test_env_var_fallback_sanitizes_hyphen(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        # engine_id "openai-compat" must map to OPENAI_COMPAT_API_KEY —
+        # shells cannot set hyphenated env-var names.
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000")
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-env"
+
+    def test_vllm_env_var_fallback(self, monkeypatch: pytest.MonkeyPatch) -> None:
+        monkeypatch.setenv("VLLM_API_KEY", "sk-vllm")
+        engine = VLLMEngine(host="http://testhost:8000")
+        with respx.mock:
+            route = respx.get("http://testhost:8000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            engine.health()
+        assert route.calls.last.request.headers["Authorization"] == "Bearer sk-vllm"
+
+    def test_explicit_api_key_beats_env_var(
+        self, monkeypatch: pytest.MonkeyPatch
+    ) -> None:
+        monkeypatch.setenv("OPENAI_COMPAT_API_KEY", "sk-env")
+        engine = OpenAICompatEngine(host="http://testhost:9000", api_key="sk-explicit")
+        assert engine._api_key == "sk-explicit"
+
+
+class TestNormalizeOpenAIBaseUrl:
+    @pytest.mark.parametrize(
+        ("url", "expected"),
+        [
+            ("http://h:8000", "http://h:8000"),
+            ("http://h:8000/", "http://h:8000"),
+            ("http://h:8000/v1", "http://h:8000"),
+            ("http://h:8000/v1/", "http://h:8000"),
+            ("http://h:8000/gateway/v1", "http://h:8000/gateway"),
+            # Only a literal trailing "/v1" is stripped — never other paths.
+            ("http://h:8000/v1x", "http://h:8000/v1x"),
+            ("http://h:8000/v2", "http://h:8000/v2"),
+        ],
+    )
+    def test_normalization(self, url: str, expected: str) -> None:
+        assert normalize_openai_base_url(url) == expected
+
+    def test_engine_requests_have_single_v1_prefix(self) -> None:
+        """End to end: a user-supplied .../v1 URL must not produce /v1/v1."""
+        host = normalize_openai_base_url("http://testhost:9000/v1")
+        engine = OpenAICompatEngine(host=host)
+        with respx.mock:
+            route = respx.get("http://testhost:9000/v1/models").mock(
+                return_value=httpx.Response(200, json={"data": []})
+            )
+            assert engine.health() is True
+        assert route.calls.last.request.url.path == "/v1/models"

tests/evals/core/test_config_timeouts.py

@@ -0,0 +1,72 @@
+"""Tests for terminal-bench harness timeout plumbing through TOML configs."""
+
+from __future__ import annotations
+
+import textwrap
+
+from openjarvis.evals.core.config import expand_suite, load_eval_config
+
+
+def _write(tmp_path, body: str):
+    path = tmp_path / "suite.toml"
+    path.write_text(textwrap.dedent(body))
+    return path
+
+
+BASE = """
+    [meta]
+    name = "timeouts"
+
+    [run]
+    output_dir = "results/"
+    {run_extra}
+
+    [[models]]
+    name = "test-model"
+
+    [[benchmarks]]
+    name = "terminalbench-native"
+    backend = "terminalbench-native"
+    {bench_extra}
+"""
+
+
+class TestTimeoutConfigPlumbing:
+    def test_run_level_timeouts_parse_and_expand(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra=(
+                    "global_agent_timeout_sec = 1200\n"
+                    "    global_timeout_multiplier = 1.5"
+                ),
+                bench_extra="",
+            ),
+        )
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec == 1200.0
+        assert suite.run.global_timeout_multiplier == 1.5
+
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec == 1200.0
+        assert rc.global_timeout_multiplier == 1.5
+
+    def test_benchmark_override_wins(self, tmp_path):
+        path = _write(
+            tmp_path,
+            BASE.format(
+                run_extra="global_agent_timeout_sec = 1200",
+                bench_extra="global_agent_timeout_sec = 300",
+            ),
+        )
+        (rc,) = expand_suite(load_eval_config(path))
+        assert rc.global_agent_timeout_sec == 300.0
+
+    def test_defaults_are_none(self, tmp_path):
+        path = _write(tmp_path, BASE.format(run_extra="", bench_extra=""))
+        suite = load_eval_config(path)
+        assert suite.run.global_agent_timeout_sec is None
+        assert suite.run.global_timeout_multiplier is None
+        (rc,) = expand_suite(suite)
+        assert rc.global_agent_timeout_sec is None
+        assert rc.global_timeout_multiplier is None

tests/evals/test_agentic_runner.py

@@ -9,6 +9,7 @@ from typing import Any, Dict, List
 import pytest
 
 from openjarvis.evals.core.agentic_runner import AgenticRunner, _extract_patch
+from openjarvis.evals.core.environment import TaskEnvironmentError
 
 # ---------------------------------------------------------------------------
 # Mock objects
@@ -50,6 +51,66 @@ class MockFailingAgent:
         raise RuntimeError("Agent error")
 
 
+class MockZeroContactAgent:
+    """Agent that returns without ever contacting the model.
+
+    Mirrors the downstream failure signature: a hung/dead in-container
+    setup yields zero LM events and zero token usage, while run_tests
+    still stamps is_resolved=False.
+    """
+
+    def ask(self, query: str) -> dict:
+        return {"content": "setup log tail ...", "usage": {}}
+
+
+class FailingTaskEnv:
+    """Task env whose __enter__ fails like a tmux/compose breakage."""
+
+    def __init__(self, metadata: Dict[str, Any]) -> None:
+        self._metadata = metadata
+
+    def __enter__(self) -> "FailingTaskEnv":
+        message = (
+            "Task 't1': required binary 'tmux' is not usable in task image "
+            "'tb__t1__client'"
+        )
+        self._metadata["harness_error"] = message
+        raise TaskEnvironmentError(message)
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+
+class ResolvingTaskEnv:
+    """Task env that stamps is_resolved into metadata like run_tests does."""
+
+    def __init__(self, metadata: Dict[str, Any], is_resolved: bool) -> None:
+        self._metadata = metadata
+        self._is_resolved = is_resolved
+
+    def __enter__(self) -> "ResolvingTaskEnv":
+        return self
+
+    def __exit__(self, *args: Any) -> None:
+        return None
+
+    def run_tests(self):
+        self._metadata["is_resolved"] = self._is_resolved
+        return self._is_resolved, {}
+
+
+class EnvDataset(MockDataset):
+    """Dataset whose create_task_env is configurable per record id."""
+
+    def __init__(self, records: List[MockRecord], env_factories: Dict[str, Any]):
+        super().__init__(records)
+        self._env_factories = env_factories
+
+    def create_task_env(self, record: MockRecord):
+        factory = self._env_factories.get(record.record_id)
+        return factory(record.metadata) if factory is not None else None
+
+
 # ---------------------------------------------------------------------------
 # Tests
 # ---------------------------------------------------------------------------
@@ -99,6 +160,59 @@ class TestAgenticRunner:
         assert len(traces) == 1
         assert not traces[0].completed
         assert "Agent error" in traces[0].response_text
+        assert traces[0].error_kind == "agent_error"
+        assert "Agent error" in (traces[0].error or "")
+
+    def test_harness_failure_recorded_and_run_continues(self):
+        """(c) one task's env breakage doesn't kill the run."""
+        records = [
+            MockRecord(record_id="r1", problem="broken env"),
+            MockRecord(record_id="r2", problem="healthy"),
+        ]
+        dataset = EnvDataset(records, {"r1": FailingTaskEnv})
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert len(traces) == 2
+        # Failed task: recorded distinctly as a harness error, not a miss.
+        assert traces[0].error_kind == "harness_error"
+        assert not traces[0].completed
+        assert "tmux" in (traces[0].error or "")
+        assert traces[0].is_resolved is None
+        # Healthy task still ran to completion.
+        assert traces[1].completed
+        assert traces[1].error_kind is None
+        assert "Response to: healthy" in traces[1].response_text
+
+    def test_zero_model_contact_flagged_as_harness_error(self):
+        """(e) zero model requests -> harness_error, not a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockZeroContactAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].error_kind == "harness_error"
+        assert "zero_model_requests" in (traces[0].error or "")
+        assert traces[0].total_input_tokens == 0
+        assert traces[0].total_output_tokens == 0
+
+    def test_genuine_model_miss_not_flagged(self):
+        """(e) control: tokens>0 + is_resolved=False is a model miss."""
+        records = [MockRecord(record_id="r1", problem="task")]
+        dataset = EnvDataset(
+            records,
+            {"r1": lambda meta: ResolvingTaskEnv(meta, is_resolved=False)},
+        )
+        runner = AgenticRunner(agent=MockAgent(), dataset=dataset)
+
+        traces = self._run_async(runner.run())
+        assert traces[0].is_resolved is False
+        assert traces[0].error_kind is None
+        assert traces[0].error is None
+        assert traces[0].total_input_tokens > 0
 
     def test_synthetic_turn_created(self):
         records = [MockRecord(record_id="r1", problem="test")]

tests/evals/test_export.py

@@ -127,6 +127,84 @@ class TestExportSummaryJson:
         assert summary["totals"]["queries"] == 0
 
 
+class TestHarnessErrorExclusion:
+    """Harness errors must never count as model misses in resolve-rate."""
+
+    def _traces(self):
+        return [
+            QueryTrace(
+                query_id="q0000",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=True,
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0001",
+                workload_type="agentic",
+                completed=True,
+                is_resolved=False,  # genuine model miss: stays in denominator
+                turns=[TurnTrace(turn_index=0, input_tokens=10, output_tokens=5)],
+            ),
+            QueryTrace(
+                query_id="q0002",
+                workload_type="agentic",
+                completed=False,
+                is_resolved=False,  # stamped by run_tests despite zero contact
+                error="zero_model_requests: the agent never contacted the model",
+                error_kind="harness_error",
+            ),
+        ]
+
+    def test_summary_excludes_harness_errors_from_accuracy(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(self._traces(), {"model": "m"}, path)
+        summary = json.loads(path.read_text())
+
+        totals = summary["totals"]
+        assert totals["harness_errors"] == 1
+        assert totals["resolved"] == 1
+        assert totals["unresolved"] == 1  # NOT 2: harness error excluded
+        assert totals["accuracy"] == 0.5  # NOT 1/3
+        # Flat table_gen metrics exclude the harness error too.
+        assert summary["metrics"]["accuracy"]["n"] == 2
+        assert summary["metrics"]["accuracy"]["mean"] == 0.5
+        # Details surfaced for diagnosis.
+        details = summary["harness_error_details"]
+        assert details[0]["query_id"] == "q0002"
+        assert "zero_model_requests" in details[0]["error"]
+
+    def test_efficiency_excludes_harness_errors(self):
+        result = _compute_efficiency(self._traces(), None, None)
+        assert result["accuracy"] == 0.5
+
+    def test_no_harness_errors_key_absent(self, tmp_path):
+        path = tmp_path / "summary.json"
+        export_summary_json(_make_traces(), {}, path)
+        summary = json.loads(path.read_text())
+        assert summary["totals"]["harness_errors"] == 0
+        assert "harness_error_details" not in summary
+
+
+class TestTraceErrorFieldRoundTrip:
+    def test_round_trip_preserves_error_fields(self):
+        trace = QueryTrace(
+            query_id="q0",
+            workload_type="agentic",
+            error="zero_model_requests: ...",
+            error_kind="harness_error",
+        )
+        restored = QueryTrace.from_dict(trace.to_dict())
+        assert restored.error == trace.error
+        assert restored.error_kind == "harness_error"
+
+    def test_old_trace_dicts_still_load(self):
+        """Backward compat: traces.jsonl written before the schema change."""
+        restored = QueryTrace.from_dict({"query_id": "q0", "workload_type": "agentic"})
+        assert restored.error is None
+        assert restored.error_kind is None
+
+
 class TestExportArtifactsManifest:
     def test_no_artifacts_dir(self, tmp_path):
         result = export_artifacts_manifest(tmp_path)

tests/evals/test_terminalbench_env.py

@@ -1,15 +1,104 @@
-"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency)."""
+"""Tests for TerminalBenchTaskEnv (mocked terminal_bench dependency).
+
+These tests install a fake ``terminal_bench`` module tree into
+``sys.modules`` so they run without the real package or a Docker daemon
+(terminal-bench is an undeclared optional dep that CI never installs).
+"""
 
 from __future__ import annotations
 
+import sys
+import types
+from contextlib import contextmanager
+from dataclasses import dataclass, field
+from types import SimpleNamespace
+from typing import Any, Dict, List
+
 import pytest
 
+from openjarvis.evals.core.environment import TaskEnvironmentError
 from openjarvis.evals.execution.terminalbench_env import TerminalBenchTaskEnv
 
-# terminal_bench is an optional dep — skip all tests if unavailable
-terminal_bench = pytest.importorskip(
-    "terminal_bench", reason="terminal_bench not installed"
-)
+# ---------------------------------------------------------------------------
+# Fake terminal_bench seam
+# ---------------------------------------------------------------------------
+
+
+@dataclass
+class FakeExecResult:
+    exit_code: int = 0
+    output: bytes = b""
+
+
+@dataclass
+class FakeContainer:
+    """Container stub whose exec_run results are configurable per binary."""
+
+    exec_results: Dict[str, FakeExecResult] = field(default_factory=dict)
+    exec_calls: List[List[str]] = field(default_factory=list)
+
+    def exec_run(self, cmd: List[str]) -> FakeExecResult:
+        self.exec_calls.append(list(cmd))
+        return self.exec_results.get(cmd[0], FakeExecResult())
+
+
+class FakeTerminal:
+    def __init__(self, events: List[str], container: FakeContainer) -> None:
+        self._events = events
+        self.container = container
+        self.create_session_error: Exception | None = None
+
+    def create_session(self, name: str, **_kwargs: Any) -> str:
+        self._events.append(f"create_session({name})")
+        if self.create_session_error is not None:
+            raise self.create_session_error
+        return f"session-{name}"
+
+
+@pytest.fixture()
+def fake_tb(monkeypatch, tmp_path):
+    """Install a fake terminal_bench tree; return the shared test state."""
+    events: List[str] = []
+    container = FakeContainer()
+    terminal = FakeTerminal(events, container)
+    state = SimpleNamespace(
+        events=events,
+        container=container,
+        terminal=terminal,
+        spin_up_error=None,
+    )
+
+    @contextmanager
+    def spin_up_terminal(**kwargs: Any):
+        events.append("compose_up")
+        try:
+            if state.spin_up_error is not None:
+                raise state.spin_up_error
+            yield terminal
+        finally:
+            events.append("compose_down")
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_terminal_pkg = types.ModuleType("terminal_bench.terminal")
+    mod_terminal = types.ModuleType("terminal_bench.terminal.terminal")
+    mod_terminal.spin_up_terminal = spin_up_terminal
+    mod_tb.terminal = mod_terminal_pkg
+    mod_terminal_pkg.terminal = mod_terminal
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal", mod_terminal_pkg)
+    monkeypatch.setitem(sys.modules, "terminal_bench.terminal.terminal", mod_terminal)
+
+    state.metadata = {
+        "task_id": "hello.world",
+        "task": SimpleNamespace(disable_asciinema=True),
+        "task_paths": SimpleNamespace(docker_compose_path=tmp_path / "compose.yaml"),
+    }
+    return state
+
+
+# ---------------------------------------------------------------------------
+# Existing behavior (now running without the real terminal_bench package)
+# ---------------------------------------------------------------------------
 
 
 class TestTerminalBenchTaskEnv:
@@ -46,3 +135,113 @@ class TestTerminalBenchTaskEnv:
         assert is_resolved is False
         assert results["error"] == "terminal_not_running"
         assert metadata["is_resolved"] is False
+
+
+# ---------------------------------------------------------------------------
+# Exception-safe __enter__ / preflight / teardown
+# ---------------------------------------------------------------------------
+
+
+class TestEnterExceptionSafety:
+    def test_success_path(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            assert fake_tb.metadata["terminal"] is fake_tb.terminal
+            assert fake_tb.metadata["session"] == "session-agent"
+            assert fake_tb.metadata["container"] == "oj-hello-world"
+            assert "compose_down" not in fake_tb.events
+        assert fake_tb.events.count("compose_down") == 1
+        assert "terminal" not in fake_tb.metadata
+
+    def test_create_session_failure_tears_down_terminal(self, fake_tb):
+        """(a) tmux failure in __enter__ -> terminal torn down, loud error."""
+        fake_tb.terminal.create_session_error = RuntimeError(
+            "tmux is not installed in the container."
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        # No leak: compose project downed BEFORE the exception escaped.
+        assert "compose_down" in fake_tb.events
+        # Actionable: names the task, the image, and the failure.
+        message = str(excinfo.value)
+        assert "hello.world" in message
+        assert "tb__hello-world__client" in message
+        assert "tmux is not installed" in message
+        # Recorded for the runner / scorers; handles cleared.
+        assert fake_tb.metadata["harness_error"] == message
+        assert "terminal" not in fake_tb.metadata
+        assert "session" not in fake_tb.metadata
+        assert "container" not in fake_tb.metadata
+        assert env._terminal is None
+        assert env._terminal_cm is None
+        assert env._logs_tmpdir is None
+
+    def test_preflight_catches_missing_tmux(self, fake_tb):
+        """(b) preflight catches missing tmux, naming the task image."""
+        fake_tb.container.exec_results["tmux"] = FakeExecResult(
+            exit_code=127,
+            output=b'exec: "tmux": executable file not found in $PATH',
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+
+        message = str(excinfo.value)
+        assert "tmux" in message
+        assert "tb__hello-world__client" in message  # task image named
+        assert "127" in message
+        # Fired BEFORE the agent session was created.
+        assert not any(e.startswith("create_session") for e in fake_tb.events)
+        assert "compose_down" in fake_tb.events
+        assert fake_tb.metadata["harness_error"] == message
+
+    def test_preflight_checks_asciinema_when_recording(self, fake_tb):
+        fake_tb.metadata["task"] = SimpleNamespace(disable_asciinema=False)
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError, match="asciinema"):
+            env.__enter__()
+        assert "disable_asciinema" in fake_tb.metadata["harness_error"]
+        assert "compose_down" in fake_tb.events
+
+    def test_preflight_skips_asciinema_when_disabled(self, fake_tb):
+        fake_tb.container.exec_results["asciinema"] = FakeExecResult(exit_code=127)
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        with env:
+            pass
+        assert ["asciinema", "--version"] not in fake_tb.container.exec_calls
+
+    def test_compose_up_failure_includes_stderr(self, fake_tb):
+        import subprocess
+
+        fake_tb.spin_up_error = subprocess.CalledProcessError(
+            returncode=1,
+            cmd=["docker", "compose", "up"],
+            stderr="no space left on device",
+        )
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(TaskEnvironmentError) as excinfo:
+            env.__enter__()
+        assert "no space left on device" in str(excinfo.value)
+
+    def test_keyboard_interrupt_not_masked(self, fake_tb):
+        fake_tb.terminal.create_session_error = KeyboardInterrupt()
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+
+        with pytest.raises(KeyboardInterrupt):
+            env.__enter__()
+        # Still cleaned up, but the interrupt is not wrapped.
+        assert "compose_down" in fake_tb.events
+
+    def test_teardown_is_idempotent(self, fake_tb):
+        env = TerminalBenchTaskEnv(fake_tb.metadata)
+        env.__enter__()
+        env.__exit__(None, None, None)
+        env.__exit__(None, None, None)
+        assert fake_tb.events.count("compose_down") == 1

tests/evals/test_terminalbench_native_backend.py

@@ -0,0 +1,302 @@
+"""Tests for the TerminalBench native backend (mocked terminal_bench).
+
+Covers: timeout kwargs threading (config -> backend -> Harness kwargs),
+loud failure on terminal-bench builds without the timeout kwargs, and the
+harness-error classification in ``summarize_benchmark_results`` —
+including the zero-model-contact vs genuine-model-miss distinction.
+"""
+
+from __future__ import annotations
+
+import sys
+import types
+from types import SimpleNamespace
+from typing import Any, Dict, List, Optional
+
+import pytest
+
+import openjarvis.evals.backends.terminalbench_native as tbn
+from openjarvis.evals.backends.terminalbench_native import (
+    summarize_benchmark_results,
+)
+
+# ---------------------------------------------------------------------------
+# Helpers / fakes
+# ---------------------------------------------------------------------------
+
+
+def make_trial(
+    task_id: str,
+    *,
+    is_resolved: bool,
+    failure_mode: str = "unset",
+    input_tokens: Optional[int] = None,
+    output_tokens: Optional[int] = None,
+) -> SimpleNamespace:
+    """Build a duck-typed terminal-bench 0.2.18 TrialResults."""
+    return SimpleNamespace(
+        task_id=task_id,
+        trial_name=f"{task_id}.1-of-1",
+        is_resolved=is_resolved,
+        failure_mode=SimpleNamespace(value=failure_mode),
+        total_input_tokens=input_tokens,
+        total_output_tokens=output_tokens,
+    )
+
+
+class FakeHarness:
+    """Records constructor kwargs; run() returns the configured results."""
+
+    captured_kwargs: Dict[str, Any] = {}
+    results: Any = SimpleNamespace(results=[])
+
+    def __init__(self, **kwargs: Any) -> None:
+        type(self).captured_kwargs = kwargs
+
+    def run(self) -> Any:
+        return type(self).results
+
+
+class OldFakeHarness:
+    """A pre-timeout-kwargs Harness signature (no **kwargs)."""
+
+    def __init__(
+        self,
+        output_path: Any = None,
+        run_id: Any = None,
+        dataset_name: Any = None,
+        dataset_version: Any = None,
+        model_name: Any = None,
+        n_concurrent_trials: Any = None,
+        cleanup: Any = None,
+        agent_name: Any = None,
+        agent_kwargs: Any = None,
+        n_tasks: Any = None,
+    ) -> None:
+        pass
+
+    def run(self) -> Any:
+        return SimpleNamespace(results=[])
+
+
+@pytest.fixture()
+def fake_tb_backend(monkeypatch):
+    """Enable the backend without the real terminal_bench package."""
+    FakeHarness.captured_kwargs = {}
+    FakeHarness.results = SimpleNamespace(results=[])
+    monkeypatch.setattr(tbn, "_HAS_TB", True)
+    monkeypatch.setattr(tbn, "Harness", FakeHarness, raising=False)
+
+    mod_tb = types.ModuleType("terminal_bench")
+    mod_agents = types.ModuleType("terminal_bench.agents")
+    mod_agent_name = types.ModuleType("terminal_bench.agents.agent_name")
+    mod_agent_name.AgentName = lambda name: name
+    mod_tb.agents = mod_agents
+    mod_agents.agent_name = mod_agent_name
+    monkeypatch.setitem(sys.modules, "terminal_bench", mod_tb)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents", mod_agents)
+    monkeypatch.setitem(sys.modules, "terminal_bench.agents.agent_name", mod_agent_name)
+    return FakeHarness
+
+
+# ---------------------------------------------------------------------------
+# Timeout kwargs threading
+# ---------------------------------------------------------------------------
+
+
+class TestTimeoutKwargs:
+    def test_default_bound_reaches_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1800.0
+        assert "global_timeout_multiplier" not in kwargs
+
+    def test_explicit_values_reach_harness(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path),
+            global_agent_timeout_sec=1234.0,
+            global_timeout_multiplier=2.0,
+        )
+        backend.run_harness("run-1")
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 1234.0
+        assert kwargs["global_timeout_multiplier"] == 2.0
+
+    def test_zero_disables_bound(self, fake_tb_backend, tmp_path):
+        backend = tbn.TerminalBenchNativeBackend(
+            output_dir=str(tmp_path), global_agent_timeout_sec=0
+        )
+        backend.run_harness("run-1")
+        assert "global_agent_timeout_sec" not in fake_tb_backend.captured_kwargs
+
+    def test_old_terminal_bench_fails_loud(
+        self, fake_tb_backend, monkeypatch, tmp_path
+    ):
+        """An old Harness without the kwargs must not hang silently."""
+        monkeypatch.setattr(tbn, "Harness", OldFakeHarness, raising=False)
+        backend = tbn.TerminalBenchNativeBackend(output_dir=str(tmp_path))
+        with pytest.raises(RuntimeError, match="global_agent_timeout_sec"):
+            backend.run_harness("run-1")
+
+
+# ---------------------------------------------------------------------------
+# Harness-error classification (zero-model-contact detection)
+# ---------------------------------------------------------------------------
+
+
+class TestSummarizeBenchmarkResults:
+    def test_genuine_model_miss_is_not_flagged(self):
+        """MANDATORY: a real miss (tokens>0, failure_mode unset) stays a miss.
+
+        terminal-bench 0.2.18 leaves failure_mode UNSET on success and on
+        genuine unresolved misses — classifying on failure_mode would flag
+        every real miss as a harness error and inflate accuracy.
+        """
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-ok", is_resolved=True, input_tokens=900, output_tokens=100
+                ),
+                make_trial(
+                    "t-miss", is_resolved=False, input_tokens=800, output_tokens=50
+                ),
+                make_trial(
+                    "t-setup-dead",
+                    is_resolved=False,
+                    failure_mode="agent_installation_failed",
+                    input_tokens=0,
+                    output_tokens=0,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2  # genuine miss stays in denominator
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5  # not 1.0 (miss kept), not 1/3 (infra out)
+        assert summary.errors == 1
+        assert [f["task_id"] for f in failures] == ["t-setup-dead"]
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_zero_contact_flagged_even_with_unset_failure_mode(self):
+        """Setup hang signature: unresolved, zero requests, failure_mode unset."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-hang", is_resolved=False, input_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert summary.scored_samples == 0
+        assert failures[0]["reason"] == "zero_model_requests"
+
+    def test_missing_token_fields_treated_as_zero_contact(self):
+        results = SimpleNamespace(results=[make_trial("t-none", is_resolved=False)])
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+
+    def test_infra_failure_mode_flagged_despite_tokens(self):
+        results = SimpleNamespace(
+            results=[
+                make_trial(
+                    "t-crash",
+                    is_resolved=False,
+                    failure_mode="unknown_agent_error",
+                    input_tokens=500,
+                    output_tokens=20,
+                ),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 1
+        assert failures[0]["reason"] == "unknown_agent_error"
+
+    def test_resolved_with_zero_tokens_not_flagged(self):
+        """Installed agents report 0 tokens on success — never flag resolved."""
+        results = SimpleNamespace(
+            results=[
+                make_trial("t-ok", is_resolved=True, input_tokens=0, output_tokens=0),
+            ]
+        )
+        summary, failures = summarize_benchmark_results(results, model="m")
+        assert summary.errors == 0
+        assert summary.correct == 1
+        assert summary.accuracy == 1.0
+
+    def test_empty_results(self):
+        summary, failures = summarize_benchmark_results(
+            SimpleNamespace(results=[]), model="m"
+        )
+        assert summary.total_samples == 0
+        assert summary.accuracy == 0.0
+        assert failures == []
+
+
+# ---------------------------------------------------------------------------
+# CLI wiring: config -> backend -> harness kwargs -> RunSummary
+# ---------------------------------------------------------------------------
+
+
+class TestRunTerminalbenchNativeWiring:
+    def _run(self, fake_tb_backend, tmp_path, trials: List[Any], **config_kwargs):
+        from rich.console import Console
+
+        from openjarvis.evals.cli import _run_terminalbench_native
+        from openjarvis.evals.core.types import RunConfig
+
+        fake_tb_backend.results = SimpleNamespace(results=trials)
+        config = RunConfig(
+            benchmark="terminalbench-native",
+            backend="terminalbench-native",
+            model="test-model",
+            output_path=str(tmp_path / "out"),
+            **config_kwargs,
+        )
+        console = Console(record=True, width=120)
+        summary = _run_terminalbench_native(config, console)
+        return summary, console.export_text()
+
+    def test_config_timeouts_reach_harness_kwargs(self, fake_tb_backend, tmp_path):
+        """(d) timeout kwargs travel config -> backend -> harness_kwargs."""
+        self._run(
+            fake_tb_backend,
+            tmp_path,
+            [],
+            global_agent_timeout_sec=901.0,
+            global_timeout_multiplier=1.5,
+        )
+        kwargs = fake_tb_backend.captured_kwargs
+        assert kwargs["global_agent_timeout_sec"] == 901.0
+        assert kwargs["global_timeout_multiplier"] == 1.5
+
+    def test_config_defaults_use_backend_bound(self, fake_tb_backend, tmp_path):
+        self._run(fake_tb_backend, tmp_path, [])
+        assert fake_tb_backend.captured_kwargs["global_agent_timeout_sec"] == 1800.0
+
+    def test_summary_counts_real_trials(self, fake_tb_backend, tmp_path):
+        """Regression: results field is ``results``, not ``trial_results``.
+
+        The old conversion read the nonexistent ``trial_results`` attribute
+        and hardcoded errors=0, rendering every run as 0 samples / 0.0.
+        """
+        trials = [
+            make_trial("t-ok", is_resolved=True, input_tokens=10, output_tokens=10),
+            make_trial("t-miss", is_resolved=False, input_tokens=10, output_tokens=2),
+            make_trial(
+                "t-hang",
+                is_resolved=False,
+                failure_mode="agent_timeout",
+                input_tokens=0,
+                output_tokens=0,
+            ),
+        ]
+        summary, output = self._run(fake_tb_backend, tmp_path, trials)
+        assert summary.total_samples == 3
+        assert summary.scored_samples == 2
+        assert summary.correct == 1
+        assert summary.accuracy == 0.5
+        assert summary.errors == 1
+        # The harness failure is reported loudly with its task id.
+        assert "t-hang" in output
+        assert "zero_model_requests" in output

tests/sdk/test_system.py

@@ -2,7 +2,7 @@
 
 from __future__ import annotations
 
-from unittest.mock import MagicMock
+from unittest.mock import MagicMock, patch
 
 import pytest
 
@@ -327,6 +327,85 @@ class TestSystemBuilder:
         assert builder._engine_key == "ollama"
 
 
+class TestSystemBuilderEngineInstance:
+    """Explicit engine injection (jarvis eval --base-url path)."""
+
+    @staticmethod
+    def _fake_engine(healthy: bool = True) -> MagicMock:
+        engine = MagicMock(
+            spec=["health", "can_serve", "generate", "list_models", "close"]
+        )
+        engine.health.return_value = healthy
+        engine._host = "http://127.0.0.1:18999"
+        return engine
+
+    def test_engine_instance_is_fluent(self):
+        builder = SystemBuilder(JarvisConfig())
+        engine = self._fake_engine()
+        result = builder.engine_instance(engine, key="my-endpoint")
+        assert result is builder
+        assert builder._engine_instance is engine
+        assert builder._engine_instance_key == "my-endpoint"
+
+    def test_resolve_engine_returns_injected_instance(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        resolved_engine, resolved_key = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        assert resolved_key == "endpoint"
+
+    def test_unhealthy_injected_instance_raises_naming_host(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with pytest.raises(RuntimeError, match=r"http://127\.0\.0\.1:18999"):
+            builder._resolve_engine(config)
+
+    def test_unhealthy_injected_instance_never_consults_discovery(self):
+        """The observed failure mode: an explicit endpoint must NOT be
+        silently replaced by whatever other engine discovery finds."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=False)
+        builder = SystemBuilder(config).engine_instance(engine)
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            with pytest.raises(RuntimeError, match="Refusing to fall back"):
+                builder._resolve_engine(config)
+        mock_get_engine.assert_not_called()
+
+    def test_healthy_injected_instance_never_consults_discovery(self):
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        builder = SystemBuilder(config).engine_instance(engine, key="endpoint")
+        with patch("openjarvis.engine._discovery.get_engine") as mock_get_engine:
+            resolved_engine, _ = builder._resolve_engine(config)
+        assert resolved_engine is engine
+        mock_get_engine.assert_not_called()
+
+    def test_build_wires_injected_engine(self):
+        """build() must use the injected engine (possibly behind security
+        wrappers) instead of running discovery."""
+        config = JarvisConfig()
+        engine = self._fake_engine(healthy=True)
+        engine.list_models.return_value = ["stub-model"]
+        builder = (
+            SystemBuilder(config)
+            .engine_instance(engine, key="endpoint")
+            .model("stub-model")
+            .telemetry(False)
+            .traces(False)
+        )
+        system = builder.build()
+        try:
+            inner = system.engine
+            while hasattr(inner, "_engine"):
+                inner = inner._engine
+            assert inner is engine
+            assert system.engine_key == "endpoint"
+        finally:
+            system.close()
+
+
 class TestJarvisSystemClose:
     def test_close_with_scheduler_store(self):
         engine = MagicMock()

tests/server/test_routes.py

@@ -487,6 +487,167 @@ class TestChatCompletions:
         assert data["choices"][0]["finish_reason"] == "stop"
 
 
+# ---------------------------------------------------------------------------
+# Identity system-prompt injection (#540)
+# ---------------------------------------------------------------------------
+
+
+def _make_capturing_engine(captured: list):
+    """Like ``_make_engine`` but records the messages each path receives.
+
+    ``engine.generate`` is a MagicMock so ``call_args`` works on the
+    direct/non-stream path. ``engine.stream`` / ``engine.stream_full`` are
+    plain async-generator FUNCTIONS, so they capture their ``messages``
+    argument into the shared *captured* list from inside the generator body
+    (``call_args`` does not apply to plain functions).
+    """
+    engine = MagicMock()
+    engine.engine_id = "mock"
+    engine.health.return_value = True
+    engine.list_models.return_value = ["test-model"]
+    engine.generate.return_value = {
+        "content": "ok",
+        "usage": {"prompt_tokens": 5, "completion_tokens": 3, "total_tokens": 8},
+        "model": "test-model",
+        "finish_reason": "stop",
+    }
+
+    async def mock_stream(messages, *, model, temperature=0.7, max_tokens=1024, **kw):
+        captured.append(messages)
+        for token in ["Hello", " ", "world"]:
+            yield token
+
+    async def mock_stream_full(
+        messages, *, model, temperature=0.7, max_tokens=1024, **kw
+    ):
+        from openjarvis.engine._stubs import StreamChunk
+
+        captured.append(messages)
+        yield StreamChunk(content="ok", finish_reason="stop")
+
+    engine.stream = mock_stream
+    engine.stream_full = mock_stream_full
+    return engine
+
+
+class TestIdentityPromptInjection:
+    """Regression for #540.
+
+    The desktop UI posts only user/assistant turns to the
+    OpenAI-compatible ``/v1/chat/completions`` endpoint, so the engine never
+    saw OpenJarvis's identity system prompt and the model answered from its
+    training identity ("I'm Claude", "I am Qwen", ...). The engine-direct
+    server handlers must now inject ``agent.default_system_prompt`` whenever
+    the client omits a system message — and must NOT inject a second one when
+    the client already supplies their own.
+    """
+
+    def test_stream_injects_identity_when_absent(self):
+        captured: list = []
+        engine = _make_capturing_engine(captured)
+        client = TestClient(create_app(engine, "test-model"))
+
+        resp = client.post(
+            "/v1/chat/completions",
+            json={
+                "model": "test-model",
+                "messages": [{"role": "user", "content": "who are you?"}],
+                "stream": True,
+            },
+        )
+        assert resp.status_code == 200
+        # Drain the stream so the generator body runs and records messages.
+        _ = resp.text
+        assert captured, "engine.stream was never called"
+        msgs = captured[-1]
+        assert msgs[0].role.value == "system"
+        assert "OpenJarvis" in msgs[0].content
+
+    def test_stream_no_double_injection_when_client_supplies_system(self):
+        captured: list = []
+        engine = _make_capturing_engine(captured)
+        client = TestClient(create_app(engine, "test-model"))
+
+        resp = client.post(
+            "/v1/chat/completions",
+            json={
+                "model": "test-model",
+                "messages": [
+                    {"role": "system", "content": "Be terse."},
+                    {"role": "user", "content": "who are you?"},
+                ],
+                "stream": True,
+            },
+        )
+        assert resp.status_code == 200
+        _ = resp.text
+        msgs = captured[-1]
+        system_msgs = [m for m in msgs if m.role.value == "system"]
+        assert len(system_msgs) == 1
+        assert system_msgs[0].content == "Be terse."
+
+    def test_direct_injects_identity_when_absent(self):
+        captured: list = []
+        engine = _make_capturing_engine(captured)
+        # No agent -> non-stream request goes through _handle_direct.
+        client = TestClient(create_app(engine, "test-model"))
+
+        resp = client.post(
+            "/v1/chat/completions",
+            json={
+                "model": "test-model",
+                "messages": [{"role": "user", "content": "who are you?"}],
+            },
+        )
+        assert resp.status_code == 200
+        assert engine.generate.called
+        msgs = engine.generate.call_args.args[0]
+        assert msgs[0].role.value == "system"
+        assert "OpenJarvis" in msgs[0].content
+
+    def test_direct_no_double_injection_when_client_supplies_system(self):
+        captured: list = []
+        engine = _make_capturing_engine(captured)
+        client = TestClient(create_app(engine, "test-model"))
+
+        resp = client.post(
+            "/v1/chat/completions",
+            json={
+                "model": "test-model",
+                "messages": [
+                    {"role": "system", "content": "Be terse."},
+                    {"role": "user", "content": "who are you?"},
+                ],
+            },
+        )
+        assert resp.status_code == 200
+        msgs = engine.generate.call_args.args[0]
+        system_msgs = [m for m in msgs if m.role.value == "system"]
+        assert len(system_msgs) == 1
+        assert system_msgs[0].content == "Be terse."
+
+    def test_stream_tools_injects_identity_when_absent(self):
+        captured: list = []
+        engine = _make_capturing_engine(captured)
+        client = TestClient(create_app(engine, "test-model"))
+
+        resp = client.post(
+            "/v1/chat/completions",
+            json={
+                "model": "test-model",
+                "messages": [{"role": "user", "content": "who are you?"}],
+                "tools": [{"type": "function", "function": {"name": "calc"}}],
+                "stream": True,
+            },
+        )
+        assert resp.status_code == 200
+        _ = resp.text
+        assert captured, "engine.stream_full was never called"
+        msgs = captured[-1]
+        assert msgs[0].role.value == "system"
+        assert "OpenJarvis" in msgs[0].content
+
+
 # ---------------------------------------------------------------------------
 # Models endpoint tests
 # ---------------------------------------------------------------------------

tests/test_vision.py

@@ -0,0 +1,94 @@
+"""Tests for vision input support: ``Message.images`` -> Ollama payload.
+
+These cover the data-flow contract that makes vision work end to end:
+a ``Message`` can carry base64 images, the engine serializer forwards them
+to Ollama's ``/api/chat`` ``images`` field, and text-only messages are
+completely unaffected. The security guardrail must preserve images when it
+rewrites a flagged message.
+"""
+
+from __future__ import annotations
+
+from types import SimpleNamespace
+
+import openjarvis.engine.ollama as ollama_mod
+from openjarvis.core.types import Message, Role
+from openjarvis.engine._base import messages_to_dicts
+
+
+def test_message_defaults_to_no_images() -> None:
+    assert Message(role=Role.USER, content="hi").images is None
+
+
+def test_messages_to_dicts_omits_images_for_text() -> None:
+    dicts = messages_to_dicts([Message(role=Role.USER, content="hi")])
+    assert "images" not in dicts[0]
+
+
+def test_messages_to_dicts_forwards_images() -> None:
+    b64 = "aGVsbG8="  # "hello"
+    dicts = messages_to_dicts(
+        [Message(role=Role.USER, content="what is this?", images=[b64])]
+    )
+    assert dicts[0]["role"] == "user"
+    assert dicts[0]["content"] == "what is this?"
+    assert dicts[0]["images"] == [b64]
+
+
+def test_messages_to_dicts_empty_images_treated_as_text() -> None:
+    dicts = messages_to_dicts([Message(role=Role.USER, content="hi", images=[])])
+    assert "images" not in dicts[0]
+
+
+def test_default_num_ctx_default_and_override(monkeypatch) -> None:
+    monkeypatch.delenv("JARVIS_NUM_CTX", raising=False)
+    assert ollama_mod._default_num_ctx() == 16384
+
+    monkeypatch.setenv("JARVIS_NUM_CTX", "8000")
+    assert ollama_mod._default_num_ctx() == 8000
+
+    # A non-integer override must fall back to the safe default, not crash.
+    monkeypatch.setenv("JARVIS_NUM_CTX", "not-an-int")
+    assert ollama_mod._default_num_ctx() == 16384
+
+
+def test_guardrails_preserves_images_when_sanitizing() -> None:
+    """A flagged message gets rewritten; its image must survive the rewrite."""
+    from openjarvis.security.guardrails import GuardrailsEngine
+
+    class _RecordingEngine:
+        """Captures the messages the guardrail forwards to the real engine."""
+
+        def __init__(self) -> None:
+            self.received: list[Message] = []
+
+        def generate(self, messages, *, model, **kwargs):
+            self.received = list(messages)
+            return {"content": "ok"}
+
+    class _AlwaysFlag:
+        """A scanner that flags everything, forcing the sanitize rewrite path."""
+
+        def scan(self, text: str):
+            finding = SimpleNamespace(
+                pattern_name="test",
+                threat_level=SimpleNamespace(value="low"),
+                description="always flags",
+            )
+            return SimpleNamespace(findings=[finding])
+
+        def redact(self, text: str) -> str:
+            return text
+
+    engine = _RecordingEngine()
+    guarded = GuardrailsEngine(
+        engine,
+        scanners=[_AlwaysFlag()],
+        scan_input=True,
+        scan_output=False,
+    )
+    msg = Message(role=Role.USER, content="suspicious", images=["aGVsbG8="])
+
+    guarded.generate([msg], model="x")
+
+    assert engine.received[0].images == ["aGVsbG8="]