fix(ci): gate live HuggingFace Hub download tests behind hub marker (#507)

The two eval-dataset suites that download real corpora from the
HuggingFace Hub at runtime were running in the default CI lane. When the
Hub was unreachable or rate-limited they failed and reddened `main` even
though no code changed — confirmed by #506 (docs-only) failing on merge
while its own PR run passed an hour earlier. They also dominated CI
wall-time (~33 min of downloads + retry backoff on the failing run).

Add a `hub` pytest marker, apply it to both suites via module-level
`pytestmark`, and exclude it from the default CI lane
(`-m "not live and not cloud and not hub"`). The tests stay runnable on
demand with `pytest -m hub`.

The ADP provider swallows per-config download errors and returns 0
records on a network failure, so a Hub outage surfaced there as
`assert 1 <= 0` (not an exception) — it could not be made non-flaky by
exception handling alone, only by gating.

Coverage holds: removing these from CI drops total from 60.92% to
~60.73% (paranoid worst case 60.18%), still above the 60% gate.

Co-authored-by: Claude Opus 4.8 (1M context) <noreply@anthropic.com>

.github/workflows/ci.yml

@@ -64,7 +64,7 @@ jobs:
 
       - name: Run tests
         run: |
-          uv run pytest tests/ -v --tb=short -m "not live and not cloud" \
+          uv run pytest tests/ -v --tb=short -m "not live and not cloud and not hub" \
             --cov=openjarvis \
             --cov-report=term-missing \
             --cov-report=xml \

pyproject.toml

@@ -174,6 +174,7 @@ markers = [
     "live_channel: requires real channel credentials (env vars)",
     "nvidia: requires NVIDIA GPU",
     "slow: long-running test",
+    "hub: downloads real datasets from the HuggingFace Hub at runtime; excluded from the default CI lane (run with -m hub)",
     "live_external: requires HERMES_AGENT_PATH and OPENCLAW_PATH; spawns real foreign-framework subprocesses",
     "modal: requires Modal token + network; runs real swebench harness on Modal",
 ]

tests/evals/datasets/test_external_agent_datasets.py

@@ -10,6 +10,14 @@ These tests download each dataset once to ~/.cache/huggingface and
 verify the provider loads, iterates, and honours the split kwarg.
 The ModuleNotFoundError skip branch is defensive — currently
 unreachable since all three provider modules exist.
+
+Marked ``hub``: they hit the live HuggingFace Hub, so they are excluded
+from the default CI lane (which runs ``-m "not live and not cloud and not
+hub"``) — a transient Hub outage or rate-limit must not redden ``main``.
+Run them on demand with ``pytest -m hub``. The ADP provider swallows
+per-config download errors and returns 0 records on a network failure
+(see ``adp.py``), so a Hub outage surfaces here as ``assert 1 <= 0``
+rather than an exception — another reason these can't run unguarded in CI.
 """
 
 from __future__ import annotations
@@ -18,6 +26,8 @@ import importlib
 
 import pytest
 
+pytestmark = pytest.mark.hub
+
 PROVIDERS = [
     ("openjarvis.evals.datasets.adp", "ADPDataset"),
     ("openjarvis.evals.datasets.toolorchestra", "ToolOrchestraDataset"),

tests/evals/test_dataset_splits_integration.py

@@ -1,4 +1,13 @@
-"""Integration test: each provider's split kwarg produces disjoint train/test slices."""
+"""Integration test: each provider's split kwarg produces disjoint train/test slices.
+
+Marked ``hub``: every provider here downloads a real corpus from the
+HuggingFace Hub, so the module is excluded from the default CI lane (which
+runs ``-m "not live and not cloud and not hub"``). A transient Hub outage
+or rate-limit raises connectivity errors (``LocalEntryNotFoundError``,
+``HfHubHTTPError``) that the gated/not-found skip branch below does not
+catch — so running these unguarded made ``main`` flaky-red. Run on demand
+with ``pytest -m hub``.
+"""
 
 from __future__ import annotations
 
@@ -6,6 +15,8 @@ import importlib
 
 import pytest
 
+pytestmark = pytest.mark.hub
+
 PROVIDERS = [
     ("openjarvis.evals.datasets.pinchbench", "PinchBenchDataset"),
     ("openjarvis.evals.datasets.liveresearch", "LiveResearchBenchDataset"),