feat: swe bench scorer

AGENTS.md

@@ -100,6 +100,7 @@ Dataset Manager --> Load Generator --> Endpoint Client --> External Endpoint
 | **DeepSeek-R1 (MLPerf)**            | `src/inference_endpoint/evaluation/scoring.py` (`LegacyMLPerfDeepSeekR1Scorer`), `examples/07_DeepSeekR1_Example/` | MLPerf DeepSeek-R1 accuracy. TensorRT-LLM is OpenAI-compatible, so it is served via `api_type: openai` / `openai_completions` (no dedicated trtllm adapter). The combined multi-subset eval (`math500`/`aime`/`gpqa`/`mmlu_pro`/`livecodebench`) is the official MLCommons `eval_accuracy.py`, run out-of-process via `uv run --project` against the isolated subproject at `src/inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/` (a uv subproject excluded from the parent wheel; mirrors the VBench pattern). The example feeds the exact MLPerf prompt via pre-tokenized `input_tokens` to `/v1/completions`.                                                                                                                                                      |
 | **VideoGen**                        | `src/inference_endpoint/videogen/`                                                                                 | Adapter for video-generation endpoints (e.g. trtllm-serve `POST /v1/videos/generations`, used by MLPerf WAN2.2-T2V-A14B). Defaults to `response_format=video_path` (server saves video to shared storage and returns path) to avoid large byte payloads. Accuracy mode also runs on `video_path`: the adapter mirrors the path into `response_output` so the event log carries it to `VBenchScorer` (see `evaluation/scoring.py`), which scores videos via VBench from a sibling `uv` subproject at `examples/09_Wan22_VideoGen_Example/accuracy/` (vbench's `transformers==4.33.2` + `numpy<2` pins are incompatible with the parent env, so it runs out-of-process via `uv run --project`). Dataset is ingested via the generic JSONL loader.                                |
 | **Compliance (submission checker)** | `src/inference_endpoint/compliance/checker.py`, `scripts/check_compliance.py`                                      | Validates a completed run's report directory against a registered ruleset. `check_submission(report_dir, ruleset, model)` reads the resolved `config.yaml` plus scorer output (`accuracy/accuracy_results.json` for accuracy, `scores.json` for the agentic perf run) and runs config-lock (deterministic + single-stream), the accuracy gate (`score >= factor x reference`, factor 0.97 for Edge-Agentic), and run validity (0 dropped turns). Server-side launch flags (`--reasoning off`, `--ctx-size`) aren't in client artifacts, so they're surfaced as manual attestations. CLI: `scripts/check_compliance.py REPORT_DIR` (exit 0 = pass).                                                                                                                             |
+| **SWE-bench**                       | `src/inference_endpoint/dataset_manager/predefined/swe_bench/`, `evaluation/scoring.py`                            | `SWEBench` predefined dataset (HuggingFace `princeton-nlp/SWE-bench_Verified` or `_Lite`; `ACCURACY_ONLY=True`). `SWEBenchScorer` sets `SKIP_ENDPOINT_PHASE=True` and bypasses the built-in accuracy phase entirely: it delegates agent execution and grading to the configured SWE-bench service via `accuracy_config.extras.swebench_service_url`. The service host owns Docker/runtime execution, artifact storage, and secret handling; the benchmark client remains the report-producing entrypoint.                                                                                                                                                                                                                                                                      |
 | **Compliance (audit tests)**        | `src/inference_endpoint/compliance/`, `commands/audit.py`                                                          | MLPerf compliance audits. `AuditTest` protocol + `AuditRunSpec`/`AuditRunArtifacts` + registry (`compliance/__init__.py`); `OutputCachingAudit` (`compliance/audit_test/output_caching_test.py`, which also owns the QPS-specific `AuditRunStats`) implements MLPerf **TEST04** output-caching detection — reference phase (distinct samples) vs. fixed-sample audit phase, comparing QPS against `threshold`. `commands/audit.py:run_audit` runs phases via `AuditTest.plan_runs`/`validate`, writing `audit_result.json`/`verify_<TEST>.txt` atomically via `compliance/result.py`. Enabled by the `audit:` YAML block; `cli._run` runs it after the main benchmark (upstream MLPerf order: perf run, then TEST04), or standalone with `audit.only: true`. Performance-only. |
 
 ### Hot-Path Architecture

ATTRIBUTION

@@ -25,4 +25,13 @@ This section lists third-party assets and their attributions.
   (instance IDs such as `django__django-16899`). Cite: Jimenez et al.,
   "SWE-bench: Can Language Models Resolve Real-world Github Issues?", ICLR 2024.
 
+### mini-swe-agent
+
+- Source: https://github.com/SWE-agent/mini-swe-agent/tree/v2.3.0
+- License: MIT
+- Copyright: Copyright (c) 2025 Kilian A. Lieret and Carlos E. Jimenez
+- Description: The SWE-bench service templates and optional Qwen tool-call
+  replacements are derived from mini-swe-agent v2.3.0. The full upstream
+  notice is retained beside those templates.
+
 For a complete list of dependencies and their licenses, see the requirements files in the `requirements/` directory.

examples/10_Agentic_Inference/README.md

@@ -194,3 +194,63 @@ Update the first `datasets` entry (`name` and `path`), `model_params.name`, and
 uv run inference-endpoint benchmark from-config \
   --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml
 ```
+
+## SWE-bench Accuracy
+
+Both `qwen_agentic_benchmark.yaml` and `kimi_agentic_benchmark.yaml` include the
+SWE-bench accuracy dataset. Run the config for the model under test with
+`--mode acc`; this skips the agentic performance dataset. The benchmark
+framework also skips its built-in endpoint phase for the SWE-bench dataset.
+Instead, `SWEBenchScorer` submits the run to a native SWE-bench service. The
+service host owns Docker, `mini-swe-agent`, and the `swebench` evaluation
+harness, and it drives requests to the configured endpoint.
+
+Keep `accuracy_config.num_repeats: 1`: the scorer performs one external
+evaluation run per benchmark. Optional `accuracy_config.extras.subset` and
+`split` are used consistently for dataset loading, preflight, and scoring.
+
+`accuracy_config.extras.swebench_service_url` points the benchmark client to
+the service. Service mode follows the LiveCodeBench-style external-service
+convention for heavyweight evaluation work and supports exactly one endpoint URL
+in `endpoint_config.endpoints`; that URL must be reachable from the service
+host. Treat the service host as trusted infrastructure: it receives the endpoint
+URL and optional endpoint API key needed to run mini-swe-agent.
+For non-loopback service deployments, bind it on a private network or start it
+with `--auth-token` and set
+`accuracy_config.extras.swebench_service_auth_token`.
+
+`accuracy_config.extras.workers` sets the agent run's parallelism (`--workers`).
+If unset, it defaults to the load pattern's `target_concurrency` (for
+`concurrency`/`agentic_inference` patterns), else 10. `max_eval_workers`
+(default 10, `--max_workers`) sets the eval harness's parallelism.
+
+Qwen tool-call runs should set `enable_swebench_toolcall_patch: true` and
+`swebench_template: qwen_tools`. Template contents are owned by the service; the
+client does not send host file paths for templates or patch code.
+
+Start the service on the host that has Docker:
+
+```bash
+uv run --project src/inference_endpoint/evaluation/swebench_service \
+  python -m swebench_service --host 0.0.0.0 --port 18080
+```
+
+Then run the matching model config from the repo root:
+
+```bash
+# Qwen
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml \
+  --mode acc
+
+# Kimi
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml \
+  --mode acc
+```
+
+`--mode acc` is required because `type: online` configs default to
+`TestMode.PERF`, which skips accuracy datasets.
+
+See `accuracy/RUNBOOK.md` for preconditions, sanity checks, and common failure
+modes.

examples/10_Agentic_Inference/accuracy/RUNBOOK.md

@@ -0,0 +1,73 @@
+# SWE-bench Accuracy Smoke-Test Runbook
+
+End-to-end validation for the SWE-bench accuracy pipeline. Unit tests mock all
+subprocesses, so running the real pipeline is the only way to catch Docker,
+HuggingFace access, or mini-swe-agent wiring issues.
+
+## 0. Preconditions
+
+- Docker daemon running on the SWE-bench service host.
+- Docker Hub auth or a pre-seeded image cache on the service host.
+- Network egress to PyPI and HuggingFace Hub from the service host.
+- Endpoint URL reachable from the service host.
+- `uv` binary on PATH (`curl -LsSf https://astral.sh/uv/install.sh | sh`).
+- Parent endpoints env already synced (`uv sync --extra dev` from repo root).
+
+## 1. Start the SWE-bench service
+
+From the repo root:
+
+```bash
+uv run --project src/inference_endpoint/evaluation/swebench_service \
+  python -m swebench_service --host 0.0.0.0 --port 18080
+```
+
+Sanity check:
+
+```bash
+curl http://localhost:18080/health
+```
+
+## 2. End-to-end test (requires live endpoint)
+
+Run the config for the model under test:
+
+```bash
+# Qwen
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml \
+  --mode acc
+
+# Kimi
+uv run inference-endpoint benchmark from-config \
+  --config examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml \
+  --mode acc
+```
+
+Both configs include a performance dataset and the SWE-bench accuracy dataset.
+`--mode acc` skips the performance dataset and is required because
+`type: online` configs default to `TestMode.PERF`, which skips accuracy
+datasets.
+
+Scorer preflight calls the service `/health` endpoint. It does not check Docker
+or pre-pull images on the benchmark client.
+
+The service is trusted infrastructure. It receives one endpoint URL and optional
+endpoint credentials, runs Docker-backed evaluations, and serves artifacts. For
+non-loopback deployments, bind it on a private network or start it with
+`--auth-token TOKEN` and set
+`accuracy_config.extras.swebench_service_auth_token: TOKEN`.
+
+Qwen SWE-bench configs may opt into `enable_swebench_toolcall_patch: true` and
+`swebench_template: qwen_tools`. That path builds a temporary minisweagent
+package overlay with replacement files packaged with the service, prepends it to
+`PYTHONPATH` for the agent run, and leaves the installed package untouched.
+Leave this flag unset for Kimi and other non-Qwen runs.
+
+## Common failure modes
+
+| Symptom                              | Likely cause                              | Fix                                                |
+| ------------------------------------ | ----------------------------------------- | -------------------------------------------------- |
+| `swebench_service_url is required`   | Client config missing service URL         | Set `accuracy_config.extras.swebench_service_url`  |
+| Service health check fails           | Service not running or unreachable        | Start the service or fix client-to-service routing |
+| Docker error during `run_evaluation` | Docker daemon not running on service host | Start Docker on the service host and retry         |

examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml

@@ -23,6 +23,15 @@ datasets:
       num_trajectories_to_issue: 990 # Should be integer multiple of 990.
       # Required benchmark default; set to true only for faster optimization/debug runs.
       stop_issuing_on_first_user_complete: false
+  - name: swe_bench
+    type: "accuracy"
+    accuracy_config:
+      eval_method: "swe_bench_scorer"
+      num_repeats: 1
+      extras:
+        swebench_service_url: "http://localhost:18080"
+        num_instances: 200
+        # workers: 8 # parallel agent workers; defaults to target_concurrency if unset.
 
 settings:
   runtime:

examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml

@@ -0,0 +1,53 @@
+name: "qwen-agentic-benchmark"
+version: "1.0"
+type: "online"
+
+model_params:
+  name: "Qwen/Qwen3.6-35B-A3B"
+  temperature: 1.0
+  top_k: 20
+  top_p: 0.95
+  repetition_penalty: 1.0
+  presence_penalty: 1.5
+  max_new_tokens: 8192
+  chat_template_kwargs:
+    preserve_thinking: true
+
+datasets:
+  - name: agentic_coding
+    type: performance
+    path: ./agentic_combined.jsonl
+    accuracy_config:
+      eval_method: agentic_inference_inline # required benchmark default.
+    agentic_inference:
+      turn_timeout_s: 14400.0
+      enable_salt: true # do not change.
+      inject_tool_delay: true # do not change.
+  - name: swe_bench
+    type: "accuracy"
+    accuracy_config:
+      eval_method: "swe_bench_scorer"
+      num_repeats: 1
+      extras:
+        swebench_service_url: "http://localhost:18080"
+        num_instances: 200
+        workers: 10
+        max_eval_workers: 10
+        enable_swebench_toolcall_patch: true
+        swebench_template: "qwen_tools"
+
+settings:
+  runtime:
+    min_duration_ms: 0
+    max_duration_ms: 36000000
+
+  load_pattern:
+    type: agentic_inference
+    target_concurrency: 8 # Submission-specific concurrency.
+
+endpoint_config:
+  endpoints:
+    - "http://localhost:30000"
+  api_type: openai
+
+report_dir: logs/qwen_agentic

pyproject.toml

@@ -33,6 +33,8 @@ source-exclude = [
     # numpy<2 / prm800k that conflict with the parent env). Invoked only via
     # `uv run --project`, never imported - keep it out of the parent wheel.
     "inference_endpoint/evaluation/legacy_mlperf_deepseek_r1/**",
+    # Isolated native uv service that owns SWE-bench Docker execution.
+    "inference_endpoint/evaluation/swebench_service/**",
 ]
 
 [project]
@@ -94,6 +96,7 @@ dependencies = [
     # Fix pytz-2024 import warning
     "pytz==2026.1.post1",
     "urllib3==2.7.0",
+    "pyyaml==6.0.3",
 ]
 
 [project.optional-dependencies]