feat: swe bench scorer#342
Conversation
|
MLCommons CLA bot All contributors have signed the MLCommons CLA ✍️ ✅ |
![gemini-code-assist[bot]](https://avatars.githubusercontent.com/in/956858?s=60&v=4){kind=small}
There was a problem hiding this comment.
Code Review
This pull request introduces support for SWE-bench accuracy evaluation by adding a new accuracy-only SWEBench dataset, a SWEBenchScorer that runs evaluations using mini-swe-agent in an isolated environment, and associated configuration templates, tests, and runbooks. Feedback on the changes focuses on improving the robustness of the SWEBenchScorer implementation, specifically by safely handling missing or null values when parsing configuration templates, benchmark configurations, and evaluation results, as well as gracefully handling cases where the Docker binary is missing from the system's PATH during preflight checks.
Important
The consumer version of Gemini Code Assist on GitHub is being sunset. Starting June 18, 2026, new organization installations will be blocked, and all code review activity will officially cease on July 17, 2026.
For more details on the timeline and next steps, please review the Help Documentation.
aa7f559 to
bb9b307
Compare
There was a problem hiding this comment.
Pull request overview
Adds first-class SWE-bench accuracy support to the benchmarking system by introducing an external (subprocess-driven) scorer and a predefined SWE-bench dataset, plus wiring in config/schema handling and examples to run the workflow.
Changes:
- Introduces
SWEBenchScorer(external evaluation viauv run --project ... mini-extra+swebenchharness) and integrates it into the scoring/benchmark lifecycle (preflight, phase skipping, reporting). - Adds predefined
swe_benchdataset with caching + anACCURACY_ONLYguard to prevent using it as a performance dataset. - Updates schema/templates/examples and adds unit tests for the scorer + dataset + benchmark wiring.
Reviewed changes
Copilot reviewed 20 out of 21 changed files in this pull request and generated 3 comments.
Show a summary per file
| File | Description |
|---|---|
| uv.lock | Locks pyyaml as a direct dependency. |
| pyproject.toml | Adds pyyaml==6.0.3 to runtime dependencies. |
| src/inference_endpoint/evaluation/scoring.py | Adds scorer preflight hooks, shared subprocess logging helper, and new SWEBenchScorer implementation. |
| src/inference_endpoint/dataset_manager/predefined/swe_bench/init.py | Adds SWE-bench predefined dataset with HF download + parquet cache and ACCURACY_ONLY=True. |
| src/inference_endpoint/dataset_manager/dataset.py | Introduces Dataset.ACCURACY_ONLY class flag. |
| src/inference_endpoint/dataset_manager/init.py | Exposes/imports SWEBench dataset. |
| src/inference_endpoint/config/schema.py | Adds swe_bench_scorer to ScorerMethod and injects concurrency → SWE-bench workers default. |
| src/inference_endpoint/commands/benchmark/execute.py | Runs scorer preflight(), prevents accuracy-only datasets as perf, skips endpoint phases for external scorers, and improves sample counting/reporting. |
| src/inference_endpoint/config/templates/online_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/offline_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| src/inference_endpoint/config/templates/concurrency_template_full.yaml | Documents swe_bench_scorer as an accuracy scorer option. |
| examples/10_Agentic_Inference/swebench_template.yaml | Adds a SWE-bench/mini-swe-agent config template used by the external scorer. |
| examples/10_Agentic_Inference/swe_bench_accuracy.yaml | Adds a runnable example config for SWE-bench external accuracy evaluation. |
| examples/10_Agentic_Inference/README.md | Documents how to sync/run the SWE-bench accuracy subproject. |
| examples/10_Agentic_Inference/qwen_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/kimi_agentic_benchmark.yaml | Adds SWE-bench accuracy dataset to an agentic benchmark example. |
| examples/10_Agentic_Inference/accuracy/RUNBOOK.md | Adds an ops runbook for validating the SWE-bench accuracy pipeline. |
| examples/10_Agentic_Inference/accuracy/pyproject.toml | Adds isolated uv subproject deps for SWE-bench evaluation tooling. |
| tests/unit/evaluation/test_swe_bench_scorer.py | Adds unit coverage for scorer behavior, config patching, and preflight. |
| tests/unit/dataset_manager/test_swe_bench_dataset.py | Adds unit coverage for dataset registration, caching, and subset mapping. |
| tests/unit/commands/test_benchmark.py | Adds tests for concurrency→workers injection, ACCURACY_ONLY enforcement, preflight propagation, and phase skipping behavior. |
💡 Add Copilot custom instructions for smarter, more guided reviews. Learn how to get started.
8d66f43 to
ae04494
Compare
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## main #342 +/- ##
=======================================
Coverage ? 81.26%
=======================================
Files ? 142
Lines ? 18763
Branches ? 0
=======================================
Hits ? 15247
Misses ? 3516
Partials ? 0 ☔ View full report in Codecov by Harness. 🚀 New features to boost your workflow:
|
da45a99 to
fe442f8
Compare
| options["swebench_service_url"] = cls._normalize_service_url( | ||
| extras.get("swebench_service_url") | ||
| ) | ||
| auth_token = extras.get("swebench_service_auth_token") |
There was a problem hiding this comment.
swebench_service_auth_token is retained as an ordinary string under accuracy_config.extras, and setup_benchmark() writes the full resolved configuration to report_dir/config.yaml without redaction. This also happens for PERF-only runs that never contact the service. Since report directories are commonly retained or shared, please exclude/redact this credential during serialization and add a test proving it never appears in config.yaml.
| base = "" | ||
| model_kwargs["api_base"] = "" | ||
|
|
||
| if request.endpoint_api_key: |
There was a problem hiding this comment.
The endpoint API key is inserted into model_kwargs. mini-swe-agent serializes the complete model configuration into each trajectory (upstream implementation), and the Qwen overlay does the same. Those trajectories persist under the service artifact root even though request.json and status.json are redacted. Please inject the key through a non-serialized mechanism, or scrub/remove it from every generated trajectory before retaining the run.
| @@ -0,0 +1,174 @@ | |||
| # SPDX-FileCopyrightText: Copyright (c) 2026 NVIDIA CORPORATION & AFFILIATES. All rights reserved. | |||
There was a problem hiding this comment.
litellm_model.py and actions_toolcall.py appear to be substantial modified copies of mini-swe-agent v2.3.0 (source 1, source 2). They currently contain only the NVIDIA Apache header. Upstream's MIT license requires retaining its copyright and permission notice in copies or substantial portions. Please preserve the upstream notice and add source attribution/third-party notices for all copied Python and YAML templates, subject to the project's OSS guidance.
There was a problem hiding this comment.
Why do we need those changes? This will create a substantial burden on us to maintain these files. Can we just use a pinned version from the mini-swe-agent?
| ) | ||
|
|
||
|
|
||
| def _terminate_process(process: subprocess.Popen[str]) -> None: |
There was a problem hiding this comment.
Cancellation kills only the mini-extra process group. mini-swe-agent starts containers with detached docker run -d and relies on object destruction for cleanup (upstream Docker environment); SIGTERM/SIGKILL does not run that cleanup. With the configured 10-hour container timeout, repeated cancellations can leave many containers consuming resources. Please label/track containers by service run and clean them in an unconditional service-side finally path for success, failure, timeout, shutdown, and cancellation.
|
|
||
| result = status.get("result") | ||
| result_path = self.report_dir / "swe_bench_results.json" | ||
| if result is None and result_path.exists(): |
There was a problem hiding this comment.
Report directories in the example configs are reused. If artifact download fails, an old swe_bench_results.json remains; this fallback can then score that old result when the current status lacks an inline result. Even when the current inline result exists, the old file is never overwritten, leaving the reported artifact inconsistent with the computed score. Please clear or namespace artifacts by run_id and always atomically write the current result. Any fallback artifact must be explicitly tied to the current run.
|
LGTM overall. Checkout the minor nits I posted. I will test it out on Kimi later. Thanks Tianmu. |
| if command == "view": | ||
| view_range = args.get("view_range") | ||
| if view_range is not None: | ||
| if not isinstance(view_range, list | tuple) or len(view_range) != 2: |
| submitted_count = result.get("submitted_instances") or 0 | ||
| resolved = result.get("resolved_instances") or 0 | ||
| if submitted_count == 0: | ||
| logger.warning("SWE-bench: submitted_instances=0; returning None score") | ||
| self.complete = False | ||
| return None, 1 |
| @classmethod | ||
| def dataset_loader_kwargs(cls, extras: dict[str, Any]) -> dict[str, Any]: | ||
| return {} | ||
|
|
||
| @classmethod | ||
| def external_sample_count(cls, extras: dict[str, Any]) -> int | None: | ||
| return None | ||
|
|
||
| @classmethod | ||
| def preflight(cls, extras: dict[str, Any]) -> None: | ||
| return None | ||
|
|
| return mean_score, n_repeats | ||
|
|
||
|
|
||
| class SWEBenchScorer(Scorer, scorer_id="swe_bench_scorer"): |
There was a problem hiding this comment.
This is a huge class, so we can probably move it to a separate file.
Plus please add documentation to give an overview of the functionality.
| PREDEFINED: ClassVar[dict[str, type["Scorer"]]] = {} | ||
| SCORER_ID: ClassVar[str] | ||
| REQUIRES_EXTRACTOR: ClassVar[bool] = True | ||
| SKIP_ENDPOINT_PHASE: ClassVar[bool] = False |
There was a problem hiding this comment.
Doc explaining what this is intended to do?
There was a problem hiding this comment.
Should this be inside templates?
|
|
||
|
|
||
| class RunManager: | ||
| def __init__(self, *, config: ServiceConfig, runner: Any): |
There was a problem hiding this comment.
Please add a type for the runner arg.
| def _effective_external_sample_count( | ||
| eval_cfg: AccuracyConfiguration, | ||
| ) -> int | None: | ||
| count = eval_cfg.scorer.external_sample_count(eval_cfg.extras) | ||
| if count is None: | ||
| return None | ||
| return min(count, eval_cfg.dataset.num_samples()) | ||
|
|
||
|
|
|
|
||
| accuracy_datasets: list[Dataset] = [] | ||
| eval_configs: list[AccuracyConfiguration] = [] | ||
| load_accuracy = test_mode in (TestMode.ACC, TestMode.BOTH) |
There was a problem hiding this comment.
Can we just use the presence of accuracy config in the yaml file to indicate that accuracy must be run. That will keep it consistent.
| def _validate_accuracy_config_for_scorer( | ||
| scorer_cls: type[Scorer], | ||
| dataset_name: str, | ||
| accuracy_config: Any, | ||
| ) -> None: | ||
| if ( | ||
| scorer_cls.SCORER_ID == ScorerMethod.SWE_BENCH.value | ||
| and accuracy_config.num_repeats != 1 | ||
| ): |
There was a problem hiding this comment.
Docstring. Plus this looks like it is only needed for swe-bench. Any reason why num_repeats !=1 isn't supported.
| if os.name == "nt": | ||
| process.terminate() | ||
| else: | ||
| os.killpg(process.pid, signal.SIGTERM) |
| ) | ||
|
|
||
|
|
||
| class SwebenchRunner: |
There was a problem hiding this comment.
nit - rename to SweBenchRunner
| def _normalize_service_url(cls, value: Any) -> str: | ||
| if value is None or str(value).strip() == "": | ||
| raise SetupError( | ||
| "accuracy_config.extras.swebench_service_url is required for " | ||
| "swe_bench_scorer. Start the SWE-bench service and pass its URL." | ||
| ) | ||
| return str(value).strip().rstrip("/") + "/" |
| def _resolve_service_template(cls, extras: dict[str, Any]) -> str: | ||
| raw = extras.get("swebench_template") | ||
| if raw is None: | ||
| raw = ( | ||
| "qwen_tools" | ||
| if cls._get_extra_bool(extras, cls.TOOLCALL_PATCH_EXTRA) | ||
| else "default" | ||
| ) | ||
| template = str(raw) | ||
| if template not in cls.SERVICE_TEMPLATES: | ||
| raise SetupError( | ||
| "accuracy_config.extras.swebench_template must be one of " | ||
| f"{sorted(cls.SERVICE_TEMPLATES)}; got {template!r}" | ||
| ) | ||
| return template |
| # Bound the raw-output read to the accuracy population so finalize never holds | ||
| # the whole run's (incl. perf) response-text corpus. | ||
| accuracy_uuids = ( | ||
| _accuracy_uuid_bound(ctx.report_dir, ctx.eval_configs) | ||
| if has_accuracy | ||
| else set() | ||
| _accuracy_uuid_bound(ctx.report_dir, eval_configs) if has_accuracy else set() | ||
| ) |
| def _redact_secret_fields(value: Any) -> Any: | ||
| if isinstance(value, dict): | ||
| redacted: dict[str, Any] = {} | ||
| for key, item in value.items(): | ||
| normalized = str(key).strip().lower().replace("-", "_") | ||
| if ( | ||
| normalized | ||
| in { | ||
| "api_key", | ||
| "access_token", | ||
| "authorization", | ||
| "auth_token", | ||
| "password", | ||
| "token", | ||
| } | ||
| or normalized.endswith(("_key", "_token", "_password")) | ||
| or "secret" in normalized | ||
| ): | ||
| redacted[key] = "<redacted>" | ||
| else: | ||
| redacted[key] = BenchmarkConfig._redact_secret_fields(item) | ||
| return redacted | ||
| if isinstance(value, list): | ||
| return [BenchmarkConfig._redact_secret_fields(item) for item in value] | ||
| return value |
What does this PR do?
Adds SWE-bench accuracy evaluation to the agentic inference workflow (per #310). A new
SWEBenchdataset loadsSWE-bench_Verified/_Lite, andSWEBenchScorerdrives the eval by shelling out tomini-swe-agentand theswebenchharness in an isolated uv subproject, bypassing the built-in accuracy phase.Agent-run parallelism (
extras.workers) defaults to the load pattern'starget_concurrencywhen unset. Includes example configs, an optional Qwen tool-call patch, and docs.Type of change
Related issues
Closes #310
Testing
Checklist