acceptodds
Under review as a conference paper at ICLR 2027

SWE-Serve: Benchmarking Agentic Engineering For Production Inference Serving

Abstract

We introduce SWE-Serve, a benchmark for evaluating agents on production inference engineering tasks. Implementing an inference feature can require coordinating multiple changes across the serving stack, including model support, runtime execution, and public APIs. Existing benchmarks provide limited coverage of production inference engineering: repository-level software engineering benchmarks do not target inference, while general terminal-agent benchmarks include only a few inference tasks. Dedicated inference benchmarks, meanwhile, focus primarily on isolated kernel generation or performance optimization rather than repository-scale production feature implementation. SWE-Serve provides 53 repository-grounded tasks derived from recent production changes to SGLang, spanning six inference engineering families. Each task executes on either CPU or a single GPU (H100) and is evaluated with hidden functional and regression tests, including, where applicable, end-to-end (E2E) serving tests and calibrated performance gates. Executable no-op and oracle controls, adversarial verifier review, and closed-book execution support task validity and evaluation integrity. Across 11 models and 31 model–effort configurations, the best-performing configuration achieves 75% mean pass@1. SWE-Serve exposes a substantial gap between completing tasks locally and achieving production correctness. On 19 tasks with end-to-end coverage, model-serving E2E tests reject roughly one third of patches that pass every other test (45.9% under the verifier versus 69.4% with E2E tests excluded from scoring), with pass rate increasing for each model’s best-performing configuration. By making the production correctness gap directly measurable, SWE-Serve enables the field to track whether future agents move beyond completing tasks locally to achieving production correctness.

Primary area:
datasets and benchmarks
Keywords:
agents, benchmark, inference engineering, large language models, software engineering, agentic coding, inference serving, end-to-end evaluation

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_kRm4OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SWE-Serve, a benchmark designed to evaluate the ability of code agents to perform software engineering tasks for inference serving systems. The benchmark consists of 53 tasks derived from actual upstream changes in SGLang, covering six categories of engineering work: model support, decoding, caching, scheduling, kernels, and service interfaces. Notably, 16 of these tasks incorporate multiple pull requests. Tasks are executed on either a CPU or a single H100 GPU and are scored based on hidden functional and regression tests; some tasks also include end-to-end tests for real-world model serving or specific performance thresholds. The authors validated the effectiveness of the tasks using several methods: establishing a baseline with no modifications, providing reference implementations, employing adversarial validator reviews, and conducting human evaluations. They evaluated 11 models across 31 different model and inference-intensity configurations within a unified agent framework, with each configuration tested in three independent trials. The best-performing configuration achieved an average pass@1 rate of approximately 75.5%. A key finding is that for the 19 tasks involving end-to-end tests, the pass rate increased from 45.9% to 69.4% when the end-to-end tests were removed while keeping the patches intact.

Critical strengths
  1. Compared to ISO-Bench and InferenceBench, which focus on optimizing inference performance, this work extends the scope to include model support, runtime states, and the implementation of public interfaces—all while requiring that patches preserve existing behavior. By incorporating real-world repositories, multi-file modifications, multi-pull-request tasks, and GPU-based execution, this benchmark offers clear, practical value within the domain.

  2. The validity of the tasks is thoroughly verified, and the core end-to-end performance gaps have been subjected to meaningful scrutiny. All 53 tasks have undergone rigorous validation, including checks against unmodified baselines and reference implementations, reviews of requirement coverage, and task-level adversarial testing. Furthermore, the authors have provided documentation regarding revisions made to the verifiers.

Critical weaknesses

Removing the 23.4 percentage-point gap attributed to end-to-end tests indicates that these tests increase the detection of failures. However, this analysis alone cannot disentangle the distinct contributions of service execution layers, requirement coverage, and test complexity. While the authors have provided a matched control group—consisting of an equal number of cases, identical tasks, and the same F2P/P2P roles—this comparison covers only 12 tasks. Furthermore, some tasks have a limited number of non-E2E tests; consequently, it is necessary to further distinguish between failures arising from cross-component interactions and those resulting from insufficient functional coverage.

Questions for authors
  1. What are the models, inference configurations, number of attempts, and filtering thresholds used for difficulty filtering and verifier revision? At what point are the tasks and the verifier frozen? Do the results presented in Tables 3 and 9 originate entirely from independent runs conducted after this freezing process, or do they include trajectories generated during the development phase?

  2. Could you provide representative patches and the root causes of failures for the primary contribution tasks? Furthermore, do the two types of tests verify the same requirements? For instance, are the failures observed in EAGLE v2, EAGLE3 overlap, and Gemma 4 core-serving primarily due to cross-component interactions, or do they stem from isolated functionalities not yet covered by end-to-end tests?

3 comments

Thanks for your thoughtful feedback and questions. We're thankful for your recognition of SWE-Serve's clear practical value and the thoroughness of both the task validation and end-to-end performance gap analysis.


Q1. Screening and evaluation protocol

A1 Independent runs and freeze point. Tables 3 and 9 use independent runs after tasks and verifiers were frozen; no development trajectories are included. Tasks and verifiers were frozen upon completion of the construction and qualification pipeline described in Figure 6 and Appendix A.1.5, including qualification with executable controls, agent-assisted review and adversarial probing (including difficulty filtering), verifier revision, and final human adjudication.

Difficulty filtering and verifier revision. The table below summarizes the models, inference settings (harness and effort), number of attempts, and filtering thresholds for these stages.

StageModelsHarness / effortAttemptsFiltering threshold
Difficulty screen 1Claude Opus 4.8 and Sonnet 4.6, GPT-5.5 and GPT-5.4-mini, GLM-5.1, Kimi K2.6, Nemotron 3 Ultra, DeepSeek V4 Pro, Qwen3.6-35B-A3BClaude Code, Codex, or Pi; default effort3 per modelExclude tasks solved by none or by 7, 8, or 9 models.
Difficulty screen 2Opus 4.8, GPT-5.5, Nemotron 3 Ultra, Qwen3.6mini-swe-agent; xhigh effort3 per modelExclude tasks solved in 11 or 12 attempts.
Final 10 tasks screenOpus 4.8, GPT-5.5, GPT-5.6 Sol, Nemotron 3 Ultramini-swe-agent; xhigh effort1 per modelExclude tasks solved by all 4 models.
Verifier revisionAgent-created patches: GPT-5.5, GPT-5.6 Sol, Opus 4.8, Nemotron 3 Ultra; Agent-assisted review: GPT-5.6 Sol and Claude Opus 5mini-swe-agent, Codex, and Claude Code; xhigh effort3 per model and harnessRevise verifier when patch and trajectory review identifies acceptance of an incorrect solution or rejection of a valid solution.

We will add this table and explicitly document the freeze point and separation between development and evaluation runs in the appendix.


Q2. E2E coverage and failure analysis.

A2 The aggregate E2E-removal gap measures failures exposed through the live model-serving path defined by the six criteria in Appendix A.1.3. Non-E2E tests do not exercise this full path. The two test types provide complementary coverage and do not necessarily verify the same requirements.

Across the 11 top-per-model configurations in Table 3 (three runs each), 25/33 EAGLE v2, 18/33 EAGLE3 overlap, and 16/33 Gemma 4 core-serving attempts passed all non-E2E tests but failed E2E tests. We inspected the patches and verifier logs to classify these 59 E2E-only failed attempts, considering an attempt a cross-component interaction failure when its failure was caused by components being connected or coordinated incorrectly. Among these 59 failed attempts, 55 (93%) involved cross-component interaction failures, 2 had other identified root causes, and 2 remain uncertain. For each task, we present the root cause of failure for the first E2E-only failing attempt, selected in Table 3 model order and then run order (1–3).

  • EAGLE v2 - Failure: Cross-component interaction. The patch passed the non-E2E worker-selection test by correctly selecting the V2 worker, but server configuration still disabled the required overlap mode for topk > 1. The E2E test exercised the serving path and detected this configuration incompatibility.
  • EAGLE3 overlap - Failure: Cross-component interaction. The patch passed non-E2E draft-token and attention-backend tests, but placed the scheduler’s release signal after the graph runner’s remaining model computation, keeping scheduling blocked after shared-metadata reads had finished. The E2E test exercised scheduling and model execution together through the serving path and detected the resulting failure to overlap them.
  • Gemma 4 core-serving - Failure: Cross-component interaction. The patch passed non-E2E model-registration and configuration checks, but the new multimodal processor supplied mixed CPU/GPU tensors to SGLang’s existing hashing helper, which required both inputs on the same device. The E2E tests exercised server initialization and detected the resulting warm-up failure.

The audit and examples show that, for these three tasks, cross-component interaction failures were involved in the large majority of attempts that passed non-E2E tests but failed E2E tests. We will add these failure analyses and a per-task outcome figure distinguishing E2E-only, non-E2E-only, and joint failures to the appendix. We will also share representative oracle patches for all tasks in a public GitHub repository.

Thank you for the clarifications and additional failure analysis, which substantially address my concerns. I maintain my rating of 3 (weak acceptance).

Thank you for your response! We appreciate your time and insightful questions, which helped us clarify the presentation and strengthen the analysis.

Reviewer_9YvdOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces SWE-Serve, a benchmark of 53 repository-level tasks built from recent merged changes to SGLang, an open-source LLM inference-serving system. Tasks cover six families (e.g., model/backend enablement, speculative decoding, caching, serving APIs) and run on CPU or a single H100. Each task is scored with hidden fail-to-pass and pass-to-pass tests, and 19 tasks also include model-serving end-to-end (E2E) tests that launch a real server with a real checkpoint. Tasks are qualified with no-op/oracle controls, agent-assisted adversarial probing, and human review, and agents are evaluated closed-book.

The authors evaluate 11 models (31 model-effort configurations, 3 runs each). The best configurations (Claude Opus 5 and GPT-5.6 Sol) reach about 75% pass@1. The main finding is a "production correctness gap": rescoring the same patches without E2E tests raises the pass rate from 45.9% to 69.4%. The paper also reports lower pass rates on tasks spanning multiple runtime domains or testing persistent state or concurrency, and shows that higher reasoning effort and native harnesses do not close these gaps.

Critical strengths
  1. New and important domain with realistic tasks. Inference engineering is a high-value area where coding agents are increasingly used, and existing benchmarks cover it poorly. The tasks use real GPU execution, real checkpoints, and real serving paths, and the qualification process (no-op/oracle controls, adversarial probing, human review, transparent rejection reasons) is careful. This is the main reason the benchmark could be a useful resource.

  2. Clean measurement of the E2E gap. Holding patches fixed and only removing E2E tests from scoring is a simple and convincing design. The matched test-removal control, task bootstrap, and the analysis without E2E-only tasks make the finding more credible. This is the paper's main empirical contribution.

Critical weaknesses
  1. Limited novelty and incomplete positioning. Grading with E2E tests is not new (e.g., SWE-Lancer uses engineer-written E2E tests), and prior work has already shown that weak tests inflate agent scores (e.g., SWE-bench+, UTBoost, SWE-ABS). None of these are discussed. The paper also does not show that SWE-Serve measures something different from existing benchmarks, for example via rank correlation with DeepSWE or SWE-bench Pro. Without this, the contribution is mainly a new domain rather than a new insight, and the claim that existing evaluations cannot reveal this gap is overstated.

  2. Score validity is not fully established. (a) Tasks come from PRs merged between December 2025 and July 2026, which overlaps with the training periods of the evaluated models, but pretraining exposure is not tested. Closed-book execution and the canary do not address this. (b) Task selection used observed agent solve rates (28 tasks removed as too easy, 13 removed for zero solves), and 23 of the 53 final tasks had verifiers revised after reviewing agent patches. If these agents overlap with the evaluated models, difficulty and test strength may be tuned to specific models. (c) With 53 tasks and 3 runs, many ranking differences are not statistically resolvable (e.g., the four configurations at 64%), and some subgroup claims rest on few tasks (only 6 concurrency tasks). Together, these issues limit how much the leaderboard and subgroup conclusions can be trusted.

Questions for authors
  1. Which models produced the solve rates used for task selection and the patches used for adversarial probing and verifier revision? Do they overlap with the evaluated models? Do rankings change on the tasks whose verifiers were not revised?
  2. Was pretraining exposure checked? For example, does pass rate differ for PRs merged before vs. after each model's training cutoff?
  3. Were future git objects removed from the task repositories (later commits, remote refs, reflog)? Were package registries such as PyPI reachable during evaluation?
  4. How stable are the E2E tests? How often does the oracle pass them across repeated runs, how many reruns were triggered, and how was "infrastructure failure" distinguished from patch failure? Could you manually audit a sample of patches that fail only E2E tests to confirm they contain real bugs rather than flaky failures or oracle-specific checks?
  5. In the matched control, the E2E-minus-control pass-rate difference is 2.1 pp with an interval of −0.5 to 4.3 pp. How should this be reconciled with the 23.4 pp headline gap? How much of the headline gap is explained simply by the number of E2E tests removed?
  6. How does SWE-Serve differ from SWE-Lancer, UTBoost, and SWE-ABS, and how do its model rankings correlate with DeepSWE or SWE-bench Pro for the 9 matched configurations?
  7. How were the persistent-state, concurrency, and runtime-domain labels produced, and was their reliability checked?
4 comments

Thank you for your careful review and thoughtful feedback. We appreciate your recognition that SWE-Serve addresses an important area poorly covered by existing benchmarks, with realistic tasks, and a careful qualification process.


Q1. Construction-model overlap and ranking robustness

A1. The models used to produce solve rates used for task selection, patches for adversarial probing and verifier revision included Claude Opus 4.8 and Sonnet 4.6, GPT-5.5 and GPT-5.4-mini, GLM-5.1, Kimi K2.6, Nemotron 3 Ultra, DeepSeek V4 Pro, Qwen3.6-35B-A3B, and GPT-5.6 Sol. Only GPT-5.6 Sol overlaps with the evaluated models. GPT-5.6 Sol produced solve rates used to screen 10 tasks, and provided revision patches.

To assess whether rankings change on tasks whose verifiers were not revised, we recomputed the leaderboard on the 25 tasks whose verifiers remained unchanged throughout construction and qualification. The 23 revised tasks reported in the paper refer to one qualification review; this subset excludes tasks revised at any stage. Rankings remain highly consistent with Table 3 (Spearman’s ρ = 0.95). The top-two tie separates by one task solve across three runs (Sol: 61/75; Opus 5: 60/75), and the four models tied at third remain within positions 3–6. The only other change is GLM-5.2 moving from eighth to tenth. We will clarify this and add the 25-task comparison to the appendix.


Q2. Potential pretraining exposure

A2. We checked this for all seven of our 11 models that publish a training-data cutoff. Our cutoff-based analysis did not detect a positive association between potential exposure and pass rate. A simple before/after split does not isolate exposure because later tasks have a lower pass rate for every model: Claude Opus 5 passes 78.6% of tasks merged before its May 2026 cutoff and 59.3% of later ones, but Laguna S 2.1 and Claude Sonnet 5, whose cutoffs precede nearly all of our tasks, drop even more on the same split (22.5 and 31.2 points). GPT-5.6 Sol, tied for the top score, passes 75.9% of the 47 tasks that became public after its cutoff, versus 75.5% overall.

We also regressed each model’s per-task pass rate, averaged over three runs, on whether any source PR was public before that model’s published training cutoff. With model and task fixed effects accounting for differences in model strength and task difficulty, the estimated association between potential exposure and pass rate was −2.3 pp (95% CI −14.4 to +10.1; 5,000 task-level resamples).

These analyses do not establish the absence of pretraining exposure. Complete training-data records are not available to us for the evaluated models, so we use published cutoffs as an imperfect proxy for potential exposure; four models lack published cutoffs. Nevertheless, GPT-5.6 Sol’s comparable performance on the 47 post-cutoff tasks shows that its high overall score does not depend on the small pre-cutoff subset. We will add this analysis and its limitations to the appendix.


Q3. Task repositories and network isolation

A3. Yes, future git objects including later commits, remote refs, and reflog are removed from the task repository. As we enforced a closed-book evaluation, package registries such as PyPI were reachable only during setup, but blocked during agent execution and remained blocked during verification. As described in section 4, only Hugging Face was allowlisted during agent execution because serving tasks require model artifacts. We will clarify these details in the revision.

Q4. E2E stability, reruns, and failure audit

A4. To assess E2E test stability, we executed the oracle solution three times on each of the 19 tasks with E2E tests, using the frozen verifiers. All 57 task executions passed every applicable E2E test; no retries were required.

We cannot reliably report the historical number of evaluation reruns because we no longer have a complete rerun log. As described in Section 4, reruns were permitted only when issues with the model-serving route, runner, hardware, or verifier infrastructure prevented a valid score. Manual review distinguished these failures from agent errors, budget exhaustion, test failures, and patch-caused crashes, which were retained as unsuccessful attempts.

We also manually audited one E2E-only failing patch from each of the 13 tasks with at least one such failure. We reused the three patches analyzed in our response to reviewer kRm4 and, for each remaining task, randomly selected one patch from the E2E-only failing attempts that contributed to the Table 3 leaderboard. For each patch, we examined the task requirements, verifier assertions, patch, and runtime logs. In all 13 cases, we identified a patch defect that violated a task requirement; we found no failures attributable to flakiness or checks requiring the oracle’s particular implementation. For example, the EAGLE v2 patch selected the correct worker but left the overlap mode required for topk > 1 disabled, causing a serving-path configuration failure. The Gemma 4 patch passed model-registration and configuration checks but supplied mixed CPU/GPU tensors to a hashing helper requiring inputs on the same device, causing server warm-up to fail. Each of the 13 unchanged patches reproduced its corresponding failure in three additional verification executions. These results support the validity and repeatability of the audited E2E test failures.


Q5. Headline gap versus matched-count control and sensitivity to number of E2E tests removed

A5. The headline gap and matched control measure different test removals. The 23.4 pp headline gap removes all 276 E2E tests across 19 tasks. The control matches removal counts within task and F2P/P2P role, limiting it to 38 tests per arm across 12 eligible tasks. These smaller removals raise the pass rate by 4.1 pp for E2E tests and 2.0 pp for non-E2E tests on average, yielding the 2.1 pp difference. This 2.1 pp difference compares two count-matched partial-removal conditions; it does not decompose the 23.4 pp full-removal gap into effects of test count and test type. Removing all E2E tests on those same 12 tasks produces a 25.0 pp gap, consistent with the 23.4 pp headline result.

The matched-control [−0.5, 4.3] pp interval describes variation across randomized test-removal selections on fixed tasks and patches. E2E removal produces the larger pass-rate increase in 92.2% of selections, while the central 95% interval includes zero. Thus, the comparison favors E2E removal in most selections, but the advantage is not uniform across selections.

To directly assess sensitivity to test count, we retained one E2E test per task, selected uniformly at random, keeping all non-E2E tests unchanged. Across 10,000 repetitions, retaining one E2E test per task lowered pass rate by an average of 14.2 pp compared with excluding all E2E tests, preserving 60.5% of the headline gap with only 19 of 276 E2E tests. Reducing the E2E suite to one test per task reduces the gap by 9.2 pp, while most of the gap remains. We will add this analysis and clarify the matched control’s scope and interval interpretation in the revision.

Q6. Prior work and cross-benchmark rankings

A6. Comparison with prior work. We agree that SWE-Lancer previously used E2E tests, and that SWE-bench+, UTBoost, and SWE-ABS showed that weak tests can inflate agent scores. SWE-Serve builds on these prior works by explicitly distinguishing non-E2E tests from model-serving E2E tests within its task verifiers. That separation enables paired evaluation of the same patches with and without E2E tests, keeping all other tests unchanged. Using this evaluation on SWE-Serve’s tasks with E2E tests, we find that approximately one-third of patches that pass every non-E2E test still fail E2E tests. Thus, our contribution goes beyond the general observation that weak tests inflate scores to identifying and measuring this specific E2E correctness gap. We will expand the related-work discussion to include these papers and clarify this distinction.

Ranking Correlation. Across the nine matched model-effort configurations, SWE-Serve’s overall rankings strongly correlate with DeepSWE’s (Spearman ρ = 0.89). The correlation is lower on SWE-Serve's 19 E2E-covered tasks (ρ = 0.68), suggesting that these tasks capture some differences in relative model performance alongside shared coding capabilities. We found no matching configurations on Scale’s SWE-bench Pro leaderboard, preventing a matched comparison. We will add the correlations to the appendix.


Q7. Persistent-state, concurrency, and runtime-domain label construction and validation

A7. As described in Section 3.2 and Appendix A.1.4, these labels were assigned before reviewing model outcomes, using task instructions and executable verifier tests. Properties were assigned only when tested: persistent state denotes state maintained across operations or lifecycle phases; concurrency denotes coordination between overlapping operations; runtime-domain labels denote which of the four domains defined in Appendix Table 4 are involved. Appendix Table 5 lists all task-level labels.

To check reliability, we examined passing implementations to determine whether they demonstrated the behaviors indicated by the assigned labels. This checks the original annotation, which used task instructions and executable verifier tests. We selected one passing implementation per task where available, prioritizing passing implementations by configuration rank in Table 3 and then by run number (1–3). For one additional task with no passing implementation under any evaluated mini-swe-agent configuration, we used the first passing implementation from the native-harness experiments. We inspected patches alongside verifier code and available runtime records, supplementing by rerunning the patch for one initially inconclusive concurrency case. All 118 label assignments across the 48 checked tasks were supported. Five tasks had no passing implementation among these configurations and could not be checked this way. We will add this analysis to the appendix.


With 53 tasks and 3 runs, many ranking differences are not statistically resolvable (e.g., the four configurations at 64%), and some subgroup claims rest on few tasks (only 6 concurrency tasks).

We agree that mean pass@1 alone does not establish a precise ordering among all top-per-model configurations, and we will clarify this interpretation. To provide a more complete understanding of differences between the evaluated configurations, Table 3 reports the 95% confidence interval across three runs, pass³ for repeated-run reliability, and resource-use metrics; Table 9 additionally reports pass@3. Section 5.1 emphasizes that configuration choice depends on more than pass@1. For example, among the four configurations at 64% mean pass@1, mean per-task cost varies by 7.6× (USD 0.95–7.24) and wall-clock time by 3.9× (25.5–99.9 minutes).

We agree that the six-task concurrency comparison warrants cautious interpretation. Nevertheless, the observed 29.0 pp performance gap highlights a potentially important challenge for coding agents and motivates further evaluation on tasks requiring concurrent coordination.

Thank you again for your detailed and constructive review! We hope the additional analyses and clarifications address your concerns about score validity and the contribution of the paired evaluation with and without E2E tests. If these responses resolve your concerns, we would be grateful if you would consider revising your rating accordingly.

Reviewer_HSCJOpenReview ↗
Rating
3
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper targets a gap between repository-level software engineering benchmarks and kernel- or performance-focused inference benchmarks: implementing production features in an LLM serving engine, which often requires coordinated changes across model support, runtime, and public APIs. SWE-Serve derives tasks from real SGLang changes across several inference-engineering families and grades them with hidden functional and regression tests, plus end-to-end serving and performance checks where applicable. Task validity is supported by no-op/oracle controls, adversarial review of the verifiers, and closed-book execution. The central message is that passing local tests is not the same as production correctness, and the end-to-end serving tests are designed to make that gap measurable.

Critical strengths
  • Task construction is careful and transparent: every task passes no-op and oracle controls, the reasons for discarding candidates are documented, and per-task outcomes are reported.
  • The benchmark fills a real gap between SWE-bench-style and kernel-level benchmarks by running end-to-end serving on real checkpoints, and the closed-book setting is empirically motivated."
Critical weaknesses
  • Task construction is careful and transparent: every task passes no-op and oracle controls, the reasons for discarding candidates are documented, and per-task outcomes are reported.
  • The benchmark fills a real gap between SWE-bench-style and kernel-level benchmarks by running end-to-end serving on real checkpoints, and the closed-book setting is empirically motivated."
Questions for authors
  1. Which models produced the solve rates and patches used for task selection and verifier revision, and do they overlap with the evaluated models?
  2. During development, could agents access the GPU and download checkpoints to run end-to-end serving themselves?
  3. How large is the end-to-end gap when tasks that have only end-to-end tests are excluded?"
2 comments

Thank you for reviewing our paper and providing thoughtful feedback. We appreciate your recognition of our careful and transparent task construction and the gap SWE-Serve fills in existing benchmarks.

Q1. Which models produced the solve rates and patches used for task selection and verifier revision, and do they overlap with the evaluated models?

A1. The models used to produce solve rates and patches used for task selection and verifier revision included Claude Opus 4.8 and Sonnet 4.6, GPT-5.5 and GPT-5.4-mini, GLM-5.1, Kimi K2.6, Nemotron 3 Ultra, DeepSeek V4 Pro, Qwen3.6-35B-A3B, and GPT-5.6 Sol. Only GPT-5.6 Sol overlaps with the evaluated models. GPT-5.6 Sol produced solve rates used to screen 10 tasks, and provided patches used for verifier revision. We will clarify this in the appendix.


Q2. During development, could agents access the GPU and download checkpoints to run end-to-end serving themselves?

A2. Yes. During agent execution, agents could download checkpoints from Hugging Face, and those working on GPU tasks had access to one H100 GPU. CPU tasks did not require GPU access. We will clarify this in the revision.


Q3. How large is the end-to-end gap when tasks that have only end-to-end tests are excluded?

A3. As described in Section 5.2 and Appendix Table 10, excluding the tasks containing only end-to-end tests results in a 22.8 percentage point gap. Across the 15 remaining tasks, the pass rate increases from 38.6% to 61.4% when end-to-end tests are removed from scoring. This is close to the 23.4-point gap across all 19 tasks, showing that the observed gap is not driven by tasks containing only end-to-end tests.

Thank you again for your thoughtful review! We hope our responses clarify the model overlap and agents’ access to GPU resources, and that the analysis excluding E2E-only tasks addresses your question about the end-to-end gap. If these responses resolve your concerns, we would be grateful if you would consider revising your rating accordingly.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.