- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SWE-Serve, a benchmark designed to evaluate the ability of code agents to perform software engineering tasks for inference serving systems. The benchmark consists of 53 tasks derived from actual upstream changes in SGLang, covering six categories of engineering work: model support, decoding, caching, scheduling, kernels, and service interfaces. Notably, 16 of these tasks incorporate multiple pull requests. Tasks are executed on either a CPU or a single H100 GPU and are scored based on hidden functional and regression tests; some tasks also include end-to-end tests for real-world model serving or specific performance thresholds. The authors validated the effectiveness of the tasks using several methods: establishing a baseline with no modifications, providing reference implementations, employing adversarial validator reviews, and conducting human evaluations. They evaluated 11 models across 31 different model and inference-intensity configurations within a unified agent framework, with each configuration tested in three independent trials. The best-performing configuration achieved an average pass@1 rate of approximately 75.5%. A key finding is that for the 19 tasks involving end-to-end tests, the pass rate increased from 45.9% to 69.4% when the end-to-end tests were removed while keeping the patches intact.
Critical strengths
-
Compared to ISO-Bench and InferenceBench, which focus on optimizing inference performance, this work extends the scope to include model support, runtime states, and the implementation of public interfaces—all while requiring that patches preserve existing behavior. By incorporating real-world repositories, multi-file modifications, multi-pull-request tasks, and GPU-based execution, this benchmark offers clear, practical value within the domain.
-
The validity of the tasks is thoroughly verified, and the core end-to-end performance gaps have been subjected to meaningful scrutiny. All 53 tasks have undergone rigorous validation, including checks against unmodified baselines and reference implementations, reviews of requirement coverage, and task-level adversarial testing. Furthermore, the authors have provided documentation regarding revisions made to the verifiers.
Critical weaknesses
Removing the 23.4 percentage-point gap attributed to end-to-end tests indicates that these tests increase the detection of failures. However, this analysis alone cannot disentangle the distinct contributions of service execution layers, requirement coverage, and test complexity. While the authors have provided a matched control group—consisting of an equal number of cases, identical tasks, and the same F2P/P2P roles—this comparison covers only 12 tasks. Furthermore, some tasks have a limited number of non-E2E tests; consequently, it is necessary to further distinguish between failures arising from cross-component interactions and those resulting from insufficient functional coverage.
Questions for authors
-
What are the models, inference configurations, number of attempts, and filtering thresholds used for difficulty filtering and verifier revision? At what point are the tasks and the verifier frozen? Do the results presented in Tables 3 and 9 originate entirely from independent runs conducted after this freezing process, or do they include trajectories generated during the development phase?
-
Could you provide representative patches and the root causes of failures for the primary contribution tasks? Furthermore, do the two types of tests verify the same requirements? For instance, are the failures observed in EAGLE v2, EAGLE3 overlap, and Gemma 4 core-serving primarily due to cross-component interactions, or do they stem from isolated functionalities not yet covered by end-to-end tests?
3 comments
Thank you for the clarifications and additional failure analysis, which substantially address my concerns. I maintain my rating of 3 (weak acceptance).
Thank you for your response! We appreciate your time and insightful questions, which helped us clarify the presentation and strengthen the analysis.
Thanks for your thoughtful feedback and questions. We're thankful for your recognition of SWE-Serve's clear practical value and the thoroughness of both the task validation and end-to-end performance gap analysis.
Q1. Screening and evaluation protocol
A1 Independent runs and freeze point. Tables 3 and 9 use independent runs after tasks and verifiers were frozen; no development trajectories are included. Tasks and verifiers were frozen upon completion of the construction and qualification pipeline described in Figure 6 and Appendix A.1.5, including qualification with executable controls, agent-assisted review and adversarial probing (including difficulty filtering), verifier revision, and final human adjudication.
Difficulty filtering and verifier revision. The table below summarizes the models, inference settings (harness and effort), number of attempts, and filtering thresholds for these stages.
We will add this table and explicitly document the freeze point and separation between development and evaluation runs in the appendix.
Q2. E2E coverage and failure analysis.
A2 The aggregate E2E-removal gap measures failures exposed through the live model-serving path defined by the six criteria in Appendix A.1.3. Non-E2E tests do not exercise this full path. The two test types provide complementary coverage and do not necessarily verify the same requirements.
Across the 11 top-per-model configurations in Table 3 (three runs each), 25/33 EAGLE v2, 18/33 EAGLE3 overlap, and 16/33 Gemma 4 core-serving attempts passed all non-E2E tests but failed E2E tests. We inspected the patches and verifier logs to classify these 59 E2E-only failed attempts, considering an attempt a cross-component interaction failure when its failure was caused by components being connected or coordinated incorrectly. Among these 59 failed attempts, 55 (93%) involved cross-component interaction failures, 2 had other identified root causes, and 2 remain uncertain. For each task, we present the root cause of failure for the first E2E-only failing attempt, selected in Table 3 model order and then run order (1–3).
The audit and examples show that, for these three tasks, cross-component interaction failures were involved in the large majority of attempts that passed non-E2E tests but failed E2E tests. We will add these failure analyses and a per-task outcome figure distinguishing E2E-only, non-E2E-only, and joint failures to the appendix. We will also share representative oracle patches for all tasks in a public GitHub repository.