- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The authors introduce ServingBench, a benchmark for evaluating whether coding agents can implement initial model support in inference engines. ServingBench contains 148 tasks reconstructed from model integration pull requests across vLLM, SGLang, TensorRT-LLM, and 10 accelerator configurations. The benchmark evaluates correctness using released checkpoints and compares successful implementations with the maintainers’ implementations, referred to as Gold, under the same serving setup. Across seven coding models, the average task resolution rate is 56.7%. Approximately 33% of resolved implementations serve more slowly than Gold, with a median serving-speed reduction of 18% among these slower implementations.
Critical strengths
- The authors construct reproducible tasks from real model integration changes across multiple inference engines and accelerator configurations. The construction procedure can incorporate future model integrations, making the benchmark useful as models and serving systems evolve.
- ServingBench evaluates correctness and serving performance using released checkpoints on actual accelerators. The authors validate that the pre-integration version fails the relevant checks and Gold passes them, and compare serving performance under the same conditions. These choices, together with the failure analysis, provide practical value beyond reporting task resolution rates.
Critical weaknesses
-
The error-detection ability of the combined correctness checks needs stronger validation. The authors combine engine-level and live-server comparisons against Gold, independent Hugging Face comparisons where available, tests from the original pull requests, and regression tests. These are important safeguards. Appendix A.5 describes a generation comparison against Gold using three fixed prompts with 32 generated tokens, allowing the first differing tokens when each belongs to the other implementation’s top-five candidates. The Hugging Face comparison separately uses the engine’s numerical correctness tests.
My concern is how reliably this combined evaluation detects incorrect implementations that could otherwise receive a passing result. The fixed-prompt Gold comparison illustrates a limitation in input coverage, while the additional tests provide safeguards whose combined effectiveness is not sufficiently established by the reported results.
The authors could evaluate a representative subset of passing implementations using additional inputs, context lengths, and generation lengths within the stated task scope, together with numerical comparisons against the reference implementations. Model-appropriate downstream evaluations could provide complementary evidence. Reporting additional failures and the proportion of original passing decisions that remain valid would strengthen confidence in the resolution metric. Gold should also be evaluated under the same additional checks to distinguish agent-specific errors from limitations shared with the reference implementation.
-
The behavioral coverage of required model-specific capabilities needs clearer evidence. Appendix B.2 states that each task prompt identifies the required capabilities and the checks used to grade them. However, the paper does not provide a benchmark-wide breakdown of the actual test inputs, assertions, or behavioral coverage for these capabilities.
For capabilities explicitly required by a task’s original integration, such as tensor or expert parallelism, quantization, tool-call parsing, multimodal processing, or reasoning controls, the authors should clarify which behaviors are exercised and how failures are detected. For example, a task requiring reasoning-output handling may need checks for separation of
reasoning_contentandcontent, missing or mixed outputs, and streaming consistency. Thinking-budget and thinking-disable controls are relevant where the original integration requires and supports them. Other serving features, including speculative decoding or LoRA, should be considered only where they are explicitly included in the task scope.A summary of the required behaviors and corresponding test assertions would make this coverage easier to assess. For representative tasks, controlled changes that break a required behavior could also show whether the checks reject partially correct implementations. The existing pre-integration-fails/Gold-passes validation is useful, but does not by itself establish sensitivity to incomplete implementations that already execute successfully.
Questions for authors
- Could the authors re-evaluate a representative subset of passing implementations using additional inputs and context or generation lengths within the task scope? Please report additional failures, numerical differences from the reference implementations, and whether the same discrepancies occur in Gold. Model-appropriate downstream results would provide useful complementary evidence.
- Could the authors expand the reporting in Appendix B.2 with the actual behaviors exercised by the capability checks? In particular, please clarify the inputs and assertions used for reasoning-output separation, supported reasoning controls, tool-call parsing, and multimodal processing where these are required by the original integration.
- How reliably do the checks reject implementations that execute successfully but incorrectly implement a required behavior? A small set of controlled faulty implementations could help answer this question. If additional validation identifies false passes, how would the reported resolution rates and comparisons between coding models change?
4 comments
(continued from the previous comment)
Q3: Faulty implementations and the effect on the results
Q3. How reliably do the checks reject implementations that execute successfully but incorrectly implement a required behavior? A small set of controlled faulty implementations could help answer this question. If additional validation identifies false passes, how would the reported resolution rates and comparisons between coding models change?
We injected faults into Gold on 29 tasks. Each faulty version still loads and serves the model but breaks a required behavior or the model math. The paper's grader rejects all six injected faults in reasoning separation, tool-call parsing, and the multimodal token budget, and the strict re-grade, which extends its comparison with Gold to longer, batched and long-context inputs, rejects all 39 text-path model-math faults it evaluated. As is standard in mutation testing, we exclude equivalent model-math faults, which leave the outputs unchanged.
| Fault family | Example | Faults | Rejected | Check |
|---|---|---|---|---|
| Reasoning separation, tool-call parsing, multimodal token budget | Chain of thought left in the answer, tool-call arguments not JSON-decoded | 6 | 6 | Paper's grader |
| Text-path model math | Causal mask, RoPE base, MoE top-k, sliding window | 39 | 39 | Strict re-grade |
Effect on the reported results. If we count every audited patch that fails the strict re-grade or the downstream evaluations as unresolved and keep the other tasks unchanged, the ranking in Table 2 stays the same, and no model's resolved rate drops by more than 3.4 points.
| Model | Resolved rate in Table 2 (%) | Change after removing flagged patches (pp) |
|---|---|---|
| Claude Opus 5 | 77.7 | −1.4 |
| DeepSeek V4.1 Flash | 71.6 | 0.0 |
| Kimi K3 | 70.3 | −0.7 |
| GPT-5.6 Sol | 60.1 | −3.4 |
| GLM-5.3 | 52.0 | −2.0 |
| Gemini 3.8 Flash | 40.5 | 0.0 |
| Gemini 3.1 Pro Preview | 24.3 | −0.7 |
We will release the strict re-grade, apply it to every resolved patch, and report Table 2 under both graders in the final version.
We hope the additional validation addresses your concerns about the reliability of the evaluation, and we would appreciate your reconsideration of the score. If you have follow-up questions, we are happy to address them and update the paper.
Best regards,
The Authors
Thank you for the detailed response and the new experiments. They answered most of my concerns, so I am raising my score to 3. I look forward to the release of the code. Thank you for the good paper. I learned a lot from reading it.
Thank you very much for your thoughtful feedback and for taking the time to reconsider our response. We really appreciate your constructive suggestions, which helped us strengthen the paper. We look forward to releasing the code and benchmark!
We thank the reviewer for the precise and constructive review and for recognizing the practical value of grading on released checkpoints and real accelerators. We respond to each question below, using Gold as a control throughout.
W1 / Q1: Re-grading with additional inputs
Strict re-grade. On a 30-task subset (18 from the paper's ablation set and 12 drawn at random), we served the resolved patches and Gold side by side on additional inputs, longer contexts (4k, 8k and 16k tokens), longer generations (up to 256 tokens) and concurrent batches, wherever a task supports them. Tokens must agree up to their first difference, where each side's token must be among the other side's five most likely tokens, as in vLLM's model tests. 90 of the 96 re-graded patches (93.8%) pass.
Numerical differences from the reference. Where both Gold runs match the Hugging Face reference, the strict re-grade compares each patch with the reference itself rather than with Gold. It compares tokens under the top-5 rule rather than raw log-probabilities, because batching and kernel choice shift log-probabilities slightly even between two runs of Gold.
Whether the same discrepancies occur in Gold. Gold is served twice as a control, and we count an agent failure only when the two Gold runs agree, so the six failures come from the patches themselves rather than from run-to-run variation in Gold.
Downstream evaluations. As you suggested, on 20 of these tasks we also compared the resolved patches with Gold on standard benchmarks suited to each model, with paired bootstrap CIs. Most patches show no significant drop from Gold, for example 93.5% on GSM8K and 95.0% on MMMU, and counting the patches that do drop as unresolved leaves the ranking unchanged (Q3).
W2 / Q2: Behaviors exercised by the capability checks
We added Appendix L, which lists the inputs and assertions of the checks for reasoning output, tool calls, image input, tensor parallelism and quantization, as summarized below. For Gold parity, the patch and Gold decode three fixed prompts greedily, and their tokens must agree under the same top-5 rule as in Q1.