- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper introduces MLServingBench, a benchmark for evaluating coding agents’ ability to build end-to-end machine learning inference serving systems under extended development budgets. The benchmark combines token-level correctness verification for greedy decoding and sampling with task-level metrics where appropriate, and scores correct implementations by their performance relative to production serving frameworks. The authors evaluate six models across seven tasks, conducting three independent 12-hour builds per model–task pair. Only 30 of the 126 main runs produce correct systems that outperform the baseline. These results highlight the difficulty of autonomously building serving systems that are both correct and efficient.
Critical strengths
The benchmark evaluates challenging, end-to-end system-building capabilities. The tasks require agents to coordinate kernel implementation, cache management, batching, scheduling, and parallel execution, going beyond localized code changes or isolated kernel optimization.
Critical weaknesses
-
The verifier’s reliability needs broader validation. Figure 5 mainly evaluates Qwen3.6-27B on generic serving, with most variants tested only once. Appendix A.6 also notes that the verifier’s overall false-rejection probability is not established. Repeated tests across models and a sensitivity analysis of the tolerance factor would strengthen confidence.
-
The InferenceBench comparison mixes differences in verification and workloads. Three of the four defective servers accepted by InferenceBench did not encounter the conditions needed to trigger their bugs. Table 3 therefore shows that the full evaluation pipeline detects more defects, but does not isolate the benefit of token-level verification. Testing both methods on the same workloads would make the comparison more convincing.
-
The ablation results have limited scope and unclear variability. All ablations use Opus 5 on generic serving, with three builds per condition and a focus on maximum scores. This makes it difficult to judge whether the trends generalize or whether small gains, such as the 3.1% improvement from 24 to 36 hours, are reliable. Reporting all run-level scores and testing another task type would strengthen the conclusions.
-
I recommand the authors to maintain a leaderboard which should include more models such as Qwen3.8-Max and Gemini4.
Questions for authors
please refer to the paper weakness.