acceptodds
Under review as a conference paper at ICLR 2027

MLServingBench

Abstract

We introduce MLServingBench, a benchmark for evaluating coding agents at a long horizon by building inference serving systems. Given a model architecture, compute topology, and workload, an agent has 12 to 36 hours in a sandbox with up to 8 H200 GPUs to build a correct, performant system. Our tasks span text generation, agentic workload, and speech and multimodal models, using production workloads. Unlike prior benchmarks that focus on localized code changes or tuning an existing engine, MLServingBench requires end-to-end system building across kernels, memory management, scheduling, and parallelism. To evaluate correctness, we introduce a token-level verifier for text model serving that compares outputs from the agent’s solution against a reference implementation, enabling low-cost and high-fidelity verification. We score correct submissions by their speedup over a production serving framework. We evaluat frontier proprietary and open models across seven tasks and find that no agent outperforms the production baseline on every task, with only 23.8% of runs producing a correct system faster than the baseline. We aim for MLServingBench to guide the development of agents that can reliably build efficient serving systems for new model architectures and workloads.

Primary area:
datasets and benchmarks
Keywords:
Language Models, Benchmark, Machine Learning Systems, Agentic Coding

Reviews

Among:
Shape:
1234this paper, Top 17%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.75 over 4 reviews, above 83% of the others.
Reviewer_dNjTOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces MLServingBench, a benchmark for evaluating coding agents’ ability to build end-to-end machine learning inference serving systems under extended development budgets. The benchmark combines token-level correctness verification for greedy decoding and sampling with task-level metrics where appropriate, and scores correct implementations by their performance relative to production serving frameworks. The authors evaluate six models across seven tasks, conducting three independent 12-hour builds per model–task pair. Only 30 of the 126 main runs produce correct systems that outperform the baseline. These results highlight the difficulty of autonomously building serving systems that are both correct and efficient.

Critical strengths

The benchmark evaluates challenging, end-to-end system-building capabilities. The tasks require agents to coordinate kernel implementation, cache management, batching, scheduling, and parallel execution, going beyond localized code changes or isolated kernel optimization.

Critical weaknesses
  1. The verifier’s reliability needs broader validation. Figure 5 mainly evaluates Qwen3.6-27B on generic serving, with most variants tested only once. Appendix A.6 also notes that the verifier’s overall false-rejection probability is not established. Repeated tests across models and a sensitivity analysis of the tolerance factor h=2h=2 would strengthen confidence.

  2. The InferenceBench comparison mixes differences in verification and workloads. Three of the four defective servers accepted by InferenceBench did not encounter the conditions needed to trigger their bugs. Table 3 therefore shows that the full evaluation pipeline detects more defects, but does not isolate the benefit of token-level verification. Testing both methods on the same workloads would make the comparison more convincing.

  3. The ablation results have limited scope and unclear variability. All ablations use Opus 5 on generic serving, with three builds per condition and a focus on maximum scores. This makes it difficult to judge whether the trends generalize or whether small gains, such as the 3.1% improvement from 24 to 36 hours, are reliable. Reporting all run-level scores and testing another task type would strengthen the conclusions.

  4. I recommand the authors to maintain a leaderboard which should include more models such as Qwen3.8-Max and Gemini4.

Questions for authors

please refer to the paper weakness.

Reviewer_rcAxOpenReview ↗
Rating
2
Confidence
1
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

MLServingBench evaluates whether coding agents can build complete inference serving systems from scratch. Seven tasks cover text, agentic, speech, and multimodal workloads on one to eight H200 GPUs. Agents receive 12-hour development budgets, with extensions to 36 hours in an ablation. Correct submissions are scored by SLO-normalized latency relative to production serving frameworks. The main methodological contribution is a token-level verifier that checks token choices or sampling behavior against a reference while allowing calibrated numerical variation. Across six model–scaffold pairs and three builds per task, 23.8% of runs produce a correct system faster than the baseline. The paper also analyzes development budgets, reasoning effort, implemented optimizations, and concrete serving defects.

Critical strengths
  1. The benchmark tests a useful capability that isolated kernel optimization does not capture. A successful solution must coordinate kernels, cache state, batching, and scheduling over realistic multi-turn workloads. Correctness and performance are measured on the same execution, and evaluation uses a fresh container and withheld workload. The 126 main builds reveal a meaningful gap between producing an optimization and producing a reliable serving system. The per-run success rates, alongside the task maxima, make this gap visible rather than reporting only successful examples.

  2. The correctness evaluation is a substantive contribution. The calibrated token-level checks are supported by controlled variants and six concrete defects in agent-generated servers (Section 4.3 and Appendix C). These examples show why end-task accuracy alone can overlook cache and concurrency bugs. The paper also distinguishes insufficient workload coverage from an insensitive quality gate: only one of the four defects accepted by InferenceBench actually triggers under its workload. This makes the evidence more informative than a simple pass/fail comparison between verifiers.

Critical weaknesses
  1. The reliability of the correctness gate is not yet characterized well enough for a benchmark whose score becomes zero on rejection. The controlled validation uses one model/task and mostly one run per variant (Appendix A.5), with the same production engines used for calibration. The paper acknowledges that the combined sampling checks do not provide an overall false-rejection probability for output-dependent trace lengths, and that aggregate consistency does not establish correct conditional sampling at every position (Appendix A.6). Inspecting rejected submissions for plausible bugs and testing selected defects is useful, but does not establish error rates on independent held-out implementations. This uncertainty directly affects the benchmark’s success rates and comparisons.

  2. The performance results need more context about the production baselines. For example, the largest reported gain is 25.87x on speech recognition, measured at a single offered load of 52 requests/s (Tables 2 and 5). Since the score is based on latency under load, such a gain can combine faster execution with a large reduction in queueing. The PDF identifies baseline versions and workloads, but does not provide the baseline tuning details and absolute latency/throughput breakdown needed to distinguish these effects. This does not invalidate the score, but limits what the headline speedups establish about the quality of the generated systems.

Questions for authors
  1. Verifier validation: Can you report the observed false-rejection rate on additional known-correct implementations or execution configurations held out from calibration, with repeated runs? How sensitive are the main success rates to the tolerance factor and the combined sampling checks?
  2. Verifier reproducibility: Can you provide the actual per-task calibration parameters, including top-k, numerical thresholds, calibration sample sizes, and the resulting rejection statistics? A sensitivity analysis on the existing rollouts would help assess whether close pass/fail decisions affect the ranking.
  3. Baseline interpretation: For speech recognition and one text task, can you provide the production baseline configuration and absolute latency, throughput, and SLO-attainment results, together with a small load sweep? In particular, what explains the 25.87x gain, and does it remain large below baseline saturation?
Reviewer_tMAKOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper introduces MLServingBench, a benchmark for evaluating coding agents' ability to build end-to-end ML inference serving systems from scratch. It includes seven diverse serving tasks covering GPU kernels, memory management, scheduling, and parallelism. The authors propose a token-level correctness verifier to detect implementation errors while accounting for numerical differences and stochastic generation. Experiments across six frontier models, with three independent runs per task, demonstrate the benchmark's difficulty and reveal substantial gaps between current coding agents and production serving systems.

Critical strengths
  1. The paper introduces a challenging benchmark for evaluating coding agents' ability to build end-to-end inference serving systems, extending beyond existing benchmarks that focus on localized code modifications.
  2. The authors conduct an extensive evaluation of six frontier models across seven diverse tasks, with three independent runs per model-task pair. This provides valuable insights into current coding agents' system-building capabilities.
Critical weaknesses
  1. The paper lacks a systematic cross-benchmark comparison of the same models. Without comparing model performance across MLServingBench and existing benchmarks, it remains unclear whether the proposed benchmark captures capabilities that are not adequately reflected by existing evaluations.
  2. The verifier's false rejection risk remains insufficiently evaluated. As acknowledged in Section 6, correct implementations may be rejected due to numerical differences, which may affect the reliability of benchmark results and model comparisons.
Questions for authors
  1. Could you compare the performance or rankings of the same models on MLServingBench and existing benchmarks, such as SWE-bench? This could help clarify whether MLServingBench provides complementary insights into coding agents' capabilities that existing benchmarks may not capture.
  2. Could you provide a Pass-to-Pass (P2P)-style evaluation, similar to SWE-bench, explaining the proportion of independently verified correct serving implementations that pass the proposed verifier? This would help quantify the false rejection risk acknowledged in Section 6.
Reviewer_9w97OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces MLServingBench, a benchmark for evaluating whether coding agents can build complete and efficient ML inference serving systems from scratch. Unlike existing benchmarks that mainly focus on code modifications or individual optimizations, MLServingBench requires agents to handle multiple parts of a serving system, including GPU kernels, memory management, scheduling, and parallelism.

The benchmark includes seven tasks covering language models, coding-agent workloads, speech, and multimodal serving. An important part of the work is its correctness verifier, which checks token-level agreement with a reference implementation while allowing for numerical differences. Only submissions that pass the correctness checks are evaluated for performance relative to production serving frameworks.

The authors evaluate six frontier models and find that building a correct and performant serving system remains challenging. The paper also studies the effects of development time and reasoning effort, and analyzes implementation failures found in agent-generated systems.

Critical strengths
  1. I think the benchmark addresses a meaningful gap in current coding-agent evaluation. Building an entire inference serving system is quite different from fixing a bug or optimizing an isolated kernel. The tasks require agents to coordinate several components, and the seven workloads cover different models and serving requirements. This makes the benchmark a useful complement to existing coding benchmarks.

  2. I find the correctness verification part particularly valuable. Simply checking final task accuracy may miss implementation errors, especially those involving caching or concurrent execution. The proposed token-level checks provide a more careful way to evaluate correctness. I also appreciate the experiments on controlled server variants and real implementation defects, which help explain why the stricter verification is needed.

Critical weaknesses
  1. My main concern is the reliability of the correctness verifier when evaluating implementations that differ substantially from the reference systems. The thresholds are calibrated using trusted serving engines, but a correct implementation with different numerical optimizations might still produce larger deviations. Since failing verification results in a score of zero, potential false rejections could affect the benchmark results. The paper discusses this limitation, but I would like to see more evidence about how often it occurs in practice.

  2. The benchmark is quite expensive to run, which may limit reproducibility and broader adoption. Also, the main results report the best score from three independent runs for each task. While the paper provides overall success rates, I think showing more task-level variation would help readers understand how reliable these results are, rather than mainly reflecting the best outcome within the available attempts.

  3. The evaluation may not fully cover the latest and strongest coding agents. Although several recent models are included, the paper also mentions that some stronger models could not be evaluated due to practical constraints. I understand this limitation, but it makes me wonder whether the main findings would still hold for the strongest currently available agents. A clearer discussion of the model selection criteria would be helpful.

Questions for authors
  1. Regarding the correctness verifier, have the authors evaluated its false-rejection behavior on correct serving implementations that use numerical optimizations different from those in the trusted calibration engines?

  2. Could the authors provide the individual results of the three runs for each model-task pair, rather than only the best score? It would also be useful to know whether the overall model rankings change when considering average performance or success rates instead of the best runs.

  3. Could the authors clarify the model selection criteria and discuss how the exclusion of some stronger or more recent coding agents might affect the conclusions? The paper mentions that certain models could not be evaluated due to safety restrictions. Is there a way to evaluate these models under a modified setup, or any evidence suggesting that the main findings would generalize to them?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.