acceptodds
Under review as a conference paper at ICLR 2027

ServingBench: Can Agents Deliver Day-0 Model Support in Production Inference Engines?

Abstract

Serving engines such as vLLM now provide day-0 model support, making major open-weight models available on the day they are released. However, supporting a new model correctly and efficiently on day 0 can be non-trivial. Engineers must implement the new model architecture, integrate it with the serving runtime, and, for some models, optimize kernels and backend code for the target accelerator. We therefore ask: can coding agents deliver correct and efficient day-0 model support? To answer this question, we introduce ServingBench, a benchmark of 148 day-0 model support tasks across 10 accelerator tiers from NVIDIA, AMD, and Google. An automated pipeline constructs executable environments from selected model-integration pull requests in vLLM, SGLang, and TensorRT-LLM. Each environment includes correctness checks and an expert performance baseline validated on the target accelerator. Evaluating seven state-of-the-art models, we find that agents resolve 56.7% of tasks on average, but 33% of resolved patches serve more slowly than the engine maintainers’ implementation, by a median of 18%. Even the strongest model in our evaluation (i.e., Claude Opus 5) delivers a correct patch at the expert’s serving speed on only 52.7% of tasks, and 32% of its resolved patches are slower, some at about half the expert’s speed. The pipeline can incorporate new model releases, allowing ServingBench to track progress on a continually evolving systems engineering task.

Primary area:
datasets and benchmarks
Keywords:
Model serving, inference systems, software engineering benchmark

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_SWhDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors introduce ServingBench, a benchmark for evaluating whether coding agents can implement initial model support in inference engines. ServingBench contains 148 tasks reconstructed from model integration pull requests across vLLM, SGLang, TensorRT-LLM, and 10 accelerator configurations. The benchmark evaluates correctness using released checkpoints and compares successful implementations with the maintainers’ implementations, referred to as Gold, under the same serving setup. Across seven coding models, the average task resolution rate is 56.7%. Approximately 33% of resolved implementations serve more slowly than Gold, with a median serving-speed reduction of 18% among these slower implementations.

Critical strengths
  1. The authors construct reproducible tasks from real model integration changes across multiple inference engines and accelerator configurations. The construction procedure can incorporate future model integrations, making the benchmark useful as models and serving systems evolve.
  2. ServingBench evaluates correctness and serving performance using released checkpoints on actual accelerators. The authors validate that the pre-integration version fails the relevant checks and Gold passes them, and compare serving performance under the same conditions. These choices, together with the failure analysis, provide practical value beyond reporting task resolution rates.
Critical weaknesses
  1. The error-detection ability of the combined correctness checks needs stronger validation. The authors combine engine-level and live-server comparisons against Gold, independent Hugging Face comparisons where available, tests from the original pull requests, and regression tests. These are important safeguards. Appendix A.5 describes a generation comparison against Gold using three fixed prompts with 32 generated tokens, allowing the first differing tokens when each belongs to the other implementation’s top-five candidates. The Hugging Face comparison separately uses the engine’s numerical correctness tests.

    My concern is how reliably this combined evaluation detects incorrect implementations that could otherwise receive a passing result. The fixed-prompt Gold comparison illustrates a limitation in input coverage, while the additional tests provide safeguards whose combined effectiveness is not sufficiently established by the reported results.

    The authors could evaluate a representative subset of passing implementations using additional inputs, context lengths, and generation lengths within the stated task scope, together with numerical comparisons against the reference implementations. Model-appropriate downstream evaluations could provide complementary evidence. Reporting additional failures and the proportion of original passing decisions that remain valid would strengthen confidence in the resolution metric. Gold should also be evaluated under the same additional checks to distinguish agent-specific errors from limitations shared with the reference implementation.

  2. The behavioral coverage of required model-specific capabilities needs clearer evidence. Appendix B.2 states that each task prompt identifies the required capabilities and the checks used to grade them. However, the paper does not provide a benchmark-wide breakdown of the actual test inputs, assertions, or behavioral coverage for these capabilities.

    For capabilities explicitly required by a task’s original integration, such as tensor or expert parallelism, quantization, tool-call parsing, multimodal processing, or reasoning controls, the authors should clarify which behaviors are exercised and how failures are detected. For example, a task requiring reasoning-output handling may need checks for separation of reasoning_content and content, missing or mixed outputs, and streaming consistency. Thinking-budget and thinking-disable controls are relevant where the original integration requires and supports them. Other serving features, including speculative decoding or LoRA, should be considered only where they are explicitly included in the task scope.

    A summary of the required behaviors and corresponding test assertions would make this coverage easier to assess. For representative tasks, controlled changes that break a required behavior could also show whether the checks reject partially correct implementations. The existing pre-integration-fails/Gold-passes validation is useful, but does not by itself establish sensitivity to incomplete implementations that already execute successfully.

Questions for authors
  1. Could the authors re-evaluate a representative subset of passing implementations using additional inputs and context or generation lengths within the task scope? Please report additional failures, numerical differences from the reference implementations, and whether the same discrepancies occur in Gold. Model-appropriate downstream results would provide useful complementary evidence.
  2. Could the authors expand the reporting in Appendix B.2 with the actual behaviors exercised by the capability checks? In particular, please clarify the inputs and assertions used for reasoning-output separation, supported reasoning controls, tool-call parsing, and multimodal processing where these are required by the original integration.
  3. How reliably do the checks reject implementations that execute successfully but incorrectly implement a required behavior? A small set of controlled faulty implementations could help answer this question. If additional validation identifies false passes, how would the reported resolution rates and comparisons between coding models change?
4 comments

We thank the reviewer for the precise and constructive review and for recognizing the practical value of grading on released checkpoints and real accelerators. We respond to each question below, using Gold as a control throughout.

W1 / Q1: Re-grading with additional inputs

Q1. Could the authors re-evaluate a representative subset of passing implementations using additional inputs and context or generation lengths within the task scope? Please report additional failures, numerical differences from the reference implementations, and whether the same discrepancies occur in Gold. Model-appropriate downstream results would provide useful complementary evidence.

Strict re-grade. On a 30-task subset (18 from the paper's ablation set and 12 drawn at random), we served the resolved patches and Gold side by side on additional inputs, longer contexts (4k, 8k and 16k tokens), longer generations (up to 256 tokens) and concurrent batches, wherever a task supports them. Tokens must agree up to their first difference, where each side's token must be among the other side's five most likely tokens, as in vLLM's model tests. 90 of the 96 re-graded patches (93.8%) pass.

Numerical differences from the reference. Where both Gold runs match the Hugging Face reference, the strict re-grade compares each patch with the reference itself rather than with Gold. It compares tokens under the top-5 rule rather than raw log-probabilities, because batching and kernel choice shift log-probabilities slightly even between two runs of Gold.

Whether the same discrepancies occur in Gold. Gold is served twice as a control, and we count an agent failure only when the two Gold runs agree, so the six failures come from the patches themselves rather than from run-to-run variation in Gold.

Downstream evaluations. As you suggested, on 20 of these tasks we also compared the resolved patches with Gold on standard benchmarks suited to each model, with paired bootstrap CIs. Most patches show no significant drop from Gold, for example 93.5% on GSM8K and 95.0% on MMMU, and counting the patches that do drop as unresolved leaves the ranking unchanged (Q3).

W2 / Q2: Behaviors exercised by the capability checks

Q2. Could the authors expand the reporting in Appendix B.2 with the actual behaviors exercised by the capability checks? In particular, please clarify the inputs and assertions used for reasoning-output separation, supported reasoning controls, tool-call parsing, and multimodal processing where these are required by the original integration.

We added Appendix L, which lists the inputs and assertions of the checks for reasoning output, tool calls, image input, tensor parallelism and quantization, as summarized below. For Gold parity, the patch and Gold decode three fixed prompts greedily, and their tokens must agree under the same top-5 rule as in Q1.

BehaviorInputAssertion
Reasoning-output separationReasoning followed by an answer or a tool call, or truncated reasoning, streamed and non-streamedExact split into reasoning and content, with no tool call parsed from the reasoning
Reasoning controlsThinking on, off, or by default, and each reasoning-effort levelReasoning split out only when thinking is on, and each effort level rendered as in the reference
Tool-call parsingSingle and parallel calls with JSON object, array, and boolean arguments, streamed in small and large chunksExact tool names and arguments, and no tool call from a plain answer
Image input32 synthetic images with a fixed instructionFirst 16 tokens compared with Gold under the top-5 rule
Tensor parallelismA tensor-parallel size equal to the device countGold parity
QuantizationThe released quantized checkpointGold parity

(continued from the previous comment)

Q3: Faulty implementations and the effect on the results

Q3. How reliably do the checks reject implementations that execute successfully but incorrectly implement a required behavior? A small set of controlled faulty implementations could help answer this question. If additional validation identifies false passes, how would the reported resolution rates and comparisons between coding models change?

We injected faults into Gold on 29 tasks. Each faulty version still loads and serves the model but breaks a required behavior or the model math. The paper's grader rejects all six injected faults in reasoning separation, tool-call parsing, and the multimodal token budget, and the strict re-grade, which extends its comparison with Gold to longer, batched and long-context inputs, rejects all 39 text-path model-math faults it evaluated. As is standard in mutation testing, we exclude equivalent model-math faults, which leave the outputs unchanged.

Fault familyExampleFaultsRejectedCheck
Reasoning separation, tool-call parsing, multimodal token budgetChain of thought left in the answer, tool-call arguments not JSON-decoded66Paper's grader
Text-path model mathCausal mask, RoPE base, MoE top-k, sliding window3939Strict re-grade

Effect on the reported results. If we count every audited patch that fails the strict re-grade or the downstream evaluations as unresolved and keep the other tasks unchanged, the ranking in Table 2 stays the same, and no model's resolved rate drops by more than 3.4 points.

ModelResolved rate in Table 2 (%)Change after removing flagged patches (pp)
Claude Opus 577.7−1.4
DeepSeek V4.1 Flash71.60.0
Kimi K370.3−0.7
GPT-5.6 Sol60.1−3.4
GLM-5.352.0−2.0
Gemini 3.8 Flash40.50.0
Gemini 3.1 Pro Preview24.3−0.7

We will release the strict re-grade, apply it to every resolved patch, and report Table 2 under both graders in the final version.

We hope the additional validation addresses your concerns about the reliability of the evaluation, and we would appreciate your reconsideration of the score. If you have follow-up questions, we are happy to address them and update the paper.

Best regards,

The Authors

Thank you for the detailed response and the new experiments. They answered most of my concerns, so I am raising my score to 3. I look forward to the release of the code. Thank you for the good paper. I learned a lot from reading it.

Thank you very much for your thoughtful feedback and for taking the time to reconsider our response. We really appreciate your constructive suggestions, which helped us strengthen the paper. We look forward to releasing the code and benchmark!

Reviewer_ooTuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ServingBench is a benchmark of 148 model "bring-up" tasks drawn from merged PRs in vLLM, SGLang, and TensorRT-LLM (plus the vLLM TPU backend). Each task gives the agent the engine just before the model-support PR, the released checkpoint, and the Hugging Face reference, and asks for a patch that first passes correctness checks and is then timed against the maintainers' merged implementation (Gold). Tasks are built by a four-stage pipeline and validated so that Base fails each check and Gold passes it. Seven models are run with mini-SWE-agent (200 steps, 4 hours, pass@1) across 10 accelerator tiers. Average resolve rate is 56.7%; a third of resolved patches are slower than Gold, median 18% (up to 97%). The paper also breaks down failures (integration vs. wrong math vs. crashes) and shows kernel work, TensorRT-LLM, multi-device, and large PRs are harder. Ablations cover the HF reference, a cross-engine Gold patch, and harness choice.

Critical strengths
  1. The construction pipeline is the real contribution. Rebuilding pre-integration engine states, checking them against both the released checkpoint and the maintainers' patch, and doing this across engines and accelerators is non-trivial engineering. It is what makes the benchmark extensible to future releases rather than a one-off task list.

  2. The task mix addresses a genuine gap. Existing coding/performance benchmarks either grade correctness on CPUs or optimize code that already works; requiring a correct checkpoint bring-up and a non-trivial serving speed on real hardware is a meaningful axis. The reference patches (1,485 lines, 45 functions, 12 classes on average) confirm the tasks are not trivial.

Critical weaknesses
  1. Contamination is unaddressed and could explain much of the headline result. Every task comes from a public merged PR in a public repo, and the evaluated models are recent enough that their training data plausibly overlaps these PRs. The mitigations (hidden Gold, truncated history, no network) only stop the agent from fetching the answer at run time; they do nothing about prior exposure. The Limitations section admits the overlap is hard to establish, but no experiment is run — no recall probe, no before/after-cutoff comparison. Without that, the resolve rates and the "gap to expert" numbers are hard to trust, and the benchmark's central claim about agent capability is unverifiable. This is the main reason I lean reject.

  2. The prompt gives away a lot of the hard part, and the measurement is narrow. The agent is told the exact interface signatures that hidden tests import, the workload shapes, and has the full HF reference. That removes much of the discovery and wiring work that "model bring-up" implies and turns the task into translating a known model into an engine's abstractions. Correctness is a single parity check (logprobs_close / cosine at a fixed threshold), and performance is one fixed workload at concurrency 32. So "slower than expert" is conditional on one operating point, and the Gold comparison is not symmetric: Gold had review cycles and unbounded time, the agent gets one 200-step run. The paper reads these numbers as a capability gap, but the setup makes it hard to separate capability from benchmark design.

Questions for authors

Can you provide any contamination evidence? Even a simple probe (ask the model to reconstruct the PR, or compare tasks merged before vs. after a model's stated cutoff) would materially change how I read the results.

How much do resolve rates drop if the prompt does not reveal the interface signatures and workload shapes? That would tell us how much of the task is actually being tested.

Why only one workload/concurrency per task? Do the efficiency conclusions hold at other operating points (e.g., higher concurrency, latency-bound)?

The Gold patches come from maintainer PRs with review and iteration; the agent gets a 4-hour single trajectory. How do you separate capability from this asymmetry in the "slower than expert" claim?

The subgroup conclusions (kernel work, TensorRT-LLM, multi-device) rest on small counts — multi-device is roughly 13 tasks. How stable are these?

The paper frames tasks as "day-0," but the HF reference may post-date the model's release. How much does this shift the difficulty relative to the original bring-up?

2 comments

We thank the reviewer for the detailed review and for recognizing the value of the construction pipeline and the task mix. We address the questions below, starting with contamination, which we agree is the key question for a benchmark built from public PRs.

W1 / Q1: Contamination

Q1. Can you provide any contamination evidence? Even a simple probe (ask the model to reconstruct the PR, or compare tasks merged before vs. after a model's stated cutoff) would materially change how I read the results.

We find no detectable association between task publication timing and success, and the seven models retain their original ranking on a new post-release task set. These results do not rule out all prior exposure, but the ranking also holds on tasks whose source PRs were opened after all evaluated models were released.

Training cutoff (before/after comparison). We split the 148 tasks at each model's stated cutoff and compared the two sides, accounting for the model and for the task's size, kernel work, engine, modality and device count. Tasks merged after the cutoff are resolved about as often as those merged before it, and their resolved patches are equally fast.

Tasks mergedResolved (%)SlowerWithin 5%FasterMedian ratio
Before the model's cutoff57.934.7%52.2%13.1%0.99
After the model's cutoff56.531.2%57.6%11.2%0.99

Resolved rates are standardized to the same mix of models and tasks. Each model's cutoff is its published knowledge cutoff or, where none is published, the data-collection date in its EU AI Act training-data summary. Appendix G lists each model's cutoff and its source and reports the regression.

New tasks. We also used the construction pipeline to build 25 new tasks, in about a day, from every qualifying bring-up PR merged between October 6 and November 6, 2026. All of them were opened after the last evaluated model was released (September 10). They span vLLM, SGLang and TensorRT-LLM on H100 and B200 GPUs and include speech, vision and embedding models. We ran all seven models on them (175 runs) and scored them with the correctness checks.

ModelRank in Table 2Pass rate on the new tasks (%)Rank on the new tasks
Claude Opus 5192.01
DeepSeek V4.1 Flash284.02
Kimi K3380.03
GPT-5.6 Sol464.04
GLM-5.3552.05
Gemini 3.8 Flash628.06
Gemini 3.1 Pro Preview720.07

The seven models rank exactly as in Table 2 (Kendall τ = 1.00). We will release this split and keep adding new tasks with the same pipeline, so every new model can be evaluated on PRs merged after its cutoff.

W2 / Q2: Prompt information

Q2. How much do resolve rates drop if the prompt does not reveal the interface signatures and workload shapes? That would tell us how much of the task is actually being tested.

We removed the workload shapes, the interface section, or both from the prompt and ran three models on 30 NVIDIA tasks (18 from the paper's ablation set and 12 drawn at random). None of these changes moves the pass rate on the correctness checks significantly (Appendix F).

Prompt changePass rate (%)Δ (pp)p
Hide the workload shapes73.3 → 68.9−4.40.39
Hide the interface section73.3 → 73.30.01.00
Hide both73.3 → 70.0−3.30.61

Hiding the interface matters little because it gives only the file path and signature of each entry point that a hidden test imports, as in FEA-Bench (Li et al., 2025), and none of the implementation. Behind these entry points, the Gold patches add 1,485 lines, 45 functions, and 12 classes on average.

(continued from the previous comment)

W2 / Q3: Other workloads and concurrency

Q3. Why only one workload/concurrency per task? Do the efficiency conclusions hold at other operating points (e.g., higher concurrency, latency-bound)?

Yes for the median ratio. We re-timed 75 resolved patches from the same 30 tasks as in Q2 on 8 workloads, a prefill-heavy and a decode-heavy shape each at concurrency 1, 8, 32 and 128. The median ratio to the human expert stays at 0.99 on every workload, and most patches keep the Slower / Within 5% / Faster class they have on the main-results workload (Appendix H).

WorkloadMedian ratio to the human expertSame speed class
Prefill-heavy, concurrency 32 (main results)0.99–
Prefill-heavy, concurrency 10.9973%
Prefill-heavy, concurrency 80.9988%
Prefill-heavy, concurrency 1280.9980%
Decode-heavy, concurrency 10.9975%
Decode-heavy, concurrency 80.9979%
Decode-heavy, concurrency 320.9983%
Decode-heavy, concurrency 1280.9981%

We measure throughput on each workload, and we will add latency metrics such as TTFT and TPOT in the next version.

W2 also asks about correctness. Besides the parity tests and the PR's own tests (§3), the new strict re-grade checks longer, batched and long-context inputs (Appendix K).

W2 / Q4: Asymmetry with Gold

Q4. The Gold patches come from maintainer PRs with review and iteration; the agent gets a 4-hour single trajectory. How do you separate capability from this asymmetry in the "slower than expert" claim?

We cannot fully eliminate the asymmetry in development time and review, but we tested how sensitive the results are to the agent's budget. A 4× step and time budget moves the pass rate from 70.0% to 75.0%, which is not statistically significant (p = 0.69).

On the maintainers' side, we timed the engine releases from about 6 and 12 months after Gold on 28 tasks drawn at random from the NVIDIA tasks whose Gold is at least a year old.

ReferenceSpeed vs Gold (median)Resolved patches slower than itTheir median slowdown
Release at +180 days1.0451%20%
Release at +365 days1.0755%25%

Gold is the first merged implementation, not the best eventual one. Agent patches match it at the median but fall further behind each later release. Like SWE-bench and SWE-fficiency, ServingBench measures what an agent delivers within a fixed budget, against the implementation the maintainers first merged (Appendix I).

Q5: Stability of the subgroup results

Q5. The subgroup conclusions (kernel work, TensorRT-LLM, multi-device) rest on small counts — multi-device is roughly 13 tasks. How stable are these?

The subgroup differences hold with task-level bootstrap CIs and in a joint logistic regression that accounts for the seven runs on each task (Appendix J).

ContrastTasksDifference, pp [95% CI]Joint odds ratio [95% CI]
Kernel work vs none34 vs 114−29.4 [−41.7, −16.8]0.34 [0.12, 0.96]
TensorRT-LLM vs vLLM25 vs 55−30.0 [−45.5, −13.4]0.19 [0.06, 0.64]
Multi-device vs single device13 vs 135−16.3 [−31.1, −2.4]0.37 [0.20, 0.70]
Largest vs smallest PR-size quartile37 vs 37−30.5 [−44.1, −17.0]0.37 [0.21, 0.65]

Every difference CI excludes zero, and every odds-ratio CI excludes one. The PR-size odds ratio is per e-fold increase in size.

Q6: Reference revisions

Q6. The paper frames tasks as "day-0," but the HF reference may post-date the model's release. How much does this shift the difficulty relative to the original bring-up?

We agree that a reference revised after the release can make a task easier. ServingBench evaluates the first integration of a released model with the reference pinned at a fixed revision, rather than an exact replay of release day (Appendix D). Table 3 (§4.3) shows how much the reference matters: removing it entirely lowers the serving score of Claude Opus 5 from 0.82 to 0.62 and of Gemini 3.8 Flash from 0.51 to 0.39. This ablation does not isolate the effect of later revisions, and we will report the revision date of each reference in the next version.

We hope the new tasks in particular address your main concern on contamination, and if so, we would appreciate it if you would consider updating your score. We are happy to address any follow-up questions and will reflect the answers in the revised paper.

Best regards,

The Authors

Reviewer_mXHJOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces ServingBench, a benchmark of 148 "day-0 model support" tasks drawn from merged model bring-up pull requests in three serving engines (vLLM, SGLang, TensorRT-LLM) plus vLLM's TPU backend, spanning 10 accelerator tiers from NVIDIA, AMD, and Google. Each task rewinds an engine to the state just before its first model-support PR (Base) and asks a coding agent to add support for a released checkpoint, matching reference outputs and serving efficiently. A four-stage automated pipeline reconstructs the pre-integration engine, admits hidden correctness checks only if Base fails and Gold (the maintainers' merged PR) passes, and measures serving speed against Gold on the target accelerator. Evaluating seven state-of-the-art models with mini-SWE-agent, the authors find agents resolve 56.7% of tasks on average, but 33% of resolved patches serve more slowly than the human expert (median 18% slowdown).

Critical strengths

Timely and well-scoped problem. Day-0 model support is a real, economically significant systems-engineering task that existing coding benchmarks do not capture.

The correctness-then-performance framing is the key contribution and it is handled carefully.

Critical weaknesses

Single trajectory per task, no variance reporting.

Small, model-limited ablations.

Questions for authors

Can you report variance across multiple trajectories (or seeds) for at least one or two models on a subset? How stable is the resolved rate and the Slower/Faster split?

1 comment

We thank the reviewer for the positive review and for recognizing day-0 model support as a timely, well-scoped problem with a carefully handled correctness-then-performance framing. Our responses to the two concerns follow.

W1 / Q1: Run-to-run variance

Q1. Can you report variance across multiple trajectories (or seeds) for at least one or two models on a subset? How stable is the resolved rate and the Slower/Faster split?

Thank you for this suggestion. We ran two more independent trajectories per task for the top two models in Table 2, Claude Opus 5 and DeepSeek V4.1 Flash, on 30 tasks with the same harness, budget, prompt and grader (Appendix E). The 30 tasks are 18 NVIDIA tasks from the paper's ablation set plus 12 sampled at random from the benchmark's NVIDIA tasks, and together they match the benchmark's modality mix and median Gold size. Both the pass rate and the Slower/Faster split are stable across the three runs.

Pass rate (%)Run 1Run 2Run 3Mean ± stdSame verdict as run 1 (run 2 / 3)
Claude Opus 583.383.380.082.2 ± 1.9100% / 97%
DeepSeek V4.1 Flash73.376.773.374.4 ± 1.990% / 93%

Each model's pass rate moves by at most one task of 30, Claude Opus 5 leads in every run, and per-task verdicts agree with run 1 on 90% to 100% of tasks.

For the Slower/Faster split, we re-timed the passing patches of all three runs next to Gold on the 19 tasks with a valid paired timing (Table 19 in Appendix E). A cell counts only patches that pass in that run and have a valid timing, so the counts are smaller than the pass rates suggest.

Slower / Within 5% / Faster (median ratio to Gold)Run 1Run 2Run 3
Claude Opus 51 / 9 / 5 (1.00)0 / 11 / 6 (1.03)2 / 10 / 3 (1.01)
DeepSeek V4.1 Flash2 / 11 / 1 (1.00)1 / 9 / 1 (1.00)2 / 9 / 2 (1.01)

The median ratio to Gold stays between 1.00 and 1.03 in every run, and patches from different runs on the same task differ by a median of 2.2% in speed.

To show the uncertainty of the main results as well, we added task-level bootstrap 95% CIs to Table 2 (Appendix E):

ModelFails (%)Serving scoreSlower (%)Within 5% (%)Faster (%)
Claude Opus 522.3 [15.5, 29.1]0.72 [0.66, 0.79]25.0 [17.6, 32.2]36.5 [28.9, 44.8]16.2 [10.4, 22.9]
DeepSeek V4.1 Flash28.4 [21.6, 35.8]0.65 [0.58, 0.72]29.7 [22.6, 38.1]37.2 [29.1, 45.2]4.7 [1.5, 8.3]
Kimi K329.7 [22.3, 37.2]0.65 [0.57, 0.71]27.0 [19.7, 34.6]35.1 [27.2, 43.3]8.1 [3.8, 13.2]
GPT-5.6 Sol39.9 [31.8, 48.0]0.54 [0.46, 0.62]18.9 [12.4, 26.1]32.4 [24.6, 40.4]8.8 [4.1, 13.9]
GLM-5.348.0 [39.9, 56.1]0.48 [0.41, 0.56]15.5 [9.6, 21.9]30.4 [22.8, 38.3]6.1 [2.3, 10.7]
Gemini 3.8 Flash59.5 [51.4, 66.9]0.36 [0.29, 0.43]12.8 [7.8, 19.2]24.3 [17.6, 31.5]3.4 [0.7, 6.0]
Gemini 3.1 Pro Preview75.7 [68.2, 82.4]0.24 [0.17, 0.31]1.4 [0.0, 4.1]21.6 [14.8, 28.1]1.4 [0.0, 4.2]
Average43.3 [38.1, 48.5]0.52 [0.47, 0.57]18.6 [14.9, 22.8]31.1 [25.6, 36.5]6.9 [4.8, 9.4]

W2: Small, model-limited ablations

W2. Small, model-limited ablations.

Thank you. We extended the ablations to more models and variants: three prompt variants with three models, a 4× budget with two models, and OpenHands with two further models. Each variant is paired with the paper's runs on the same tasks and models (Appendix F), and none of the changes in pass rate is statistically significant under an exact McNemar test.

ChangeModelsTasksPass rate (%)Δ (pp)p
Hide the workload shapesClaude Opus 5, DeepSeek V4.1 Flash, GPT-5.6 Sol3073.3 → 68.9−4.40.39
Hide the interface sectionClaude Opus 5, DeepSeek V4.1 Flash, GPT-5.6 Sol3073.3 → 73.30.01.00
Hide bothClaude Opus 5, DeepSeek V4.1 Flash, GPT-5.6 Sol3073.3 → 70.0−3.30.61
4× step and time budgetClaude Opus 5, DeepSeek V4.1 Flash2070.0 → 75.0+5.00.69
OpenHands instead of mini-SWE-agentDeepSeek V4.1 Flash, GPT-5.6 Sol2057.5 → 52.5−5.00.73

These results complement Table 3 in the paper, which reports the serving score of Claude Opus 5 and Gemini 3.8 Flash under the original ablations.

We thank the reviewer again for the support and the concrete requests. If you have any follow-up questions, we would be happy to address them and revise the paper as needed during the discussion period.

Best regards,

The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to Reviewers and Summary of Revisions

We thank the reviewers and the AC for their careful and constructive reviews. We appreciate that the reviewers find day-0 model support a timely problem (mXHJ, ooTu), view the construction pipeline as a core contribution (ooTu, SWhD), and value grading on released checkpoints and real accelerators (mXHJ, ooTu, SWhD).

Below we summarize the new results by topic. All changes in the revised PDF (§4 and Appendices E to L) are marked in blue, and the new experiments took about 1,930 accelerator-hours and $16k in accelerator and API costs.

Contamination (ooTu). We compared tasks merged before and after each model's training cutoff, and tasks merged after the cutoff are resolved about as often as earlier ones (§4.4, Appendix G). We also built 25 new tasks from PRs opened after every evaluated model was released, and the seven models rank on them exactly as in Table 2.

Variance (mXHJ). We reran the top two models, Claude Opus 5 and DeepSeek V4.1 Flash, two more times on 30 tasks (§4.4, Appendix E). Across the three runs, each model's pass rate moves by at most one task, the speed classes barely change, and Claude Opus 5 leads in every run.

Validity of the checks (SWhD). We re-graded the resolved patches of 30 tasks on longer, batched, and long-context inputs, compared them with Gold on downstream benchmarks, and injected faults into Gold (§4.4, Appendix K). Most patches pass both checks, and counting the rest as unresolved leaves the model ranking unchanged. The paper's grader rejects every injected behavior fault, and the strict re-grade rejects every text-path model-math fault.

Other workloads and later releases (ooTu). We re-timed resolved patches on 8 workloads with concurrency from 1 to 128, and we timed the maintainers' releases 6 and 12 months after Gold (Appendices H and I). The median resolved patch matches the human expert's speed on every workload, and the later releases are faster still.

Ablations (mXHJ, ooTu). We hid the interface or the workload shapes from the prompt, gave agents a 4× budget, and ran DeepSeek V4.1 Flash and GPT-5.6 Sol with a second harness (§4.4, Appendix F). None of these changes the pass rate by more than 5 points, and none is statistically significant.

Thank you again for the thoughtful feedback. We hope these results address the main concerns in the reviews. If there are any follow-up questions, we are happy to address them and to update the paper accordingly during the discussion phase.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.