acceptodds
Under review as a conference paper at ICLR 2027

LMSysDojo: Building Scalable Analytical Agents for Optimizing Complex LLM Serving Systems

Abstract

Complex real-world computing systems are hard to optimize because their performance emerges from many interacting components and depends on the hardware and workloads they serve. We introduce **LMSysDojo**, a toolkit that reconstructs reproducible optimization playgrounds from merged performance pull requests in vLLM and SGLang. Agents optimize live deployments on real GPUs, and each patch's improvement is normalized against the human expert's fix on the same setup. On the held-out splits of our **LMSysDojo-100** benchmark, a Claude Code agent using Opus 5 achieves –% of the expert's gain. A key bottleneck is that a patch's worth is known only after costly deployment and measurement, and a vanilla coding agent exposes no ex-ante signal for choosing among candidate ideas beforehand. Inspired by how systems researchers justify optimizations, we propose *gap analysis*, which separates diagnosis from implementation: an analyzer locates a bottleneck, proposes a mechanism, and predicts its gain before an implementer builds it. *Parallel gap analysis* selects among competing analyses by predicted gain; *structured gap analysis* expresses each prediction as a program over profiles, experimental evidence, and stated assumptions; and *iterative gap analysis* checks and refines these programs before selection. Our combined approach reaches –% of the expert's gain, improving over the single agent by – percentage points. Across variants, predictive signals turn analysis into a new scaling axis under a fixed evaluation budget, with gains continuing through analyses. We will release the toolkit, benchmark, and -setup corpus.

Primary area:
datasets and benchmarks
Keywords:
LLM serving, systems optimization, agents, benchmark

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_3JyQOpenReview ↗
Rating
4(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LMSysDojo, a new benchmark and framework for optimizing LLM serving systems like vLLM and SGLang. It does two main things:

  1. The Benchmark: Mines around 40k historical PRs down to 900 reproducible setups, creating a 100-item benchmark (split into dev, pre-cutoff, and post-cutoff sets) with sealed expert fixes.
  2. Gap Analysis Framework: Splits the task so an analyzer predicts a performance gain before a separate implementer writes the code. They test parallel (K=10K=10), structured (executable analyze() code), and iterative variants (with sensitivity and assumption checks). The iterative approach boosts held-out E2E normalized gain from roughly 57% (vanilla Claude Code) up to 73-83%. They also run some Gemini and Sonnet role-ablation checks.
Critical strengths

The filtering, temporal splits, and noise floor (1.075x) are super careful. Their open-ended null result—150 baseline attempts and none beat the noise floor—is a fantastic justification for why we need these calibrated setups in the first place.

Proving that analyzer quality is what actually matters while implementer quality barely moves the needle is a super useful takeaway that goes way beyond just this paper.

Showing that predicting gains beforehand beats an LLM ranker looking at finished diffs is clean proof that the ex-ante signal has real value.

The qualitative section is great about calling out mis-sized analyses, flawed premises, and patches that wouldn't normally merge upstream.

Critical weaknesses

The "fixed evaluation budget" framing is a bit misleading since it only matches compute at the final deploy stage. Parallel (K=10K=10) and iterative runs burn way more tokens, dollars, and GPU hours than vanilla. We really need a per-item cost breakdown across the board.

The paper claims vanilla Claude Code "states no prediction." But vanilla has measure.sh and could easily be asked to output a predicted speedup in a config file. Without testing vanilla with a self-reported prediction, we can't tell if the gains come from the actual gap-analysis architecture or just from forcing the model to estimate a number.

The parallel curve is still going up at K=10K=10, and iterative uses up to 4x the analyzer sessions. It's unclear if the extra multi-step checks actually add value compared to just bumping up KK in pure parallel sampling.

The structured analyzer gets SQL access to the profile data (corpus/). Its wins might just come from having better data access rather than the required program structure. A free-form analyzer with corpus access would clear this up.

Looking at Table 14, ranking consistency is only 65-76% with wide CIs, and pick-minus-average gains are pretty small. The high Spearman correlation mostly comes from between-item headroom. The text makes it sound like ranking is super precise, but the numbers suggest structure mostly helps by improving pool quality instead.

The authors admit their sensitivity check is pretty loose (τ=0.01\tau = 0.01 lets a -20% projection pass), yet it still stops 36-46% of analyses. We need a breakdown of these stops—if most are just crashes or zero-leverage, it's basically just an execution gate, not a real sensitivity test.

The workloads are designed to be the "smallest workload that enters the changed path" and setup cards include explicit themes. That gives the agent a lot of hints that real-world debugging never provides. An ablation with generic workloads would help show how much this measures true diagnosis vs guided search.

Questions for authors

What is the actual per-item cost (tokens, dollars, GPU-hours) for each row in Table 1?

How does vanilla perform if you give it a self-reported predicted gain plus selection?

How do the sensitivity check stops break down by reason?

How does a free-form analyzer with corpus/ access compare to the structured one?

How does parallel selection compare to iterative when you match the total number of analyzer sessions?

How does perfomance change with a generic workload or no setup card theme?

7 comments
AuthorsOpenReview ↗

Response to Reviewer 3JyQ (Part 1/3)

Dear Reviewer 3JyQ,

We thank you for the detailed and constructive review. We appreciate the opportunity to report the cost of every configuration, add the controls the review proposes, and clarify what the checks and the setup cards do.

Note: Please also see our General Response (Parts 1–4) above (and the revised PDF, with updates in blue) for the shared experiments and tables referenced below: LMSysDojo-Ext100 (G1, Table R1), cost breakdown & scaling to K = 50 (G2, Tables R2–R4), Vanilla + self-select & + corpus access controls (G3, Table 1), and Analysis RL (G4, Table R5).

W1

The "fixed evaluation budget" framing is a bit misleading since it only matches compute at the final deploy stage. Parallel (K) and iterative runs burn way more tokens, dollars, and GPU hours than vanilla. We really need a per-item cost breakdown across the board.

Agreed: the fixed budget is one final 2.0 GPU-h deployed verification per item (§4; §5 in the revision), while agent compute scales with K. Per item, vanilla costs USD 7.49 (6.3M / 93k tokens, 0.16 sandbox GPU-h), K = 10 parallel gap analysis costs USD 57–60 (48.8–51.8M / 873–943k tokens, 1.2 h concurrent wall clock, 0.92 GPU-h) vs. USD 75–85 for ten vanilla sessions, and K = 10 iterative costs USD 94 (16.9 analyzer sessions, 1.27 GPU-h); full breakdown in G2 (Table R2; referenced in Table 1 and reported in §C.8, Table 14 & Fig. 16 of the revision).

W2

The paper claims vanilla Claude Code "states no prediction." But vanilla has measure.sh and could easily be asked to output a predicted speedup in a config file. Without testing vanilla with a self-reported prediction, we can't tell if the gains come from the actual gap-analysis architecture or just from forcing the model to estimate a number.

Prompting vanilla to also output a predicted gain in DELIVERY.yaml alongside its patch and report (using the same sandbox, ./measure.sh and selection rule as gap analysis, so it is essentially coupled gap analysis + implementation; Vanilla + self-select in G3, Table 1) lifts K = 10 from 57.0 / 60.1 to 62.4 / 70.0 and matches parallel gap analysis at equal K = 10 (64.7 / 73.2, Holm p = 1.00). What decoupling changes is timing and per-candidate cost (ranking estimates before coding so only the winner is built): an analysis costs USD 5.28 vs. USD 8.46 for a full session that also implements, so at matched budgets 30 analyses reach 94.2 post-cutoff at USD 162.9 vs. 81.6 for 20 self-selected runs at USD 169.2 (G2.2, Table R3), while also enabling pre-build checks (W3) and analyzer RL (G4). We added this control to Table 1 and state the pre-implementation distinction precisely in the revision.

W3

The parallel curve is still going up at K, and iterative uses up to 4x the analyzer sessions. It's unclear if the extra multi-step checks actually add value compared to just bumping up K in pure parallel sampling.

At matched agent dollars, increasing parallel K is more efficient in the measured range (e.g., K = 20 structured + parallel reaches 91.1 at USD 116 with 20 sessions, vs. 82.7 at USD 94 with 16.9 sessions for K = 10 iterative). At fixed K (maximizing gain per deployed verification), even "the two simplest checks the program affords" (§3) add +2.2 to +5.4 points at every tested K from 10 to 50 (+3.2 pooled at K = 10, Holm p = 0.001; Table R7) and reach the highest measured score, 103.8 at K = 50 vs. 100.7 for structured parallel (full curves, ≈USD 420 fixed-K break-even, and richer future gates in G2.2, Tables R3–R4).

W4

The structured analyzer gets SQL access to the profile data (corpus/). Its wins might just come from having better data access rather than the required program structure. A free-form analyzer with corpus access would clear this up.

Both analyzers already receive the same profiles (prof.sqlite vs. the SQL handle C, §C.2). Adding corpus/ to free-form analysis at K = 1 gives 56.6 / 61.9, compared with 55.1 / 62.2 without it (+1.0 pooled, 95% CI [−7.3, +9.7], Holm p = 1.00; G3, Table 1, Table R7). No statistically significant benefit from the additional SQL interface was detected in this control; instead, the structured representation makes predictions executable and checkable, enabling iterative gap analysis to use sensitivity checks and assumption audits to guide revisions before implementation (G3, Table 1).


(Continued in Part 2/3)

AuthorsOpenReview ↗

Response to Reviewer 3JyQ (Part 2/3)

W5

Looking at Table 14, ranking consistency is only 65-76% with wide CIs [...] the numbers suggest structure mostly helps by improving pool quality instead.

Table R11 separates pool quality (mean of all ten and mean of delivered, a post-hoc diagnostic) from within-item selection (pick) across the held-out K = 10 pools (§B.1, Eq. 7):

held-out poolmean of all tenmean of deliveredpick
pre-cutoff, gap analysis55.160.564.7
pre-cutoff, structured64.667.170.7
post-cutoff, gap analysis62.268.373.2
post-cutoff, structured62.166.177.3

Both mechanisms contribute, and their balance differs by split: pre-cutoff, structure mainly raises pool quality (64.6 vs. 55.1), whereas post-cutoff, pool means are identical (62.1 vs. 62.2) and structured selection adds +11.2 points over its delivered mean (vs. +4.9 for free-form). Across all 170 held-out items, ranking by predicted gain adds +11.8 (free-form) and +11.1 (structured) points over a single analysis (Holm p < 0.001; Table R7); Table R11 is added in §D.2 (Table 22) of the revision, framing within-item ranking as a useful, imperfect signal (65–76% pairwise consistency vs. 50% chance).

W6

The authors admit their sensitivity check is pretty loose [...] We need a breakdown of these stops—if most are just crashes or zero-leverage, it's basically just an execution gate, not a real sensitivity test.

Table R12 breaks down the 700 held-out structured analyses (ten per item, τ = 0.01):

outcomecount (share)delivered E2E geomean
pass400 (57.1%)1.67×
stop: program did not run40 (5.7%)1.47×
↳ handle misuse (A, E or C used as a dict, iterable or callable)22
↳ SQL error (missing table, bad SQL)9
↳ assume.yaml unparseable9
stop: zero leverage on every metric62 (8.9%)1.39×
↳ predicts exactly 0% on every metric42
↳ nonzero prediction, rounded below the tracer's step20
stop: claim-to-leverage ratio below τ198 (28.3%)1.20×
↳ a claim of 1% or more on some metric155
↳ every claim under 1%43

One third of the 300 stops are execution or zero-leverage failures (40 + 62, which alone leave the post-cutoff passed-set mean unchanged at 61.9 vs. 62.1), while two thirds (198) fail the claim-to-leverage ratio (median claimed E2E change 2.5% vs. 20.5% among passes; applying τ = 0.01 lifts the passed-set mean to 66.4, and +4 to +10 points across τ ∈ [0.0025, 0.2]). Thus the sensitivity check is a mechanical conditioning gate rather than a pure crash check, while premise errors are caught separately by the assumption audit (failing 96 of 700 analyses, only 45 overlapping; passing both reaches 67.3% post-cutoff, §D.3). Table R12 is added to §D.3 (Table 23) in the revision.

W7

The workloads are designed to be the "smallest workload that enters the changed path" and setup cards include explicit themes. [...] An ablation with generic workloads would help show how much this measures true diagnosis vs guided search.

  1. Staged cards carry no theme. Any theme section is stripped at staging (themed: false on every run; stated in §C.2 with concrete setup-card examples in §2 and §C.2 of the revision). What §1–§2 call a "scenario theme" is a one-line user-facing restatement of the traffic that never names the bottleneck, subsystem or fix; e.g.:
  • "A knowledge-base assistant answering single-turn questions over 20 shared 6000-token briefs, ten concurrent users, one GPU." (vLLM 47782)
  • "A general-purpose chat assistant on one GPU, serving 512 concurrent users short mixed-length single-turn conversations." (vLLM 18161)
  • "A document question-answering helper on one GPU, where eight concurrent users share long 32k-token reference texts for brief answers." (vLLM 25752)
  1. Real-world optimization targets a known traffic regime. Operators know the traffic they serve, and in our survey of 50 serving papers (§B.4, Table 9), mechanisms only pay off under a stated workload regime (e.g., Marconi on shared prefixes, NanoFlow on large batches, FlexPipe 3× faster on bursty CV=4 traffic and 2.7× slower on smooth traffic). Specifying the traffic defines the problem; diagnosing where time goes is left to the agent (≈25% of delivered patches share zero modified files with the reference PR, and 22–27% of those still beat the expert, Table 13).
  2. Why we do not swap in generic workloads. A workload must exercise the PR's changed path to guarantee real optimization headroom and a valid expert denominator; the workloads themselves follow standard serving shapes (decode-heavy, long prefill, multimodal, chat and speculative decoding cover 80% of items). In our open-ended study without calibrated headroom (§A.6), agents chose their own workloads across >1,000 GPU runs and 0 of 150 attempts cleared the 1.075× noise floor, making weak diagnosis indistinguishable from zero headroom.

(Continued in Part 3/3)

AuthorsOpenReview ↗

Response to Reviewer 3JyQ (Part 3/3)

Questions

Q1 What is the actual per-item cost (tokens, dollars, GPU-hours) for each row in Table 1?

Because only the selected proposal is implemented, K = 10 parallel gap analysis costs less per item (USD 57–60, 0.92 sandbox GPU-h) than ten vanilla sessions (USD 75–85, 1.60–1.92 GPU-h), while K = 10 iterative costs USD 94 across 16.9 analyzer sessions, all sharing one 2.0 GPU-h final verification (W1; full table in G2, Table R2, referenced in Table 1 and reported in §C.8, Table 14 of the revision).

Q2 How does vanilla perform if you give it a self-reported predicted gain plus selection?

It confirms that any informative gain estimate unlocks multi-sample selection, matching parallel gap analysis at equal K = 10 (62.4 / 70.0 vs. 64.7 / 73.2), while showing that the architectural benefit of decoupling is obtaining that estimate before paying to implement every candidate, yielding higher gain at matched budget (W2; G3, Table 1, Table R3).

Q3 How do the sensitivity check stops break down by reason?

Two thirds of the 300 stops (198) fail the claim-to-leverage threshold rather than crashing (40) or having zero leverage (62), and the threshold step accounts for the passed-set enrichment, confirming the gate acts as a conditioning check rather than a crash filter (W6; Table R12, §D.3 Table 23 in the revision).

Q4 How does a free-form analyzer with corpus/ access compare to the structured one?

At K = 1, adding corpus access changes free-form analysis by +1.0 point pooled, with no statistically significant difference (Holm p = 1.00). Both analyzers receive the same underlying profiles; the structured representation additionally makes predictions executable and checkable, thus further enabling our iterative gap analysis.

Q5 How does parallel selection compare to iterative when you match the total number of analyzer sessions?

At matched analyzer sessions or agent dollars, broader parallel sampling wins in the measured range (e.g., 91.1 at 20 sessions vs. 82.7 at 16.9 sessions for K = 10 iterative; W3, G2.2).

However, iterative checks explore analytical depth within each candidate track: even using "the two simplest checks the program affords" (§3), checks and retries improve the pick at every tested K from 10 to 50 and reach 103.8 at K = 50, motivating richer pre-build verifiers on the same structured interface as future work. Among the fixed K = 10 configurations in Table R4, iterative has the lowest total cost per point when verification costs more than approximately USD 420; when K can also vary, the full cost–performance curves in Table R3 determine the preferred configuration.

Q6 How does perfomance change with a generic workload or no setup card theme?

All reported runs omit the theme section and withhold the bottleneck and fix. We have not run a controlled generic-workload ablation: changing the workload can remove the known optimization opportunity. The open-ended study provides related context, with none of 150 attempts clearing the noise floor, but does not isolate the effect of changing workloads (W7).


We hope these analyses and controls address your concerns, and we would be glad to discuss any of them further.

Best Regards,

Authors of Submission 33699

Reviewer_fqthOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper presents LMSysDojo, a toolkit to reconstructs reproducible optimization playgrounds from development history of LLM serving systems like vLLM and SGLang, and LMSysDojo-100, a benchmark suite to test agents on it. The paper additionally presents structured gap analysis method for agents to perform better at these tasks. The paper reports that the best agent achieves 73--83% of the gain compared to human experts.

Critical strengths
  1. The paper proposed a benchmark suite to test agents' capability of developing/optimizing LLM serving systems, taken from real-world development history, with a detailed explanation about collection process in the appendix. This would be a great resource for testing agents in real-world settings.
  2. The evaluation fixes the final deployed-evaluation allowance and per-attempt time limits, providing practical setting to evaluate agents' capability in real-world settings.
Critical weaknesses
  1. While the Table 1 and Figure 5 report better performance from structured gap analysis, it is not clear if the gain comes from better harness design or simply from larger token consumption.
  2. The benchmark uses common metrics for scoring regardless of each PR's original optimization objective. Although five metrics are reported separately, scoring is not tailored to task-specific goals such as TTFT optimization. So it is not clear if the scores in this benchmark reflects real benefits of implementations.
Questions for authors
  1. Can you report the amount of token consumed for each method?
  2. Does structured gap analysis perform better than other methods even with the similar amount of token budget?
  3. Is it possible to have task-specific metric selection in scoring?
  4. It was not clear to me what we are supposed to learn from analysis in Figure 4. Higher standard deviation does not necessarily mean the performance gain would be bigger.
5 comments
AuthorsOpenReview ↗

Response to Reviewer fqth (Part 1/2)

Dear Reviewer fqth,

We thank you for the positive assessment and the precise questions. We appreciate the opportunity to show where the gain comes from, add scoring on each item's target metrics, and clarify what Fig. 4 establishes.

Note: Please also see our General Response (Parts 1–4) above (and the revised PDF, with updates in blue) for the shared experiments and tables referenced below: LMSysDojo-Ext100 (G1, Table R1), cost breakdown & scaling to K = 50 (G2, Tables R2–R4), Vanilla + self-select & + corpus access controls (G3, Table 1), and Analysis RL (G4, Table R5).

W1

While the Table 1 and Figure 5 report better performance from structured gap analysis, it is not clear if the gain comes from better harness design or simply from larger token consumption.

Comparing token consumption, cost and pooled held-out gain across the 70 LMSysDojo-100 held-out items (full breakdown in G2–G3, Tables R2–R3):

policytokens in (M) / out (k)USD / itemE2E normalized gain
Vanilla (1 session)6.3 / 937.4959.1
Vanilla, random 1 of 10 sessions63.0 / 93074.9060.4
Vanilla + ranker (K = 10)63.0 / 93175.0261.6
Vanilla + self-select (K = 10)71.2 / 1,05084.6065.8
Gap analysis + parallel (K = 10)48.8 / 87357.2668.3
Structured + parallel (K = 10)51.8 / 94360.2673.7

10× tokens without selection barely moves performance (59.1 → 60.4). Even against multi-sample vanilla with selection (+ ranker and + self-select), Structured + parallel consumes 27% fewer input tokens than Vanilla + self-select (51.8M vs. 71.2M) because only the selected analysis is implemented, while scoring 7.9 points higher (73.7 vs. 65.8; at higher matched budgets in Table R3, 30 analyses reach 94.2 at USD 162.9 vs. 81.6 for 20 self-selected runs at USD 169.2). Structure further enables executable pre-build checks that lift the pick by +2.2 to +5.4 points across K = 10–50 (Table R3).

W2. The benchmark uses common metrics for scoring regardless of each PR's original optimization objective.

Although five metrics are reported separately, scoring is not tailored to task-specific goals such as TTFT optimization. So it is not clear if the scores in this benchmark reflects real benefits of implementations.

Each setup card already declares the metrics its PR targeted (79 of 100 items declare two; 52 declare TTFT, 58 ITL/TPOT, 55 throughput, 21 E2E). Table R10 rescores every sample on the geometric mean of its item's declared metrics, keeping the same picks as Table 1:

Table R10 (normalized gain on each item's declared metrics; mean E2E in parentheses):

setupdev (n = 30)pre-cutoff (n = 35)post-cutoff (n = 35)Ext100 pre-cutoff (n = 50)Ext100 post-cutoff (n = 50)
Vanilla76.8 (75.8)60.9 (57.0)65.2 (60.1)100.3 (95.1)65.5 (64.6)
Gap analysis81.9 (78.1)59.5 (55.1)66.2 (62.2)100.6 (96.3)72.0 (68.1)
+ parallel (K = 10)90.1 (83.3)70.1 (64.7)78.5 (73.2)112.1 (109.8)86.9 (82.8)
Structured gap analysis77.5 (79.9)68.7 (64.6)65.7 (62.1)96.2 (96.6)67.0 (66.1)
+ parallel (K = 10)104.0 (104.9)73.5 (70.7)84.2 (77.3)111.9 (110.9)83.8 (80.2)
+ iterative (K = 10)112.6 (113.5)75.9 (73.1)89.6 (82.7)114.0 (111.0)90.5 (86.1)

On LMSysDojo-100's held-out splits, every configuration scores 3–7 points higher on its declared metrics than on mean E2E (on Ext100, within 6 points), and the method ordering is unchanged. We keep mean E2E as the headline because every item reports it, while reporting all five metrics alongside exposes trade-offs across non-target metrics; Table R10 is included in §D (Table 19) of the revision.

(Continued in Part 2/2)

AuthorsOpenReview ↗

Response to Reviewer fqth (Part 2/2)

Questions

Q1 Can you report the amount of token consumed for each method?

Reported in W1 above and in full per-session/per-item detail in G2.1 (Table R2; referenced in Table 1 and reported in §C.8, Table 14 of the revision): an analyzer session uses ≈25% fewer tokens than a full vanilla session, and implementing only the selected proposal reduces K = 10 token consumption by 18–27% relative to multi-sample vanilla.

Q2 Does structured gap analysis perform better than other methods even with the similar amount of token budget?

Against coupled multi-sample vanilla, yes, because decoupling avoids coding unselected proposals (W1). More broadly, structured and iterative analysis are not inherently cheaper in raw agent tokens than single-run or free-form baselines, and on agent cost alone broader parallel sampling is cheaper; rather, they break even and become the most cost-effective option once the cost of a deployed verification V is large (above ≈USD 180 for structured + parallel and ≈USD 420 for iterative on the fixed K = 10 menu; G2.2, Table R4). Beyond these numbers, the deeper role of structure is to open a paradigm for scaling analytical depth before touching the system: as stated in §3, we intentionally tested the loop with "the two simplest checks the program affords" and already observe consistent gains across K, leaving richer verifiers and critiques on the same executable interface to future work. Because live deployment, traffic exposure and regression testing in real-world serving carry high cost V, investing extra analysis tokens to raise candidate quality before deployment is a practical lever, and we expanded our discussion in the revision (§3, §5, §B.2, §C.8 Fig. 16, §F) to highlight this trade-off and future direction.

Q3 Is it possible to have task-specific metric selection in scoring?

Yes (W2, Table R10): rescoring on each PR's declared metrics raises held-out scores on LMSysDojo-100 while preserving the exact method ranking, confirming that the benchmark gains reflect each PR's intended optimization goal rather than an artifact of E2E scoring.

Q4 It was not clear to me what we are supposed to learn from analysis in Figure 4. Higher standard deviation does not necessarily mean the performance gain would be bigger.

Fig. 4 identifies the dominant source of run-to-run outcome variability; it does not claim that higher variance implies greater performance gains. Holding the analysis fixed and rerunning only the implementer reduces the median within-item standard deviation of log speedup from 0.067 to 0.010 for free-form analysis and from 0.072 to 0.012 for structured analysis, a 97–98% reduction in squared median spread. Thus, in this setting, most outcome variability is associated with which analysis is produced, while repeated implementation of the same analysis contributes little. This establishes the importance of the analysis stage, as clarified in §3 of the revision.


We hope these answers address your questions, and we would be glad to provide further detail on any of them.

Best Regards,

Authors of Submission 33699

Dear Reviewer fqth,

As the discussion period draws to a close, we would appreciate hearing whether you have any remaining questions or concerns. We would be happy to use the remaining time to address them and provide any further clarification.

Thank you again for your careful review and constructive suggestions, which have helped us strengthen the paper!

Best regards,

Authors of Submission 33699

Reviewer_XB9aOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The authors present LMSysDojo, a toolkit that turns merged performance PRs of serving engines into sealed, expert-calibrated playgrounds, and LMSysDojo-100, a benchmark supporting temporal post-cutoff evaluation. The paper also proposes gap analysis for predicting a fix's gain before implementing it and structured gap analysis to formulate an LLM agent's optimization claim as an executable source-traced program. Overall, the work makes interesting contributions, but can be significantly strengthened by enlarging the benchmark, reporting uncertainty, and providing compute-matched results.

Critical strengths
  1. A carefully engineered benchmark. Reconstructing live serving deployments from merged PRs and grading against the expert's re-measured fix addresses the headroom problem the authors discuss, and the benchmark provides thorough controls including sealing of git history, transcript audit, output-equivalence probe, and workload checks.
  2. Temporal post-cutoff split. Carrying merge dates and reporting pre- and post-cutoff held-out splits separately controls for contamination from a model's training data, and the benchmark can be refreshed for future models.
  3. Gap analysis is well-motivated with promising results. The fixed-analysis experiment (97–98% reduction in within-item spread, Figure 4) and the role ablation (downgrading the analyzer costs 8–12 points, downgrading the implementer costs nothing, Table 15) together make a strong case that the analysis stage is where variance and value live.
Critical weaknesses
  1. The held-out benchmark is small, and most reported differences are not tested for significance. The main results rest on 35 items per held-out split. Table 1 gives no confidence intervals. Only the largest contrast is reported and discussed in the text (structured+parallel vs. vanilla, p = 0.003 pooled over 70 items); the smaller steps the paper also builds claims on are left to the pairwise matrix in Figure 19, which is uncorrected for ten comparisons per panel and never discussed: iterative vs. structured+parallel (+2.4 / +5.4 points), structured vs. unstructured parallel (+6.0 / +4.1), and gap analysis at K = 1 vs. vanilla (−1.9 / +2.1). The two held-out splits also differ in engine, model family, and expert gain (all p < 0.01, Table 7), so pre/post differences conflate time with item mix, and the Gemini and role-ablation results each use only the 35 post-cutoff items. The pre-cutoff split samples only 65 items from a larger qualified pool, so the benchmark could be larger; at minimum, Table 1 needs item-bootstrap intervals and corrected tests.

  2. The comparison is not compute-matched. The evaluation budget is fixed at one deployed measurement per item, but the agent budget is not: vanilla gets one 1.5-hour session, parallel gap analysis gets 10 analyzer sessions plus an implementer, and iterative gets up to 40 analyzer sessions plus an audit per analysis, i.e., 10–40× more compute. The natural control is missing: run K = 10 vanilla sessions, have each report its own locally measured or predicted speedup (the agent already has ./measure.sh and the GPUs), and implement the highest. The claim that vanilla "exposes no ex-ante signal" is a prompt choice, not a property of the agent. Without this baseline, the paper cannot separate "sampling analyses beats sampling patches" from "sampling 10× with any self-reported selection signal helps."

  3. Missing cost reporting. The results do not report tokens, dollars, or total GPU-hours per configuration per item, so the "analysis as a scaling axis" claim has no cost axis.

  4. Large dev-to-held-out gap suggests overfitting of prompts and τ. On dev, structured+parallel reaches 104.9% and iterative 113.5% vs. vanilla 75.8% (Table 12), against 71–83% held out. The paper states this but should discuss it further given the magnitude: how much of the held-out gain survives when τ and prompt details are perturbed?

  5. Benchmark construction is agent-driven with limited human validation. Scan, segment, parse, and measure are all performed by agents; the reported validation is 96.8% blind re-rating agreement on eligibility and two-rater labelling of categorical fields, but appears to lack human validation. Also the measure stage’s acceptance rule checks that the fix’s effect reproduces, but no human verifies that the setup card and workload match the PR’s scenario.

Questions for authors
  1. How much of the dev-to-held-out gap (113% vs. 73–83%) is attributable to prompt and τ tuning? Does the held-out result survive perturbing τ?
  2. Was any human validation done on the 100 setup cards and workloads beyond agent re-rating?
7 comments
AuthorsOpenReview ↗

Response to Reviewer XB9a (Part 1/4)

Dear Reviewer XB9a,

We thank you for the careful and constructive review. We have expanded the held-out evaluation, added confidence intervals and corrected tests, and run the proposed self-selection control with a detailed cost analysis.

Note: Please also see our General Response (Parts 1–4) above (and the revised PDF, with updates in blue) for the shared experiments referenced below (G1 Ext100 Table R1; G2 cost & K=50 Tables R2–R4; G3 controls Table 1; G4 RL Table R5).

W1. The held-out benchmark is small, and most reported differences are not tested for significance.

The main results rest on 35 items per held-out split. Table 1 gives no confidence intervals [...] Figure 19 is uncorrected [...] and the Gemini and role-ablation results each use only the 35 post-cutoff items. [...]

On 100 new held-out items in LMSysDojo-Ext100 (bringing held-out evaluation to 170 items), all main conclusions hold: every parallel configuration beats vanilla by +14.7 to +21.5 points while K = 1 stays close to vanilla, and pooled across all 170 items structured + parallel leads vanilla by +15.0 points (Holm p = 0.003; G1, Table R1). Below we report item-bootstrap intervals, Holm-corrected paired tests, and the expanded Gemini and role ablations (85 post-cutoff items total).

1. Item-bootstrap 95% intervals (Table R6). Intervals across items are wide because expert headroom varies substantially from item to item; the paired tests in Table R7 remove this shared item variation.

Table R6 (E2E normalized gain with item-bootstrap 95% intervals):

setupLMSysDojo-100 pre (n = 35)LMSysDojo-100 post (n = 35)Ext100 pre (n = 50)Ext100 post (n = 50)
Vanilla57.0 [38.1, 76.8]60.1 [40.8, 81.3]95.1 [76.3, 115.6]64.6 [48.1, 82.6]
+ ranker (K = 10)58.3 [39.2, 78.3]65.6 [45.4, 87.7]——
+ self-select (K = 10)62.4 [42.6, 83.1]70.0 [49.2, 92.9]——
Gap analysis55.1 [37.1, 74.0]62.2 [42.2, 84.0]96.3 [76.8, 117.3]68.1 [50.9, 86.6]
+ corpus access56.6 [38.4, 75.8]61.9 [41.9, 83.6]——
+ parallel (K = 10)64.7 [46.3, 83.8]73.2 [50.4, 96.2]109.8 [87.6, 132.0]82.8 [63.0, 102.5]
Structured gap analysis64.6 [45.4, 83.5]62.1 [40.4, 85.7]96.6 [75.5, 119.3]66.1 [47.6, 86.0]
+ parallel (K = 10)70.7 [51.4, 89.6]77.3 [53.6, 100.6]110.9 [88.0, 133.2]80.2 [60.0, 99.8]
+ iterative (K = 10)73.1 [53.5, 92.3]82.7 [58.2, 106.8]111.0 [87.7, 133.7]86.1 [65.2, 106.4]

2. Holm-corrected paired tests and split composition (Table R7). Table R7 tests the twelve method and control contrasts as one Holm-corrected family (two-sided paired sign-flip test on per-item log differences; the three steps you name come first; expert E2E geomeans on Ext100's halves are 1.34 and 1.37):

contrastwhat the paper claimsLMSysDojo-100 (70): Δ points [95% CI]pHolm pall held-out (170): Δ points [95% CI]Holm p
+ iterative − structured + parallelthe checks lift the pick+3.2 [+1.7, +4.7]< 0.0010.001+3.2 [+1.9, +4.5]< 0.001
structured + parallel − gap analysis + parallelnot claimed (structure enables checks)+5.3 [−3.7, +14.9]0.251.00+2.6 [−5.4, +10.9]1.00
gap analysis − vanillanot claimed (comparable at K = 1)−0.7 [−6.2, +4.2]0.791.00+0.6 [−3.9, +5.2]1.00
gap analysis + parallel − gap analysisselection on predicted gain helps+9.9 [+4.9, +15.1]< 0.0010.001+11.8 [+7.3, +16.4]< 0.001
structured + parallel − vanillabeats the single agent+14.5 [+4.9, +24.8]0.0030.025+15.0 [+6.5, +23.9]0.003
gap analysis + parallel − vanillanot claimed separately+9.2 [+0.9, +17.9]0.0300.24+12.4 [+5.0, +20.1]0.005
structured + parallel − structuredselection on predicted gain helps+8.7 [+3.5, +14.0]< 0.0010.009+11.1 [+6.5, +15.7]< 0.001
structured − gap analysisnot claimed (described in §B.3)+6.5 [−0.7, +15.9]0.130.80+3.3 [−4.2, +11.1]1.00
structured − vanillanot claimed+5.8 [−2.3, +14.4]0.211.00+4.0 [−4.1, +12.2]1.00
+ self-select − gap analysis + parallelno significant difference (W2)−2.5 [−11.2, +6.6]0.591.00——
structured + parallel − self-selectnot claimed (W2)+7.9 [−1.2, +17.4]0.0890.62——
+ corpus access − gap analysisno detected interface benefit at K = 1+1.0 [−7.3, +9.7]0.821.00——

Every claimed contrast (bold rows) survives Holm correction on both the 70 items (Holm p ≤ 0.025) and all 170 held-out items (Holm p ≤ 0.003). Regarding pre/post item mix: post-cutoff is a census while pre-cutoff was stratified on workload × engine up to SGLang availability in the pre-cutoff pool (§A.5), which is why we report splits separately and draw conclusions only from paired within-item comparisons, never cross-split level differences.

(Continued in Part 2/4)

AuthorsOpenReview ↗

Response to Reviewer XB9a (Part 2/4)

W1 (continued)

3. Gemini and role ablations on the 50 new post-cutoff items of Ext100 (Tables R8, R9; 85 post-cutoff items total).

Table R8 (Gemini on LMSysDojo-Ext100, post-cutoff, normalized gain):

setupPro E2ETTFTITLTPSRPSFlash E2ETTFTITLTPSRPS
Vanilla44.749.629.844.444.463.268.059.863.165.5
Gap analysis56.065.841.057.058.268.971.657.068.669.3
+ parallel (K = 5)62.865.258.361.862.676.989.871.277.378.0
Structured gap analysis53.349.946.053.254.366.978.860.866.367.6
+ parallel (K = 5)62.165.856.863.064.277.382.171.573.874.4

Table R9 (role ablation on LMSysDojo-Ext100, post-cutoff, normalized gain; analyzer → implementer; Opus rows from Table R1):

setupgap E2ETTFTITLTPSRPSstructured E2ETTFTITLTPSRPS
Opus, single agent64.689.653.764.164.5—————
Sonnet, single agent34.031.720.736.036.7—————
Opus → Opus68.194.854.865.265.566.190.955.467.267.4
+ parallel (K = 10)82.899.159.683.181.080.2125.759.884.978.9
Sonnet → Sonnet57.461.549.360.361.554.852.341.556.558.3
+ parallel (K = 10)67.665.951.970.470.661.054.553.864.465.7
Sonnet → Opus58.153.248.857.960.554.949.044.857.959.5
+ parallel (K = 10)75.168.958.579.881.764.974.153.568.469.8
Opus → Sonnet70.172.656.867.468.966.569.257.266.768.0
+ parallel (K = 10)83.787.164.582.683.776.996.657.177.978.7

Both replicate the submission's findings: structured + parallel beats vanilla on Gemini 3.1 Pro (62.1 vs. 44.7) and Gemini 3.8 Flash (77.3 vs. 63.2), and downgrading the analyzer to Sonnet costs 10.0–11.2 points at K = 1 while downgrading the implementer changes scores by +2.0 and +0.4.

W2. The comparison is not compute-matched.

The evaluation budget is fixed at one deployed measurement per item, but the agent budget is not: vanilla gets one 1.5-hour session, parallel gap analysis gets 10 analyzer sessions plus an implementer, and iterative gets up to 40 analyzer sessions plus an audit per analysis, i.e., 10–40× more compute. The natural control is missing: run K = 10 vanilla sessions, have each report its own locally measured or predicted speedup (the agent already has ./measure.sh and the GPUs), and implement the highest. [...]

Selection unlocks multi-sample gains, and decoupling keeps gap analysis more cost-effective than sampling full attempts (full cost accounting and controls in G2–G3, Table 1, Tables R2–R4):

  • Actual compute ratio is 7.6–12.6×, not 10–40× (Table R2). Because only the selected analysis is implemented and iterative averages 16.9 analyzer sessions at K = 10, K = 10 parallel gap analysis costs USD 57–60 (1.2 h concurrent wall clock) and iterative USD 94, vs. USD 7.49 for one vanilla session and USD 85 for ten self-selected vanilla sessions.
  • Gap analysis beats the natural self-select control at matched cost (G3, Table R3). Self-selection confirms that a selection signal makes 10× sampling useful (65.8 pooled vs. 60.4 random and 59.1 single) and matches parallel gap analysis at equal K = 10 (68.3, Holm p = 1.00), corresponding to the two routes in Eq. 8 (§B.2). Because self-selection pays to implement and locally probe all K patches, gap analysis achieves higher gain at matched agent budgets: 30 analyses reach 94.2 post-cutoff at USD 162.9, vs. 81.6 for 20 self-selected vanilla sessions at USD 169.2 (and at K = 10, structured + parallel scores +7.9 points higher at 71% of self-select's cost). Including final verification V (Table R4), structured + parallel is also cheaper per point than single-session vanilla once V exceeds ≈USD 180.

The revision adds the self-select row to Table 1 and clarifies pre-implementation selection, which also enables pre-build checks and analyzer RL (G4).

(Continued in Part 3/4)

AuthorsOpenReview ↗

Response to Reviewer XB9a (Part 3/4)

W3. Missing cost reporting.

The results do not report tokens, dollars, or total GPU-hours per configuration per item, so the "analysis as a scaling axis" claim has no cost axis.

Per item, vanilla costs USD 7.49 (6.3M / 93k tokens, 0.16 sandbox GPU-h), K = 10 parallel gap analysis costs USD 57–60 (48.8–51.8M / 873–943k tokens, 0.92 GPU-h) vs. USD 85 for ten self-selected vanilla sessions, and scaling parallel analysis from K = 1 (≈USD 10) to K = 50 (USD 269–284) raises post-cutoff gain from ≈62% to ≈100% of the expert's under a single 2.0 GPU-h verification (full session/item breakdown and curves in G2, Tables R2–R4; revised §5 Fig. 8, §B.2 Table 10, §C.8 Table 14 & Fig. 16).

W4. Large dev-to-held-out gap suggests overfitting of prompts and τ.

On dev, structured+parallel reaches 104.9% and iterative 113.5% vs. vanilla 75.8% (Table 12), against 71–83% held out. The paper states this but should discuss it further given the magnitude: how much of the held-out gain survives when τ and prompt details are perturbed?

Our conclusions rest on held-out results. The strongest evidence against dev-set overfitting comes from replication on 100 new items under the frozen setup:

  • Frozen settings generalize to 100 new items, with consistent benefits across five other models. On LMSysDojo-Ext100, constructed after the prompts, budgets and τ were frozen, Structured + parallel beats Vanilla by +15.8 and +15.6 points on the pre- and post-cutoff splits without any retuning. These results replicate the +13.7 and +17.2-point gains on the original held-out splits (G1, Table R1). Parallel selection also improves over a single analysis across five additional models: Sonnet, Gemini 3.1 Pro, Gemini 3.8 Flash, and Qwen3.5 at two sizes (Tables 2, 15, R5, R8, R9).

Two further points clarify the split differences and tuning procedure:

  1. Much of the level difference comes from the split itself. Even unmodified single-session Vanilla drops by 16–19 points from dev (75.8) to the held-out splits (57.0 and 60.1), despite dev and pre-cutoff being a random division of the same 65 items (§A.5). Conversely, on Ext100's pre-cutoff split (sampled after everything was frozen), Vanilla scores 95.1 and Structured + parallel / Iterative score 110.9 / 111.0 (Table R1), showing >100% normalized levels reflect split composition.
  2. Neither prompts nor τ underwent fine-grained sweeps. One K = 10 pass over dev requires 300 full builds and benchmarks (≈USD 3,000 and up to 600 GPU-hours, Table R2), making hyperparameter or prompt sweeps impractical. The prompts are brief, general role instructions with no item-specific hints (§C.4; e.g., "Nobody is telling you what is slow here or where to look"; "Read it before you theorise"). And τ enters only the iterative row (never the parallel rows): it was set once to a round order of magnitude (0.01, ≈1% claimed change per unit of traced leverage), and replaying thresholds across an 80-fold range (0.0025 to 0.2) on the post-cutoff pools consistently enriches passing analyses by +4 to +10 points over the 62.1 pool mean.

(Continued in Part 4/4)

Comments on the paper

4 comments
AuthorsOpenReview ↗

General Response and Summary of Revisions (Part 1/4)

Dear Area Chair and Reviewers,

We thank all three reviewers for their careful and constructive reviews. We appreciate their recognition of LMSysDojo as a "carefully engineered benchmark" (XB9a) and "a great resource for testing agents in real-world settings" (fqth), and of the finding that "the analysis stage is where variance and value live" (XB9a), a "super useful takeaway that goes way beyond just this paper" (3JyQ). To address the reviewers' primary concerns and further strengthen the soundness of our conclusions, we substantially expanded both the scale of our evaluation (G1–G3) and its analytical depth through reinforcement learning (G4):

  1. Larger held-out evaluation (G1; XB9a). Because 35 items per held-out split is small, we built a second held-out benchmark of 100 new items, LMSysDojo-Ext100 (50 pre-cutoff, 50 post-cutoff, bringing held-out evaluation to 170 items), and the results replicate on it.

  2. Cost-effectiveness and scaling to K = 50 (G2; XB9a, fqth, 3JyQ). Because agent compute increases with K, we report the cost breakdown and scaling complexity of every configuration (G2.1), extend the scaling curve to K = 50 where selection reaches the expert's gain with one scored deployment, and analyze the break-even verification cost at which richer analysis pays off (G2.2).

  3. Natural baselines and further ablations (G3; XB9a, fqth, 3JyQ). We ran both proposed controls (Vanilla + self-select outputting a predicted gain alongside its patch, and free-form analysis with corpus/ SQL access), expanded the Gemini and role ablations to 85 post-cutoff items via Ext100, and added task-specific declared-metric scoring and gate/ranking diagnostics.

We also add a reinforcement learning experiment to further verify LMSysDojo's signal quality and the importance of the analysis stage (G4). Using an elastic asynchronous RL system that coordinates policy serving, training and a GPU pool of live serving rollouts (4.7× valid training throughput over synchronous RL), training an open 9B analyzer on measured speedups from corpus items outside the benchmark more than doubles its held-out gain, matching an untrained 27B analyzer at K=8.

Reviewer-specific analyses appear in the individual replies. New tables are numbered R1–R12; table, figure and section numbers without an R refer to the submitted paper (locations in the revised PDF, highlighted in blue, are listed under Summary of revisions).

G1. Second held-out benchmark: LMSysDojo-Ext100

We constructed 100 new items from the corpus (50 merged before the cutoff and 50 after, none shared with LMSysDojo-100) and ran Table 1's frozen protocol, prompts, budgets and τ without modification. Because LMSysDojo-100 already contains every post-cutoff item passing its E2E screen (§A.5), Ext100 applies one uniform corpus screen to both halves (sealed setup, reproduced expert fix, and ≥1.075× expert gain on the PR's declared metric; post-cutoff takes all qualifying 1–4 GPU items plus one seeded 8-GPU draw, pre-cutoff is a seeded draw among items with all four profiles, and unbuildable environments advance to the next draw without reading scores), using the same construction and human review process as LMSysDojo-100. Because the screens differ, we compare methods within each benchmark rather than baseline levels across them. On 21 Ext100 items the expert clears 1.075× on the declared metric rather than mean E2E; rescoring on declared metrics leaves every Ext100 cell within 6 points of its E2E cell (Table R10, reply to fqth).

Table R1 (LMSysDojo-Ext100, normalized gain; K = 1 rows average three runs per item):

setuppre E2ETTFTITLTPSRPSpost E2ETTFTITLTPSRPS
Vanilla95.1106.888.385.885.964.689.653.764.164.5
Gap analysis96.3108.586.986.887.268.194.854.865.265.5
+ parallel (K = 10)109.8139.395.089.489.382.899.159.683.181.0
Structured gap analysis96.6121.486.789.089.266.190.955.467.267.4
+ parallel (K = 10)110.9120.0102.396.996.280.2125.759.884.978.9
+ iterative (K = 10)111.0147.699.997.697.686.1113.066.088.683.3
  • Parallel selection beats vanilla on both splits: +14.7 to +15.9 points pre-cutoff and +15.6 to +21.5 post-cutoff, while K = 1 stays close to vanilla.
  • Claims hold across all 170 held-out items: pooling both benchmarks, structured + parallel exceeds vanilla by +15.0 points (Holm p = 0.003), and selection on predicted gain adds +11.8 points for free-form and +11.1 for structured analysis (both Holm p < 0.001; Table R7, reply to XB9a).

(Continued in Part 2/4)

AuthorsOpenReview ↗

General Response and Summary of Revisions (Part 2/4)

G2. Cost-effectiveness of analysis-time scaling

Every configuration receives one official deployed verification V per item and differs in the candidate work done beforehand (§B.2, Eq. 8). We examine two questions in turn: what each additional candidate costs as we scale K (G2.1), and what scaling returns and under what conditions richer analysis is more cost-effective (G2.2).

G2.1 Cost breakdown and scaling complexity (Tables R2a, R2b)

Table R2a (per-session cost; analyzer and implementer split the single agent's 1.5 h allowance, totaling ≈1.3× a vanilla session):

sessionUSDtokens in (M) / out (k)wall clock (h)GPU-h
S: single-agent session (analyzes, implements, measures)7.496.3 / 930.950.16
S′: single-agent session that also outputs a predicted gain (self-select)8.467.1 / 1050.950.19
A: analyzer session, free-form5.284.5 / 810.550.08
A_c: analyzer session, free-form with corpus/5.434.6 / 830.560.08
A_s: analyzer session, structured5.584.8 / 880.570.08
A_r: analyzer retry, a fresh structured session given the check's verdict and the stopped analysis's artifacts3.643.1 / 570.370.05
I: implementer session4.463.8 / 630.450.12
C: assumption audit, a short read-only agent session (the sensitivity check is a CPU trace)0.740.35 / 150.06—
R: ranker, one API call over the K diffs0.120.02 / 0.70.01—
scoring: the one deployed measurement——0.252.0 (exclusive 8×H100 node)

Table R2b (policy execution cost per item, excluding V from agent USD; extra builds used only to characterize pools/oracles are post-hoc experimental costs, §C.6):

configurationKcost per itemagent USDtokens in (M) / out (k)wall clock (h)GPU-h (sandbox / scoring)
Vanilla1S + V7.496.3 / 930.950.16 / 2.0
+ self-select10K·S′ + V84.6071.2 / 1,0501.41.92 / 2.0
+ ranker10K·S + R + V75.0263.0 / 9311.41.60 / 2.0
Gap analysis1A + I + V9.748.3 / 1441.00.20 / 2.0
+ corpus access1A_c + I + V9.898.4 / 1461.00.20 / 2.0
+ parallel10K·A + I + V57.2648.8 / 8731.20.92 / 2.0
Structured gap analysis1A_s + I + V10.048.6 / 1511.00.20 / 2.0
+ parallel10K·A_s + I + V60.2651.8 / 9431.20.92 / 2.0
+ iterative10K·(A_s + 0.69 A_r + 1.17 C) + I + V94.0377.3 / 1,5122.41.27 / 2.0
  • Decoupling changes scaling complexity from K·S′ to K·A + I. Because only the winning analysis is implemented, each extra candidate costs one analyzer session (USD 5.28–5.58) rather than a full coding session (USD 7.49–8.46): at K = 10, USD 57–60 and 0.92 sandbox GPU-h vs. USD 75–85 and 1.60–1.92 GPU-h for ten vanilla sessions.
  • Flat wall clock for parallel analysis. Running K analyzers concurrently finishes in 1.2 h (vs. 0.95 h for one vanilla session).
  • Bounded overhead for iterative checks. At K = 10, iterative gap analysis uses 16.9 analyzer sessions on average (well below the cap of 40), costing USD 94 and 2.4 h.

G2.2 Returns to scaling (to K = 50) and when richer analysis pays off (Tables R3, R4)

Table R3 extends post-cutoff scaling to K = 50 (implementing and scoring a single winner per item):

K11020304050
Vanilla + self-select: USD8.584.6169.2253.8338.4423.0
↳ normalized gain60.170.081.690.094.995.6
Gap + parallel: USD9.757.3110.1162.9215.7268.5
↳ normalized gain62.273.285.594.299.199.6
Structured + parallel: USD10.060.3116.1171.9227.7283.5
↳ normalized gain62.177.391.195.798.9100.7
+ iterative: USD13.494.0183.6273.2362.8452.4
↳ normalized gain63.882.793.997.9101.9103.8

Table R4 reports total cost per point, (agent USD + V) / gain, across verification prices V for the fixed K = 10 menu (post-cutoff):

configurationagent USDgainV = USD 0V = USD 200V = USD 500V = USD 1,000
Vanilla7.4960.10.123.458.4416.76
+ self-select (K = 10)84.6070.01.214.078.3515.49
+ ranker (K = 10)75.0265.61.144.198.7716.39
Gap analysis + parallel (K = 10)57.2673.20.783.517.6114.44
Structured + parallel (K = 10)60.2677.30.783.377.2513.72
+ iterative (K = 10)94.0382.71.143.567.1813.23
  • Scaling reaches expert-level gain. At K = 50, parallel and structured parallel reach 99.6 and 100.7 for USD 269–284, and iterative reaches 103.8, whereas fifty self-selected vanilla sessions cost USD 423 for 95.6.

(Continued in Part 3/4)

AuthorsOpenReview ↗

General Response and Summary of Revisions (Part 3/4)

G2.2 (continued)

  • When richer analysis is worth its cost. Which method is most economical depends on the verification price V and whether K is fixed or free:
  • Cheap verification (V ≈ 0): one vanilla session has the lowest cost per point; when spending more agent budget, increasing parallel breadth K is more dollar-efficient than per-track retries in the measured range.
  • Costly verification (large V) or fixed K: iterative checks add +2.2 to +5.4 points at every K from 10 to 50 (+3.2 pooled at K = 10, Holm p = 0.001) and reach the highest score (103.8). On the fixed K = 10 menu (Table R4), structured + parallel becomes cheapest per point above V ≈ USD 180, and iterative takes over above V ≈ USD 420 (when K also varies, Table R3 governs). Our benchmark V uses 2 GPU-hours, whereas live production rollouts carry much higher traffic and regression costs. To explore analytical depth, we tested “the two simplest checks the program affords” (§3), leaving richer pre-build verifiers on the same structured interface to future work.

G3. Natural baselines and further ablations

We added the two requested controls to Table 1: (1) Vanilla + self-select (K = 10), where each of ten Vanilla sessions, in addition to delivering its patch and report in DELIVERY.yaml, also outputs a predicted gain like the analyzer under the same sandbox, ./measure.sh and selection rule as parallel gap analysis (missing = 0%, §C.6; 92% of deliveries report one), differing only in that each session also implements its candidate before selection; and (2) Gap analysis + corpus access (K = 1), which gives the free-form analyzer the structured SQL handle C.

Table 1 with the two new rows (normalized gain; new rows in bold; structured rows of Table 1 are unchanged, e.g., Structured + parallel: USD 60.26, 70.7 pre / 77.3 post E2E):

setupUSD / itempre E2ETTFTITLTPSRPSpost E2ETTFTITLTPSRPS
Vanilla7.4957.061.249.262.568.260.161.644.460.761.3
+ ranker (K = 10)75.0258.363.050.563.569.065.669.650.066.065.7
+ self-select (K = 10, new)84.6062.463.854.667.870.570.073.855.368.769.2
Gap analysis9.7455.158.345.561.467.162.265.651.361.562.9
+ corpus access (new)9.8956.659.747.861.967.261.964.051.961.562.6
+ parallel (K = 10)57.2664.764.057.567.473.373.272.959.171.672.3
  • Selection converts extra samples into gains. Pooled over the 70 held-out items, picking a random session of ten vanilla runs scores 60.4 (vs. 59.1 for one run), whereas self-selection reaches 65.8 (+6.7 points).
  • Decoupling matches self-selection at equal K and beats it at matched budget. Self-selection is essentially vanilla prompted to output a quantitative speedup (equivalently, coupled gap analysis and implementation): at K = 10 it matches parallel gap analysis (65.8 vs. 68.3, −2.5 points, Holm p = 1.00), while structured + parallel scores 73.7 (+7.9 points at 71% of the cost; p = 0.089, Holm p = 0.62; Table R7). Because self-selection implements all K candidates rather than only the selected analysis, it spends USD 85 vs. USD 57; at comparable budgets (Table R3), 30 free-form analyses reach 94.2 (USD 162.9) vs. 81.6 for 20 self-selected vanilla sessions (USD 169.2).
  • No significant gain from SQL access alone. Adding corpus/ to free-form analysis changes K = 1 by +1.0 point pooled (95% CI [−7.3, +9.7], Holm p = 1.00). Both analyzers receive the same profiles; structure's distinct value is making predictions executable and checkable, thereby enabling our iterative gap analysis.
  • Further ablations across models, roles, metrics and checks (individual replies). On Ext100's 50 post-cutoff items (85 post-cutoff items total), structured + parallel again beats vanilla on Gemini 3.1 Pro (62.1 vs. 44.7) and Gemini 3.8 Flash (77.3 vs. 63.2; Table R8, reply to XB9a), and role swaps confirm the analyzer drives the outcome (−10.0 to −11.2 points for a Sonnet analyzer vs. +0.4 to +2.0 for a Sonnet implementer; Table R9, reply to XB9a). Rescoring on each PR's declared metrics preserves all method rankings at +3 to +7 higher points on LMSysDojo-100 (Table R10, reply to fqth), and Tables R11–R12 (reply to 3JyQ) break down pool quality vs. within-item ranking and sensitivity-gate stops.

(Continued in Part 4/4)

AuthorsOpenReview ↗

General Response and Summary of Revisions (Part 4/4)

G4. Analysis RL and asynchronous training infrastructure

Fig. 4 identifies analysis as the dominant source of outcome variability, while the role ablation in Table 15 shows that downgrading the analyzer hurts substantially more than downgrading the implementer. Role separation lets us train the analyzer independently: we train an open Qwen3.5-9B analyzer with full-parameter GRPO on 100 one-GPU corpus items disjoint from LMSysDojo-100 (8 analyses/item) while freezing claude-opus-5 as the implementer, using LMSysDojo for both training and held-out evaluation (§4, Table 3, and Appendix G in the revision).

Table R5 (analysis RL on unseen LMSysDojo-100, E2E normalized gain of parallel gap analysis):

analyzerheld-out K = 1K = 2K = 4K = 8dev K = 1K = 2K = 4K = 8
Qwen3.5-27B (untrained)5.67.311.616.37.89.714.920.2
Qwen3.5-9B1.52.44.87.41.82.95.08.9
Qwen3.5-9B + analysis RL4.17.211.516.33.77.913.919.4
  • 9B analyzer matches untrained 27B. Rewarded solely by clipped median log speedup over E2E, TTFT and ITL under the validity and faithful-implementation checks (predicted gain never enters the reward), the 9B analyzer more than doubles its held-out score from 1.5% to 4.1% at K = 1 and from 7.4% to 16.3% at K = 8, matching untrained Qwen3.5-27B.
  • Dense learning signal via decoupling. With the frozen Opus implementer (restricted during training to faithful implementation), the untrained 9B analyzer already delivers patches in 64% of 712 rollouts, with 67% in groups having reward spread; by contrast, in the same setup, an exploratory whole-solver 27B run timed out in 204 of 268 groups (8 rollouts each), and no group showed reward spread.
  • Elastic asynchronous GPU RL infrastructure (4.7× throughput). RL over live serving deployments faces long, uneven rollouts (median 42 min) and shifting three-way GPU contention among policy serving, 110k-token training, and benchmark measurement. Across 14 nodes (112 H100s, ≈10 nodes forming the rollout pool for live serving deployments and the rest running 9B policy serving and training), our asynchronous loop combines prefix-pinned serving (97% prefix-cache hit rate), in-place weight hot-swapping across active sessions (median 48 s), and elastic node draining and role-switching as bottlenecks shift, training 4.7× as many valid groups per hour (29.7 vs. 6.2) as a synchronous loop on the same nodes.

Summary of revisions

  • Cost analysis and scaling (Tables R2–R4 → revised §3, §5 Fig. 8, §B.2 Table 10, §C.8 Table 14 & Fig. 16): Per-session/per-item costs, post-cutoff scaling to K = 50, and real-world verification break-even.
  • LMSysDojo-Ext100 (Tables R1, R8, R9 → revised §A.5, §D Table 16, §D.4–D.5 Tables 25–26): Main results plus Gemini and role ablations on 100 extended held-out items.
  • Statistical tests (Tables R6, R7 → revised §D Tables 17–18): Item-bootstrap 95% intervals and Holm-corrected paired sign-flip tests over 70 and 170 held-out items.
  • New controls and diagnostics (Table 1, Tables R10–R12 → revised §5 Table 1, §C.8, §D Table 19, §D.2–D.3 Tables 22–23):Vanilla + self-select, free-form + corpus/, declared-metric scoring, stop-reason diagnostics, and pool vs. ranking breakdown.
  • Analysis RL and RL infrastructure (Table R5 → revised §4 Fig. 5, §5 Table 3, Appendix G): GRPO analyzer training and the elastic asynchronous GPU training system.
  • Text clarifications (revised §2, §3, §A.3, §C.2, §D, §F): Pre-implementation selection, parallel breadth vs. iterative analytical depth under costly verification, setup-card specification and examples, expert checklist audit, and dev-to-held-out discussion.

We sincerely thank the Area Chair and all three reviewers again for their time and constructive feedback.

Best Regards,

Authors of Submission 33699

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.