- Rating
- 4(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces LMSysDojo, a new benchmark and framework for optimizing LLM serving systems like vLLM and SGLang. It does two main things:
- The Benchmark: Mines around 40k historical PRs down to 900 reproducible setups, creating a 100-item benchmark (split into dev, pre-cutoff, and post-cutoff sets) with sealed expert fixes.
- Gap Analysis Framework: Splits the task so an analyzer predicts a performance gain before a separate implementer writes the code. They test parallel (), structured (executable analyze() code), and iterative variants (with sensitivity and assumption checks). The iterative approach boosts held-out E2E normalized gain from roughly 57% (vanilla Claude Code) up to 73-83%. They also run some Gemini and Sonnet role-ablation checks.
Critical strengths
The filtering, temporal splits, and noise floor (1.075x) are super careful. Their open-ended null result—150 baseline attempts and none beat the noise floor—is a fantastic justification for why we need these calibrated setups in the first place.
Proving that analyzer quality is what actually matters while implementer quality barely moves the needle is a super useful takeaway that goes way beyond just this paper.
Showing that predicting gains beforehand beats an LLM ranker looking at finished diffs is clean proof that the ex-ante signal has real value.
The qualitative section is great about calling out mis-sized analyses, flawed premises, and patches that wouldn't normally merge upstream.
Critical weaknesses
The "fixed evaluation budget" framing is a bit misleading since it only matches compute at the final deploy stage. Parallel () and iterative runs burn way more tokens, dollars, and GPU hours than vanilla. We really need a per-item cost breakdown across the board.
The paper claims vanilla Claude Code "states no prediction." But vanilla has measure.sh and could easily be asked to output a predicted speedup in a config file. Without testing vanilla with a self-reported prediction, we can't tell if the gains come from the actual gap-analysis architecture or just from forcing the model to estimate a number.
The parallel curve is still going up at , and iterative uses up to 4x the analyzer sessions. It's unclear if the extra multi-step checks actually add value compared to just bumping up in pure parallel sampling.
The structured analyzer gets SQL access to the profile data (corpus/). Its wins might just come from having better data access rather than the required program structure. A free-form analyzer with corpus access would clear this up.
Looking at Table 14, ranking consistency is only 65-76% with wide CIs, and pick-minus-average gains are pretty small. The high Spearman correlation mostly comes from between-item headroom. The text makes it sound like ranking is super precise, but the numbers suggest structure mostly helps by improving pool quality instead.
The authors admit their sensitivity check is pretty loose ( lets a -20% projection pass), yet it still stops 36-46% of analyses. We need a breakdown of these stops—if most are just crashes or zero-leverage, it's basically just an execution gate, not a real sensitivity test.
The workloads are designed to be the "smallest workload that enters the changed path" and setup cards include explicit themes. That gives the agent a lot of hints that real-world debugging never provides. An ablation with generic workloads would help show how much this measures true diagnosis vs guided search.
Questions for authors
What is the actual per-item cost (tokens, dollars, GPU-hours) for each row in Table 1?
How does vanilla perform if you give it a self-reported predicted gain plus selection?
How do the sensitivity check stops break down by reason?
How does a free-form analyzer with corpus/ access compare to the structured one?
How does parallel selection compare to iterative when you match the total number of analyzer sessions?
How does perfomance change with a generic workload or no setup card theme?
7 comments
Response to Reviewer 3JyQ (Part 2/3)
W5
Looking at Table 14, ranking consistency is only 65-76% with wide CIs [...] the numbers suggest structure mostly helps by improving pool quality instead.
Table R11 separates pool quality (mean of all ten and mean of delivered, a post-hoc diagnostic) from within-item selection (pick) across the held-out K = 10 pools (§B.1, Eq. 7):
| held-out pool | mean of all ten | mean of delivered | pick |
|---|---|---|---|
| pre-cutoff, gap analysis | 55.1 | 60.5 | 64.7 |
| pre-cutoff, structured | 64.6 | 67.1 | 70.7 |
| post-cutoff, gap analysis | 62.2 | 68.3 | 73.2 |
| post-cutoff, structured | 62.1 | 66.1 | 77.3 |
Both mechanisms contribute, and their balance differs by split: pre-cutoff, structure mainly raises pool quality (64.6 vs. 55.1), whereas post-cutoff, pool means are identical (62.1 vs. 62.2) and structured selection adds +11.2 points over its delivered mean (vs. +4.9 for free-form). Across all 170 held-out items, ranking by predicted gain adds +11.8 (free-form) and +11.1 (structured) points over a single analysis (Holm p < 0.001; Table R7); Table R11 is added in §D.2 (Table 22) of the revision, framing within-item ranking as a useful, imperfect signal (65–76% pairwise consistency vs. 50% chance).
W6
The authors admit their sensitivity check is pretty loose [...] We need a breakdown of these stops—if most are just crashes or zero-leverage, it's basically just an execution gate, not a real sensitivity test.
Table R12 breaks down the 700 held-out structured analyses (ten per item, τ = 0.01):
| outcome | count (share) | delivered E2E geomean |
|---|---|---|
| pass | 400 (57.1%) | 1.67× |
| stop: program did not run | 40 (5.7%) | 1.47× |
| ↳ handle misuse (A, E or C used as a dict, iterable or callable) | 22 | |
| ↳ SQL error (missing table, bad SQL) | 9 | |
↳ assume.yaml unparseable | 9 | |
| stop: zero leverage on every metric | 62 (8.9%) | 1.39× |
| ↳ predicts exactly 0% on every metric | 42 | |
| ↳ nonzero prediction, rounded below the tracer's step | 20 | |
| stop: claim-to-leverage ratio below τ | 198 (28.3%) | 1.20× |
| ↳ a claim of 1% or more on some metric | 155 | |
| ↳ every claim under 1% | 43 |
One third of the 300 stops are execution or zero-leverage failures (40 + 62, which alone leave the post-cutoff passed-set mean unchanged at 61.9 vs. 62.1), while two thirds (198) fail the claim-to-leverage ratio (median claimed E2E change 2.5% vs. 20.5% among passes; applying τ = 0.01 lifts the passed-set mean to 66.4, and +4 to +10 points across τ ∈ [0.0025, 0.2]). Thus the sensitivity check is a mechanical conditioning gate rather than a pure crash check, while premise errors are caught separately by the assumption audit (failing 96 of 700 analyses, only 45 overlapping; passing both reaches 67.3% post-cutoff, §D.3). Table R12 is added to §D.3 (Table 23) in the revision.
W7
The workloads are designed to be the "smallest workload that enters the changed path" and setup cards include explicit themes. [...] An ablation with generic workloads would help show how much this measures true diagnosis vs guided search.
- Staged cards carry no theme. Any theme section is stripped at staging (
themed: falseon every run; stated in §C.2 with concrete setup-card examples in §2 and §C.2 of the revision). What §1–§2 call a "scenario theme" is a one-line user-facing restatement of the traffic that never names the bottleneck, subsystem or fix; e.g.:
- "A knowledge-base assistant answering single-turn questions over 20 shared 6000-token briefs, ten concurrent users, one GPU." (vLLM 47782)
- "A general-purpose chat assistant on one GPU, serving 512 concurrent users short mixed-length single-turn conversations." (vLLM 18161)
- "A document question-answering helper on one GPU, where eight concurrent users share long 32k-token reference texts for brief answers." (vLLM 25752)
- Real-world optimization targets a known traffic regime. Operators know the traffic they serve, and in our survey of 50 serving papers (§B.4, Table 9), mechanisms only pay off under a stated workload regime (e.g., Marconi on shared prefixes, NanoFlow on large batches, FlexPipe 3× faster on bursty CV=4 traffic and 2.7× slower on smooth traffic). Specifying the traffic defines the problem; diagnosing where time goes is left to the agent (≈25% of delivered patches share zero modified files with the reference PR, and 22–27% of those still beat the expert, Table 13).
- Why we do not swap in generic workloads. A workload must exercise the PR's changed path to guarantee real optimization headroom and a valid expert denominator; the workloads themselves follow standard serving shapes (decode-heavy, long prefill, multimodal, chat and speculative decoding cover 80% of items). In our open-ended study without calibrated headroom (§A.6), agents chose their own workloads across >1,000 GPU runs and 0 of 150 attempts cleared the 1.075× noise floor, making weak diagnosis indistinguishable from zero headroom.
(Continued in Part 3/3)
Response to Reviewer 3JyQ (Part 3/3)
Questions
Q1 What is the actual per-item cost (tokens, dollars, GPU-hours) for each row in Table 1?
Because only the selected proposal is implemented, K = 10 parallel gap analysis costs less per item (USD 57–60, 0.92 sandbox GPU-h) than ten vanilla sessions (USD 75–85, 1.60–1.92 GPU-h), while K = 10 iterative costs USD 94 across 16.9 analyzer sessions, all sharing one 2.0 GPU-h final verification (W1; full table in G2, Table R2, referenced in Table 1 and reported in §C.8, Table 14 of the revision).
Q2 How does vanilla perform if you give it a self-reported predicted gain plus selection?
It confirms that any informative gain estimate unlocks multi-sample selection, matching parallel gap analysis at equal K = 10 (62.4 / 70.0 vs. 64.7 / 73.2), while showing that the architectural benefit of decoupling is obtaining that estimate before paying to implement every candidate, yielding higher gain at matched budget (W2; G3, Table 1, Table R3).
Q3 How do the sensitivity check stops break down by reason?
Two thirds of the 300 stops (198) fail the claim-to-leverage threshold rather than crashing (40) or having zero leverage (62), and the threshold step accounts for the passed-set enrichment, confirming the gate acts as a conditioning check rather than a crash filter (W6; Table R12, §D.3 Table 23 in the revision).
Q4 How does a free-form analyzer with corpus/ access compare to the structured one?
At K = 1, adding corpus access changes free-form analysis by +1.0 point pooled, with no statistically significant difference (Holm p = 1.00). Both analyzers receive the same underlying profiles; the structured representation additionally makes predictions executable and checkable, thus further enabling our iterative gap analysis.
Q5 How does parallel selection compare to iterative when you match the total number of analyzer sessions?
At matched analyzer sessions or agent dollars, broader parallel sampling wins in the measured range (e.g., 91.1 at 20 sessions vs. 82.7 at 16.9 sessions for K = 10 iterative; W3, G2.2).
However, iterative checks explore analytical depth within each candidate track: even using "the two simplest checks the program affords" (§3), checks and retries improve the pick at every tested K from 10 to 50 and reach 103.8 at K = 50, motivating richer pre-build verifiers on the same structured interface as future work. Among the fixed K = 10 configurations in Table R4, iterative has the lowest total cost per point when verification costs more than approximately USD 420; when K can also vary, the full cost–performance curves in Table R3 determine the preferred configuration.
Q6 How does perfomance change with a generic workload or no setup card theme?
All reported runs omit the theme section and withhold the bottleneck and fix. We have not run a controlled generic-workload ablation: changing the workload can remove the known optimization opportunity. The open-ended study provides related context, with none of 150 attempts clearing the noise floor, but does not isolate the effect of changing workloads (W7).
We hope these analyses and controls address your concerns, and we would be glad to discuss any of them further.
Best Regards,
Authors of Submission 33699
Response to Reviewer 3JyQ (Part 1/3)
Dear Reviewer 3JyQ,
We thank you for the detailed and constructive review. We appreciate the opportunity to report the cost of every configuration, add the controls the review proposes, and clarify what the checks and the setup cards do.
W1
Agreed: the fixed budget is one final 2.0 GPU-h deployed verification per item (§4; §5 in the revision), while agent compute scales with K. Per item, vanilla costs USD 7.49 (6.3M / 93k tokens, 0.16 sandbox GPU-h), K = 10 parallel gap analysis costs USD 57–60 (48.8–51.8M / 873–943k tokens, 1.2 h concurrent wall clock, 0.92 GPU-h) vs. USD 75–85 for ten vanilla sessions, and K = 10 iterative costs USD 94 (16.9 analyzer sessions, 1.27 GPU-h); full breakdown in G2 (Table R2; referenced in Table 1 and reported in §C.8, Table 14 & Fig. 16 of the revision).
W2
Prompting vanilla to also output a predicted gain in
DELIVERY.yamlalongside its patch and report (using the same sandbox,./measure.shand selection rule as gap analysis, so it is essentially coupled gap analysis + implementation;Vanilla + self-selectin G3, Table 1) lifts K = 10 from 57.0 / 60.1 to 62.4 / 70.0 and matches parallel gap analysis at equal K = 10 (64.7 / 73.2, Holm p = 1.00). What decoupling changes is timing and per-candidate cost (ranking estimates before coding so only the winner is built): an analysis costs USD 5.28 vs. USD 8.46 for a full session that also implements, so at matched budgets 30 analyses reach 94.2 post-cutoff at USD 162.9 vs. 81.6 for 20 self-selected runs at USD 169.2 (G2.2, Table R3), while also enabling pre-build checks (W3) and analyzer RL (G4). We added this control to Table 1 and state the pre-implementation distinction precisely in the revision.W3
At matched agent dollars, increasing parallel K is more efficient in the measured range (e.g., K = 20 structured + parallel reaches 91.1 at USD 116 with 20 sessions, vs. 82.7 at USD 94 with 16.9 sessions for K = 10 iterative). At fixed K (maximizing gain per deployed verification), even "the two simplest checks the program affords" (§3) add +2.2 to +5.4 points at every tested K from 10 to 50 (+3.2 pooled at K = 10, Holm p = 0.001; Table R7) and reach the highest measured score, 103.8 at K = 50 vs. 100.7 for structured parallel (full curves, ≈USD 420 fixed-K break-even, and richer future gates in G2.2, Tables R3–R4).
W4
Both analyzers already receive the same profiles (
prof.sqlitevs. the SQL handleC, §C.2). Addingcorpus/to free-form analysis at K = 1 gives 56.6 / 61.9, compared with 55.1 / 62.2 without it (+1.0 pooled, 95% CI [−7.3, +9.7], Holm p = 1.00; G3, Table 1, Table R7). No statistically significant benefit from the additional SQL interface was detected in this control; instead, the structured representation makes predictions executable and checkable, enabling iterative gap analysis to use sensitivity checks and assumption audits to guide revisions before implementation (G3, Table 1).(Continued in Part 2/3)