- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether code LLMs that can correctly implement software changes can also predict their downstream operational consequences when those consequences are not immediately observable. The authors introduce a benchmark based on synthetic service instances with known resource structure and derived ground-truth capacity behavior. The benchmark contains 11 task types: 4 implementation (“do”) tasks and 7 consequence-prediction (“judge”) tasks, including 4 matched do–judge pairs defined on the same service instances. The evaluation is conducted primarily on Qwen2.5-Coder models across six model scales, with performance reported using chance-corrected accuracy relative to the exact best constant-predictor baseline. The main empirical finding is a persistent gap between implementation and judgment performance across model scales, with quantitative judgment tasks remaining particularly challenging.
Critical strengths
Clear benchmark and task design. The service instances are well specified, and the distinction between “do” and “judge” tasks is easy to understand.
Clear research objective. The paper asks a focused question: whether models that can implement software changes can also predict their downstream operational consequences.
Objective ground truth. Judge-task labels are derived from an explicit capacity law rather than subjective human annotation, which makes the benchmark more reproducible.
Useful diagnostic analysis. The stripped arithmetic control helps identify whether failures come from the final computation or from mapping the service description into the correct capacity representation.
Critical weaknesses
The matched-pair evidence is uneven. Two of the four pairs show large do–judge gaps, while the visit-structure gap is substantially smaller and the offloading pair shows no gap. More discussion of why the separation varies so strongly across task pairs would strengthen the main claim.
Task difficulty may confound the do–judge comparison. Some implementation tasks, such as writing an index migration, appear considerably simpler than their corresponding judgment tasks, which require multi-step quantitative reasoning. It is therefore not fully clear how much of the observed gap is due to generation versus judgment rather than differences in intrinsic task difficulty.
The capacity law’s validation is not shown. The paper repeatedly states that judge labels come from a capacity law “validated on running services,” and this validation is central to the claim of objective ground truth. However, no validation results appear in the main text or appendix:
The benchmark is synthetic and simplified. While this provides clean ground truth, it omits many effects present in real systems, which limits the extent to which the results can be generalized to realistic software-engineering settings.
Cross-family validation is limited. The main scaling study is conducted only on Qwen2.5-Coder models. Although the authors include a DeepSeek-Coder 1.3B evaluation, this is only a single-scale comparison, so it provides limited evidence that the observed behavior generalizes across model families.
Questions for authors
How do you separate the do–judge gap from differences in intrinsic task difficulty? For example, could you add harder do tasks or simpler quantitative judge tasks so that the two forms are more comparable in difficulty?
In the offloading pair, both the do and judge scores are 0.0, and async conversion scores 0.0 even at 32B. What causes this: a grading or harness issue, an underspecified task, or a genuine model failure?
How was the capacity law validated as ground truth? Please describe the running services and workloads used, and how closely the law’s predictions matched the measurements.
4 comments
Response to Reviewer GAdZ (part 2/2)
(2/2)
W1 (why the matched-pair gaps differ). The pair gaps differ because the do members differ (§4, Table 3). At 3B every judge member lies within 22 percentage points of its best constant (−8.2%, +0.0%, +0.0%, +21.9%), while the do members range from +87.3% to +0.0%, and the gaps follow the same order. Index migration is written correctly on 87.3% of matched instances while the threshold judge is below chance, a gap of 95.6 points. Pool configuration scores +69.3% and the pool-binding judge +0.0%, a gap of 69.3 points.
The visit-structure do member must reproduce the exact visit sequence, the most demanding do item for small models (+38.8% at 3B). Its judge member names a stage, the categorical form that improves with scale (Table 4). The gap is 16.9 points, and its interval [+6.9, +41.0] still excludes zero. The offloading pair is explained in Q2.
Q1 / W2 (separating the gap from intrinsic task difficulty; new §6.1, Table 5, Appendix H). We added both kinds of items the reviewer proposes, on the same evaluated instances, in runs of Qwen2.5-Coder 1.5B, 3B, 7B, 14B and 32B and DeepSeek-Coder 1.3B and 6.7B, in which every comparison is within a run (float16 activations; AWQ int4 weights at 14B and 32B). Harder do items require the consequence: the index migration only if it is needed at a stated future table size, the handler that offloads the step whose removal most raises capacity, and the concurrency limit for a stated load. Simpler quantitative judge items ask for one stage's demand, a single multiplication, and for that stage's throughput alone. Code twins answer each quantitative judge question by writing a Python function answer(), executed and graded with the judge item's tolerance.
At 32B the specified change scores +0.859 and the stage demand +0.973, while the harder do items score +0.008, the code twins −0.046, the stated answers −0.058, and the stage's throughput, which needs , +0.068. The same order holds at every scale (Table 5): stage demand rises from +0.324 at 1.5B to +0.973 at 32B, while no consequence column exceeds +0.008 in any run. The 1.5B, 3B, 7B and 14B runs write the migration itself correctly on 97.5% to 100.0% of items, yet answer -- no index needed on all 55 index-if-needed instances. Performance therefore tracks whether the consequence must be derived, not the output form.
The stripped arithmetic control points the same way. It computes the same argmax over the same numbers with the service removed, and 1.5B and 3B score +57.4% and +61.5% there, against +11.7% and +30.0% on the capacity question (Table 6). Judge items are also not uniformly hard: at 32B, choosing the best intervention scores +83.1%, above three of the four do types, while every quantitative judge type is at or below chance (Tables 4 and 9). The gap survives removing the easiest do type, index migration (+0.334, [+0.285, +0.454]).
W5 (cross-family evidence). The second family now covers DeepSeek-Coder 1.3B, 6.7B and 33B (§7.1, Table 7). The do–judge gap is +0.141, +0.335 and +0.308 under the oracle answer locator, positive at every scale, and in the control runs DeepSeek-Coder at 1.3B and 6.7B shows the same pattern as Qwen, with the specified change at +0.294 and +0.703 and the do items that need the consequence at −0.203 and −0.087 (Appendix H).
W4 (what the synthetic setting measures). The generated specifications give exact labels at scale, and §3.1 shows that the labels hold on running services in two languages and three concurrency models. The benchmark measures whether a model applies a validated operational law to fully specified services, which is the computation a capacity review performs.
We are glad to answer any further questions during the discussion.
Thank you for your response. I maintain my score.
Thank you for taking the time to follow up on our response.
Because your initial review noted minor or fixable soundness concerns and raised three specific empirical questions, all of which we implemented directly in the revision ((1) 1,126 live Python/Go service runs validating the capacity law to a 0.7% median error in §3.1, (2) isolating the prompt-API and local-deferral causes behind both 0.0 cells, with the stated-API async conversion reaching 76/80 at 32B in §4, and (3) adding difficulty-matched controls across five Qwen scales and two DeepSeek scales in §6.1 alongside the three-scale DeepSeek sweep in §7.1). Could you let us know which remaining limitation or aspect of the new results weighs most heavily in your current assessment?
If concerns remain regarding the scope of synthetic service specifications (W4) or any detail of the new experiments, we would be very glad to clarify before the discussion period closes on November 18.
Response to Reviewer GAdZ (part 1/2)
(1/2)
We thank the reviewer for the careful reading and the precise questions. Each one led to a new measurement in the revision.
Q3 (validation of the capacity law as ground truth: services, workloads and agreement; new §3.1, Table 2, Appendix G, Table 15). We realise the evaluated instances as HTTP services in three runtimes: Python
asynciowith one single-worker thread per stage, so that all stages share one interpreter; Python with one single-worker process per stage; and Go with one locked OS thread per stage. In each service, a visit consumes its stated cost as CPU time on its stage's server. The outbound call is a sleep that holds an admission slot, and the admission pool is a semaphore.For the workloads, a separate Go load generator drives each service in one of two modes. Closed-loop keeps a fixed number of requests in flight, which measures saturation throughput. Open-loop issues Poisson arrivals at the item's stated load.
The agreement over 1,126 runs is as follows.
Appendix G states which of the two measurement passes gives each row; the archive contains both.
The measurements also give two precise statements of the law, which we now use as refined labels (Appendix F). When stages share one interpreter, demands add. Capacity then falls from the first added row of an unindexed scan: measured capacity at 0.8T⋆ is 0.83–0.87 of that at 0.5T⋆, where T⋆ is the derived threshold table size. The index-threshold type is therefore defined on the parallel runtimes.
A small pool with long outbound calls caps throughput below the stage bound, by a median 35.1% on 13 maximum-throughput instances and 52.1% on 20 runtime-change instances. On those instances the running service matches the closed network to a median 2.4% and 1.4%.
Under both refinements every quantitative judge cell stays at or below chance at every scale. The quantitative macro is −13.0%, −12.7%, −10.7%, −7.6%, −4.7% and −4.1% from 0.5B to 32B (Table 13).
Q2 (offloading 0.0 and async conversion 0.0 at 32B). The two zeros have different causes, and the revision locates both.
For async conversion, the cause is the item's prompt. The submitted item named the hand-off call
ctx.enqueue(op, n), but unlike the handler item it did not list the context methods and operation names. Models therefore wrote their own APIs: at 32B, 58 of 80 outputs call context methods that do not exist, against 0 of 80 handler outputs on the item that lists them. The revised item states the same API as the handler item, and its harness accepts the hand-off call whether or not it is awaited (§4, Appendix D). With the API stated, the 32B run of §6.1 writes the conversion correctly on 76 of 80 instances (95%) and no output calls a non-existent method, against 51 such outputs of 80 on the submitted item in the same run. At 14B the stated API cuts such outputs from 58 to 0, and 67 of 80 conversions are correct. The async-conversion zero is therefore a property of the submitted prompt, and the corrected item is among the best-solved do items.For the offloading judge, the cause is a genuine and specific model behaviour. The type has three variants with different correct answers. Deferring work to a queue on the same machine never raises capacity. Offloading raises it exactly when it removes the demand that binds. Batching raises it when the reduction lowers that demand. At 3B the model answers
noto every variant. At 32B it answers the offloading variant correctly on 80% of the instances where offloading raises capacity and 60% of those where it does not, and answersyesto deferral on 80% (Table 14). The aggregate zero is therefore the sum of correct offloading answers and a systematic error on deferral. The running services confirm all 77 labels of these instances.Continued in part 2/2 below.