acceptodds
Under review as a conference paper at ICLR 2027

Generation and Judgment Diverge in Code Models: A Matched Benchmark with Derived Ground Truth

Abstract

Code-model benchmarks usually evaluate outputs whose correctness can be checked immediately, but many software decisions concern consequences that emerge only under future load. We ask whether models that can implement a change can also predict its system-level consequence. We introduce a matched benchmark built from synthetic service specifications with known resource decompositions. Each instance yields implementation tasks, graded by execution, and judgment tasks, graded against an operational capacity law validated on running services; four matched pairs probe the same underlying knowledge. Across six Qwen2.5-Coder scales from 0.5B to 32B parameters, chance-corrected implementation scores range from +0.275 to +0.616, whereas judgment scores range from -0.044 to +0.153, leaving a positive gap at every scale. Quantitative judgment remains at or below chance in all 24 scale-by-type cells, while categorical judgment improves with scale. A stripped arithmetic control shows that 1.5B and 3B models can solve the final argmax once the service representation is removed, localising much of the deficit to constructing the quantities required by the capacity law. Evaluation details also matter: raw accuracy can reverse model-family ordering, and a marker-strict answer parser can introduce a 45-point artifact. We derive the exact best-constant baseline for tolerance-scored numeric outputs, separate answer extraction from grading, and reproduce the main separation in a second model family.

Primary area:
datasets and benchmarks
Keywords:
code language models, software engineering, code generation, model judgment, LLM evaluation, datasets and benchmarks, system capacity, chance-corrected evaluation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_GAdZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies whether code LLMs that can correctly implement software changes can also predict their downstream operational consequences when those consequences are not immediately observable. The authors introduce a benchmark based on synthetic service instances with known resource structure and derived ground-truth capacity behavior. The benchmark contains 11 task types: 4 implementation (“do”) tasks and 7 consequence-prediction (“judge”) tasks, including 4 matched do–judge pairs defined on the same service instances. The evaluation is conducted primarily on Qwen2.5-Coder models across six model scales, with performance reported using chance-corrected accuracy relative to the exact best constant-predictor baseline. The main empirical finding is a persistent gap between implementation and judgment performance across model scales, with quantitative judgment tasks remaining particularly challenging.

Critical strengths

Clear benchmark and task design. The service instances are well specified, and the distinction between “do” and “judge” tasks is easy to understand.

Clear research objective. The paper asks a focused question: whether models that can implement software changes can also predict their downstream operational consequences.

Objective ground truth. Judge-task labels are derived from an explicit capacity law rather than subjective human annotation, which makes the benchmark more reproducible.

Useful diagnostic analysis. The stripped arithmetic control helps identify whether failures come from the final computation or from mapping the service description into the correct capacity representation.

Critical weaknesses

The matched-pair evidence is uneven. Two of the four pairs show large do–judge gaps, while the visit-structure gap is substantially smaller and the offloading pair shows no gap. More discussion of why the separation varies so strongly across task pairs would strengthen the main claim.

Task difficulty may confound the do–judge comparison. Some implementation tasks, such as writing an index migration, appear considerably simpler than their corresponding judgment tasks, which require multi-step quantitative reasoning. It is therefore not fully clear how much of the observed gap is due to generation versus judgment rather than differences in intrinsic task difficulty.

The capacity law’s validation is not shown. The paper repeatedly states that judge labels come from a capacity law “validated on running services,” and this validation is central to the claim of objective ground truth. However, no validation results appear in the main text or appendix:

The benchmark is synthetic and simplified. While this provides clean ground truth, it omits many effects present in real systems, which limits the extent to which the results can be generalized to realistic software-engineering settings.

Cross-family validation is limited. The main scaling study is conducted only on Qwen2.5-Coder models. Although the authors include a DeepSeek-Coder 1.3B evaluation, this is only a single-scale comparison, so it provides limited evidence that the observed behavior generalizes across model families.
Questions for authors

How do you separate the do–judge gap from differences in intrinsic task difficulty? For example, could you add harder do tasks or simpler quantitative judge tasks so that the two forms are more comparable in difficulty?

In the offloading pair, both the do and judge scores are 0.0, and async conversion scores 0.0 even at 32B. What causes this: a grading or harness issue, an underspecified task, or a genuine model failure?

How was the capacity law validated as ground truth? Please describe the running services and workloads used, and how closely the law’s predictions matched the measurements.
4 comments
AuthorsOpenReview ↗

Response to Reviewer GAdZ (part 1/2)

(1/2)

We thank the reviewer for the careful reading and the precise questions. Each one led to a new measurement in the revision.

Q3 (validation of the capacity law as ground truth: services, workloads and agreement; new §3.1, Table 2, Appendix G, Table 15). We realise the evaluated instances as HTTP services in three runtimes: Python asyncio with one single-worker thread per stage, so that all stages share one interpreter; Python with one single-worker process per stage; and Go with one locked OS thread per stage. In each service, a visit consumes its stated cost as CPU time on its stage's server. The outbound call is a sleep that holds an admission slot, and the admission pool is a semaphore.

For the workloads, a separate Go load generator drives each service in one of two modes. Closed-loop keeps a fixed number of requests in flight, which measures saturation throughput. Open-loop issues Poisson arrivals at the item's stated load.

The agreement over 1,126 runs is as follows.

LabelAgreement
maximum throughput (stage law)median error 0.7%; 97.5% within the 8% tolerance
throughput after runtime changemedian error 0.4%; 97.5% within tolerance
with the instance's admission pool (closed network)median error 1.9%; 93.8% within tolerance
first stage to saturate80/80
does the change raise capacity77/77
index threshold (19 instances × 4 table sizes)crossing within median 0.7%; 19/19 within tolerance
does the pool bind (requests wait for a slot)69/79; 65/68 where the stated load is not within 25% of the binding point

Appendix G states which of the two measurement passes gives each row; the archive contains both.

The measurements also give two precise statements of the law, which we now use as refined labels (Appendix F). When stages share one interpreter, demands add. Capacity then falls from the first added row of an unindexed scan: measured capacity at 0.8T⋆0.8T^{\star} is 0.83–0.87 of that at 0.5T⋆0.5T^{\star}, where T⋆T^{\star} is the derived threshold table size. The index-threshold type is therefore defined on the parallel runtimes.

A small pool with long outbound calls caps throughput below the stage bound, by a median 35.1% on 13 maximum-throughput instances and 52.1% on 20 runtime-change instances. On those instances the running service matches the closed network to a median 2.4% and 1.4%.

Under both refinements every quantitative judge cell stays at or below chance at every scale. The quantitative macro is −13.0%, −12.7%, −10.7%, −7.6%, −4.7% and −4.1% from 0.5B to 32B (Table 13).

Q2 (offloading 0.0 and async conversion 0.0 at 32B). The two zeros have different causes, and the revision locates both.

For async conversion, the cause is the item's prompt. The submitted item named the hand-off call ctx.enqueue(op, n), but unlike the handler item it did not list the context methods and operation names. Models therefore wrote their own APIs: at 32B, 58 of 80 outputs call context methods that do not exist, against 0 of 80 handler outputs on the item that lists them. The revised item states the same API as the handler item, and its harness accepts the hand-off call whether or not it is awaited (§4, Appendix D). With the API stated, the 32B run of §6.1 writes the conversion correctly on 76 of 80 instances (95%) and no output calls a non-existent method, against 51 such outputs of 80 on the submitted item in the same run. At 14B the stated API cuts such outputs from 58 to 0, and 67 of 80 conversions are correct. The async-conversion zero is therefore a property of the submitted prompt, and the corrected item is among the best-solved do items.

For the offloading judge, the cause is a genuine and specific model behaviour. The type has three variants with different correct answers. Deferring work to a queue on the same machine never raises capacity. Offloading raises it exactly when it removes the demand that binds. Batching raises it when the reduction lowers that demand. At 3B the model answers no to every variant. At 32B it answers the offloading variant correctly on 80% of the instances where offloading raises capacity and 60% of those where it does not, and answers yes to deferral on 80% (Table 14). The aggregate zero is therefore the sum of correct offloading answers and a systematic error on deferral. The running services confirm all 77 labels of these instances.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer GAdZ (part 2/2)

(2/2)

W1 (why the matched-pair gaps differ). The pair gaps differ because the do members differ (§4, Table 3). At 3B every judge member lies within 22 percentage points of its best constant (−8.2%, +0.0%, +0.0%, +21.9%), while the do members range from +87.3% to +0.0%, and the gaps follow the same order. Index migration is written correctly on 87.3% of matched instances while the threshold judge is below chance, a gap of 95.6 points. Pool configuration scores +69.3% and the pool-binding judge +0.0%, a gap of 69.3 points.

The visit-structure do member must reproduce the exact visit sequence, the most demanding do item for small models (+38.8% at 3B). Its judge member names a stage, the categorical form that improves with scale (Table 4). The gap is 16.9 points, and its interval [+6.9, +41.0] still excludes zero. The offloading pair is explained in Q2.

Q1 / W2 (separating the gap from intrinsic task difficulty; new §6.1, Table 5, Appendix H). We added both kinds of items the reviewer proposes, on the same evaluated instances, in runs of Qwen2.5-Coder 1.5B, 3B, 7B, 14B and 32B and DeepSeek-Coder 1.3B and 6.7B, in which every comparison is within a run (float16 activations; AWQ int4 weights at 14B and 32B). Harder do items require the consequence: the index migration only if it is needed at a stated future table size, the handler that offloads the step whose removal most raises capacity, and the concurrency limit for a stated load. Simpler quantitative judge items ask for one stage's demand, a single multiplication, and for that stage's throughput alone. Code twins answer each quantitative judge question by writing a Python function answer(), executed and graded with the judge item's tolerance.

At 32B the specified change scores +0.859 and the stage demand +0.973, while the harder do items score +0.008, the code twins −0.046, the stated answers −0.058, and the stage's throughput, which needs X=1/DX=1/D, +0.068. The same order holds at every scale (Table 5): stage demand rises from +0.324 at 1.5B to +0.973 at 32B, while no consequence column exceeds +0.008 in any run. The 1.5B, 3B, 7B and 14B runs write the migration itself correctly on 97.5% to 100.0% of items, yet answer -- no index needed on all 55 index-if-needed instances. Performance therefore tracks whether the consequence must be derived, not the output form.

The stripped arithmetic control points the same way. It computes the same argmax over the same numbers with the service removed, and 1.5B and 3B score +57.4% and +61.5% there, against +11.7% and +30.0% on the capacity question (Table 6). Judge items are also not uniformly hard: at 32B, choosing the best intervention scores +83.1%, above three of the four do types, while every quantitative judge type is at or below chance (Tables 4 and 9). The gap survives removing the easiest do type, index migration (+0.334, [+0.285, +0.454]).

W5 (cross-family evidence). The second family now covers DeepSeek-Coder 1.3B, 6.7B and 33B (§7.1, Table 7). The do–judge gap is +0.141, +0.335 and +0.308 under the oracle answer locator, positive at every scale, and in the control runs DeepSeek-Coder at 1.3B and 6.7B shows the same pattern as Qwen, with the specified change at +0.294 and +0.703 and the do items that need the consequence at −0.203 and −0.087 (Appendix H).

W4 (what the synthetic setting measures). The generated specifications give exact labels at scale, and §3.1 shows that the labels hold on running services in two languages and three concurrency models. The benchmark measures whether a model applies a validated operational law to fully specified services, which is the computation a capacity review performs.

We are glad to answer any further questions during the discussion.

Thank you for your response. I maintain my score.

Thank you for taking the time to follow up on our response.

Because your initial review noted minor or fixable soundness concerns and raised three specific empirical questions, all of which we implemented directly in the revision ((1) 1,126 live Python/Go service runs validating the capacity law to a 0.7% median error in §3.1, (2) isolating the prompt-API and local-deferral causes behind both 0.0 cells, with the stated-API async conversion reaching 76/80 at 32B in §4, and (3) adding difficulty-matched controls across five Qwen scales and two DeepSeek scales in §6.1 alongside the three-scale DeepSeek sweep in §7.1). Could you let us know which remaining limitation or aspect of the new results weighs most heavily in your current assessment?

If concerns remain regarding the scope of synthetic service specifications (W4) or any detail of the new experiments, we would be very glad to clarify before the discussion period closes on November 18.

Reviewer_NgwYOpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper constructs a benchmark comprising four implementation tasks and seven system consequence prediction tasks. Implementation tasks are graded through execution or structural checks, whereas judgment tasks are evaluated using operational capacity law. Six instruction-tuned Qwen2.5-Coder models (0.5B, 1.5B, 3B, 7B, 14B, and 32B) were evaluated on the benchmark, consistently achieving higher scores on implementation tasks.

Critical strengths
  1. The paper studies whether code models understand workload and system constraints, which is vital in real-world programming.
Critical weaknesses
  1. The benchmark design is flawed in proving the paper's central do-judge comparison. The benchmark should consist of tasks of similar difficulty from the sets of "do" tasks and "judge" tasks. The example in the paper, adding an index and predicting its consequence, involves connected stages and does not establish comparable difficulty. For instance, CRUXEval uses the same function and execution relation for input/output prediction, a plausible basis for comparability. Task formulation is biased toward simplified implementation. Real-world programming tasks are almost always performance-aware. The "do" task should involve why a change is needed and under what workload and system constraints it must be done. "Add an index" is the last step of a "do" task that only requires knowledge of database schema and SQL syntax. Also, the paper does not explain how the benchmark is realistic and practical or how it sufficiently represents the sets of "do" and "judge" tasks.

  2. Model coverage and grading. The experiments do not use frontier models. Whether the gap persists in SOTA frontier models is unknown. Is this a benchmark/study that is limited to small models? The accuracy and fairness of the grading cannot be verified.

Questions for authors
  1. How do you establish the comparable difficulty of "do" and "judge" tasks? If "do" tasks are easier, the results will show a gap with higher "do" scores; if "judge" tasks are easier, the results will show the opposite. Fairness in task selection in terms of difficulty is crucial for your conclusion.

  2. The "do" tasks can be formulated in many ways. Why did you choose to formulate them in the form of "add an index"? This seems too simple to me.

  3. How does the benchmark effectively represent the sets of "do" and "judge" tasks? Does it thoroughly represent each set?

  4. Could you share the anonymous repository? I would like to check the details of the benchmark, including the tasks. I also do not thoroughly understand how the grading is done and would like to check.

  5. Do the gap and conclusion hold for frontier models?

5 comments
AuthorsOpenReview ↗

Response to Reviewer NgwY (1/2)

(1/2)

We thank the reviewer for the precise questions about comparability, grading and coverage.

Q1 (comparable difficulty of do and judge tasks). The two members of a pair share one service instance, and every quantity needed for the judge answer is printed in the prompt; the paired gaps are computed on identical instances with an instance-level bootstrap (§4, Table 3). To test difficulty directly, the revision adds three control types on the same evaluated instances (new §6.1, Table 5, Appendix H), run with Qwen2.5-Coder 1.5B, 3B, 7B, 14B and 32B and DeepSeek-Coder 1.3B and 6.7B (float16 activations; AWQ int4 weights at 14B and 32B), with every comparison within a run.

The first is the CRUXEval construction the reviewer describes. Each judge question is posed as writing a Python function answer(), and its returned value is executed and graded with the judge item's own grader and tolerance. At 32B the code twins of the four quantitative judge types score −0.046, near chance like the stated answers (−0.058), while the do items that specify the change score +0.859. At 1.5B, 3B, 7B and 14B the code twins score −0.084, −0.074, −0.016 and −0.084, against +0.258, +0.450, +0.742 and +0.672 for the change. Framing consequence prediction as executable code generation therefore does not close the gap.

The second is a pair of single-step judge items, simpler than any original judge type. One stage's total demand, a single multiplication of two stated numbers, rises from +0.324 at 1.5B to +0.973 at 32B. The throughput of the same stage alone, which needs the capacity relation X=1/DX=1/D, stays between −0.068 and +0.243 at every scale (+0.068 at 32B). The boundary therefore lies where the capacity relation enters.

The stripped arithmetic control agrees. It asks for the same argmax over the same numbers with the service removed, and 1.5B and 3B models score +57.4% and +61.5% there, against +11.7% and +30.0% as a capacity question (Table 6). Judge items whose answer is a choice are within reach: at 32B, choosing the intervention that most raises capacity scores +83.1%, higher than three of the four do types (Tables 4 and 9).

Q2 (why index migration is a do task, and performance-aware do tasks). Index migration is one of four do types, alongside a handler that must reproduce an exact visit sequence, an asynchronous conversion, and an admission-control configuration. Each is the implementation step of a standard capacity remedy, and its paired judge item asks under what workload the remedy matters; Figure 2 shows one instance with both items and the derivation of the label (329,468 rows).

The revision adds the performance-aware do tasks the reviewer asks for, where the artefact is correct only if the model decides why and under what workload the change is needed. They are the index migration only if the missing index costs capacity at a stated future table size, the handler that offloads the one step whose removal most raises capacity, and the concurrency limit sized for a stated load. These items stay between −0.188 and +0.008 at every scale from 1.5B to 32B, near or below chance, while the same runs write the specified change at +0.258 to +0.859. The 1.5B, 3B, 7B and 14B runs write the migration itself correctly on 97.5% to 100.0% of items, yet answer -- no index needed on all 55 index-if-needed instances. A model that edits correctly thus fails as soon as the edit depends on deriving the consequence, which is the capability the judge items measure.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Response to Reviewer NgwY (2/2)

(2/2)

Q3 (coverage of the do and judge sets, and their realism). §3 ("Why these tasks") and Table 1 now state the coverage. The do types are the edits a capacity review recommends. The seven judge types are the questions such a review answers: which resource saturates, the sustainable load, whether a change matters and when, which change to make, the effect of the concurrency model, and whether the admission pool binds. Instances span five resource kinds, three runtimes, visit counts from 1 to 20 and per-visit costs from 0.05 to 90 ms.

The ground truth is now validated on running Python and Go services across three concurrency models (new §3.1, Table 2). Measured saturation throughput agrees with the law to a median of 0.7%, with 97.5% of runs within the grading tolerance. All 77 capacity-change decisions and all 80 bottleneck stages match their labels.

Q4 (anonymous code release and how grading is done). The anonymised supplementary archive attached to the submission contains the item generator and all item sets, the grading harness with its execution sandbox, the raw generations and per-item scores of every reported run, the running-service rig and its measurements, and the scripts that produce every table.

The harness reproduces the per-item scores of the submitted runs exactly: 880/880 items at each of the six Qwen2.5-Coder scales. Appendix D describes each grader; Appendix C gives the exact prompts and decoding settings.

Q5 (frontier models). Our runs cover open-weight code models up to 32B and 33B, chosen because they allow a controlled scale sweep under fixed greedy decoding and runs that anyone can reproduce from the released weights. Within that range the gap is present in every run we report. It holds in two code-model families, Qwen2.5-Coder at six scales up to 32B and DeepSeek-Coder at 1.3B, 6.7B and 33B (§7.1, Table 7), and at the largest scale of each family: +0.463 at Qwen2.5-Coder-32B, larger than the +0.319 at 0.5B, and +0.308 at DeepSeek-Coder-33B under an oracle answer locator.

The 32B control run of §6.1 (AWQ int4 weights) draws the same boundary: the specified change +0.859 and one stage's demand +0.973, against +0.008 for do items that need the consequence and −0.046 for the code twins of the quantitative judge types, and no consequence score exceeds +0.008 at any of the five Qwen2.5-Coder scales from 1.5B to 32B.

At 32B the gap appears without any of the failure signatures of small models: no constant answers, no position bias, no missing answers and no short outputs. The median judge response is 474 tokens (§7). The benchmark, harness and graders are released, so a frontier model can be scored under the identical protocol.

We are glad to answer any further questions during the discussion.

Area_Chair_bfvhOpenReview ↗

Need to respond to authors today

Today is the last day of the discussion period. The authors have prepared a detailed response to your review. Please reply to them as soon as possible to ensure there is a chance for back and forth discussion before the discussion period ends.

Reviewer_AdD6OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper provides a benchmark for LLMs generating codes. The benchmark assess the performance by using two types of tasks, do tasks and judge tasks. The paper experimentally compared similar tasks in different task types to analyze the performance differences depending on task types.

Critical strengths

The motivation of the paper provided in the introduction is understandable. There are important properties of codes that can be evaluated only after deployment.

Critical weaknesses

It is hard to understand the details of the benchmark. I can understand that it consists of different kind of tasks do and judge. However, it is totally unclear for me to understand the number of instances, and how are these benchmarks tasks are used. The explanations are only given by short texts like those in Section 3. These are insufficient to evaluate the values of them.

Experimental results are also hard to understand. The paper provides some experimental results. However, I cannot understand what we can see from these results. This is mainly due to the lack of the detailed explanations of benchmarks. Moreover, the experiments are conducted with only a single family of models Qwen-2.5. Therefore, it is unclear what happens if we use different LLMs.

Questions for authors

In the reproducibility checklist, the paper says that the submission includes supplementary materials to reproduce the results. However, they are not provided in this submission.

3 comments
AuthorsOpenReview ↗

Response to Reviewer AdD6

We thank the reviewer for pointing out where the benchmark description needed to be explicit. The revision makes every count and procedure visible in the main text.

Number of instances, and how the tasks are used (Table 1, Figure 2). New Table 1 lists all 11 item types. For each type it gives the answer form, the grader and tolerance, the exact chance baseline, and the counts in each set. The released set has 10,733 items generated from 1,400 service instances. The evaluated subset has 880 items, 80 per type, drawn from 657 instances, and every model answers exactly this subset. The instance-matched subset has 976 items forming 488 do/judge pairs on 170 instances and is used for the paired analysis of §4.

New Figure 2 walks through one instance. Its specification has three stages with visit counts and per-visit costs, a runtime and an admission pool. The do item asks for the index migration and is graded by parsing the SQL. The judge item asks for the table size at which the missing index starts to reduce maximum throughput. The derivation of its label gives 329,468 rows, scored within a 12% tolerance. Appendix C gives the exact system prompts and decoding settings, and Appendix D gives every grader.

Reading the experimental results (Tables 3, 4 and 6, Figure 3). The central reading is that, on the same service instances, models produce the artefact far more reliably than they predict its consequence. The deficit is concentrated in judge items that require a quantity. Each results table now states its comparison. Table 4 gives do and judge scores at each of six scales; the gap is positive at every scale, categorical judgment rises from +0.050 at 0.5B to +0.432 at 32B, and the four quantitative judge types stay at or below chance in all 24 scale-by-type cells. Table 3 repeats the comparison on identical instances. Table 6 shows that the arithmetic alone is solved once the service description is removed (+57.4% and +61.5% at 1.5B and 3B, against +11.7% and +30.0% as a capacity question). Figure 3 shows the same results at a glance: (a) do against judge at every scale, (b) categorical against quantitative judge types with all 24 quantitative cells, and (c) the four matched pairs.

New §6.1 (Table 5) adds difficulty-matched controls on the same instances: at 32B the specified change (+0.859) and one stage's demand, a single multiplication (+0.973), are solved, while do items that need the consequence (+0.008) and quantitative judge questions answered as executed code (−0.046) stay near chance, and no consequence score exceeds +0.008 at any of five scales from 1.5B to 32B. The new §3.1 adds the validation of the ground truth on running Python and Go services. Measured throughput agrees with the law to a median of 0.7% over 1,126 runs.

A second model family (§7.1, Table 7). We now evaluate a second code-model family, DeepSeek-Coder at 1.3B, 6.7B and 33B, with the same items, graders and baselines. Its do scores are +0.303, +0.381 and +0.381, and its do–judge gap under the oracle answer locator of §7.1 (Table 7), which keeps the best-scoring line of each output, is +0.141, +0.335 and +0.308. The separation is positive at every scale of both families.

Supplementary material. The anonymised supplementary archive is attached to the submission. It contains the item generator and all item sets, the grading harness and execution sandbox, the raw generations and per-item scores of every reported run, the running-service rig with its measurements, and the scripts that regenerate every table and figure. The harness reproduces the per-item scores of the submitted runs exactly (880/880 items at each Qwen2.5-Coder scale).

We are glad to answer any further questions during the discussion.

Thank you for the response. I will keep my score. I think the most important weakness of the paper is its presentation. I've checked the revised paper, but it is still very hard to understand. For example, the paragraph on Judge tasks (line 154) is still hard to understand because it is just an enumeration of undefined technical terms, with no further explanations. The paragraph on Do tasks (line 161) also has the same issue. "Why these tasks." (line 203) is also hard to understand due to main undefined words.

Thank you for checking the revision and for pointing to the exact paragraphs. We agree that §3 used systems terms before defining them. We have uploaded a new PDF in which the four paragraphs of §3 (Service instances, Judge tasks, Do tasks and Why these tasks, now lines 147 to 203) define every term in plain words where it first appears. No number, table, figure or experiment has changed.

In short, each instance is a web service that handles a request by calling three to five stages in order, such as a database lookup, a table scan, a CPU computation or an outbound network call. The prompt states how often one request visits each stage and how long one visit takes; their product is the stage's demand, the time the stage spends on one request (an outbound call only waits and uses no local server). The prompt also states whether the stages share one Python interpreter, and so run one at a time, or run in parallel, and the size of the admission pool, the largest number of requests the server admits at once. As load grows, the local stage with the largest demand is the first to be busy all the time (it saturates), and the highest request rate the service can sustain is its capacity, also called its maximum sustainable throughput.

The judge tasks (your line 154) ask for consequences of this specification: which stage saturates first, the capacity in requests per second, whether a proposed change raises it, the table size at which a missing index starts to lower it, which stage made ten times faster helps most, the capacity after the stages move into parallel processes, and whether at a stated request rate the requests in flight fill the admission pool, so that new ones must wait for a slot. The do tasks (line 161) ask for the code of a change to the same service: the request handler that makes the listed calls in order, the SQL statement that adds the missing index, the handler rewritten to hand one stage to a background queue, and the server setting that caps concurrent requests at the pool size. Why these tasks (line 203) now states the link directly: each do task is the code of a standard capacity fix, and each judge task asks what that fix, or the service it is applied to, does to capacity. Pairing the two on one service asks whether a model that writes a fix correctly can also tell whether and when the fix matters.

Figure 2 works through one instance from the prompt to the label, and §3.1 gives the formulas and their check against running services. If another passage is still hard to follow, we would be glad to know which one, and we will revise it before the discussion closes.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of revision

(1/2)

We thank the three reviewers for careful and constructive reviews. We have uploaded a revised PDF and an anonymised supplementary archive. The benchmark, the scale sweep and every submitted number are unchanged, and the title is unchanged; the revision adds the evidence the reviews asked for, and the abstract states the main result explicitly.

The capacity law is now validated on running services (GAdZ; new §3.1, Table 2, Appendix G). We realise the evaluated instances as HTTP services in three runtimes: Python with all stages sharing one interpreter, Python with one process per stage, and Go with one OS thread per stage. Each visit consumes its stated cost as CPU time, the outbound call is a sleep and the admission pool is a semaphore. A separate Go load generator drives each service closed-loop (saturation) or open-loop with Poisson arrivals at the item's stated load.

Over 1,126 runs, maximum throughput agrees with the law to a median error of 0.7%, with 97.5% of runs within the 8% grading tolerance, and throughput after the runtime change to a median error of 0.4%. The capacity-change decisions (defer, offload, batch) match the label in 77/77 cases and the first stage to saturate in 80/80. On parallel runtimes, capacity is flat below the derived index threshold T⋆T^{\star} and falls as 1/T1/T above it, where TT is the table size; the measured crossing lies within a median 0.7% of the label (19/19 within tolerance). With the instance's own admission pool, the median error against the closed-network form of the law is 1.9%. Appendix G states which of the two measurement passes gives each row, and the archive contains both.

The measurements give two precise statements of the law, which we use as refined labels (Appendix F, Table 13). When all stages share one interpreter, demands add, so an unindexed scan costs capacity from its first row, and the index-threshold type is defined on the parallel runtimes. A small admission pool combined with long outbound calls caps throughput below the stage bound, as the closed network predicts. Against the refined labels every quantitative judge cell is still at or below chance; the quantitative judge macro is −13.0%, −12.7%, −10.7%, −7.6%, −4.7% and −4.1% from 0.5B to 32B (Table 13).

Difficulty-matched controls on the same instances (NgwY, GAdZ; new §6.1, Table 5, Appendix H). Three further item types separate the output form from the content. Code twins pose each judge question as writing a Python function answer(), whose return value is executed and graded with the judge item's own grader and tolerance, the CRUXEval construction. Consequence-dependent do items ask for an artefact that is correct only if the consequence is derived: the index migration if it is needed at a stated future table size, the handler that offloads the step whose removal most raises capacity, and the concurrency limit for a stated load. Single-step judge items ask for one stage's demand, v×sv\times s, and for its throughput alone. The runs cover Qwen2.5-Coder at 1.5B, 3B, 7B, 14B and 32B and DeepSeek-Coder at 1.3B and 6.7B, with float16 activations and AWQ int4 weights at 14B and 32B; every comparison is within a run.

At Qwen2.5-Coder-32B the specified change and the single computation are solved, while every output that needs the capacity consequence stays near chance. The do items that specify the change score +0.859 and one stage's demand +0.973. The do items that need the consequence score +0.008, the code twins of the quantitative judge types −0.046, the stated answers −0.058, and that stage's throughput, which needs the relation X=1/DX=1/D, +0.068. The same separation holds at every scale: stage demand rises from +0.324 at 1.5B to +0.973 at 32B, while no consequence column exceeds +0.008 in any of the five Qwen runs, and DeepSeek-Coder at 1.3B and 6.7B shows the same pattern (Appendix H). Performance therefore tracks whether the task requires deriving the capacity consequence, regardless of output form.

Continued in part 2/2 below.

AuthorsOpenReview ↗

Summary of revision (part 2)

(2/2)

Diagnosing the two zero-score cells (GAdZ; §4, Appendices D and F). For async conversion, the submitted item named the hand-off call without listing the context methods that the handler item lists. At 32B, 58 of 80 outputs call context methods that do not exist, against 0 of 80 on the handler item. The revised item states the same API as the handler item, and with it the 32B run writes the conversion correctly on 76 of 80 instances (95%) with no call to a non-existent method, against 51 such outputs of 80 on the submitted item in the same run. At 14B the stated API cuts such outputs from 58 to 0, and 67 of 80 conversions are correct. For the offloading judge, the zero is the sum of two opposite behaviours. At 32B the model answers the offloading variant correctly on 80% of the instances where offloading raises capacity and 60% of those where it does not, and it answers yes to deferring work to a queue on the same machine, which never raises capacity, on 80% of instances (Table 14).

Clarity (AdD6, NgwY). Table 1 is a benchmark specification table that lists the 11 item types with their answer forms, graders and chance baselines. It gives the counts of every set: 10,733 released items from 1,400 instances, 880 evaluated items from 657 instances, and 976 paired items forming 488 pairs on 170 instances. Figure 2 is a worked example that shows one instance, its do item, its judge item and the derivation of the label. Figure 3 shows the main results at all six scales, on the same runs as Tables 4 and 9. Appendix C gives the exact prompting protocol, including the precision of every run, and Appendix D gives every grader.

Code and data (AdD6, NgwY). The supplementary archive contains the generator and all item sets including the control items, the grading harness and its execution sandbox, the generations and per-item scores of every reported run, the running-service rig with all measurements, and the scripts behind every table. The harness reproduces the per-item scores of the submitted runs exactly: 880 of 880 items at each of the six Qwen2.5-Coder scales.

A second family at three scales (all reviewers; §7.1, Table 7). DeepSeek-Coder at 1.3B, 6.7B and 33B is scored with the same items, graders and baselines. Its do scores are +0.303, +0.381 and +0.381, and its do–judge gap under the oracle answer locator is +0.141, +0.335 and +0.308. The separation is positive at every scale of both families, and DeepSeek-Coder at 1.3B and 6.7B shows the same pattern as Qwen in the control runs (specified change +0.294 and +0.703, do items that need the consequence −0.203 and −0.087; Appendix H).

We are glad to answer any further questions during the discussion.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.