acceptodds
Under review as a conference paper at ICLR 2027

LoopArena: Benchmarking Models as Runtime Controllers for Loop Engineering

Abstract

Loop Engineering is emerging as a practice for organizing development work around coding agents. Instead of writing each prompt by hand, practitioners design loops that monitor progress, assign work, run checks, and decide what the agent should do next. Even with a capable coding agent, a loop may trust a stale progress note, skip needed verification, spend its budget in the wrong direction, or stop before the task is safe to submit. Yet the final outcome of one end-to-end run cannot tell whether success or failure reflects the loop's guidance or the coding agent's ability to carry out the task. We introduce LoopArena, a benchmark for evaluating how well one model can guide a separate coding agent through a long-running task. The model under evaluation is the Controller: after each coding round, it receives a structured summary of the run and instructs a separate, fixed coding agent, the Worker, on what to do or verify next, or decides whether to stop. LoopArena evaluates this ability in three complementary settings that differ in execution scope and cost. Type I consists of 160 low-cost control questions; its execution cost is incurred once during benchmark construction, so each new Controller is evaluated through a single four-way choice with no code execution. Type II evaluates Controller guidance on 80 selected task slices, while Type III evaluates the corresponding 80 full tasks from their original states. On full tasks, the best observed Strict Success Rate is 36.25%, leaving substantial room for improvement in long-horizon loop control. Across Controllers, the paired reduction in estimated inference cost averages 60.6%, and Type II produces a positively correlated ordering under the main Core criterion (Spearman's ρ = 0.7353). We release the benchmark data and evaluation code at https://anonymous.4open.science/r/LoopArena.

Primary area:
datasets and benchmarks
Keywords:
LLM agents, coding agents, agent evaluation, agent orchestration, long-horizon agent control, software engineering benchmarks, loop engineering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_752UOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LoopArena evaluates language models as runtime controllers that guide a separate, fixed coding worker. A reporter summarizes the worker's progress into an Evidence Packet, and the controller issues instructions to advance, verify, or stop the task. The benchmark comprises 160 execution-validated contract-selection questions (Type I), 80 task slices (Type II), and their corresponding 80 full tasks (Type III), drawn from three existing coding benchmarks. The authors evaluate six controllers, include no-control and fixed-control baselines, and run three repeats per executable task. The best full-task strict success rate is 36.25%. Type II reduces estimated inference cost by 60.6% on average and has a Spearman ranking correlation of 0.7353 with Type III. A smaller experiment examines a second worker configuration.

Critical strengths
  1. A useful and clearly scoped evaluation target. Holding the coding worker and execution interface fixed enables comparisons of models in a supervisory role: interpreting progress, assigning subsequent work, requesting verification, and deciding when to stop. This is a meaningful evaluation setting for agent orchestration and gives the benchmark potential practical value.

  2. A substantive effort to connect evaluation cost with execution-based evidence. The three settings provide complementary levels of evaluation, and the paired slice/full-task construction enables direct cost comparisons. Freezing Type I candidates before replay, requiring agreement across two replay schedules, including a fixed-control baseline, and reporting repeated executable runs strengthen the methodology. The detailed prompts and component-level cost accounting also make the protocol easier to assess.

Critical weaknesses
  1. The reliability and interpretability of Type I labels need stronger validation. In Appendix B.1, the winning contract depends on downstream success under a selected continuation controller, with control cycles and worker turns breaking ties. Agreement across two replay schedules is useful, but the paper does not quantify how often labels depend on these tie-breakers or whether the winning choice is reliably identifiable from the public evidence. Table 9 is particularly striking: every model scores only 26.87%–34.33% on LoopsBench questions, while GPT-5.5 scores 87.50% on SCBench. These results could reflect genuine difficulty, but the reported audits do not sufficiently distinguish that explanation from incomplete decision context or unstable execution-based labels. This matters because interpretable, reliable labels are central to the value of the low-cost decision benchmark.

  2. The reliability of the executable model comparisons remains insufficiently established. The reported Type II–III correlation of 0.7353 is based on only six controllers with tied scores, and the best Type III result exceeds fixed control by 5 percentage points. Three repeats and the stability counts are helpful, but paired uncertainty estimates for these comparisons are missing. In addition, Appendix F.4 reports frequent controller output-budget exhaustion for GLM, which raises a question about how much its low rank reflects instruction-generation failures under this interface. Although these are valid failures under the stated protocol, their contribution to the rankings should be quantified. These uncertainties limit how confidently the benchmark can distinguish control ability and support lower-cost controller selection.

Questions for authors
  1. Clarification of Type I construction and audits: What fraction of retained questions have exactly one successful candidate, and what fraction require cycle/turn tie-breaking? What did the post-freeze audits find about whether the public Evidence Packets support the labels, particularly for the LoopsBench subset? If independent replay checks beyond the two construction schedules were performed, please report their label agreement.

  2. Validation using existing executable results: Could you report confidence intervals for the controller-versus-fixed-control success-rate differences and the Type II–III ranking correlation, using resampling that preserves task pairing and accounts for the three repeats within each task? These intervals would help determine whether the reported differences support dependable controller comparisons.

  3. Clarification of protocol failures: How many task runs fail due to controller output-budget exhaustion or invalid contracts, particularly for GLM? Please explain how the 271 Type III exhaustion events map to checkpoint episodes and the 240 task runs, including whether execution continues across checkpoints after such a failure. This would help assess the contribution of interface failures to the reported rankings.

4 comments

Thank you for the suggestions, which helped strengthen the paper. Line numbers below refer to the revised PDF. We added audit findings (lines 881–893), paired CIs (1381–1403), and protocol-failure statistics (1273–1306). Changes are marked in blue.

Public evidence supports the Type I labels (W1/Q1)

Our audits confirmed that the public Evidence Packets supported the reference choices for all 160 retained questions, including the 67 LoopsBench items. These audits took place during benchmark construction, after the reference answers were frozen. For the LoopsBench subset highlighted in Table 12 (26.87%–34.33% accuracy), the following examples show the public clues that distinguish the reference choice from the alternatives.

  • In type1_0142, the packet says that one set of tests has passed and a second test file is still unchecked. C asks the Worker to inspect that file. The other options stop. C succeeds in both replays, while the other options fail in both.

  • In type1_0139, all four options require running tests. C alone also requires running the public functional verifier provided in the task description. B, C, and D all succeed in both replays, but C needs only 2 and 3 subsequent control cycles. B and D need 4–5 each.

The reference answer comes from replay results. A question enters the benchmark only when two matched replay schedules select the same unique winner. The table reports how the winner was determined in each replay, covering all 160 retained questions.

Deciding criterionReplay 1Replay 2
Only one option succeeds43 (26.875%)41 (25.625%)
Multiple options succeed; decided by Controller cycles64 (40%)86 (53.75%)
Controller cycles tied; decided by Worker turns53 (33.125%)33 (20.625%)

Paired uncertainty estimates for model comparisons (W2/Q2)

Results from all eight models, including Kimi K3 and GPT-5.6 Luna, support a positive association between Type II and Type III performance (Spearman correlation 0.7854; 95% CI: [0.3473, 0.9572]). We computed 95% percentile confidence intervals from 10,000 paired bootstrap samples. In each sample, we drew 80 tasks with replacement, stratified by source, and kept all three runs for every model and fixed control in both settings.

Fixed control provides a common non-adaptive reference for comparing models as Controllers. In Type III, five of the eight models (Qwen3.6, Qwen3.7, GPT-5.5, Opus-4.8, and Kimi K3) show positive differences in 80.67%–94.67% of the 10,000 paired bootstrap samples. Although their 95% confidence intervals include zero, most resampled differences lie above zero.

ΔSSR = Controller − fixed control. All differences and interval endpoints are in percentage points. Each setting uses 80 tasks × 3 executions per condition. These are pointwise 95% intervals.

ModelType II [95% CI]Type III [95% CI]
Qwen3.6+0.83 [-4.17, 5.83]+2.50 [-1.25, 6.67]
Qwen3.7+1.67 [-3.33, 6.67]+2.92 [-0.42, 6.25]
DeepSeek-V4-4.17 [-10.42, 1.67]-2.92 [-8.33, 2.50]
GLM-5.2-14.58 [-22.50, -6.67]-7.08 [-11.67, -2.92]
GPT-5.5+1.67 [-5.83, 8.75]+5.00 [-1.67, 11.68]
Opus-4.8+3.75 [-2.92, 10.42]+2.50 [-2.50, 7.92]
Kimi K3+1.67 [-4.58, 7.50]+3.33 [-3.75, 10.42]
GPT-5.6 Luna-2.08 [-7.92, 3.75]-0.42 [-7.08, 6.25]

Protocol failures and GLM's task outcomes (W2/Q3)

Producing an executable instruction within the output budget is part of Controller performance, so runs with protocol failures are retained in the reported success rates. Protocol failures affected 96 Type II and 114 Type III GLM runs; all of these runs failed.

Each setting has 240 runs per model. The table reports runs with output-budget exhaustion / runs with invalid Contracts / distinct runs with either failure.

ModelType II failuresType III failures
Qwen3.60 / 4 / 40 / 4 / 4
Qwen3.70 / 4 / 40 / 7 / 7
DeepSeek-V42 / 19 / 2118 / 39 / 49
GLM-5.290 / 6 / 96111 / 18 / 114
GPT-5.50 / 0 / 00 / 0 / 0
Opus-4.80 / 0 / 00 / 3 / 3
Kimi K327 / 0 / 2737 / 3 / 40
GPT-5.6 Luna0 / 0 / 00 / 6 / 6

Appendix F.4 reports 271 GLM requests that reached the 20,480-token cap. They occurred at 271 checkpoints across 111 of the 240 Type III runs. An output-limit hit ends the current task checkpoint. In SCBench’s multi-checkpoint tasks, execution can continue at the next checkpoint from the inherited code state, while the earlier failure remains recorded. We count one outcome per full run.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer 752U,

Thank you again for your time and thoughtful review. We would be grateful if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Thank you for your consideration.

Thank you for the detailed response. The added audit findings, paired confidence intervals, and task-level protocol-failure statistics directly address my questions. In particular, the distinction between checkpoint-level failures and full-task outcomes resolves my concern about the 271 GLM output-limit events.

The expanded eight-model comparison and paired bootstrap analysis strengthen the evidence for a positive association between Type II and Type III performance. The interval remains fairly broad, so I would retain a cautious interpretation of how reliably Type II reproduces fine-grained controller rankings. I also appreciate the explicit acknowledgment that the positive success-rate differences over fixed control have confidence intervals containing zero; these results support a more qualified conclusion about the benefits of adaptive control.

The Type I examples and criterion breakdown are helpful. They clarify that most questions distinguish execution efficiency among successful alternatives. One remaining clarification would be useful: who performed the audits of the 160 questions, and were the frozen reference answers visible during those judgments? This would help assess how independently the audits establish that the public evidence supports the selected answers. Additional replay testing would be useful future validation of efficiency-based labels, but I do not consider it a prerequisite for acceptance.

These additions strengthen my assessment of the paper. I maintain my recommendation of 3: Weak acceptance. The remaining limitations primarily concern the interpretation and generalization of the scores, while the benchmark provides a useful and substantive contribution to evaluating runtime controller models.

Dear Reviewer 752U,

Thank you for your careful follow-up and supportive assessment.

Two experts in programming conducted the human audits of all 160 Type I questions. The frozen reference answers were withheld from both experts throughout their assessments. They reviewed the public Evidence Packets and the four candidate Contracts to assess which option was best supported by the available evidence, recording their reasoning and any ambiguity or missing information.

Thank you again for your constructive feedback and careful consideration of our work.

Reviewer_ZhBrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper introduces LoopArena, a benchmark for evaluating large language models as runtime controllers of long-horizon coding agents. Unlike conventional coding benchmarks that evaluate the overall agent system, LoopArena focuses on a model's ability to supervise a separate coding agent (the Worker), determine its next assignment, request verification, and decide when to terminate execution. The framework uses a Reporter to summarize the Worker's progress into structured Evidence Packets, which the Controller uses to generate Loop Contracts specifying subsequent actions.

LoopArena provides three complementary evaluation settings: Type I consists of 160 multiple-choice questions whose correct answers are determined through executable replays; Type II evaluates runtime control on 80 condensed coding tasks; and Type III evaluates the corresponding 80 full-length tasks. The benchmark draws from three existing coding benchmarks and compares six Controller models while keeping the Worker and execution environment fixed.

Experimental results show that the best Controller achieves a 36.25% Strict Success Rate on full tasks. Type II reduces estimated inference costs by an average of 60.6% relative to Type III while producing a positively correlated Controller ranking (Spearman's ρ = 0.7353). The paper aims to establish a systematic and cost-efficient framework for evaluating runtime control capabilities independently of the coding agent's implementation ability.

Critical strengths
  1. This paper focuses on loop engineering within the context of coding agents, which is the capability to guide another agent through the processes of iterative development, verification, and termination. This is an aspect that cannot be directly measured by traditional coding benchmarks. Investigating how a model can function as a controller to direct an agent in completing tasks is a meaningful endeavor, and one that needs independent evaluation. To this end, the authors propose a three-tier evaluation framework that scales in terms of both granularity and cost, acheiving an effective balance between the two. Specifically, Type I evaluations utilize downstream execution results (rather than relying solely on subjective judgment) to determine ground-truth answers, while Type II evaluations offer a more cost-effective, executable environment for assessing complete tasks, which makes evaluation more flexible for different scenarios and budgets.

  2. The experimental design is quite comprehensive and provides detailed information regarding reproducibility. This paper evaluates six controller models against two baselines (no control and fixed control), and reports both task success rates and inference costs. Furthermore, it offers an in-depth analysis of the behavior of specific models and the configurations of the workers, which would be helpful when using the proposed benchmark to evaluate other models in the future.

Critical weaknesses
  1. This paper retains 160 Type I questions, but it doesn't report how many candidate questions were generated in the beginning, or the proportion that met the replay-based acceptance criteria. Given that questions are rejected if 1) all four candidates fail the task, 2) a schedule has no unique winner under the ordering, or 3) if the two replay schedules yield inconsistent results, the actual yield rate of the construction process may be significantly lower than what the final dataset size suggests. Please provide additional details regarding the total number of questions attempted, the overall acceptance rate, and the specific distribution of the various reasons for rejection. Such statistics are crucial for evaluating the construction costs and potential selection bias, as the retained problems might be disproportionately skewed toward control decisions that yield stable outcomes and are easily distinguishable.

  2. The validity of both Type I and Type II requires more thorough verification:

    (1) The reference answers for Type I rely on replay results. The labels are influenced by stochastic execution, the specific subsequent controller selected, and the replay seed. When multiple candidates succeed, the result is determined by downstream control cycles and worker turns, which reflects subtle differences in execution efficiency, rather than necessarily the quality of the decision-making itself. Furthermore, model rankings differ between Type I and Type III (e.g., Qwen3.6-35B-A3B ranks lowest in Type I but ties for third in Type III), leaving it unclear whether Type I can effectively predict performance on complete tasks.

    (2) Type II begins from prepared intermediate workspaces, where states derived from source task checkpoints or reference implementations, rather than states generated during the controller’s actual runtime. These prepared states may differ significantly from actual runtime states in terms of task difficulty and the control decisions required. Consequently, a positive correlation in overall rankings across the six models (ρ = 0.7353) is insufficient to conclude that Type II and Type III measure the same type of control capability. We recommend evaluating rank consistency separately based on task origin and development stage, incorporating a broader range of controllers (ideally including models not previously assessed in this study), and clarifying the extent to which these prepared intermediate states affect the representativeness of Type II.

Questions for authors
  1. Are the retained questions skewed toward control decisions with stable, easily distinguishable outcomes? For example, among retained questions, what fraction had only one successful candidate, and what fraction had a winner decided by downstream control cycles or Worker turns?
  2. When several candidates succeed, why are downstream control cycles and Worker turns an appropriate measure of decision quality rather than of execution efficiency?
  3. How are the rejections distributed across the three reasons: (1) all four candidates fail, (2) a schedule has no unique winner, and (3) the two replay schedules disagree?
  4. How sensitive are the Type I labels to stochastic execution, the choice of continuation controller, and the replay seeds?
  5. Does the agreement hold across a broader set of controllers, including models not evaluated in the paper?
4 comments

Thank you for the review. Line numbers below refer to the revised PDF. We added construction and audit evidence (831–893) and model comparisons (1347–1378). Changes are marked in blue.

Type I construction yield and exclusions (W1/Q3)

During construction, we generated 1,002 distinct candidate questions and retained 160 (15.97%). The table accounts for all candidates.

OutcomeQuestions% of 1,002 candidates
All four options failed212.10%
Execution incomplete or results unusable30029.94%
Excluded during candidate quality screening35635.53%
No unique winner383.79%
Different winners across the two replays12712.67%
Retained16015.97%

All three trajectory stages remain represented, with retention rates ranging from 13.88% to 18.45%:

StageCandidatesRetainedRetention rate
Early374 (37.33%)69 (43.13%)18.45%
Middle275 (27.45%)42 (26.25%)15.27%
Late353 (35.23%)49 (30.63%)13.88%

Replay costs are incurred during construction. The frozen answers are reusable: each new model makes one choice per question, with no Worker execution.

Why efficiency informs control quality (W2/Q1/Q2)

Control quality includes reaching the same verified goal with less follow-up supervision and execution work. A Contract sets both the next action and its completion checks, affecting how much further intervention the Worker needs. We prioritize task success; when several options succeed under matched replay conditions, we compare subsequent Controller cycles, using Worker turns to break any remaining tie.

In type1_0139, B, C, and D succeed in both replays. C is the only option that requires the public functional verifier in addition to the regular tests. C takes 2 and 3 control cycles, B takes 5 and 4, and D takes 5 in both replays.

For all 160 retained questions:

Deciding criterionReplay 1Replay 2
Only one option succeeds43 (26.875%)41 (25.625%)
Multiple options succeed; decided by Controller cycles64 (40%)86 (53.75%)
Controller cycles tied; decided by Worker turns53 (33.125%)33 (20.625%)

Even on these 160 replay-consistent questions, the highest-scoring model (GPT-5.5) answered only 99/160 correctly (61.875%), showing that agreement on the winner across two replays does not make the correct choice obvious from the packet.

Replays and audits support Type I labels (W2/Q4)

Two checks support the Type I labels: the replays agree on the winner, and audits confirm that the public Evidence Packets support the reference choices. All four options in each replay share the same checkpoint, randomly selected continuation Controller, Worker, budget, and seed. We retain a question only when two replay schedules with different seeds select the same unique winner. During construction, audits covered all 160 retained questions after the reference answers were frozen.

Type I evaluates individual control decisions (W2)

Type I evaluates individual control decisions; Type III tests control over an entire task, so their rankings need not be identical (lines 1111–1113). Type I asks models to select a Contract from fixed alternatives. Type III also requires them to write instructions, handle the consequences of earlier decisions, and manage the full-task budget. Qwen3.6, for example, ranks last in Type I and ties for fourth among the eight models in Type III.

Comparing control from a common starting point (W2)

Type II gives every model the same code and progress, allowing a fair comparison of stage-level control. Each task is a development stage from its paired Type III task and still requires the model to guide, verify, and correct the work. Type III also tests how models reach and handle the intermediate states they create. Across eight models on the 80 task pairs, the Spearman correlation is 0.7854, with positive correlations within all three task sources. Appendix B describes the shared slices, which balance early, middle, and late stages within each source. Rank correlations are also positive in most development-stage groups.

Agreement extends to additional models (W2/Q5)

Adding Kimi K3 and GPT-5.6 Luna raises the Spearman correlation from 0.7353 to 0.7854, further supporting Type II’s ability to track full-task control performance. Both completed 160 Type I questions and three runs per task on all 80 Type II/III task pairs. Kimi ranks joint second in Type II and second in Type III; Luna is sixth in both.

ModelType I acc.Type II SSRType III SSR
Kimi K361.25%56.67% (136/240)34.58% (83/240)
GPT-5.6 Luna47.50%52.92% (127/240)30.83% (74/240)
AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer ZhBr,

Thank you again for your time and thoughtful review. We would be grateful if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Thank you for your consideration.

Thanks for your response! These results make the claims more solid, so I raise my soundness score from 2 to 3.

Dear Reviewer ZhBr,

Thank you very much for following up and updating your assessment. We are glad the added results strengthened our claims, and we appreciate your constructive feedback.

Best regards, The Authors

Reviewer_wEwrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces LoopArena, a benchmark that evaluates whether language models can supervise coding agents during long-horizon software engineering tasks. The design separates runtime control from code execution. The coding Worker is held fixed, and different Controller models are compared against it. A Reporter summarizes execution progress into Evidence Packets. Based on these packets, the Controller chooses one of three actions: continue implementation, request verification, or stop. The benchmark consists of three evaluation settings. Type I evaluates individual control decisions through 160 execution-validated contract-selection questions. Type II evaluates continuous control on 80 task slices. Type III evaluates the corresponding 80 full tasks. Experiments with six Controller models show that the best full-task Strict Success Rate reaches 36.25%. Type II reduces estimated inference cost by 60.6% on average relative to Type III, while maintaining a positive ranking correlation with Type III (Spearman's ρ=0.7353).

Critical strengths

1.The paper addresses an evaluation gap in coding-agent research. Most existing benchmarks measure only the final performance of a coding agent or the full agent system. This makes it hard to separate execution ability from runtime supervision. LoopArena keeps the Worker and the execution environment fixed. It therefore offers a more focused way to compare Controller models. This evaluation perspective is meaningful, especially as coding agents are increasingly used for long-running development tasks. 2.The multi-level evaluation design is practical and well motivated. I especially value the way Type I labels are grounded in actual downstream execution. This avoids relying entirely on model-based judgments. The paired Type II and Type III tasks also provide a useful setting. They support the study of the trade-off between evaluation cost and fidelity. The detailed execution protocols and the additional analyses in the appendix help readers understand the benchmark.

Critical weaknesses

1.The representativeness of the Type I benchmark needs further discussion. In Section 3.3 and Appendix B.1, the authors keep only questions for which the two matched replay schedules identify the same unique winning Contract. This procedure improves label consistency. However, it may also favor decision points with relatively clear outcomes, while excluding more ambiguous control situations. For this reason, it is unclear how well the retained questions reflect the diversity and difficulty of decisions in real agent supervision. A clearer discussion of the filtering process and its potential influence on the benchmark would strengthen this part of the paper. 2.The relationship between Type II and Type III performance needs further interpretation. The reported Spearman correlation of 0.7353 shows positive agreement between Controller rankings. However, this result is based on only six models. Type II starts from prepared intermediate workspaces, and earlier development stages are already completed there. This differs from Type III. In Type III, the Controller must manage the entire process. This process also includes potential errors accumulated during earlier stages. The results support Type II as a useful lower-cost evaluation setting. Still, the paper could discuss more clearly how well performance on condensed tasks reflects full-task control ability.

Questions for authors

1.Could the authors clarify how the Type I filtering procedure affects the composition of the benchmark? It would be helpful to know how often candidate questions were excluded. The reasons include inconsistent or non-unique winners. It would also help to know whether this filtering favors certain types of control decisions. 2.Could the authors further discuss the differences between Type II and Type III Controller rankings? The number of evaluated models is relatively small. An explanation of the observed ranking disagreements would be useful. The possible effects of task-slice selection should also be discussed. These points would help readers understand the scope and limitations of Type II as a lower-cost evaluation. 3.Could the authors clarify the trade-off between control effectiveness and execution cost? Tables 2 and 11 show that some Controller-guided configurations achieve higher success rates than fixed control. They also require more Worker turns and greater inference costs. A discussion based on the existing results would help explain when adaptive guidance is beneficial. It would also show how its additional execution cost should be interpreted.

4 comments

Thank you for the helpful suggestions. Line numbers below refer to the revised PDF. We added construction statistics (831–854), model comparisons (1347–1378), and cost analysis (1218–1229), marked in blue.

Type I difficulty after replay filtering (W1/Q1)

Agreement on the winner across two replays does not make the correct Contract obvious from the public Evidence Packet. Both replays must identify the same unique winner; model accuracy is not a selection criterion. GPT-5.5, the highest-scoring model, answered only 99 of the 160 retained questions correctly (61.875%). This substantial headroom shows that the retained, execution-grounded subset remains challenging for the evaluated models.

During construction, we generated 1,002 distinct candidate questions and retained 160 (15.97%).

OutcomeQuestions% of 1,002
All four options failed212.10%
Execution incomplete or results unusable30029.94%
Excluded during candidate quality screening35635.53%
No unique winner383.79%
Different winners across the two replays12712.67%
Retained16015.97%

All three trajectory stages remain represented, with retention rates ranging from 13.88% to 18.45%:

StageCandidatesRetainedRetention rate
Early37469 (43.13%)18.45%
Middle27542 (26.25%)15.27%
Late35349 (30.63%)13.88%

Together, the coverage across trajectory stages and the substantial remaining performance headroom support Type I’s value as a challenging assessment of individual control decisions.

Type II links stage-level and full-task performance (W2/Q2)

Type II tests control capabilities needed for the full task while giving every model the same starting state for a fair comparison. Each slice comes from its paired Type III task and retains its code dependencies, cumulative checks, and control loop. Models must still decide how to proceed, what to verify or repair, and when to stop. Type III additionally tests how they reach intermediate states, handle the consequences of earlier decisions, and manage the full-task budget.

We added Kimi K3 and GPT-5.6 Luna, each completing 160 Type I questions and three runs per task on all 80 Type II/III task pairs:

ModelType I acc.Type II SSRType III SSR
Kimi K361.25%56.67% (136/240)34.58% (83/240)
GPT-5.6 Luna47.50%52.92% (127/240)30.83% (74/240)

Across all eight models, the Spearman correlation is 0.7854, with agreement on 21 of 24 model pairs after excluding tied comparisons. Kimi ranks joint second in Type II and second in Type III; Luna ranks sixth in both.

All three reversals involve Opus-4.8: it leads GPT-5.5, Qwen3.7, and Kimi in Type II but trails them in Type III. For the two reversals in the original six-model table, LoopsBench contributed the largest shift: Opus's lead over GPT-5.5 changed from six successful runs to a deficit of three; against Qwen3.7, it changed from a lead of three to a deficit of eight.

Across all eight models, Type II reduced estimated inference cost by 62.0% on average. As described in Appendix B, we fixed one slice per task before evaluation, covering early, middle, and late stages within each source. This samples control at different points in development while giving every model the same starting states.

Interpreting success and execution cost (Q3)

All models share tasks, Worker, and budget limits; actual execution varies by control policy. Fixed control always continues. Adaptive Controllers use execution evidence to decide when to implement, repair, verify, or stop.

Execution and cost. Adaptive control does not necessarily add work or cost. In Type II, Qwen3.6 used fewer Worker turns on average (103.37 vs. 105.82 per run) and cost less (USD 4.21 vs. USD 5.62 per run) than fixed control, with a slightly higher observed success rate (55.83% vs. 55.00%).

In Type III, GPT-5.5 achieved a success rate of 36.25%, averaging 355.94 Worker turns and USD 34.68 per run, compared with 31.25%, 211.14 turns, and USD 9.47 for fixed control. The USD 25.21 increase per run consists of USD 16.95 for Worker execution, USD 6.18 for Controller calls, and USD 2.08 for Reporter calls. This gain was concentrated on SCBench’s full tasks with cumulative checks: GPT-5.5 succeeded in 16/63 runs versus 3/63 for fixed control (Table 19).

Model prices also matter. In Type III, Qwen3.6 and Opus-4.8 both reached 33.75% success at estimated costs of USD 11.58 and USD 20.63 per run, respectively. Controller calls accounted for USD 6.96 of the USD 9.05 gap (USD 0.20 vs. USD 7.16). Across 240 runs, Opus’s Controller calls used 320.5M input and 4.6M output tokens, versus Qwen3.6's 143.0M and 6.1M. Opus's input and output token prices were 18.9 and 15.7 times Qwen3.6's, respectively (Table 13). Tables 3 and 14 report full-task outcomes and inference costs by role.

The appropriate choice depends on the application’s success requirements and cost constraints.

AuthorsOpenReview ↗

Follow-up on our response

Dear Reviewer wEwr,

Thank you again for your time and thoughtful review. We would be grateful if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.

Thank you for your consideration.

AuthorsOpenReview ↗

a polite follow up

Esteemed Reviewer wEwr,

As discussion period finishes really soon, please kindly give us a chance by checking our rebuttal and sharing any further questions you may have before discussions are cut off and we lose the access to the response button.

Kind regards, Authors

AuthorsOpenReview ↗

Thank you.

Esteemed Reviewer wEwr,

We would like to thank you again for helping us improve our work. While discussion period ends, rest assured we will incorporate rebuttal, your requests and act on any further feedback you may pass on us in your final comments here or in final justification box.

Warmest regards, Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.