- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LoopArena evaluates language models as runtime controllers that guide a separate, fixed coding worker. A reporter summarizes the worker's progress into an Evidence Packet, and the controller issues instructions to advance, verify, or stop the task. The benchmark comprises 160 execution-validated contract-selection questions (Type I), 80 task slices (Type II), and their corresponding 80 full tasks (Type III), drawn from three existing coding benchmarks. The authors evaluate six controllers, include no-control and fixed-control baselines, and run three repeats per executable task. The best full-task strict success rate is 36.25%. Type II reduces estimated inference cost by 60.6% on average and has a Spearman ranking correlation of 0.7353 with Type III. A smaller experiment examines a second worker configuration.
Critical strengths
-
A useful and clearly scoped evaluation target. Holding the coding worker and execution interface fixed enables comparisons of models in a supervisory role: interpreting progress, assigning subsequent work, requesting verification, and deciding when to stop. This is a meaningful evaluation setting for agent orchestration and gives the benchmark potential practical value.
-
A substantive effort to connect evaluation cost with execution-based evidence. The three settings provide complementary levels of evaluation, and the paired slice/full-task construction enables direct cost comparisons. Freezing Type I candidates before replay, requiring agreement across two replay schedules, including a fixed-control baseline, and reporting repeated executable runs strengthen the methodology. The detailed prompts and component-level cost accounting also make the protocol easier to assess.
Critical weaknesses
-
The reliability and interpretability of Type I labels need stronger validation. In Appendix B.1, the winning contract depends on downstream success under a selected continuation controller, with control cycles and worker turns breaking ties. Agreement across two replay schedules is useful, but the paper does not quantify how often labels depend on these tie-breakers or whether the winning choice is reliably identifiable from the public evidence. Table 9 is particularly striking: every model scores only 26.87%–34.33% on LoopsBench questions, while GPT-5.5 scores 87.50% on SCBench. These results could reflect genuine difficulty, but the reported audits do not sufficiently distinguish that explanation from incomplete decision context or unstable execution-based labels. This matters because interpretable, reliable labels are central to the value of the low-cost decision benchmark.
-
The reliability of the executable model comparisons remains insufficiently established. The reported Type II–III correlation of 0.7353 is based on only six controllers with tied scores, and the best Type III result exceeds fixed control by 5 percentage points. Three repeats and the stability counts are helpful, but paired uncertainty estimates for these comparisons are missing. In addition, Appendix F.4 reports frequent controller output-budget exhaustion for GLM, which raises a question about how much its low rank reflects instruction-generation failures under this interface. Although these are valid failures under the stated protocol, their contribution to the rankings should be quantified. These uncertainties limit how confidently the benchmark can distinguish control ability and support lower-cost controller selection.
Questions for authors
-
Clarification of Type I construction and audits: What fraction of retained questions have exactly one successful candidate, and what fraction require cycle/turn tie-breaking? What did the post-freeze audits find about whether the public Evidence Packets support the labels, particularly for the LoopsBench subset? If independent replay checks beyond the two construction schedules were performed, please report their label agreement.
-
Validation using existing executable results: Could you report confidence intervals for the controller-versus-fixed-control success-rate differences and the Type II–III ranking correlation, using resampling that preserves task pairing and accounts for the three repeats within each task? These intervals would help determine whether the reported differences support dependable controller comparisons.
-
Clarification of protocol failures: How many task runs fail due to controller output-budget exhaustion or invalid contracts, particularly for GLM? Please explain how the 271 Type III exhaustion events map to checkpoint episodes and the 240 task runs, including whether execution continues across checkpoints after such a failure. This would help assess the contribution of interface failures to the reported rankings.
4 comments
Follow-up on our response
Dear Reviewer 752U,
Thank you again for your time and thoughtful review. We would be grateful if you could take a look at our response and let us know whether it addresses your concerns. We would be happy to clarify any remaining questions.
Thank you for your consideration.
Thank you for the detailed response. The added audit findings, paired confidence intervals, and task-level protocol-failure statistics directly address my questions. In particular, the distinction between checkpoint-level failures and full-task outcomes resolves my concern about the 271 GLM output-limit events.
The expanded eight-model comparison and paired bootstrap analysis strengthen the evidence for a positive association between Type II and Type III performance. The interval remains fairly broad, so I would retain a cautious interpretation of how reliably Type II reproduces fine-grained controller rankings. I also appreciate the explicit acknowledgment that the positive success-rate differences over fixed control have confidence intervals containing zero; these results support a more qualified conclusion about the benefits of adaptive control.
The Type I examples and criterion breakdown are helpful. They clarify that most questions distinguish execution efficiency among successful alternatives. One remaining clarification would be useful: who performed the audits of the 160 questions, and were the frozen reference answers visible during those judgments? This would help assess how independently the audits establish that the public evidence supports the selected answers. Additional replay testing would be useful future validation of efficiency-based labels, but I do not consider it a prerequisite for acceptance.
These additions strengthen my assessment of the paper. I maintain my recommendation of 3: Weak acceptance. The remaining limitations primarily concern the interpretation and generalization of the scores, while the benchmark provides a useful and substantive contribution to evaluating runtime controller models.
Dear Reviewer 752U,
Thank you for your careful follow-up and supportive assessment.
Two experts in programming conducted the human audits of all 160 Type I questions. The frozen reference answers were withheld from both experts throughout their assessments. They reviewed the public Evidence Packets and the four candidate Contracts to assess which option was best supported by the available evidence, recording their reasoning and any ambiguity or missing information.
Thank you again for your constructive feedback and careful consideration of our work.
Thank you for the suggestions, which helped strengthen the paper. Line numbers below refer to the revised PDF. We added audit findings (lines 881–893), paired CIs (1381–1403), and protocol-failure statistics (1273–1306). Changes are marked in blue.
Public evidence supports the Type I labels (W1/Q1)
Our audits confirmed that the public Evidence Packets supported the reference choices for all 160 retained questions, including the 67 LoopsBench items. These audits took place during benchmark construction, after the reference answers were frozen. For the LoopsBench subset highlighted in Table 12 (26.87%–34.33% accuracy), the following examples show the public clues that distinguish the reference choice from the alternatives.
In
type1_0142, the packet says that one set of tests has passed and a second test file is still unchecked. C asks the Worker to inspect that file. The other options stop. C succeeds in both replays, while the other options fail in both.In
type1_0139, all four options require running tests. C alone also requires running the public functional verifier provided in the task description. B, C, and D all succeed in both replays, but C needs only 2 and 3 subsequent control cycles. B and D need 4–5 each.The reference answer comes from replay results. A question enters the benchmark only when two matched replay schedules select the same unique winner. The table reports how the winner was determined in each replay, covering all 160 retained questions.
Paired uncertainty estimates for model comparisons (W2/Q2)
Results from all eight models, including Kimi K3 and GPT-5.6 Luna, support a positive association between Type II and Type III performance (Spearman correlation 0.7854; 95% CI: [0.3473, 0.9572]). We computed 95% percentile confidence intervals from 10,000 paired bootstrap samples. In each sample, we drew 80 tasks with replacement, stratified by source, and kept all three runs for every model and fixed control in both settings.
Fixed control provides a common non-adaptive reference for comparing models as Controllers. In Type III, five of the eight models (Qwen3.6, Qwen3.7, GPT-5.5, Opus-4.8, and Kimi K3) show positive differences in 80.67%–94.67% of the 10,000 paired bootstrap samples. Although their 95% confidence intervals include zero, most resampled differences lie above zero.
ΔSSR = Controller − fixed control. All differences and interval endpoints are in percentage points. Each setting uses 80 tasks × 3 executions per condition. These are pointwise 95% intervals.
Protocol failures and GLM's task outcomes (W2/Q3)
Producing an executable instruction within the output budget is part of Controller performance, so runs with protocol failures are retained in the reported success rates. Protocol failures affected 96 Type II and 114 Type III GLM runs; all of these runs failed.
Each setting has 240 runs per model. The table reports runs with output-budget exhaustion / runs with invalid Contracts / distinct runs with either failure.
Appendix F.4 reports 271 GLM requests that reached the 20,480-token cap. They occurred at 271 checkpoints across 111 of the 240 Type III runs. An output-limit hit ends the current task checkpoint. In SCBench’s multi-checkpoint tasks, execution can continue at the next checkpoint from the inherited code state, while the earlier failure remains recorded. We count one outcome per full run.