acceptodds
Under review as a conference paper at ICLR 2027

LoopsBench: From Harness Engineering to Loop Engineering in Coding Agent Evaluation

Abstract

Long-horizon coding requires agents to implement interdependent requirements, track unfinished work, and preserve completed functionality. Existing benchmarks largely emphasize final outcomes, providing limited insight into this process. We introduce LOOPSBENCH, a benchmark for evaluating loop engineering: the mechanisms that organize agent work across successive steps. It comprises 112 tasks across 8 programming languages and 9 domains, with more than 5,300 separately testable development units organized into dependency DAGs recovered from source materials. The evaluator activates each unit’s hidden tests once its prerequisite units pass their tests and reruns activated tests on subsequent code snapshots to track progress and regressions. We also analyze recorded plans, code changes, test activity, and evaluation outcomes. Across the evaluated models and harnesses, external continuation generally improves resolution. The strongest configuration, Opus-4.7 with Claude Code and external continuation, resolves 25.00% of tasks. Further analysis shows incomplete agreement between recorded plans and recovered dependencies, longer patches than the gold reference on resolved units, and regressions across the four configurations examined in the execution analysis.

Primary area:
datasets and benchmarks
Keywords:
coding agents, long-horizon development, dependency graph

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_qL5aOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • This paper introduces LoopsBench, a benchmark for evaluating how agents solve long-horizon coding tasks with interdependent requirements over successive steps. Each task is split into separately testable units organized as a DAG. A unit’s hidden tests are activated once all of its prerequisite units pass, and activated tests are rerun on later code snapshots to track progress and regressions.
  • Task generations is done via selection various sources such as course labs, PR sequences and research papers followed with filtering.
  • The paper evaluates a range of models, harnesses, and loop mechanisms. The best configuration resolves 25% of tasks, continuation generally helps, and plans made by agents only partly match the recovered dependencies.
Critical strengths
  1. I like the core idea of evaluating the development process instead of just the final outcome. Activating hidden tests as prerequisites pass, then rerunning them on later snapshots, shows when each requirement gets done and whether later edits break it. This is quite novel as compared to other coding benchmarks. The tasks are also hard, so there’s plenty of headroom.
  2. The benchmark is built carefully and covers a lot of ground. Each DAG unit’s tests are checked to fail before the unit is implemented and pass after, a large share of units and dependencies were reviewed by humans, and the evaluation spans many models, harnesses, and loop mechanisms.
Critical weaknesses
  1. I think the paper could be made much stronger by including qualitative analysis of how different models and harnesses actually solve the tasks. A few case studies or failure categorization would help inform practitioners on the current state: how harnesses manage context, what survives compaction, how a restarted session picks up remaining work, how a coordinator splits units among workers, and where agents typically fail. That would help explain why configurations differ, not just by how much.
  2. The paper is positioned benchmark for loop engineering, but the experiments compare complete off-the-shelf systems (model + harness + loop mechanism) without isolating loop components or running a full model × harness grid. It’s therefore unclear whether the benchmark measures loop engineering, harness quality, model capability, or their interaction. Clarifying the intended use, and aligning the experiments with it, would strengthen the paper.
Questions for authors
  1. How much do regressions actually contribute to task failure? Does the setup allow to separate temporary regressions from ones still present at the end, and report what fraction of unresolved tasks fail because of persistent regressions rather than incomplete units? A breakdown of failure causes across models and harnesses (e.g., wrong order, early stopping, timeouts, regressions) would also make the benchmark much more diagnostic.
2 comments
AuthorsOpenReview ↗

Response to Reviewer qL5a

We thank the reviewer for highlighting the need to analyze execution behavior and disentangle loop effects from model and harness capabilities.

Q1. I think the paper could be made much stronger by including qualitative analysis of how different models and harnesses actually solve the tasks. A few case studies or failure categorization would help inform practitioners on the current state: how harnesses manage context, what survives compaction, how a restarted session picks up remaining work, how a coordinator splits units among workers, and where agents typically fail. That would help explain why configurations differ, not just by how much.

How much do regressions actually contribute to task failure? Does the setup allow to separate temporary regressions from ones still present at the end, and report what fraction of unresolved tasks fail because of persistent regressions rather than incomplete units? A breakdown of failure causes across models and harnesses (e.g., wrong order, early stopping, timeouts, regressions) would also make the benchmark much more diagnostic.

A1. We audited 300 traces from 60 of 112 tasks across five model–harness configurations. The main issues were incomplete dependency tracking, insufficient requirement coverage, and failure to preserve satisfied test obligations. Claude Code, Codex, and GitHub Copilot more often recovered dependencies, whereas Qwen Code more often followed a sequential pattern.

Table R1. Failure taxonomy across five model–harness configurations.

Failure CategoryCodex + GPT-5.5CC + Opus-4.7Qwen Code + Qwen3.6-PlusCC + GPT-5.4Copilot + GPT-5.4Total
A. Requirement Fulfillment and Implementation
A1. Omitted requirements15 (25.0%)13 (21.7%)22 (36.7%)18 (30.0%)20 (33.3%)88 (29.3%)
A2. Incorrect implementation31 (51.7%)29 (48.3%)35 (58.3%)32 (53.3%)35 (58.3%)162 (54.0%)
B. Dependency and Execution-State Management
B1. Dependency handling failure20 (33.3%)17 (28.3%)28 (46.7%)17 (28.3%)25 (41.7%)107 (35.7%)
B2. Context/session recovery failure12 (20.0%)10 (16.7%)20 (33.3%)14 (23.3%)16 (26.7%)72 (24.0%)
B3. Multi-agent coordination and integration failure4 (6.7%)3 (5.0%)5 (8.3%)4 (6.7%)4 (6.7%)20 (6.7%)
C. Verification and Regression
C1. Incomplete verification39 (65.0%)37 (61.7%)44 (73.3%)41 (68.3%)43 (71.7%)204 (68.0%)
C2. Persistent regression6 (10.0%)8 (13.3%)10 (16.7%)8 (13.3%)8 (13.3%)40 (13.3%)

Note: Categories use multi-label counting, so percentages do not sum to 100%.

Incorrect implementation and incomplete verification dominated. Qwen Code + Qwen3.6-Plus showed more omitted requirements and dependency failures, while Claude Code + Opus-4.7 showed fewer. A MySQL run needed an accumulator correction for an upper-bound Fibonacci test. An Autodiff run passed local tests but failed final Transformer, gradient, and batch-processing checks. Compaction and restart use requirement files, saved plans, Git state, and checkpoints, but may lose reasoning and tool outputs. Coordinators split and integrate work, yet parallel workers may leave global requirements unsatisfied. Most GPT-5.4 regressions were recovered. In unresolved classifiable tasks, incomplete development predominated, persistent regressions were less frequent, and exclusive-regression cases were rare and unconfirmed. We separately track prerequisite-order violations, premature stopping, and timeouts.


Q2. The paper is positioned benchmark for loop engineering, but the experiments compare complete off-the-shelf systems (model + harness + loop mechanism) without isolating loop components or running a full model × harness grid. It’s therefore unclear whether the benchmark measures loop engineering, harness quality, model capability, or their interaction. Clarifying the intended use, and aligning the experiments with it, would strengthen the paper.

A2. LoopsBench measures loop behavior via dependency planning, development progress, and preservation of completed functionality, which reflect model–harness–policy interactions. Table 2 compares models under a fixed harness, harnesses under fixed GPT-5.4, and external continuation with model and harness fixed under the same task-specific wall-clock budget. Continuation raises Resolve Rate from 13.39% to 18.75% for GPT-5.4 with Codex, testing a loop-level intervention. Token use is reported separately. RQ2 analyzes development under the fixed-model setting. RQ3 descriptively compares execution profiles, treating cross-model differences as configuration properties. A full model × harness grid would better test interactions and generality. Conclusions cover evaluated configurations and policies. The benchmark supports future component-level studies.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Reviewer_iZp3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

LoopsBench is a benchmark of 112 long-horizon coding tasks (PR sequences, course labs, research evolutions; 8 languages), decomposed into ~5,300 testable units with recovered dependency DAGs. Hidden unit tests activate once prerequisites pass and are rerun on later snapshots to track progress and regressions. Tasks are built with Claude Code and human-audited. Across models, harnesses, and loop mechanisms, the best configuration (Opus-4.7 + Claude Code + continuation) resolves 25% of tasks.

Critical strengths
  • Dependency-aware evaluation with regression tracking captures progress and breakage of working code that end-state benchmarks miss; this drives my lean toward acceptance.
  • Validated tasks, reproducible environments, and multi-harness adapters make it a reusable community resource.
Critical weaknesses
  • Table 2 reports identical RR and TPR for Opus-4.5 and gpt-5.4 under Claude Code while Depth differs, which casts doubt on the results.

  • Claude Code with Opus 4.7 built the tasks and is also the top configuration, so rankings may be biased.

Questions for authors
  • Why do the Opus-4.5 and gpt-5.4 rows in Table 2 match on RR and TPR?

  • Can you report variance or confidence intervals for the main results?

  • Do rankings hold if a task subset is rebuilt with a different model?

2 comments
AuthorsOpenReview ↗

Response to Reviewer iZp3

We thank the reviewer for identifying the Table 2 inconsistency and for asking about uncertainty estimates and the robustness of our rankings to the task-construction model.

Q1. Table 2 reports identical RR and TPR for Opus-4.5 and gpt-5.4 under Claude Code while Depth differs, which casts doubt on the results. Why do the Opus-4.5 and gpt-5.4 rows in Table 2 match on RR and TPR?

A1. We apologize for the typographical error in Table 2. The identical RR and TPR values reported for Opus-4.5 and GPT-5.4 under Claude Code were caused by an inadvertent copy error when preparing the table, rather than reflecting the actual experimental results. We have corrected the Claude Code + GPT-5.4 row in Table 2. The corrected values are 12.50%, 17.86%, 35.87%, 46.33%, and 0.42, respectively. We apologize for the oversight and have rechecked the corresponding entries to ensure the accuracy of the revised table.


Q2. Can you report variance or confidence intervals for the main results?

A2. As noted in Section 3.1, we ran three independent attempts per task for the within-family comparisons with outer continuation. Table 11 reports pass@3. Using the same runs, we also computed the sample variance across the three run-level scores for RR, TPR, and Depth. Each run covered all 112 tasks. We used percentage points for RR and TPR and the original 0–1 scale for Depth.

ConfigurationPass@3 (%)RR varianceTPR varianceDepth variance
GPT-5.5 with Codex23.210.270.850.00029
GPT-5.2 with Codex15.180.270.370.00014
GPT-5 with Codex9.820.000.690.00017
GPT-4o with Codex4.460.270.960.00002
Opus-4.7 with Claude Code26.790.801.000.00063
Opus-4.5 with Claude Code19.640.271.560.00013
Sonnet-4.5 with Claude Code14.290.000.740.00022
Sonnet-4 with Claude Code8.930.270.450.00026
Qwen3.6-Plus with Qwen Code11.611.061.870.00012
Qwen3.5-Plus with Qwen Code9.820.001.200.00004
Qwen3-Max with Qwen Code2.680.000.220.00010
Qwen2.5-72B with Qwen Code0.890.270.210.00001

Q3. Claude Code with Opus 4.7 built the tasks and is also the top configuration, so rankings may be biased. Do rankings hold if a task subset is rebuilt with a different model?

A3. We agree that using Claude Code with Opus-4.7 in task construction may introduce potential model-specific bias. To examine this, we reconstructed an independent subset of 20 tasks using the same construction pipeline, but with Codex instead of Claude Code with Opus-4.7, followed by manual auditing. We then evaluated these tasks using a controlled 2 × 2 design, crossing two harnesses (Claude Code and Codex) with two models (GPT-5.4 and Opus-4.7). Notably, Claude Code with Opus-4.7 remained the best-performing configuration on this independently constructed subset. This result suggests that the leading performance is not merely an artifact of using Claude Code and Opus-4.7 during task construction, and provides evidence that our main finding is robust to changing the task-construction agent.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

Reviewer_w1HSOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces LoopsBench, a benchmark for long-horizon coding agents. It breaks complex tasks into dependent development units and tracks progress and regressions throughout execution, rather than only checking the final result. The authors evaluate multiple models and agent frameworks on 112 tasks.

Critical strengths

Tracking intermediate progress and regressions is valuable and goes beyond what most existing coding benchmarks measure.

The benchmark covers different languages, task types, models, and agent frameworks. The results provide useful insights into long-running coding agents.

Critical weaknesses

The DAGs are central to the benchmark, but the validation mainly checks existing edges. Missing dependencies or alternative valid development orders could affect the reported metrics.

Some loop mechanisms are evaluated with different underlying models, making it hard to tell whether improvements come from the loop design or the model itself.

Questions for authors

Have you measured how many real dependencies might be missing from the recovered DAGs?

Could you compare different loop mechanisms using the same model and compute budget?

2 comments
AuthorsOpenReview ↗

Response to Reviewer w1HS

We thank the reviewer for asking about the completeness of the recovered dependency DAGs and about controlled comparisons of loop mechanisms under matched models and budgets.

Q1. The DAGs are central to the benchmark, but the validation mainly checks existing edges. Missing dependencies or alternative valid development orders could affect the reported metrics. Have you measured how many real dependencies might be missing from the recovered DAGs?

A1. Our human validation explicitly checked for missing dependencies. For Course Labs and Research Evolutions, we verified the recovered dependencies against the prerequisites and methodological inheritance documented in the original assignments and papers. For PR Sequences, we examined the source artifacts for prerequisites absent from the recovered DAG, rather than checking only existing edges. As reported in Appendix A.2, the audit exhaustively covered 96 tasks and sampled nodes and edges from each of the remaining 16 large PR Sequence tasks, covering 1,674 development units in total. No missing dependencies were identified within the audited material. The five issues found were semantic mismatches between tests and requirements, all of which were corrected. This finding applies to the audited material; we do not claim that sampling establishes completeness for every unreviewed unit.


Q2. Some loop mechanisms are evaluated with different underlying models, making it hard to tell whether improvements come from the loop design or the model itself. Could you compare different loop mechanisms using the same model and compute budget?

A2. Table 2 already includes a fixed-model comparison across six agent harnesses using GPT-5.4 on the same 112 tasks, with the same task-specific wall-clock budgets and both with and without outer continuation. Under outer continuation, Resolve Rate ranges from 7.14% to 18.75%, demonstrating performance differences while holding the underlying model fixed. Table 4 also contains model-matched comparisons: Codex goal mode and Ralph both use GPT-5.5, achieving 20.54% and 8.04% Resolve Rate, respectively; Claude Code goal mode and dynamic workflows both use Opus-4.7, achieving 17.86% and 24.11%. Regarding resources, our evaluation controls wall-clock budgets, while actual token consumption is reported separately in Appendix C.3. We impose no total token limit: apart from the task timeout or the configured Ralph loop round limit, runs terminate naturally when the agent decides to stop. These experiments therefore do not enforce identical total token expenditure. Accordingly, the cross-model entries in Table 4 characterize execution profiles rather than establish an isolated causal effect of loop design. A strictly compute-matched comparison would additionally require a shared total inference budget, including all worker calls for dynamic workflows.

Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.