- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
- This paper introduces LoopsBench, a benchmark for evaluating how agents solve long-horizon coding tasks with interdependent requirements over successive steps. Each task is split into separately testable units organized as a DAG. A unit’s hidden tests are activated once all of its prerequisite units pass, and activated tests are rerun on later code snapshots to track progress and regressions.
- Task generations is done via selection various sources such as course labs, PR sequences and research papers followed with filtering.
- The paper evaluates a range of models, harnesses, and loop mechanisms. The best configuration resolves 25% of tasks, continuation generally helps, and plans made by agents only partly match the recovered dependencies.
Critical strengths
- I like the core idea of evaluating the development process instead of just the final outcome. Activating hidden tests as prerequisites pass, then rerunning them on later snapshots, shows when each requirement gets done and whether later edits break it. This is quite novel as compared to other coding benchmarks. The tasks are also hard, so there’s plenty of headroom.
- The benchmark is built carefully and covers a lot of ground. Each DAG unit’s tests are checked to fail before the unit is implemented and pass after, a large share of units and dependencies were reviewed by humans, and the evaluation spans many models, harnesses, and loop mechanisms.
Critical weaknesses
- I think the paper could be made much stronger by including qualitative analysis of how different models and harnesses actually solve the tasks. A few case studies or failure categorization would help inform practitioners on the current state: how harnesses manage context, what survives compaction, how a restarted session picks up remaining work, how a coordinator splits units among workers, and where agents typically fail. That would help explain why configurations differ, not just by how much.
- The paper is positioned benchmark for loop engineering, but the experiments compare complete off-the-shelf systems (model + harness + loop mechanism) without isolating loop components or running a full model × harness grid. It’s therefore unclear whether the benchmark measures loop engineering, harness quality, model capability, or their interaction. Clarifying the intended use, and aligning the experiments with it, would strengthen the paper.
Questions for authors
- How much do regressions actually contribute to task failure? Does the setup allow to separate temporary regressions from ones still present at the end, and report what fraction of unresolved tasks fail because of persistent regressions rather than incomplete units? A breakdown of failure causes across models and harnesses (e.g., wrong order, early stopping, timeouts, regressions) would also make the benchmark much more diagnostic.
2 comments
Thank you again for your thoughtful review. As the discussion period draws to a close, we wanted to ask whether our response addressed your concerns. If any questions remain, we’d be happy to discuss them further.
Response to Reviewer qL5a
We thank the reviewer for highlighting the need to analyze execution behavior and disentangle loop effects from model and harness capabilities.
Q1. I think the paper could be made much stronger by including qualitative analysis of how different models and harnesses actually solve the tasks. A few case studies or failure categorization would help inform practitioners on the current state: how harnesses manage context, what survives compaction, how a restarted session picks up remaining work, how a coordinator splits units among workers, and where agents typically fail. That would help explain why configurations differ, not just by how much.
How much do regressions actually contribute to task failure? Does the setup allow to separate temporary regressions from ones still present at the end, and report what fraction of unresolved tasks fail because of persistent regressions rather than incomplete units? A breakdown of failure causes across models and harnesses (e.g., wrong order, early stopping, timeouts, regressions) would also make the benchmark much more diagnostic.
A1. We audited 300 traces from 60 of 112 tasks across five model–harness configurations. The main issues were incomplete dependency tracking, insufficient requirement coverage, and failure to preserve satisfied test obligations. Claude Code, Codex, and GitHub Copilot more often recovered dependencies, whereas Qwen Code more often followed a sequential pattern.
Table R1. Failure taxonomy across five model–harness configurations.
Note: Categories use multi-label counting, so percentages do not sum to 100%.
Incorrect implementation and incomplete verification dominated. Qwen Code + Qwen3.6-Plus showed more omitted requirements and dependency failures, while Claude Code + Opus-4.7 showed fewer. A MySQL run needed an accumulator correction for an upper-bound Fibonacci test. An Autodiff run passed local tests but failed final Transformer, gradient, and batch-processing checks. Compaction and restart use requirement files, saved plans, Git state, and checkpoints, but may lose reasoning and tool outputs. Coordinators split and integrate work, yet parallel workers may leave global requirements unsatisfied. Most GPT-5.4 regressions were recovered. In unresolved classifiable tasks, incomplete development predominated, persistent regressions were less frequent, and exclusive-regression cases were rare and unconfirmed. We separately track prerequisite-order violations, premature stopping, and timeouts.
Q2. The paper is positioned benchmark for loop engineering, but the experiments compare complete off-the-shelf systems (model + harness + loop mechanism) without isolating loop components or running a full model × harness grid. It’s therefore unclear whether the benchmark measures loop engineering, harness quality, model capability, or their interaction. Clarifying the intended use, and aligning the experiments with it, would strengthen the paper.
A2. LoopsBench measures loop behavior via dependency planning, development progress, and preservation of completed functionality, which reflect model–harness–policy interactions. Table 2 compares models under a fixed harness, harnesses under fixed GPT-5.4, and external continuation with model and harness fixed under the same task-specific wall-clock budget. Continuation raises Resolve Rate from 13.39% to 18.75% for GPT-5.4 with Codex, testing a loop-level intervention. Token use is reported separately. RQ2 analyzes development under the fixed-model setting. RQ3 descriptively compares execution profiles, treating cross-model differences as configuration properties. A full model × harness grid would better test interactions and generality. Conclusions cover evaluated configurations and policies. The benchmark supports future component-level studies.