- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3(+1)
Read review
Paper summary
In this paper the authors study a practical problem of determining when to stop an agentic loop system without access to its true quality of the repeated generated artifact. Overall, the authors demonstrate an important point that quality blind assessors cannot determine the true quality of the artifact generated. Now this is an important insight to the community specially in current times where lot of research and engineering work is being spent to come up with auto-research. This observation is also what motivates their approach of measuring coherence of updates to determine when to stop dynamically. Through experiments across different agentic loop baselines, they show that their method achieves 8% high quality while using almost 50% the iterations.
Critical strengths
-
The proposed work by the authors are timely and solve and important problem. the observation that additional iterations do not always improve quality AND quality blind assessors cannot determine the true quality of the artifact generated are particularly interesting.
-
The computation efficiency argument is supported through experiments. For e.g in table 2, loopstop achieves very similar accuracy but with much lower costs (faster latency) making the approach useful for practical applications.
-
Expanded details on design ablations are provided in appendix I improving the transparency of the experimental methodology. For e.g. they discuss cross-fitted baseline comparisons, paired per-seed results, and scenario-level evaluations.
Critical weaknesses
-
The theory analysis primarily measures changes in the artifacts rather than their quality. For e.g., in Theorem 1 assumes an quality function, allowing the construction of an adversarial evaluator using (1-q). While this analyses the limitation of quality-blind detection, it does not demonstrate that settling is a reliable indicator of when an agent should stop. For e.g., a loop can consistently degrade artifact quality while producing coherent updates, which would not necessarily trigger the proposed stopping condition.
-
On some datasets like ARC-AGI, BigCodeBench-Hard, MLAgentBench-Kaggle, the improvements are relatively small and are often within the reported standard errors for the quality. Furthermore, BigCodeBench-Hard and MLAgentBench-Kaggle use only 3–5 seeds as the authors also acknowledge due to resource constraints, making it difficult to establish statistical significance. Interestingly, also as shown in Table I.4 with cross-fitted thresholds, PoU outperforms/at part with LoopStop on ARC-AGI and BigCodeBench-Hard. This still preserves the benefits of reduced loops though stopping criterion may not work as effectively for loops exhibiting slow but consistent improvements.
-
Baseline's performance comparison are done using a single fixed hyparameteres as discussed in appendix I.7. The authors are encouraged to discuss more if this choice is optimum for other baselines for fair comparison. Also, given loopstop terminates early than other baselines, the authors are also encouraged to see how other baselines perform when terminated early at similar number of loops. This would be a useful baseline to understand if other baselines are able to reach similar quality but degrade further due to increased iterations.
Questions for authors
-
Appendix I.7 mentions that the design choices were made using trajectories from the evaluated benchmarks. As an experiment, can the authors provide results where the algorithm is developed on one set of benchmark families and evaluated on entirely held-out families to better establish its generalizability?
-
Refer to weakness-3.
6 comments
Response to Reviewer dAfp (2 of 2)
W3 and Q2: Baseline Thresholds and Early Termination
In the main results, as in the original version, each baseline uses the threshold on its grid with the highest quality averaged over all families. Choosing a threshold for each family would require that family's true quality. PoR then uses a patience of 50 rounds, which is the largest of its grid. With this patience, PoR delivers a quality of 49.8% at 71.0 rounds, and LoopStop delivers a quality of 52.4% at 25.3 rounds (Table R1 in the General Response). A longer patience approaches running to the budget, which delivers a quality of 50.0% at 73.3 rounds. A single patience does not suit every scenario, since the best patience of a scenario ranges from 3 to 50 rounds within cart-pole alone. PoR also runs nearly every loop for most of the budget (Table H.15 and Section GR.2 in the General Response). On the scenarios that apply every change, PoR uses 73.2 rounds, and on those whose evaluator can reject, it uses 69.4 rounds.
Capped at LoopStop's rounds, every baseline delivers less than LoopStop on every family except TheoremQA, where PoR and PoU are level with LoopStop. We cap each baseline at LoopStop's mean rounds on the family, i.e., we stop the baseline no later than that round. As a reference, a fixed budget stops every loop at the cap. The fixed budget delivers a quality of 50.6% at 25.8 rounds, and LoopStop delivers a quality of 52.4% at 25.3 rounds. LoopStop delivers more because it chooses the stopping round of each loop, whereas the fixed budget stops every loop at the same round.
Baselines capped at LoopStop's rounds (new Table H.13 in Appendix H.6.1). The first table gives the true quality (%), and the second table gives the rounds used, with the standard error over seeds.
True quality (%)
| Family | Fixed budget at the cap (reference) | PoR | PoU | RLF | LoopStop |
|---|---|---|---|---|---|
| Cart-pole | 23.9 ± 1.3 | 23.9 ± 1.3 | 23.9 ± 1.3 | 20.8 ± 1.2 | 27.8 ± 1.7 |
| Cruise control | 48.6 ± 1.4 | 48.6 ± 1.4 | 48.6 ± 1.4 | 40.7 ± 1.5 | 51.9 ± 1.6 |
| Ship heading | 30.5 ± 1.4 | 30.5 ± 1.4 | 29.4 ± 1.4 | 18.8 ± 1.0 | 33.4 ± 1.5 |
| Two-mass spring | 29.0 ± 0.9 | 29.0 ± 0.9 | 28.9 ± 0.9 | 24.3 ± 1.0 | 35.4 ± 1.0 |
| ARC-AGI | 65.1 ± 0.1 | 65.1 ± 0.1 | 65.1 ± 0.1 | 65.2 ± 0.1 | 65.6 ± 0.1 |
| BigCodeBench-Hard | 57.4 ± 0.2 | 57.4 ± 0.2 | 57.4 ± 0.2 | 54.1 ± 0.2 | 58.4 ± 0.2 |
| MLAgentBench-Kaggle | 47.0 ± 1.4 | 47.0 ± 1.4 | 47.0 ± 1.4 | 44.6 ± 1.0 | 48.0 ± 0.9 |
| TravelPlanner | 61.3 ± 0.1 | 61.3 ± 0.1 | 61.3 ± 0.1 | 59.6 ± 0.1 | 62.3 ± 0.1 |
| GPQA Diamond | 66.2 ± 0.4 | 66.2 ± 0.4 | 66.2 ± 0.4 | 65.5 ± 0.4 | 67.6 ± 0.4 |
| SuperGPQA | 41.7 ± 0.3 | 41.7 ± 0.3 | 41.7 ± 0.3 | 41.4 ± 0.2 | 42.2 ± 0.3 |
| SciBench | 66.1 ± 0.1 | 66.1 ± 0.1 | 66.1 ± 0.1 | 65.3 ± 0.2 | 66.3 ± 0.1 |
| TheoremQA | 70.2 ± 0.1 | 70.2 ± 0.1 | 70.2 ± 0.1 | 68.8 ± 0.2 | 70.2 ± 0.2 |
| Mean | 50.6 ± 0.2 | 50.6 ± 0.2 | 50.5 ± 0.2 | 47.4 ± 0.2 | 52.4 ± 0.3 |
Rounds used
| Family | Fixed budget at the cap (reference) | PoR | PoU | RLF | LoopStop |
|---|---|---|---|---|---|
| Cart-pole | 27.0 ± 0.0 | 27.0 ± 0.0 | 27.0 ± 0.0 | 5.8 ± 0.1 | 26.1 ± 1.0 |
| Cruise control | 17.0 ± 0.0 | 17.0 ± 0.0 | 17.0 ± 0.0 | 10.6 ± 0.3 | 16.2 ± 0.4 |
| Ship heading | 48.0 ± 0.0 | 48.0 ± 0.0 | 45.5 ± 0.3 | 11.9 ± 0.8 | 47.7 ± 1.7 |
| Two-mass spring | 40.0 ± 0.0 | 40.0 ± 0.0 | 39.7 ± 0.1 | 10.6 ± 0.7 | 39.5 ± 1.4 |
| ARC-AGI | 23.0 ± 0.0 | 23.0 ± 0.0 | 23.0 ± 0.0 | 5.0 ± 0.0 | 22.9 ± 0.1 |
| BigCodeBench-Hard | 29.0 ± 0.0 | 29.0 ± 0.0 | 29.0 ± 0.0 | 5.1 ± 0.0 | 28.4 ± 0.1 |
| MLAgentBench-Kaggle | 25.0 ± 0.0 | 25.0 ± 0.0 | 25.0 ± 0.0 | 6.3 ± 0.1 | 24.8 ± 0.7 |
| TravelPlanner | 20.0 ± 0.0 | 20.0 ± 0.0 | 20.0 ± 0.0 | 5.1 ± 0.0 | 19.5 ± 0.0 |
| GPQA Diamond | 19.0 ± 0.0 | 19.0 ± 0.0 | 19.0 ± 0.0 | 5.0 ± 0.0 | 18.6 ± 0.1 |
| SuperGPQA | 19.0 ± 0.0 | 19.0 ± 0.0 | 19.0 ± 0.0 | 5.0 ± 0.0 | 18.8 ± 0.0 |
| SciBench | 21.0 ± 0.0 | 21.0 ± 0.0 | 21.0 ± 0.0 | 5.0 ± 0.0 | 20.2 ± 0.0 |
| TheoremQA | 21.0 ± 0.0 | 21.0 ± 0.0 | 21.0 ± 0.0 | 5.0 ± 0.0 | 20.4 ± 0.0 |
| Mean | 25.8 ± 0.0 | 25.8 ± 0.0 | 25.5 ± 0.0 | 6.7 ± 0.1 | 25.3 ± 0.2 |
Q1: Held-Out Benchmark Families
Yes, the seven families added in the revision played no part in LoopStop's design. These families are ship heading, two-mass spring, TravelPlanner, GPQA Diamond, SuperGPQA, SciBench and TheoremQA. On these families, LoopStop delivers the most quality or is within a standard error of the best rule (Table R1 in the General Response). PoR and PoU use more rounds there. In addition, if every method uses the setting with the highest mean quality over the other families, LoopStop delivers the most quality on every family left out. On the mean, LoopStop then uses more rounds than RLF and fewer than PoR and PoU (Table R5 in the General Response). In the revised paper, we have added this evaluation in the new Appendix H.7.3.
response to rebuttal and followup
I thank the authors for their detailed response and rebuttal efforts. The expanded evaluation across benchmark, more seeds and qualit round comparisons help address some of my original concerns.
I have two followup questions:
for W1: I appreciate the additional analysis of coherent quality degradation in Table H.1. However, the table primarily examines coherence after the best-quality round. Could the authors provide a similar analysis of coherence before the quality peak (for only a subset of the tasks which could be feasible to run in the remaining time), comparing the statistics during improving, plateauing, and degrading phases? This would help further establish how reliably directional coherence distinguishes ongoing quality improvements from convergence.
for W3 : The leave 1 family out results are encouraging. though the avg rounds increase from 25.3 to 41.3, while mean quality improves only from 52.4% to 52.7%. Could the authors clarify which hyperparameters primarily drive this increase?
Also, as a suggestion, the authors are encouraged to explore the existing hyperparameter sweep and explore whether alternative settings provide a better quality–efficiency tradeoff, particularly for previously unseen tasks? This could be very useful for the community in general in this area of research.
Response to Reviewer dAfp (1 of 2)
W1: Coherent Updates That Degrade Quality
A coherent decline can delay LoopStop's stop. On both controller families, whose quality decreases after its peak, LoopStop still delivers higher quality than every baseline (Table R1 in the General Response). PoR and PoU use more rounds there. By Theorem 1, a quality-blind rule observes the same updates whether they improve or degrade the quality. Such a rule thus cannot tell a coherent decline from a coherent improvement and can at most detect that the quality has stopped changing. Theorem D.1 shows that the bound on the remaining change is smallest once the updates share no direction, which is what LoopStop detects (see Section GR.3 in the General Response). LoopStop therefore continues a loop while its updates share a direction, and such a decline can cost quality under every quality-blind rule. In some controller loops, the changes after the best round still share a direction, so LoopStop does not stop them immediately. The new Table H.1 measures these loops on every controller scenario. The six scenarios whose changes after the peak keep the strongest direction have a mean yk of at least 1.45. On five of these six scenarios, LoopStop still avoids between 0.31 and 0.65 of the quality lost between the peak and the budget. The reason is that the direction fades within the budget, after which LoopStop stops the loop. In the revised paper, we have added the new Appendix H.1.2 on coherent decline.
W2: Small Improvements and Slowly Improving Loops
As mentioned in Section GR.1 in the General Response, we have run new experiments with both more benchmark families and more seeds in the revision. On most families other than the controllers, the quality increases and then holds or keeps increasing slowly, instead of decreasing after a peak. The quality that any stopping rule can gain over running to the budget is therefore small there, and LoopStop's benefit is to reach the same quality in fewer rounds. On these families, LoopStop delivers higher or comparable quality at fewer rounds than running to the budget (Table R1 in the General Response). LoopStop and running to the budget are within a standard error of each other on MLAgentBench-Kaggle. There, LoopStop delivers a quality of 48.0% at 24.8 rounds, and running to the budget delivers a quality of 47.2% at 60.0 rounds. To reduce the standard errors, we have increased the seeds to 20 on BigCodeBench-Hard and to 10 on MLAgentBench-Kaggle. On every family, LoopStop delivers higher quality than every baseline threshold that uses at most as many rounds as LoopStop (the new Figure H.3 in Appendix H.6.1). Section GR.2 in the General Response includes a summary.
The cross-fitted PoU of the original Table I.4 (Table G.4 in the revised paper) chooses the threshold of each scenario from the true quality of the family's other scenarios. A scenario is a combination of evaluator kind and model size. The cross-fitted PoU thus uses the true quality, which a quality-blind rule does not observe. On the mean, the cross-fitted PoU delivers a quality of 49.1% at 37.7 rounds, and LoopStop delivers a quality of 52.4% at 25.3 rounds (Table G.4). The cross-fitted PoU delivers slightly more than LoopStop on ARC-AGI, BigCodeBench-Hard, TravelPlanner and SciBench, at more rounds. On BigCodeBench-Hard and TravelPlanner, the cross-fitted PoU equals the best budget per scenario, which also uses the true quality, at fewer rounds. The cross-fitted PoU exceeds that reference on ARC-AGI, at more rounds. Exceeding the reference is possible because the reference stops every loop of a scenario at the same round, whereas a rule stops each loop at its own round. In the revised paper, we have extended the cross-fitted Table G.4 (previously I.4) to all families and increased seeds.
A loop that improves through frequent changes keeps their direction, so LoopStop continues it. In contrast, a loop that improves through sparse accepted changes among many rejected rounds can be stopped by those rounds before its best round. On cruise control, LoopStop stops 0.90 of the loops with the 4B model evaluator and 0.95 of the loops with the 9B model evaluator before their best round (Table H.1). With the 9B evaluator, LoopStop then delivers a quality of 60.9% at 23.4 rounds. The best budget per scenario, which uses the true quality, delivers a quality of 82.1% at 41.0 rounds (Table H.26). Over the scenarios of cruise control, LoopStop gives up at most 21.2 percentage points against the best budget per scenario. Every baseline gives up more in its worst scenario (Table 2). LoopStop can also give up quality if the improvement pauses for several rounds and then resumes, since it may stop during the pause. We mention this case as a limitation in the revised paper, with more details in Section GR.3 of the General Response.