- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2(+1)
Read review
Paper summary
DAGLoop addresses the mismatch between evaluating a complete candidate method and deciding which individual design choices to retain or revise. It translates each choice into a testable expectation, records its prerequisites, and constructs a directed acyclic graph of experiments. A component is tested against a control that retains its prerequisites. Components that supply intermediate properties receive property checks, while inseparable decisions can be tested jointly.
Critical strengths
- The novelty is interesting: the integration of dependency-preserving experimental comparisons with automated revision, rather than the basic ideas of ablation or dependency graphs themselves.
Critical weaknesses
- It is unclear whether DAGLoop improves results because of its experiment design or because it does more work. The system adds memory, a review agent, and extra experiments, but the evaluation does not separate their benefits or give both systems the same computing budget.
- Almost all of the average improvement comes from one task. DAGLoop performs better on 14 tasks and worse on 18; without the online-serving task, its average advantage is small. One run per task is not enough to show whether the improvement is reliable.
Questions for authors
- Do any existing tests show that keeping required components in place improves results? Can you separate this benefit from the effects of memory, the review agent, or simply running more experiments?
- Do you have repeated-run results showing that the improvement is reliable?
5 comments
Response to reviewer 5Msg [2/2]
Q2: Does the advantage persist without online serving, given the original 14 wins and 18 losses? Do repeated runs show that the improvement is reliable?
A2: Thank you for raising both concentration and repeatability. The repeated comparisons show gains beyond Serving and Flux. With DeepSeek-V4.1-Flash reasoning Off, both systems report the mean of two runs per task across all 36 tasks; DAGLoop wins on 26 tasks and loses on 10, with a mean advantage of 0.0488 after excluding Serving and Flux.
In the original GLM-5.2 reasoning Max experiments, the workflow returned the latest result rather than the best accepted complete candidate. We corrected this selection behavior and reran DAGLoop on all 36 AutoLab tasks, covering both previous wins and previous losses. Terminus-2 was unaffected, and its reference scores remain unchanged.
| GLM-5.2 reasoning Max comparison | DAGLoop | Terminus-2 | Wins / ties / losses |
|---|---|---|---|
| Original submission | 0.6113 | 0.5831 | 14 / 4 / 18 |
| Corrected, all 36 tasks rerun | 0.6459 | 0.5831 | 19 / 7 / 10 |
The four-setting comparison is:
| Setting | DAGLoop | Terminus-2 | Wins / ties / losses |
|---|---|---|---|
| GLM-5.2 reasoning Max | 0.6459 | 0.5831 | 19 / 7 / 10 |
| GLM-5.2 reasoning Off | 0.5430 | 0.4266 | 27 / 1 / 8 |
| DeepSeek-V4-Flash reasoning Off | 0.5258 | 0.4202 | 24 / 0 / 12 |
| DeepSeek-V4.1-Flash reasoning Off | 0.5819 | 0.4926 | 26 / 0 / 10 |
For each of the three reasoning Off settings, both DAGLoop and Terminus-2 report the mean of two recorded runs per task across all 36 AutoLab tasks (Section 4.1). The corrected GLM-5.2 reasoning Max result is reported separately from those repeated-run comparisons.
The mean paired differences under task exclusions are:
| Setting | All 36 | Without Serving (35) | Without Serving and Flux (34) |
|---|---|---|---|
| GLM-5.2 reasoning Max | +0.0628 | +0.0435 | +0.0154 |
| GLM-5.2 reasoning Off | +0.1164 | +0.1044 | +0.1016 |
| DeepSeek-V4-Flash reasoning Off | +0.1057 | +0.0936 | +0.0743 |
| DeepSeek-V4.1-Flash reasoning Off | +0.0892 | +0.0662 | +0.0488 |
After excluding Serving and Flux, DAGLoop retains a higher mean score than Terminus-2 on the remaining 34 tasks in all four settings. All 36 tasks remain in the full-suite comparison; exclusions are sensitivity analyses.
Thank you again for asking about reliability. We have added the repeated comparisons and task-exclusion analyses above.
Welcome for more discussions
Dear Reviewer #5Msg,
Thank you very much for your time and valuable comments. Your main concerns were whether the gains come from the experimental design rather than other workflow additions or extra work, and whether they persist beyond a single large-gain task and a single run. For your convenience, we summarize our responses below:
-
Sources of improvement. We added four-policy ablations on six tasks, retaining Planner/Critic and sharing preparation and configured budget limits. Relative to full DAGLoop, independent component experiments, chronological history, and whole-candidate-only feedback reduce the mean score by 0.0372, 0.0201, and 0.0464, respectively. We also clarified that both agents may complete their research workflows without an additional agent wall-clock cutoff.
-
Repeatability and task coverage. We added three model/reasoning settings with two runs per task for both systems. With DeepSeek-V4.1-Flash and reasoning disabled, DAGLoop wins on 26 of 36 tasks and retains a mean advantage of 0.0488 after excluding Serving and Flux. The mean advantage remains positive after these exclusions in all four reported settings.
As the discussion period approaches its end, would you kindly review our responses and let us know whether any questions remain? We would be happy to provide further clarification and welcome further discussion.
Best regards and thanks,
The Authors of Submission 19430
Response to reviewer 5Msg [1/2]
Q1: What tests show that keeping required components in place improves results, independently of memory, the review agent, and extra experiments? Were DAGLoop and Terminus-2 evaluated under the same computing budget?
A1: Thank you for pointing out the need to distinguish the sources of the improvement. We have added ablations that separately change prerequisite preservation, memory presentation, and intermediate experimental feedback while retaining Planner/Critic. The full workflow has the highest mean score on each of the six ablation tasks (Section 4.3).
To examine these effects, we compare four policies on six AutoLab tasks using DeepSeek-V4.1-Flash with reasoning disabled. We selected these six tasks for their mid-range scores and ran each policy twice per task. For each task, all policies reuse the same completed literature retrieval and baseline reproduction. From this fixed preparation snapshot, each policy runs the subsequent experimentation and method-revision workflow twice; each entry averages the two final scores. This includes the full system A, and the cohort is reported separately from the main benchmark results. All four policies use the same configured budget limits as the main experiments.
Relative to the full system, independent experiments, chronological history, and whole-candidate-only feedback reduce the mean by 0.0372, 0.0201, and 0.0464. These differences are computed from unrounded means. B still performs component experiments, and D retains Planner/Critic, providing targeted comparisons for prerequisite preservation and intermediate feedback.
Both agents follow the same completion rule: each runs until its configured research budget is exhausted or its workflow concludes. Each paired comparison uses the same model and reasoning setting and retains the official task verifiers and task-specific training and evaluation constraints. We impose no additional agent wall-clock cutoff on either system, allowing both workflows to complete their own research process.
Thank you again for suggesting these component comparisons.