acceptodds
Desk rejected at ICLR 2027

DAGLoop: Aligning Method Design with Experimental Feedback in Automated AI Research

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Automated AI research agents aim to improve machine learning methods by iteratively proposing, implementing, and evaluating candidate methods. Existing work uses the outcome of each evaluation to determine how the next candidate should be revised for performance improvement. However, a candidate often combines multiple design decisions, and its overall outcome alone does not reveal which decisions produced their expected effects, whether their required conditions were satisfied, or which decisions should be retained or revised. Inspired by ablation studies in the machine learning community, we propose DAGLoop to test the expected effect of a design decision by comparing implementations with and without that decision while preserving the other decisions it requires. Specifically, DAGLoop translates each design decision’s rationale into a testable expectation and pairs it with a targeted experiment, forming an experimental specification for assessing the expected effect under its required conditions. Using these specifications, we formulate ablation planning as the construction of a directed acyclic graph (DAG) from the candidate method’s design: nodes associate decisions with tests, and edges encode dependencies. The DAG specifies which decisions each experiment must retain and supports joint tests when an expected effect requires several decisions to act together. Outcomes are recorded with the tested decisions, expectations, and conditions to form structured research memory. This memory guides the agent in retaining experimentally supported decisions, revising unsupported ones, and planning further experiments when the evidence is unresolved. Experiments on dozens of research tasks demonstrate substantial performance improvements achieved by DAGLoop across diverse executable research settings.

Primary area:
foundation or frontier models, including LLMs
Keywords:
automated AI research, experimental feedback, ablation studies, directed acyclic graphs, agent memory

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_5MsgOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

DAGLoop addresses the mismatch between evaluating a complete candidate method and deciding which individual design choices to retain or revise. It translates each choice into a testable expectation, records its prerequisites, and constructs a directed acyclic graph of experiments. A component is tested against a control that retains its prerequisites. Components that supply intermediate properties receive property checks, while inseparable decisions can be tested jointly.

Critical strengths
  • The novelty is interesting: the integration of dependency-preserving experimental comparisons with automated revision, rather than the basic ideas of ablation or dependency graphs themselves.
Critical weaknesses
  • It is unclear whether DAGLoop improves results because of its experiment design or because it does more work. The system adds memory, a review agent, and extra experiments, but the evaluation does not separate their benefits or give both systems the same computing budget.
  • Almost all of the average improvement comes from one task. DAGLoop performs better on 14 tasks and worse on 18; without the online-serving task, its average advantage is small. One run per task is not enough to show whether the improvement is reliable.
Questions for authors
  • Do any existing tests show that keeping required components in place improves results? Can you separate this benefit from the effects of memory, the review agent, or simply running more experiments?
  • Do you have repeated-run results showing that the improvement is reliable?
5 comments
AuthorsOpenReview ↗

Response to reviewer 5Msg [1/2]

Q1: What tests show that keeping required components in place improves results, independently of memory, the review agent, and extra experiments? Were DAGLoop and Terminus-2 evaluated under the same computing budget?

A1: Thank you for pointing out the need to distinguish the sources of the improvement. We have added ablations that separately change prerequisite preservation, memory presentation, and intermediate experimental feedback while retaining Planner/Critic. The full workflow has the highest mean score on each of the six ablation tasks (Section 4.3).

To examine these effects, we compare four policies on six AutoLab tasks using DeepSeek-V4.1-Flash with reasoning disabled. We selected these six tasks for their mid-range scores and ran each policy twice per task. For each task, all policies reuse the same completed literature retrieval and baseline reproduction. From this fixed preparation snapshot, each policy runs the subsequent experimentation and method-revision workflow twice; each entry averages the two final scores. This includes the full system A, and the cohort is reported separately from the main benchmark results. All four policies use the same configured budget limits as the main experiments.

  • A: Full uses the complete DAGLoop workflow, including prerequisite-preserving experiments and structured memory.
  • B: Independent tests components from a common baseline without retaining ancestor changes or their acceptance prerequisites.
  • C: History replaces the model-facing structured memory view with chronological observations, retaining DAG experiments and the underlying evidence records.
  • D: Whole-only retains Planner/Critic but uses complete-candidate evaluation and revision without intermediate component experiments or property checks.
TaskA: FullB: IndependentC: HistoryD: Whole-only
Adaptive Compression0.60420.56380.58520.5495
AES-128 CTR0.63810.61200.61800.6029
Hash Join0.50310.46630.48840.4566
Flash Attention0.44430.40670.44300.3968
NTT CUDA0.54170.48810.50020.4864
MSM CUDA0.49310.46470.46950.4542
Mean (6)0.53740.50020.51730.4911

Relative to the full system, independent experiments, chronological history, and whole-candidate-only feedback reduce the mean by 0.0372, 0.0201, and 0.0464. These differences are computed from unrounded means. B still performs component experiments, and D retains Planner/Critic, providing targeted comparisons for prerequisite preservation and intermediate feedback.

Both agents follow the same completion rule: each runs until its configured research budget is exhausted or its workflow concludes. Each paired comparison uses the same model and reasoning setting and retains the official task verifiers and task-specific training and evaluation constraints. We impose no additional agent wall-clock cutoff on either system, allowing both workflows to complete their own research process.

Thank you again for suggesting these component comparisons.

AuthorsOpenReview ↗

Response to reviewer 5Msg [2/2]

Q2: Does the advantage persist without online serving, given the original 14 wins and 18 losses? Do repeated runs show that the improvement is reliable?

A2: Thank you for raising both concentration and repeatability. The repeated comparisons show gains beyond Serving and Flux. With DeepSeek-V4.1-Flash reasoning Off, both systems report the mean of two runs per task across all 36 tasks; DAGLoop wins on 26 tasks and loses on 10, with a mean advantage of 0.0488 after excluding Serving and Flux.

In the original GLM-5.2 reasoning Max experiments, the workflow returned the latest result rather than the best accepted complete candidate. We corrected this selection behavior and reran DAGLoop on all 36 AutoLab tasks, covering both previous wins and previous losses. Terminus-2 was unaffected, and its reference scores remain unchanged.

GLM-5.2 reasoning Max comparisonDAGLoopTerminus-2Wins / ties / losses
Original submission0.61130.583114 / 4 / 18
Corrected, all 36 tasks rerun0.64590.583119 / 7 / 10

The four-setting comparison is:

SettingDAGLoopTerminus-2Wins / ties / losses
GLM-5.2 reasoning Max0.64590.583119 / 7 / 10
GLM-5.2 reasoning Off0.54300.426627 / 1 / 8
DeepSeek-V4-Flash reasoning Off0.52580.420224 / 0 / 12
DeepSeek-V4.1-Flash reasoning Off0.58190.492626 / 0 / 10

For each of the three reasoning Off settings, both DAGLoop and Terminus-2 report the mean of two recorded runs per task across all 36 AutoLab tasks (Section 4.1). The corrected GLM-5.2 reasoning Max result is reported separately from those repeated-run comparisons.

The mean paired differences under task exclusions are:

SettingAll 36Without Serving (35)Without Serving and Flux (34)
GLM-5.2 reasoning Max+0.0628+0.0435+0.0154
GLM-5.2 reasoning Off+0.1164+0.1044+0.1016
DeepSeek-V4-Flash reasoning Off+0.1057+0.0936+0.0743
DeepSeek-V4.1-Flash reasoning Off+0.0892+0.0662+0.0488

After excluding Serving and Flux, DAGLoop retains a higher mean score than Terminus-2 on the remaining 34 tasks in all four settings. All 36 tasks remain in the full-suite comparison; exclusions are sensitivity analyses.

Thank you again for asking about reliability. We have added the repeated comparisons and task-exclusion analyses above.

AuthorsOpenReview ↗

Welcome for more discussions

Dear Reviewer #5Msg,

Thank you very much for your time and valuable comments. Your main concerns were whether the gains come from the experimental design rather than other workflow additions or extra work, and whether they persist beyond a single large-gain task and a single run. For your convenience, we summarize our responses below:

  1. Sources of improvement. We added four-policy ablations on six tasks, retaining Planner/Critic and sharing preparation and configured budget limits. Relative to full DAGLoop, independent component experiments, chronological history, and whole-candidate-only feedback reduce the mean score by 0.0372, 0.0201, and 0.0464, respectively. We also clarified that both agents may complete their research workflows without an additional agent wall-clock cutoff.

  2. Repeatability and task coverage. We added three model/reasoning settings with two runs per task for both systems. With DeepSeek-V4.1-Flash and reasoning disabled, DAGLoop wins on 26 of 36 tasks and retains a mean advantage of 0.0488 after excluding Serving and Flux. The mean advantage remains positive after these exclusions in all four reported settings.

As the discussion period approaches its end, would you kindly review our responses and let us know whether any questions remain? We would be happy to provide further clarification and welcome further discussion.

Best regards and thanks,
The Authors of Submission 19430

Reviewer_CKqcOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes DAGLoop, an automated research system designed to align candidate method revision with decision-level experimental feedback on executable machine learning and algorithm optimization tasks. Observing that a candidate method typically bundles multiple design decisions—and that evaluating only the complete candidate conflates individual effects, while independent single-change ablations can break necessary upstream prerequisites—DAGLoop introduces a structured workflow:

  1. Experimental Specifications: Before running code, a planner (checked by a critic) translates each design decision's rationale into a testable expectation, required conditions, and either a performance test (measuring task-metric improvement) or a property check (verifying an intermediate property needed by downstream decisions).
  2. Dependency DAG & Prerequisite-Preserving Controls: Decisions and their dependencies are organized into a directed acyclic graph (DAG). Each component is tested in topological order by comparing a control implementation that retains all ancestor prerequisites against a treatment implementation that adds the active decision (or a merged group of tightly coupled decisions tested jointly).
  3. Structured Research Memory & Rule-Based Revision: Execution outcomes, checkpoints, and prerequisite contexts are stored in research memory to guide debugging, retaining supported components, revising failed ones, or replanning the graph, before finally assembling all accepted components for official task evaluation.

The authors evaluate DAGLoop on 36 tasks from the AutoLab benchmark (25 CPU and 11 GPU tasks) using glm-5.2 and compare it against locally run Terminus-2 (averaged over two runs), as well as reporting standalone results on 18 FML-Bench tasks against local baselines. On AutoLab, DAGLoop achieves a higher mean score (0.6113 vs. 0.5831 overall; 0.6344 vs. 0.6247 on CPU; 0.5588 vs. 0.4885 on GPU).

Critical strengths
  1. Clear Identification of the Attribution Gap in Bundled Agent Revisions: The paper rightly highlights an important practical flaw in existing code-and-research agents: proposing multi-part candidate revisions and relying solely on complete-candidate evaluator scores makes it impossible to know which constituent change helped, which hurt, and whether intermediate prerequisites were even satisfied. Distinguishing between property checks (for enabling upstream representations/structures) and performance tests (for downstream task gains), while allowing tightly coupled changes to be grouped into joint nodes, is a sensible and well-motivated engineering design.

  2. Broad Executable Evaluation Across Diverse System and ML Tasks: Evaluating the system across all 36 executable tasks in AutoLab (spanning C/Rust systems optimization, CUDA kernels, and model fine-tuning) and 18 tasks in FML-Bench demonstrates that the engineering pipeline can execute end-to-end across heterogeneous codebases, achieving strong gains on complex multi-stage tasks such as llm_online_serving (1.0000 vs. 0.0006) and msm_pippenger (0.6725 vs. 0.3824).

Critical weaknesses
  1. Lack of Truly Principled Attribution and Backtracking Algorithms (Rule-Based State Machine vs. Intelligent Credit Assignment): Stripped of the heavy notation in §3 and Appendices A–D, DAGLoop essentially translates standard human experimental hygiene (incremental implementation, control-variable testing, and invalidating downstream caches when upstream code changes) into a deterministic if-else workflow for LLM agents. While useful as an engineering scaffold, the system lacks a genuinely intelligent or principled algorithm for backtracking and credit assignment when non-trivial experimental failures occur:

    • No algorithmic diagnosis for cross-branch negative interference: When components pass their local tests in isolation along separate DAG branches but degrade performance when assembled into the complete candidate M+M^+, DAGLoop has no principled mechanism to attribute the conflict or identify which subset of interacting decisions is responsible.
    • Vulnerability to upstream false positives (bad local optima): If an upstream decision z1z_1 passes a weak or misspecified property check p1p_1, DAGLoop locks z1z_1 into the control P2P_2 and forces the agent to repeatedly revise the downstream decision z2z_2. If z1z_1 is actually the root cause trapping z2z_2 in a dead end, the system has no quantitative belief-updating or counterfactual backtracking rule to suspect and roll back z1z_1; it merely dumps the history text into the LLM prompt (Revise(M, L)) and hopes the LLM guesses how to replan.
    • Superficial theoretical framing: Although §3.2 cites Shapley value literature (Olsen et al., 2024) to motivate marginal contributions δi∣C\delta_{i|C}, DAGLoop never estimates multi-coalition marginal effects or interactions. Propositions 1 and 2 in Appendix D are mathematically trivial (showing that two numbers f(∅)f(\emptyset) and f({1,2})f(\{1,2\}) cannot determine a third unconstrained number f({1})f(\{1\}), and that the ancestor set in a DAG is the minimal upward-closed set) and do not provide algorithmic guarantees for search efficiency or attribution accuracy.
  2. Unconvincing Empirical Advantage (Lower Task Win-Rate than Baseline, Unmatched Compute Budgets, and Zero System Ablations):

    • Losing on the majority of non-tied tasks: Despite a higher arithmetic mean (0.6113 vs. 0.5831), Table 2 and §6 reveal that out of 36 AutoLab tasks, DAGLoop scores lower than Terminus-2 on 18 tasks, ties on 4, and wins on only 14. The higher mean is heavily driven by a few extreme outliers (e.g., llm_online_serving where Terminus-2 gets 0.0006 and flux2_klein_lora where Terminus-2 gets 0.0000, likely due to baseline execution/formatting failures rather than search quality). Under a paired sign test or median comparison, DAGLoop does not outperform Terminus-2.
    • Uncontrolled compute and token budgets: Section 4.1 states that "Both agents run to completion without agent wall-clock truncation", and DAGLoop is only run once per task. Because DAGLoop executes an elaborate multi-agent loop (Planner, Critic, topological per-node control/treatment runs, up to 6 attempts per component, and 3 outer revisions), how many total LLM tokens, evaluator launches, and GPU/CPU hours did DAGLoop consume compared to Terminus-2? Without matched compute/token budgets, it is unclear whether gains on tasks like msm_pippenger stem from DAG-based attribution or simply spending significantly more compute and LLM calls.
    • Complete absence of ablation studies on DAGLoop itself: Ironically, for a paper centered on ablation planning, the paper contains zero ablation studies of its own system components (acknowledged in §5: "The contributions of tests that retain prerequisites and research memory remain unmeasured"). We do not know how much of the performance comes from the upfront Planner/Critic decomposition, the property checks, the prerequisite-preserving DAG controls, or the structured memory.
Questions for authors
  1. Constructive Feedback — Leveraging Established Meta-Research and Experimental Design (DoE) Principles: Since the core premise of this paper is applying human meta-research principles to AI agents, could the framework be strengthened by incorporating classical Design of Experiments (DoE) and statistical attribution methodologies rather than relying purely on sequential one-at-a-time DAG traversal and LLM-prompted replanning? Specifically:

    • Fractional Factorial / Group Testing for Bundled Changes: When humans face many candidate modifications and cannot afford full one-by-one ablations, they use fractional factorial designs, orthogonal arrays, or group testing (e.g., Plackett–Burman designs) to screen main effects and pairwise interactions in far fewer runs.
    • Principled Backtracking via Bayesian / Bandit Attribution: When complete-candidate assembly fails or downstream nodes repeatedly exhaust their attempt budget, could the agent maintain an explicit probabilistic belief (or bandit value estimate) over upstream nodes and edge dependencies—automatically triggering counterfactual leave-one-out rollbacks on accepted ancestors rather than relying on unguided LLM replanning? How do the authors view integrating such quantitative meta-research principles into DAGLoop?
  2. Clarification on Compute Cost, Token Usage, and Fair Budget Matching: Can you report the exact distribution of (a) total LLM input/output tokens, (b) number of code executions / sandbox launches, and (c) wall-clock time per task for DAGLoop versus Terminus-2 on AutoLab? What is the performance comparison when Terminus-2 is granted the exact same token/execution budget as DAGLoop?

  3. Clarification on Win/Tie/Loss Breakdown (14 Wins vs. 18 Losses) and System Ablations: Why does DAGLoop underperform Terminus-2 on 18 out of 36 tasks (e.g., data_select_ifeval: 0.3774 vs. 0.8581; sstable_compaction_rs: 0.0696 vs. 0.3479; scaling_law: 0.3402 vs. 0.6054)? Does strict topological step-by-step gating prematurely lock the agent into suboptimal decompositions or waste budget on brittle LLM-written property checks? Furthermore, can you provide an ablation on a representative subset of AutoLab tasks isolating (i) Planner + Critic without DAG intermediate tests, (ii) independent ablations vs. prerequisite-preserving DAG tests, and (iii) structured memory vs. full-history context?

5 comments
AuthorsOpenReview ↗

Response to reviewer CKqc [1/4]

Q1: What is the contribution beyond a rule-based implementation of standard experimental hygiene?

A1: Thank you for asking us to make the contribution precise. DAGLoop addresses a mismatch between method evaluation and method revision: a complete-candidate score assesses a bundle of decisions, while revision must decide which decisions to retain or change. Our contribution is a research workflow that connects each decision's rationale to an executable experiment and carries the resulting evidence into method revision (Sections 2.1–2.4).

  • Experimental specifications derived from design rationales. Each decision is associated with an expected effect, required conditions, and a measurable observable. Distinguishing intermediate property checks from direct performance tests lets an enabling decision be assessed for the property it supplies before downstream gains appear.

  • Experiments that preserve prerequisites and test coupled decisions jointly. The DAG constructs controls containing all declared ancestor prerequisites and allows decisions whose effect requires them to operate together to be tested as a group. Compared with a complete-candidate score or a single-change test from a common baseline, this supplies experimental evidence for a specified decision or group in its required context.

  • Experimental evidence that guides method revision. Memory binds each observation to its specification, prerequisite implementations, and checkpoint. Revision uses these records to retain supported decisions, revisit unsupported ones, regroup coupled decisions, and revalidate affected descendants when an upstream context changes.

Q2: When separately accepted branches reduce performance after assembly, how does DAGLoop identify the conflicting decisions and decide what to revise?

A2: Thank you for asking how assembly regressions are diagnosed. DAGLoop uses an LLM to formulate an evidence-backed hypothesis about the conflicting branches, then tests the proposed revision through new experiments. Complete-candidate evaluation separately checks the assembled method; a regression is recorded without replacing the retained qualified best.

Method feedback was enabled in the GLM-5.2 reasoning Max experiments, where recorded assembly regressions triggered formal Method-revision requests. The diagnosis proceeds as follows:

  1. Establish a comparable regression. Check the actual integration starting checkpoint, measurement validity, baseline identity, metric direction, and evaluation-protocol compatibility so that the diagnosis uses comparable evidence.
  2. Form a testable revision hypothesis. Given the current Method, retained incumbent, and verified comparison evidence, the review proposes one hypothesis and specifies which components to retain, modify, add, or remove. It must supply evidence references, an expected observation, a validation plan, and the affected dependencies.
  3. Validate the revision before adoption. Host validation checks evidence references and plan completeness. The revised implementation is then evaluated before it can replace the retained incumbent.

Q3: If an upstream component passes a weak property check but causes repeated downstream failures, how is it suspected, tested counterfactually, and revised or rolled back?

A3: Thank you for asking how accepted upstream decisions can be reconsidered. Accepted upstream claims remain revisable when verified evidence contradicts them. Previously accepted components were also reopened and revised through the formal Method path in the recorded runs. The revision instructions require retaining the counterexample, revising the upstream component that owns the faulty claim and its acceptance conditions, and revalidating the affected descendants.

The existing revision path operates on the recorded experiment context:

  1. Inspect the failed downstream experiment together with the active component and ancestor histories, including their expected properties, observed checks, and checkpoint references.
  2. When a verified counterexample identifies an upstream fault, request formal Method/ComponentPlan revision of the component that owns it, its acceptance conditions, and the affected dependencies. Preserve the counterexample and original observations.
  3. Revalidate the changed upstream component and affected descendants under the revised controls. Reuse prior acceptance only where the specification, prerequisite implementation, and checkpoint context remain compatible; preserve evidence for unaffected branches.

In SSTable, the numerical property check passed for an implementation using the correct u32 field, while the frozen specification declared u64. Implementation review detected this inconsistency, withheld component qualification, and requested specification revision. Ancestor selection currently uses the LLM's diagnosis; posterior belief updates and systematic counterfactual selection are proposed extensions.

AuthorsOpenReview ↗

Response to reviewer CKqc [2/4]

Q4: Why invoke Shapley values when DAGLoop measures only local effects, and what do Propositions 1–2 establish about attribution or search efficiency?

A4: Thank you for asking us to clarify the theoretical framing. Equation (4) defines a conditional difference at a fixed feasible configuration C: f(C∪{i})−f(C). It compares implementations with and without decision i in that tested prerequisite context. A Shapley estimate instead averages marginal effects over coalitions, which is a different quantity.

  • Proposition 1 explains why a baseline score and a complete-candidate score cannot determine the sign of an unmeasured intermediate effect. Operationally, that missing control must be measured before the local difference can be evaluated.
  • Proposition 2 identifies the smallest context closed under declared prerequisites, so an experiment retains indirect ancestors as well as direct parents. Its role is to define executable controls.

We have revised the explanation around Equation (4) to describe the comparison at a fixed feasible configuration and removed the Shapley citation (Section 2.2). These propositions formalize experimental design requirements: the observations needed to evaluate a conditional difference and the prerequisite closure needed for an executable control. Quantitative claims about final performance are supported separately by the benchmark and policy comparisons.

Thank you for pointing out this distinction. We have revised the explanation around Equation (4) accordingly.

Q5: How broad and reliable is the improvement given the original 14 wins versus 18 losses and one DAGLoop run per task? How much does the higher mean depend on near-zero baseline scores for Serving and Flux?

A5: Thank you for highlighting the difference between a higher mean and broad task-level improvement. The corrected full-suite rerun improves the GLM-5.2 reasoning Max win/loss breakdown, while repeated comparisons in additional settings provide broader evidence. After excluding Serving and Flux, DAGLoop still has a higher mean score than Terminus-2 on the remaining 34 tasks in all four settings.

For GLM-5.2 reasoning Max, we corrected the selection behavior that returned the latest result rather than the best accepted complete candidate and reran DAGLoop on all 36 AutoLab tasks; Terminus-2 was unaffected.

GLM-5.2 reasoning Max comparisonDAGLoopTerminus-2Wins / ties / losses
Original submission0.61130.583114 / 4 / 18
Corrected, all 36 tasks rerun0.64590.583119 / 7 / 10

The expanded comparison is:

SettingDAGLoopTerminus-2Wins / ties / lossesSign-test p
GLM-5.2 reasoning Max0.64590.583119 / 7 / 100.1360
GLM-5.2 reasoning Off0.54300.426627 / 1 / 80.0019
DeepSeek-V4-Flash reasoning Off0.52580.420224 / 0 / 120.0652
DeepSeek-V4.1-Flash reasoning Off0.58190.492626 / 0 / 100.0113

For GLM-5.2 reasoning Max, the corrected DAGLoop result is from one run per task, while Terminus-2 reports the mean of two runs per task. In all three reasoning Off settings—GLM-5.2, DeepSeek-V4-Flash, and DeepSeek-V4.1-Flash—both systems report the mean of two runs per task across all 36 AutoLab tasks.

Sign-test p-values are two-sided exact values excluding ties, without adjustment for multiple comparisons. For GLM-5.2 reasoning Max, the median paired score difference is approximately +0.00276. The two-sided sign test reaches p<0.05 in the GLM-5.2 reasoning Off and DeepSeek-V4.1-Flash reasoning Off settings; the other two comparisons do not reach that threshold.

Removing the large-gain tasks gives the following mean paired differences:

SettingAll 36Without Serving (35)Without Serving and Flux (34)
GLM-5.2 reasoning Max+0.0628+0.0435+0.0154
GLM-5.2 reasoning Off+0.1164+0.1044+0.1016
DeepSeek-V4-Flash reasoning Off+0.1057+0.0936+0.0743
DeepSeek-V4.1-Flash reasoning Off+0.0892+0.0662+0.0488

After excluding Serving and Flux, DAGLoop retains a higher mean score than Terminus-2 on the remaining 34 tasks in all four model/reasoning settings. All 36 tasks remain in the full-suite comparison; exclusions are sensitivity analyses.

Saved GLM-5.2 reasoning Max Terminus-2 receipts identify disallowed_app_entry:eval_images_5 for a Flux trial and correctness_failed for a Serving trial. Both record timeout=false and exception=null: the Flux trial was rejected before LoRA quality evaluation, and the Serving trial before performance scoring. The Serving reference of 0.00055 averages 0 and 0.0011; the identified correctness failure explains the zero trial. These are official delivery failures, so we retain their rewards in the full-suite results and separately report performance after excluding Serving and Flux.

AuthorsOpenReview ↗

Response to reviewer CKqc [3/4]

Q6: Could additional compute explain the gains? What are the per-task LLM input/output tokens, execution or sandbox-launch counts, and wall-clock times, and how does performance compare under matched token and execution budgets?

A6: Thank you for asking us to clarify the budget and completion protocol. Terminus-2 was allowed to run until its configured research budget was exhausted or it chose to finish. We imposed no additional agent wall-clock cutoff on either system. When an agent chooses to finish, extending an external deadline alone does not cause it to resume research. Our evaluation focuses on the quality of the method returned by a completed research workflow, and on whether the closed loop between component experiments and method design improves that outcome.

Within each pair, the model, reasoning setting, official task verifier, and task-specific training and evaluation constraints are held fixed. The GLM-5.2 reasoning Max GPU comparison uses H800 hardware (Section 4.1). This specifies the completion protocol; equality of realized token and execution counts is a separate experimental control. The within-system ablations examine research-policy changes, rather than supplying a resource-matched head-to-head result.

Q7: Can system ablations compare Planner/Critic without intermediate tests, independent versus prerequisite-preserving experiments, and structured memory versus full-history context?

A7: Thank you for pointing out the need for system ablations and specifying the three comparisons. We added four policies that retain Planner/Critic. Removing intermediate feedback gives the largest observed aggregate decline, followed by independent component testing and then chronological history (Section 4.3).

The experiment uses six AutoLab CPU/CUDA systems tasks with DeepSeek-V4.1-Flash and reasoning Off. We selected these six tasks for their mid-range scores and ran each policy twice per task. All policies reuse the same completed literature retrieval and baseline reproduction for each task. From that fixed preparation snapshot, each policy runs the subsequent experimentation and method-revision workflow twice; the table reports the mean of the two final scores. The full-system runs belong to this separate ablation cohort. All four policies use the same configured budget limits as the main experiments.

  • A: Full uses prerequisite-preserving experiments and structured memory.
  • B: Independent tests components from a common baseline without retaining ancestor changes or their acceptance prerequisites.
  • C: History retains DAG experiments and the evidence ledger, but replaces the model-facing structured memory view with chronological observations.
  • D: Whole-only retains Planner/Critic and revises complete candidates without intermediate component experiments or property checks.
TaskA: FullB: IndependentC: HistoryD: Whole-only
Adaptive Compression0.60420.56380.58520.5495
AES-128 CTR0.63810.61200.61800.6029
Hash Join0.50310.46630.48840.4566
Flash Attention0.44430.40670.44300.3968
NTT CUDA0.54170.48810.50020.4864
MSM CUDA0.49310.46470.46950.4542
Mean (6)0.53740.50020.51730.4911
  • Intermediate feedback: A versus D compares full DAGLoop with Planner/Critic plus whole-candidate feedback. D removes intermediate component experiments and property checks together; its mean is lower by 0.0464.
  • Prerequisite preservation: A versus B changes whether component controls retain ancestor changes and acceptance prerequisites; the mean decreases by 0.0372.
  • Memory presentation: A versus C replaces the structured view with chronological observations while retaining DAG experiments and the evidence ledger; the mean decreases by 0.0201.

Differences use unrounded means. C is chronological experimental history under the same context window and configured presentation ceiling as A; actual trajectories and prompt lengths can differ. The result therefore compares the structured view with bounded chronological history. C receives chronological attempts, metrics, and failures together with baseline observations and the current best result; the underlying evidence ledger is retained.

Thank you again for specifying these ablations. We have added the four-policy comparison above.

Reviewer_EiYHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents DAGLoop, an automated research system that turns design rationales into testable expectations and organizes experiments around a dependency DAG. It preserves prerequisites when testing individual changes and stores experimental evidence with its implementation context to guide revision. The system is evaluated on executable research tasks spanning program optimization and model development.

Critical strengths

I liked the distinction between checking an intermediate property and requiring an immediate task-level improvement. This gives the system a reasonable way to retain prerequisites that would otherwise be discarded too early. Recording the implementation and prerequisite context alongside each result is also useful.

Critical weaknesses

My main concern is that the experiments do not isolate the proposed mechanisms. The comparison with Terminus-2 changes planning, implementation review, experiment scheduling, and memory together, so it is hard to tell what explains the gains. Section 5 acknowledges this limitation. Ablating dependency-preserving experiments and structured memory separately, within the same implementation and under matched budgets, would provide much stronger support for the paper’s central claims.

The component acceptance rule may prevent the system from finding useful combinations. Equation (8) requires positive improvement for a performance component, and Section 3.5 evaluates the complete candidate only after all components are accepted. Some changes are harmful on their own but beneficial together, even without an obvious prerequisite relationship. Grouping decisions can address this, but the paper does not show how the planner discovers a missed interaction or recovers from an unhelpful decomposition in an actual task.

The validity of the experimental specifications still depends heavily on the language model’s judgment. Graph checks can establish acyclicity and consistency with declared dependencies, but they cannot establish that the dependencies are complete or that an intermediate observable meaningfully tests the rationale. The critic handles these questions, yet its reliability is not evaluated. A property check could pass without establishing a useful prerequisite for downstream improvement. I would like to see actual specifications and an independent assessment of whether their tests support the conclusions recorded in memory.

Questions for authors

How much does performance change when dependency-preserving experiments and structured memory are removed separately under the same research budget?

Can the authors show a task where the initial decomposition blocks a beneficial combination, and explain how the system detects the interaction and revises the grouping?

Have the generated dependencies and property checks been independently reviewed? How often does the critic accept a test that passes numerically but does not adequately test the stated rationale?

4 comments
AuthorsOpenReview ↗

Response to reviewer EiYH [1/3]

Q1: Can within-system ablations isolate prerequisite-preserving experiments and structured memory from the other workflow changes? How much does performance change when each is removed under the same research budget?

A1: Thank you for pointing out the need to distinguish the sources of the improvement. We have added ablations that separately change prerequisite preservation, memory presentation, and intermediate experimental feedback while retaining Planner/Critic. Removing prerequisite preservation produces a larger mean decline than replacing the structured memory view with bounded chronological history (Section 4.3).

The experiment uses six AutoLab tasks with DeepSeek-V4.1-Flash and reasoning Off. We selected these six tasks for their mid-range scores and ran each policy twice per task. All policies reuse the same completed literature retrieval and baseline reproduction for each task. From that fixed preparation snapshot, each policy runs the subsequent experimentation and method-revision workflow twice; the table reports the mean of the two final scores. The full-system runs belong to this separate ablation cohort. All four policies use the same configured budget limits as the main experiments; each runs until its budget is exhausted or its workflow concludes on its own.

  • A: Full uses prerequisite-preserving experiments and structured memory.
  • B: Independent tests components from a common baseline without retaining ancestor changes or their acceptance prerequisites.
  • C: History retains DAG experiments and the evidence ledger, but replaces the model-facing structured memory view with bounded chronological observations.
  • D: Whole-only retains Planner/Critic and revises complete candidates without intermediate component experiments or property checks.
TaskA: FullB: IndependentC: HistoryD: Whole-only
Adaptive Compression0.60420.56380.58520.5495
AES-128 CTR0.63810.61200.61800.6029
Hash Join0.50310.46630.48840.4566
Flash Attention0.44430.40670.44300.3968
NTT CUDA0.54170.48810.50020.4864
MSM CUDA0.49310.46470.46950.4542
Mean (6)0.53740.50020.51730.4911

The full mean is 0.5374, compared with 0.5002 for independent component tests and 0.5173 for chronological history: decreases of 0.0372 and 0.0201. Whole-only feedback gives 0.4911, a decrease of 0.0464. Differences use unrounded means. The comparison holds preparation and backbone fixed while allowing each policy's research trajectory to develop under its own workflow.

AuthorsOpenReview ↗

Response to reviewer EiYH [2/3]

Q2: Could the positive-gain rule in Equation (8) discard individually harmful changes that help jointly? Can a concrete task show how a blocking decomposition is detected and revised to test the combination?

A2: Thank you for requesting concrete evidence about recovery from an unhelpful decomposition. Equation (8) applies to a node that may contain jointly tested decisions, so its criterion does not require every decision in a group to improve individually. Following your request, we examined the historical runs and traced two recorded examples in which diagnostic feedback led to revised grouping and revalidation.

For a group S={a,b}, the comparison is J(P+a+b)>J(P), with external prerequisites retained in P. Neither a nor b must first pass an individual performance test, and grouping does not require a pre-existing prerequisite edge between them (Section 2.2).

A failed local test can prompt formal replanning or regrouping within the revision budget; the original failed component does not have to pass before that revision can occur.

FML-Bench: federated learning with PFLlib. In a recorded SCAFFOLD run, the initial plan split zero initialization, local gradient correction, client-state updates, and server aggregation into separate nodes. The correction node could only read zero control variates, while the updates needed to make the correction effective were downstream and outside its allowed scope. The correction term therefore remained zero. The recorded diagnosis identified this blocking boundary and requested regrouping. Subsequent plans combined local training, state updates, aggregation, and redistribution into a jointly tested feedback-loop node. Further optimization introduced damping and produced a positive component-level result.

StageRecorded observation and action
Initial local testA faithful correction implementation scored 0.450728 against a control of 0.451235 and failed the positive-gain test.
Diagnosis and regroupingThe diagnosis identified zero control variates and requested regrouping; subsequent plans combined the complete feedback loop into one node.
First joint testThe jointly implemented loop scored 0.428602 against 0.452524, so search continued.
Revised joint methodA formal revision introduced a correction scale of 0.05; an accepted performance node reached 0.491369 against 0.452400.

The last row is a component-level improvement of approximately 3.90 percentage points after regrouping, damping, and implementation revision.

AutoLab: MSM Pippenger (msm_pippenger_bls12_381_cuda). The plan separated bucket accumulation from the final Jacobian-to-affine conversion. A combined implementation measured 68.834396 ms against a 533.971741 ms control, but implementation review marked it drifted because the conversion belonged to a separately scheduled downstream node. Method feedback identified the conversion as necessary for producing the required output and requested a formal revision of the experimental boundary. The approved plan merged affine-invert into pippenger-buckets, retaining the field and point-operation prerequisites. Subsequent component revalidation passed with 56.885601 ms against its matched 532.681458 ms control.

Both examples concern component-level validation. The recorded sequence makes explicit what blocked the test, which grouping changed, and which measurements supported subsequent acceptance.

AuthorsOpenReview ↗

Response to reviewer EiYH [3/3]

Q3: How are incomplete dependencies and property checks that fail to test the stated rationale identified? Have generated specifications been independently reviewed, and how often does the Critic accept a numerically passing but inadequate test?

A3: Thank you for asking about the validity of the tests and the Critic's reliability. The saved records show specification and evidence problems being detected, corrected, and rechecked. Smallest Game Player and IFEval reached approved plans after Critic-requested revisions, while Adaptive Compression reached component qualification after execution-stage review required the missing probe evidence.

We audited saved plan reviews from two DeepSeek-V4.1-Flash cohorts, each covering the same 36 AutoLab tasks. The records show the Critic returning plans for revision when the proposed observable failed to test the declared property or when a test consumed an undeclared dependency. At least one substantive plan return occurred in 24/36 and 21/36 task runs, respectively (45/72 overall); review-response and protocol failures were not counted as substantive returns. This internal audit counts plan returns; the false-acceptance rate among Critic-approved, numerically passing tests remains unmeasured.

Plan review checks the meaning of the test. In Smallest Game Player, the proposed feature node used rule_top1_fraction >= 0.30. The Critic noted that this heuristic-agreement threshold could accept incorrect feature extraction and reject a correct extractor. The review required a check of the node's declared intermediate properties. In the subsequent approved plan, the feature check became measured_feature_contract_failures == 0, explicitly counting nonfinite features, incorrect legality masks, input modification, and nondeterministic outputs. The heuristic-agreement fraction remained a reported diagnostic rather than acceptance evidence for downstream usefulness.

Dependency review checks what the test actually consumes. In IFEval, a proposed source-only attribution check required the official candidate's readout while declaring no prerequisite. The Critic identified this undeclared consumed output. The approved plan explicitly depended on if-source-stratified-select and checked that both readouts were finite and carried matching protocol fingerprints.

A numerical pass does not automatically qualify the observation. In Adaptive Compression, the frozen run-length component specification required two assertions on visible seed-2024 data: positive leave-one-out improvement in bits per byte (bpb), and mean mixture weight above 1/3. Its scalar failed_check_count counted failures of those assertions.

RecordSaved result
Numerical checkfailed_check_count=0, with the numerical verdict met; reported leave-one-out improvement was 0.007863716 bpb and mean weight was 0.999961544.
Execution-stage comparison reviewinsufficient_evidence: the captured wrapper did not supply the probe source needed to establish calls to the actual candidate class.
Component qualification and memoryThe contract failed; memory preserved the attempt as execution_failed, with contract_passed=false.
Subsequent attemptReview traced the captured probe to imports of the candidate's own _RunLengthModel and its predict()/update() calls. The review passed, and the component and memory recorded qualified, with contract_passed=true.

The retained candidate checkpoint matches the candidate hash recorded by the successful attempt. The probe uses reference models for the other mixture members, so this observation concerns the run-length component in that stated test context. Final task improvement remains a separate complete-candidate evaluation.

AuthorsOpenReview ↗

Welcome for more discussions

Dear Reviewer #EiYH,

Thank you very much for your time and constructive comments. Your main questions concerned whether the proposed mechanisms explain the gains, how a blocking decomposition can be revised, and how the generated tests are checked. For your convenience, we summarize our responses below:

  1. Mechanism isolation. We added six-task ablations with shared preparation, Planner/Critic, and configured budget limits. The full system has a mean score of 0.5374, compared with 0.5002 for independent component tests and 0.5173 for chronological history, supporting the contributions of prerequisite preservation and structured memory presentation.

  2. Recovery from a blocking decomposition. We clarified that Equation (8) can assess jointly tested decisions without requiring each to improve individually. Recorded PFLlib/SCAFFOLD and AutoLab MSM trajectories show how diagnostic feedback identified a blocking component boundary, prompted regrouping, and led to successful component revalidation.

  3. Specification and evidence checks. We provided saved reviews showing the Critic identifying an inadequate observable in Smallest Game Player and an undeclared dependency in IFEval. The Adaptive Compression trajectory separately shows execution-stage review withholding qualification despite a numerical pass, then accepting the component after the missing probe evidence was supplied.

As the discussion period approaches its end, would you kindly review our responses and let us know whether any questions remain? We would be happy to provide further clarification and welcome further discussion.

Best regards and thanks,
The Authors of Submission 19430

Comments on the paper

1 comment
Area_Chair_26YBOpenReview ↗

Please review the authors’ responses and join the discussion

Dear Reviewers,

The authors have responded to all reviews. Please read their responses and the other reviewers’ comments, then post a brief follow-up indicating which concerns, if any, remain and whether the responses have changed your assessment. If your concerns have been resolved, a short confirmation would also be helpful.

Thank you for your time and for contributing to the discussion.

Best regards, Your AC

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.