acceptodds
Under review as a conference paper at ICLR 2027

NodeGrad: Attribution-Guided Recursive Self-Improvement of Multi-Agent Workflows

Abstract

Improving a multi-agent workflow requires identifying where to intervene and whether the resulting gain justifies the cost of discovery. End-to-end feedback offers little guidance, while removing a structurally necessary node can invalidate the workflow. We introduce NodeGrad, an attribution-guided research loop for recursive workflow improvement. Calibrated, interface-preserving interventions distinguish sensitivity to degradation from upgrade potential, and joint probes reveal conditional benefits between nodes. A language-model controller uses these signals to propose edits, while external tests select versioned updates. Attribution is recomputed after accepted edits so that search follows changing workflow bottlenecks. We further evaluate controller updates through fresh research episodes with equal budgets and fixed evaluation rules. On SWE-bench Lite, repeated attribution improved the resolved rate from 66.0% to 82.3%, and the recursive controller reached 86.0%. Independent patch tests showed that attribution helped prioritize promising nodes, while budget and transfer analyses characterized the conditions under which recursive improvement remained effective. NodeGrad thus connects local diagnosis to tested workflow and controller updates with explicit accounting of research and deployment costs.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LLM agents, multi-agent workflows, attribution-guided search, workflow optimization, recursive self-improvement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mwQPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes NodeGrad, an attribution-guided framework for recursively improving multi-agent workflows under explicit research and deployment budgets. Instead of relying solely on end-to-end performance, NodeGrad uses calibrated, interface-preserving interventions to distinguish node fragility from upgrade potential and to identify beneficial interactions between nodes. An LLM-based research controller uses these signals to propose workflow modifications, which are validated by external tests. Attribution is recomputed following accepted updates, and a recursive extension further optimizes the research controller itself.

The authors evaluate NodeGrad on SWE-bench Lite. Repeated attribution improves the resolved rate from 66.0% to 82.3%, while the recursive controller achieves 86.0%. Additional experiments examine mechanism ablations, correlations between attribution scores and independent patch gains, research-budget sensitivity, controller transfer, and token costs.

Critical strengths

1. A well-motivated attribution mechanism for directing workflow optimization.

The distinction between degradation sensitivity, upgrade potential, and pairwise interaction is conceptually meaningful. Unlike node-removal approaches, interface-preserving interventions allow the authors to study improvement opportunities without invalidating the workflow structure. The independent patch experiment further supports the practical relevance of attribution-guided targeting, reporting a Spearman correlation of 0.77 between attribution scores and patch gains. Although limited in scale, this provides useful evidence that the attribution signal is not merely reproducing its own intervention outcomes.

2. A systematic evaluation of iterative improvement, recursion, and computational cost.

The paper compares unguided search, random targeting, one-time attribution, repeated attribution, and recursive controller updates. The improvements in resolved rate are substantial, and the budget-sensitivity experiments clarify that controller-level improvement is not uniformly beneficial under limited research budgets. Explicitly accounting for unsuccessful research attempts and acknowledging negative transfer are valuable aspects of the evaluation. These analyses make the paper more informative than a simple demonstration of improved benchmark performance.

Critical weaknesses

1. Insufficient specification of the statistical validation procedure and concerns about its feasibility.

The paper's candidate acceptance mechanism relies on independent confirmation data and statistically controlled improvement estimates. However, Section 4.1 reports only 23 development tasks, while Appendix A.2 describes multiple disjoint data partitions for search, candidate confirmation, and controller evaluation. Appendix A.3 introduces a conservative Hoeffding-based reference rule and acknowledges that it may be too conservative for a small repository benchmark. Nevertheless, the manuscript does not clearly specify whether this reference rule or an alternative statistical procedure was actually used in the experiments.

More importantly, the paper does not report the actual confirmation-batch sizes, error allocations, number of acceptance attempts, or observed acceptance statistics. It is therefore difficult to assess whether the proposed acceptance criteria could be reliably satisfied with the available independent evaluation data. Although the appendix describes the intended evaluation protocol and reproducibility manifest, the corresponding experimental configurations and acceptance records are not sufficiently documented. Since independently validated update selection is a central component of NodeGrad, this lack of detail materially limits confidence in the reported improvements and their reproducibility.

2. Limited evidence for robust and generalizable recursive controller improvement.

The recursive controller achieves an additional 3.7 percentage points over repeated attribution on SWE-bench Lite, but the total token usage across research and 300 deployments increases from 81.66 million to 97.14 million. More importantly, Section 4.3 reports that the updated controller improves workflow utility by 9.5 percentage points in fresh research episodes but performs 4.7 percentage points worse than the initial controller on unseen task groups. Although the authors appropriately acknowledge this negative transfer, the contrasting results raise uncertainty about whether the recursive procedure produces a broadly improved research strategy or one specialized to the development distribution.

Furthermore, the paper does not provide sufficient uncertainty estimates across independent research runs to establish the stability of these improvements. The evaluation is also restricted to SWE-bench Lite, while the independent attribution-prediction experiment examines only six nodes at a single workflow snapshot. Consequently, the evidence supports the effectiveness of attribution-guided optimization in the evaluated setting, but remains insufficient to establish robust and generalizable recursive controller improvement. This limitation is consequential because recursive self-improvement is a central motivation and claimed contribution of the paper.

Questions for authors

1. Statistical acceptance and data partitioning (clarification). Could the authors provide the actual numbers of independent tasks or repositories assigned to search, candidate confirmation, controller development, fresh-episode evaluation, and transfer evaluation? In particular, which statistical procedure was used to accept candidate edits, and what were the corresponding batch sizes, error allocations, and observed acceptance bounds? Given the 23 development tasks reported in Section 4.1, this information is essential for understanding how the acceptance criteria were satisfied.

2. Robustness and generalization (clarification). Could the authors provide uncertainty estimates and paired evaluation results across independent research runs for the improvements attributed to re-attribution and recursive controller updates? In particular, how consistent are the 3.7-point main-benchmark improvement and the 4.7-point transfer degradation across different research seeds and task groups? This would help determine whether controller improvement is robust or strongly dependent on the evaluation distribution.

Reviewer_e3FcOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Improving a multi-agent workflow requires identifying where to intervene and whether the resulting gain justifies the cost of discovery. Existing approaches use language models to construct and search executable workflows guided by end-to-end feedback. End-to-end scores provide limited localization, and removing a structurally necessary node reveals little about its upgrade value. NodeGrad measures each node’s upgrade potential through calibrated interventions and lets a language-model controller propose edits that external tests accept, while recursively updating the controller itself under equal budgets. On SWE-bench Lite the resolved rate improved from 66.0% to 86.0% with explicit accounting of research and deployment costs.

Critical strengths

The paper identifies a real limitation of current workflow improvement: end-to-end scores provide limited localization, and deletion value need not predict the benefit of improving a node. As an alternative, it proposes calibrated, interface-preserving weak and strong variants that distinguish sensitivity to degradation from upgrade potential. Attribution-guided search outperforms unguided and random-target search under the same research budget (82.3% vs. 74.7% and 72.7%), which supports the value of attribution for target selection.

Critical weaknesses

W1. The central claims are not directly tested.

  • "Deletion value need not predict the benefit of improving that node." No deletion-based or leave-one-out attribution is included as a baseline. The comparisons are only against unguided LLM search and random-target search, which show that attribution helps target selection but not that upgrade/degradation attribution is better than deletion-based attribution.
  • "Structural necessity, sensitivity to degradation, and benefit from an upgrade are different properties." The paper does not report whether fragility Di,tD_ {i,t} and upgrade potential Ui,tU_ {i,t} produce different node rankings, and Di,tD_ {i,t} is not used in any reported experiment.
  • "Independent patch tests showed that attribution helped prioritize promising nodes." The evidence is a Spearman correlation of 0.77 over six nodes on a single workflow snapshot. This is not significant at the 0.05 level (two-sided critical value ≈ 0.886 for n = 6), and the highest-ranked node (Localizer) was not the best patch target.
  • "Joint probes exposed opportunities that single-node measurements missed." The only support is a +4.0-point difference in one comparison. No values of Iij,t+I^+ _{ij,t} or examples of synergistic pairs are reported.

W2. The evaluation protocol and statistical support are insufficient.

  • Data partitions. Sec. 4.1 states that Lite contains 300 test and 23 development tasks, and that search, confirmation, controller evaluation, and repository-held-out transfer use disjoint task groups while the test set remains hidden during search. Twenty-three tasks cannot support these partitions, and the source of the remaining data is not specified.
  • Acceptance rule. Under the confirmation rule in App. A.3, the reward-bound radius is about 0.57 for mj=23m_ j = 23, even with a single attempt at αj=0.05\alpha_ j = 0.05. It is therefore unclear how any candidate was accepted, and the paper does not state which procedure was actually used.
  • Variance. Each method appears to be a single run, with no seeds or confidence intervals reported. The main comparison, 86.0% vs. 82.3%, corresponds to 258 vs. 247 resolved tasks, and in Fig. 5(b) the difference between the recursive controller and repeated attribution ranges from −3.0 to +3.7 points across budgets.
Questions for authors
  1. Recursive controller. On unseen task groups, aTa_ T achieved 72.8% compared with 77.5% for a0a_ 0. Do the authors have evidence that controller updates improve research ability beyond the development task distribution? Results on another benchmark or workflow topology would help assess the "recursive self-improvement" claim in the title and abstract.

  2. Cost and baselines. The recursive controller improves on the uniform-strong reference by 6.0 points while using 97.14M vs. 33.57M total tokens.

    • Could the authors report the uniform-strong reference given the same research budget, and at least one of ADAS, AFlow, or DSPy under the same node contracts?
    • Could the authors report price-weighted cost, since tokens are summed across models of different prices?
    • Which edits were accepted (node, edit type, final model routes)? If most accepted edits are model-route upgrades, how does the method differ from incrementally approaching the uniform-strong reference?
Reviewer_4d8POpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes NodeGrad, an attribution-guided framework for optimizing multi-agent workflows. It uses calibrated node interventions to estimate degradation sensitivity, upgrade potential, and pairwise interactions, guiding an LLM-based controller to propose and evaluate workflow modifications. The framework also supports recursive improvement of the research controller itself.

Experiments on SWE-bench Lite report improvements from 66.0% to 82.3% with repeated attribution and 86.0% with recursive controller optimization. Additional analyses examine attribution effectiveness, search efficiency, controller transfer, and token costs. However, concerns remain regarding the statistical feasibility and reproducibility of the evaluation protocol.

Critical strengths
  1. It recognizes that removing a structurally necessary workflow component is not an appropriate proxy for determining whether that component is worth improving. The proposed use of interface-preserving interventions directly addresses this issue.

  2. It compares unguided search, random targeting, one-time attribution, repeated attribution, and recursive controller updates rather than reporting a single optimized workflow.

Critical weaknesses
  1. The method lacks a sufficiently justified connection between attribution and actual optimization gains. NodeGrad estimates upgrade potential through predefined capability interventions, but uses these scores to guide independently generated workflow edits. The two are not necessarily correlated, and the six-node experiment in Figure 4 provides limited evidence that attribution reliably identifies actionable optimization opportunities.

  2. The costs are not adequately considered in target selection. Equation 8 evaluates candidate interventions based on performance improvement relative to additional deployment cost. However, it does not account for the research expenses associated with measuring node responses, constructing candidate edits, and testing their effectiveness. Although the paper tracks these costs separately, it remains unclear how the proposed strategy efficiently allocates the limited research budget.

  3. Limited evidence for effective recursive self-improvement. The controller-update mechanism lacks sufficient implementation detail and comparison against simpler controller optimization. More importantly, the updated controller performs 4.7 pp worse on unseen task groups, raising concerns about overfitting and the generality of the claimed self-improvement.

Questions for authors
  1. What evidence supports using predefined capability-intervention scores to predict gains from independently generated workflow edits?

  2. How does the target-selection strategy account for research costs, rather than only deployment costs?

  3. What specific controller modifications produced the reported recursive improvement, and why did performance deteriorate on unseen tasks?

  4. How were the development tasks partitioned, and what statistical procedure was actually used to accept workflow updates?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.