- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
SPRUNE adapts agentic workflows by estimating how much each operator, prompt, and edge contributes to task performance. The authors place this approach within a credit-and-act (CRACT) framework, consisting of a credit estimator, an accumulation rule, and an action set. Their implementation estimates Owen values by permutation sampling and maintains running scores with an exponential moving average. Components with persistently low scores are archived but continue to be evaluated, so they can be restored later. Variant D also rewrites prompts and adds operators. The evaluation covers six benchmarks, three open-weight backbones, and seven competing workflow methods, alongside analyses of credit trajectories and sensitivity to configuration choices. For the 9B and 27B backbones, the reported SPRUNE scores are the highest test scores found during configuration search, a choice that substantially limits the performance comparison.
Critical strengths
I find the joint treatment of operators, prompts, and edges useful. It gives the method several distinct ways to modify a workflow: an operator can remain active while its instruction or connections change. The running scores also make archive and restore decisions easier to inspect, since the reader can follow the quantities used to make them. This is a concrete benefit of connecting attribution to adaptation. The paper would be stronger with an ablation of the three component types, but the formulation itself is worth exploring.
Keeping archived components available for later use is a sensible response to noisy early estimates. The examples show when components leave and return, making the behavior easier to understand than a final workflow diagram alone would. I also appreciate the analysis of unsuccessful runs. Variant D sometimes performs much worse than variant A, and the examples of rewritten instructions help explain what can go wrong. The authors are candid about test-based selection, missing runs, and incomplete cost measurements. These disclosures make the results easier to assess, even though the underlying experimental problems still need to be addressed.
Critical weaknesses
My main concern is the use of test scores to select configurations. On the 9B and 27B backbones, SPRUNE gets many configuration trials and reports the best test result, while each baseline uses one configuration. Appendix D.1 states that validation-selected, three-seed results are not included. The gap is large enough to matter: on 9B HumanEval, variant D reaches 80.9 at its best but has a median of 16.0. I cannot tell from the current table how well SPRUNE would perform with proper validation-based selection.
I am also concerned about Equation (6). When estimated credits sum to zero, the equation sets every individual credit to zero. Yet positive and negative contributions can cancel even when the game is not constant. This can remove useful information before the EMA and change later archive or restore decisions. Calling the operation a numerical safeguard does not account for this case. The authors should clarify whether the code uses this branch and measure its effect.
The restoration statistics are harder to interpret than the paper suggests. Scores start at zero, below the archive threshold, so some early removals are induced by initialization. Restoring these components may largely undo those removals. Without a matched comparison against irreversible pruning, the performance benefit remains unclear. There is a related gap in the interpretability claim: seeing which score triggered an action does not establish that the score faithfully measures the component's contribution.
The cost and baseline comparisons also need more detail. Coalition evaluations add many model calls, but comparable token counts for the baselines are missing. Appendix H describes benchmark adaptations and a GPTSwarm entry point that does not use the optimized-swarm runner. It is unclear which of the original optimization procedures these comparisons retain.
Questions for authors
Q1. Please report results with configurations chosen on validation data and evaluated on the held-out test set over at least three seeds. Do the main comparisons still hold when methods receive comparable tuning opportunities and compute budgets? Q2. Why does Equation (6) set all credits to zero when their sum is zero? Positive and negative contributions can cancel while intermediate coalitions still have nonzero scores. Is this branch used in the experiments, and what changes if the original estimates are kept? Q3. How much does restoration help once early archiving caused by zero initialization is avoided? A comparison with irreversible pruning and no pruning, using threshold initialization or a warm-up period, would help separate these effects. Q4. Can you check whether component interventions support the credit rankings, and whether those rankings depend on the grouping or execution rules? Please also explain which optimization steps the adapted baselines retain and report comparable costs for optimization and inference.
3 comments
Response to Reviewer cq5L (2/2)
Q3. Restoring without initialization effects. We agree that the restore statistics alone cannot separate the benefit of restoring from the effect of initializing scores at zero below τ. Appendix J therefore adds the requested comparison (Table 15, Figure 10). It runs variant A with the default configuration on DROP, HotpotQA, and HumanEval with all three LLMs. There are four settings: reversible archiving initialized at zero, as in the main runs, or at τ; irreversible archiving initialized at τ; and no archiving. Initialization at τ and at zero yields the same scores within noise on all nine cells. With Qwen3.5-9B, no archiving loses 5.7 to 8.2 points (DROP: 68.3 against 76.5). Irreversible archiving is only 0.9 to 1.4 points lower, within seed noise. With the two larger LLMs, no archiving differs by −0.6 to +1.1 points. These controls show that the gain with Qwen3.5-9B comes from archiving itself, while restoring adds little in this setting. Section 5.3 and the limitations now state this.
We also ran initialization at τ on GSM8K and MATH with the two larger LLMs. It matches the normal run on GSM8K with Qwen3.8-27B (89.9 against 90.5) and on MATH with Gemma 4 31B (93.6 against 93.9). It is lower on MATH with Qwen3.8-27B (76.1 against 84.3) and on GSM8K with Gemma 4 31B (84.5 against 91.7, one normal seed). The lower means stem from three seeds whose final graph passes no draft to the answering operator. The remaining seeds score at the level of the normal runs. These runs will also be added to Appendix J in the final revision.
Interpretability. We agree with the reviewer that knowing which score triggered an action does not by itself establish that the score measures the component's contribution. We therefore tested this directly. In a pre-registered removal test on 31 final workflows covering all benchmarks and LLMs, each live operator, prompt, and edge was removed in turn (225 removals). The median drop is 0.3 points, although 13 removals cost more than 10 points. Credit and drop correlate weakly: the mean Spearman correlation is 0.09 (95% CI [−0.03, 0.21]), against 0.01 for a placebo removal. In an exploratory analysis, ordering by player type alone gives 0.16, and the correlation within operators is 0.02. In the controls with Qwen3.5-9B, archiving on running scores shuffled within each player type performs as well as archiving on the credit (Table 15). Based on these results, the revised paper presents the credit as a record of the signal behind each decision, rather than a causal importance measure (Section 5.3, Section 6, Appendix N).
Q4. Interventions, grouping, baselines, and costs. The intervention check is the removal test described above. For grouping, we compared exact Owen and Shapley values on six small workflows (Qwen3.5-9B, three live operators). One of them has no live prompt, so the two values are identical. In four of the other five, the rankings agree exactly, and in the fifth, Kendall's τ_b is 0.55 (Appendix J). The execution rules were held fixed.
Among the baselines, AFlow runs its full search (20 rounds) on all six benchmarks. MaAS runs its own optimization and test pipeline on HumanEval, GSM8K, and MATH, and ADAS runs its search on its own DROP split (not ranked). The GPTSwarm HumanEval script, by contrast, evaluates a fixed agent, because its edge optimization appears only in scripts for other benchmarks. Likewise, the DyLAN scripts run a fixed team, because its team selection is released only for MMLU. Rebuilt cells run no search (Appendix H, Table 13).
On costs, the frozen Sprune workflow uses fewer test tokens per task than the median rebuilt baseline in all 18 columns (median factor 4.2; Table 11). Table 11 also reports search tokens per method. Search is the larger part of Sprune's cost: 0.26 to 7.0M tokens with the selected configuration, against at least 1.2 to 34.8M tokens per cell for one AFlow seed. Appendix F.4 reports the cost of the full selection over 36 configurations.
We thank the reviewer again for the suggestions behind these controls, and we hope that the revised analyses answer the reviewer's questions. We would be grateful if the reviewer would consider updating the score.
Follow-up: validation selection, controls, and costs
We thank the reviewer again for the review. We have uploaded the revised PDF. Configurations are now selected on the validation split, Appendix J adds the restoration controls and the removal test, and Table 11 reports token costs per method. We would be grateful to know whether our responses address the reviewer's concerns. We are happy to answer any further questions before the discussion period ends.
Response to Reviewer cq5L (1/2)
We thank the reviewer for the careful and constructive review. We are especially grateful for the points on the cancellation case in Eq. 6 and on the zero initialization below τ. Appendix J now contains the matched controls that the reviewer proposed. These controls and the new analyses also helped us state several claims more precisely, as noted below. We respond to each comment in turn.
Ablation of the three component types. We agree that an ablation of operators, prompts, and edges would strengthen the paper. A dedicated per-type ablation is not part of the revision; the closest evidence comes from existing analyses. Operators receive 84% to 106% of the credit in every column, while prompts receive little (Figure 5). The frozen graphs of variant A keep on average 3.85 of four operators, but only 1.20 of four prompts and 1.95 of twelve edges (Appendix F.1). In the removal test discussed below, the player type also explains most of the relation between credit and drop (Appendix J).
Q1. Selection on test scores. Table 2 of the revision reports validation-selected configurations with three test seeds in all 18 columns, as the reviewer requested (Section 5.1). For the 9B HumanEval example raised by the reviewer, the validation-selected configuration scores 93.4±1.8, second in its column. The 36 configurations of this column range from 68.7 to 95.4 (median 89.3; Tables 2 and 8). The lowest of them is a variant D search that keeps a proposed operator whose instruction does not request a solution (Appendix F.3). This is consistent with the failure cases the reviewer mentions. Overall, Sprune ranks among the top three in 9 columns, and AgentSquare ranks first in 14. The validation selection itself also holds up. The selected configuration is at or above the upper quartile of its column in 9 of 18 columns. Selection on the test split would add a median of only 1.5 points (Appendix E).
On comparable tuning and compute, the most direct check is the default configuration, which requires no hyperparameter selection. With the default hyperparameters of the selected variant, Sprune stays within one point of the selected configuration in 13 of 18 columns (Table 7). In most columns, the comparison therefore does not depend on extra tuning; the largest difference is on MATH with Qwen3.5-9B (85.9 against 70.9). On the baseline side, the rebuilt workflows run no search, and the original implementations use their published budgets (Table 13). We report the costs of search and inference under Q4.
Q2. Eq. 6. We agree with the reviewer that this branch is used and that it discards information. Appendix A.2 now states when it applies, how often, and with what effect. An ordering adds the players one at a time, from the empty coalition to the full one. The marginal contributions along it therefore add up to v_T(P) − v_T(∅), and so do the estimates (Appendix A.2). The sum is therefore zero exactly when the full workflow scores the same as the empty one, that is, when it fails the task. As the reviewer notes, smaller coalitions can still score in this case. The branch applies in 22.7% of the updates behind Table 2. In up to 68.9% of these, some model output on the task scores above zero, so some coalition may lose its credit. In such an update, every running score shrinks by the factor 1−α, and the archive step is skipped. Variants B–D also skip the rewrite and the expansion. The branch thus never archives a player directly, but it moves all scores toward zero.
We also ran the default variant-A configuration with the raw estimates kept. Relative to the normal runs with the same seeds, the mean test score changes by +2.1 (GSM8K) and +1.6 (MBPP) with Qwen3.5-9B. With Qwen3.8-27B, it changes by −0.9 (GSM8K) and −2.4 points (MATH). The MATH decrease stems from one seed (72.6) whose final graph passes no draft to the answering operator. The other two seeds (85.8 and 87.2) exceed the normal run (84.3). Appendix N lists the branch as a limitation, and the runs with raw estimates will be added to Appendix J in the final revision.