acceptodds
Under review as a conference paper at ICLR 2027

Sprune: Training-Free Credit-and-Act for Agentic Workflows

16 views

Abstract

Agentic workflows are graphs of operators, prompts, and edges that pass drafts between operators. Automated methods improve such workflows mostly by searching over candidate graphs and returning one, which leaves open how much each component contributes and discards components that later tasks may still need. A growing number of methods instead tie actions to per-component credit, but this coupling has not been named or treated as a design space of its own. We call it credit-and-act (CrAct) and describe it by three parts: a credit estimator, an accumulation rule, and an action set. We then present Sprune, a training-free CrAct method that values operators, prompts, and edges jointly with Owen values over three a priori unions, estimated by permutation sampling and smoothed into running scores by an exponential moving average. The running score archives weak components without deleting them, restores them when their score recovers, and, in a second variant, selects prompts to rewrite and decides whether operators proposed by a language model stay. Because every action reads this score, the attribution that Sprune reports is the signal behind its decisions. Across six benchmarks in code generation, mathematical reasoning, and question answering and three open-weight backbones, Sprune is competitive with seven workflow-search methods that share its backbone. An analysis of the credit recorded in nearly a thousand runs, most of them from the hyperparameter search, shows that operators carry most of the credit, that credit rankings are stable across seeds on five of the six benchmarks, and that most operators ending with the highest credit had been archived and restored along the way. Code is available at https://anonymous.4open.science/r/sprune-iclr.

Primary area:
foundation or frontier models, including LLMs
Keywords:
agentic workflows, LLM agents, credit assignment, Owen value, Shapley value, interpretability, workflow optimization, training-free

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_cq5LOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

SPRUNE adapts agentic workflows by estimating how much each operator, prompt, and edge contributes to task performance. The authors place this approach within a credit-and-act (CRACT) framework, consisting of a credit estimator, an accumulation rule, and an action set. Their implementation estimates Owen values by permutation sampling and maintains running scores with an exponential moving average. Components with persistently low scores are archived but continue to be evaluated, so they can be restored later. Variant D also rewrites prompts and adds operators. The evaluation covers six benchmarks, three open-weight backbones, and seven competing workflow methods, alongside analyses of credit trajectories and sensitivity to configuration choices. For the 9B and 27B backbones, the reported SPRUNE scores are the highest test scores found during configuration search, a choice that substantially limits the performance comparison.

Critical strengths

I find the joint treatment of operators, prompts, and edges useful. It gives the method several distinct ways to modify a workflow: an operator can remain active while its instruction or connections change. The running scores also make archive and restore decisions easier to inspect, since the reader can follow the quantities used to make them. This is a concrete benefit of connecting attribution to adaptation. The paper would be stronger with an ablation of the three component types, but the formulation itself is worth exploring.

Keeping archived components available for later use is a sensible response to noisy early estimates. The examples show when components leave and return, making the behavior easier to understand than a final workflow diagram alone would. I also appreciate the analysis of unsuccessful runs. Variant D sometimes performs much worse than variant A, and the examples of rewritten instructions help explain what can go wrong. The authors are candid about test-based selection, missing runs, and incomplete cost measurements. These disclosures make the results easier to assess, even though the underlying experimental problems still need to be addressed.

Critical weaknesses

My main concern is the use of test scores to select configurations. On the 9B and 27B backbones, SPRUNE gets many configuration trials and reports the best test result, while each baseline uses one configuration. Appendix D.1 states that validation-selected, three-seed results are not included. The gap is large enough to matter: on 9B HumanEval, variant D reaches 80.9 at its best but has a median of 16.0. I cannot tell from the current table how well SPRUNE would perform with proper validation-based selection.

I am also concerned about Equation (6). When estimated credits sum to zero, the equation sets every individual credit to zero. Yet positive and negative contributions can cancel even when the game is not constant. This can remove useful information before the EMA and change later archive or restore decisions. Calling the operation a numerical safeguard does not account for this case. The authors should clarify whether the code uses this branch and measure its effect.

The restoration statistics are harder to interpret than the paper suggests. Scores start at zero, below the archive threshold, so some early removals are induced by initialization. Restoring these components may largely undo those removals. Without a matched comparison against irreversible pruning, the performance benefit remains unclear. There is a related gap in the interpretability claim: seeing which score triggered an action does not establish that the score faithfully measures the component's contribution.

The cost and baseline comparisons also need more detail. Coalition evaluations add many model calls, but comparable token counts for the baselines are missing. Appendix H describes benchmark adaptations and a GPTSwarm entry point that does not use the optimized-swarm runner. It is unclear which of the original optimization procedures these comparisons retain.

Questions for authors

Q1. Please report results with configurations chosen on validation data and evaluated on the held-out test set over at least three seeds. Do the main comparisons still hold when methods receive comparable tuning opportunities and compute budgets? Q2. Why does Equation (6) set all credits to zero when their sum is zero? Positive and negative contributions can cancel while intermediate coalitions still have nonzero scores. Is this branch used in the experiments, and what changes if the original estimates are kept? Q3. How much does restoration help once early archiving caused by zero initialization is avoided? A comparison with irreversible pruning and no pruning, using threshold initialization or a warm-up period, would help separate these effects. Q4. Can you check whether component interventions support the credit rankings, and whether those rankings depend on the grouping or execution rules? Please also explain which optimization steps the adapted baselines retain and report comparable costs for optimization and inference.

3 comments
AuthorsOpenReview ↗

Response to Reviewer cq5L (1/2)

We thank the reviewer for the careful and constructive review. We are especially grateful for the points on the cancellation case in Eq. 6 and on the zero initialization below τ. Appendix J now contains the matched controls that the reviewer proposed. These controls and the new analyses also helped us state several claims more precisely, as noted below. We respond to each comment in turn.

Ablation of the three component types. We agree that an ablation of operators, prompts, and edges would strengthen the paper. A dedicated per-type ablation is not part of the revision; the closest evidence comes from existing analyses. Operators receive 84% to 106% of the credit in every column, while prompts receive little (Figure 5). The frozen graphs of variant A keep on average 3.85 of four operators, but only 1.20 of four prompts and 1.95 of twelve edges (Appendix F.1). In the removal test discussed below, the player type also explains most of the relation between credit and drop (Appendix J).

Q1. Selection on test scores. Table 2 of the revision reports validation-selected configurations with three test seeds in all 18 columns, as the reviewer requested (Section 5.1). For the 9B HumanEval example raised by the reviewer, the validation-selected configuration scores 93.4±1.8, second in its column. The 36 configurations of this column range from 68.7 to 95.4 (median 89.3; Tables 2 and 8). The lowest of them is a variant D search that keeps a proposed operator whose instruction does not request a solution (Appendix F.3). This is consistent with the failure cases the reviewer mentions. Overall, Sprune ranks among the top three in 9 columns, and AgentSquare ranks first in 14. The validation selection itself also holds up. The selected configuration is at or above the upper quartile of its column in 9 of 18 columns. Selection on the test split would add a median of only 1.5 points (Appendix E).

On comparable tuning and compute, the most direct check is the default configuration, which requires no hyperparameter selection. With the default hyperparameters of the selected variant, Sprune stays within one point of the selected configuration in 13 of 18 columns (Table 7). In most columns, the comparison therefore does not depend on extra tuning; the largest difference is on MATH with Qwen3.5-9B (85.9 against 70.9). On the baseline side, the rebuilt workflows run no search, and the original implementations use their published budgets (Table 13). We report the costs of search and inference under Q4.

Q2. Eq. 6. We agree with the reviewer that this branch is used and that it discards information. Appendix A.2 now states when it applies, how often, and with what effect. An ordering adds the players one at a time, from the empty coalition to the full one. The marginal contributions along it therefore add up to v_T(P) − v_T(∅), and so do the estimates (Appendix A.2). The sum is therefore zero exactly when the full workflow scores the same as the empty one, that is, when it fails the task. As the reviewer notes, smaller coalitions can still score in this case. The branch applies in 22.7% of the updates behind Table 2. In up to 68.9% of these, some model output on the task scores above zero, so some coalition may lose its credit. In such an update, every running score shrinks by the factor 1−α, and the archive step is skipped. Variants B–D also skip the rewrite and the expansion. The branch thus never archives a player directly, but it moves all scores toward zero.

We also ran the default variant-A configuration with the raw estimates kept. Relative to the normal runs with the same seeds, the mean test score changes by +2.1 (GSM8K) and +1.6 (MBPP) with Qwen3.5-9B. With Qwen3.8-27B, it changes by −0.9 (GSM8K) and −2.4 points (MATH). The MATH decrease stems from one seed (72.6) whose final graph passes no draft to the answering operator. The other two seeds (85.8 and 87.2) exceed the normal run (84.3). Appendix N lists the branch as a limitation, and the runs with raw estimates will be added to Appendix J in the final revision.

AuthorsOpenReview ↗

Response to Reviewer cq5L (2/2)

Q3. Restoring without initialization effects. We agree that the restore statistics alone cannot separate the benefit of restoring from the effect of initializing scores at zero below τ. Appendix J therefore adds the requested comparison (Table 15, Figure 10). It runs variant A with the default configuration on DROP, HotpotQA, and HumanEval with all three LLMs. There are four settings: reversible archiving initialized at zero, as in the main runs, or at τ; irreversible archiving initialized at τ; and no archiving. Initialization at τ and at zero yields the same scores within noise on all nine cells. With Qwen3.5-9B, no archiving loses 5.7 to 8.2 points (DROP: 68.3 against 76.5). Irreversible archiving is only 0.9 to 1.4 points lower, within seed noise. With the two larger LLMs, no archiving differs by −0.6 to +1.1 points. These controls show that the gain with Qwen3.5-9B comes from archiving itself, while restoring adds little in this setting. Section 5.3 and the limitations now state this.

We also ran initialization at τ on GSM8K and MATH with the two larger LLMs. It matches the normal run on GSM8K with Qwen3.8-27B (89.9 against 90.5) and on MATH with Gemma 4 31B (93.6 against 93.9). It is lower on MATH with Qwen3.8-27B (76.1 against 84.3) and on GSM8K with Gemma 4 31B (84.5 against 91.7, one normal seed). The lower means stem from three seeds whose final graph passes no draft to the answering operator. The remaining seeds score at the level of the normal runs. These runs will also be added to Appendix J in the final revision.

Interpretability. We agree with the reviewer that knowing which score triggered an action does not by itself establish that the score measures the component's contribution. We therefore tested this directly. In a pre-registered removal test on 31 final workflows covering all benchmarks and LLMs, each live operator, prompt, and edge was removed in turn (225 removals). The median drop is 0.3 points, although 13 removals cost more than 10 points. Credit and drop correlate weakly: the mean Spearman correlation is 0.09 (95% CI [−0.03, 0.21]), against 0.01 for a placebo removal. In an exploratory analysis, ordering by player type alone gives 0.16, and the correlation within operators is 0.02. In the controls with Qwen3.5-9B, archiving on running scores shuffled within each player type performs as well as archiving on the credit (Table 15). Based on these results, the revised paper presents the credit as a record of the signal behind each decision, rather than a causal importance measure (Section 5.3, Section 6, Appendix N).

Q4. Interventions, grouping, baselines, and costs. The intervention check is the removal test described above. For grouping, we compared exact Owen and Shapley values on six small workflows (Qwen3.5-9B, three live operators). One of them has no live prompt, so the two values are identical. In four of the other five, the rankings agree exactly, and in the fifth, Kendall's τ_b is 0.55 (Appendix J). The execution rules were held fixed.

Among the baselines, AFlow runs its full search (20 rounds) on all six benchmarks. MaAS runs its own optimization and test pipeline on HumanEval, GSM8K, and MATH, and ADAS runs its search on its own DROP split (not ranked). The GPTSwarm HumanEval script, by contrast, evaluates a fixed agent, because its edge optimization appears only in scripts for other benchmarks. Likewise, the DyLAN scripts run a fixed team, because its team selection is released only for MMLU. Rebuilt cells run no search (Appendix H, Table 13).

On costs, the frozen Sprune workflow uses fewer test tokens per task than the median rebuilt baseline in all 18 columns (median factor 4.2; Table 11). Table 11 also reports search tokens per method. Search is the larger part of Sprune's cost: 0.26 to 7.0M tokens with the selected configuration, against at least 1.2 to 34.8M tokens per cell for one AFlow seed. Appendix F.4 reports the cost of the full selection over 36 configurations.

We thank the reviewer again for the suggestions behind these controls, and we hope that the revised analyses answer the reviewer's questions. We would be grateful if the reviewer would consider updating the score.

AuthorsOpenReview ↗

Follow-up: validation selection, controls, and costs

We thank the reviewer again for the review. We have uploaded the revised PDF. Configurations are now selected on the validation split, Appendix J adds the restoration controls and the removal test, and Table 11 reports token costs per method. We would be grateful to know whether our responses address the reviewer's concerns. We are happy to answer any further questions before the discussion period ends.

Reviewer_quYMOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

SPRUNE estimates the contributions of operators, prompts, and edges using sampled Owen values, accumulates these scores, and uses them to archive or restore workflow components. An extended variant also rewrites prompts and adds operators. Evaluation covers six benchmarks and three backbones.

Critical strengths

Jointly attributing performance to operators, prompts, and connections is a useful direction. Reversible component removal could also help avoid permanently discarding components based on noisy early estimates.

Critical weaknesses

The figures are poorly polished and look unfinished. Figure 4(b), in particular, has a crowded layout, small labels, and a legend overlapping the tops of the bars. The visual presentation needs substantial improvement.

The evaluation does not convincingly establish the method’s effectiveness or relevance to current agent systems. On the 9B and 27B backbones, SPRUNE reports the best test score across many configurations, while each baseline uses one configuration. This compromises the comparison, especially given the large performance variation. Moreover, the reliance on dated benchmarks such as GSM8K and MATH provides limited evidence about more complex, interactive agent workflows.

Questions for authors

How is SPRUNE intended to identify a reliable configuration in practice, given its sensitivity to hyperparameters and the absence of test outcomes during deployment?

What important capability of contemporary agents does the evaluation establish beyond improving final-answer scores on traditional reasoning benchmarks?

3 comments
AuthorsOpenReview ↗

Response to Reviewer quYM (1/2)

We thank the reviewer for the careful review, including the concrete remarks on Figure 4(b) and the practical question of how to choose a configuration without test outcomes. Below, we respond to each comment in turn and indicate where the revised paper addresses it.

Figures. We agree with the reviewer that the figures needed improvement, and the revision reworks all result figures. The credit figure (Figure 4 in the submission, now Figure 5) keeps its four panels. It now has larger labels, centered titles, and legends placed above the bars, so that no element overlaps. All other result figures have been redrawn from the revised runs. Figure 4 now follows three of the runs behind Table 2 and shows their live graphs. In addition, new figures show the main comparison per benchmark (Figure 1), the variants and hyperparameters (Figures 6 and 7), and test cost against score (Figure 9). Others show the controls (Figure 10), the average per LLM (Figure 12), and the seed stability of the credit (Figure 13).

Selection on the test split and performance variation. We agree with the reviewer that a fair comparison requires selecting the configuration without test scores. Sprune now selects its configuration on the validation split only. Table 2 reports the mean over three test seeds in all 18 columns (Section 5.1; general response). Under this protocol, the variation is small. Across the 36 configurations of a column, the interquartile range of test scores has a median of 1.4 points. It exceeds 5 points only on MATH with the two Qwen LLMs (14.4 and 10.0 points; Section 5.2, Table 8). No single hyperparameter value changes the score by one point or more relative to the column mean (Table 9). Across seeds, the standard deviation of the selected configuration is at most 3.3 points, and below one point in 11 of 18 columns (Table 2).

With this selection, Sprune ranks among the top three in 9 of 18 columns. Among methods with all six cells finished, its six-benchmark average is the second highest with each LLM (Figure 12). Only AgentSquare, which remains the strongest baseline, is higher.

Benchmarks. We use the six benchmarks of the compared methods, following AFlow. This way, all eight methods are evaluated on the same tasks and splits, and original implementations can be run where they exist (Table 13). The MATH split is restricted to Level 5, the hardest level, in four categories (605 tasks, 486 for test; Table 12). Table 12 also states one known caveat per benchmark, for example that GSM8K is near-saturated and subject to contamination. We agree with the reviewer that interactive, long-horizon agents need benchmarks of their own. The revision states this scope explicitly: Appendix N lists it as a limitation and leaves interactive, agentic benchmarks to future work.

AuthorsOpenReview ↗

Response to Reviewer quYM (2/2)

Q1. Choosing a reliable configuration without test outcomes. In practice, Sprune selects its configuration on the validation split, as Table 2 does. Only validation scores enter the selection, and the selected workflow is then frozen for the test split. The revised results indicate that this selection is reliable. The selected configuration lies a median of 0.6 points above the column median, and at or above the upper quartile in 9 of 18 columns. Selection on the test split would add a median of only 1.5 points (Appendix E). Even without hyperparameter selection, the default hyperparameters of the selected variant remain within one point of the selected configuration in 13 of 18 columns (Table 7). The largest differences are on MATH with Qwen3.5-9B (85.9 against 70.9 for the default) and MBPP with Qwen3.8-27B (74.1 against 67.7).

Q2. Capabilities beyond final-answer scores. We do not claim a new capability on interactive tasks. Beyond final scores, the evaluation documents how the workflow is built and what it costs. Each archive and restore decision reads a running score that Sprune reports, so every edit can be traced to the scores behind it (Figure 4, Appendix O). A removal test shows that this credit is a record of the decisions rather than a causal importance measure, as the paper states (Section 5.3). The edits are also reversible. The operator that ends with the highest running score had been archived and later restored in 46% to 63% of the runs, depending on the variant. Moreover, 642 of the 648 seed-0 searches restore at least one component (Section 5.3, Appendix F.1). Finally, the frozen Sprune workflow is cheaper at test time. It uses fewer tokens per task than the median rebuilt baseline in all 18 columns, by a median factor of 4.2 (Table 11, Figure 9).

These properties are aimed at agent harnesses, which are still largely tuned by hand (Section 1). Their evaluation on interactive benchmarks is left to future work (Appendix N).

We hope that the redrawn figures and the validation-based results resolve the reviewer's main concerns, and we would appreciate it if the reviewer would consider revising the score in light of the revision.

AuthorsOpenReview ↗

Follow-up: revised figures and validation-based selection

We thank the reviewer again for the review. We have uploaded the revised PDF. All result figures are redrawn, and every configuration is now selected on the validation split only (Section 5.1). We would be grateful to know whether our responses address the reviewer's concerns. We are happy to answer any further questions before the discussion period ends.

Reviewer_fRSrOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper names the pattern of estimating per-component credit and editing a workflow based on it "credit-and-act" (CrAct), described by a credit estimator, an accumulation rule, and an action set. Within this frame it proposes Sprune: operators, prompts, and candidate edges are players in a cooperative game; their credit is the Owen value over three a priori unions, estimated by permutation sampling and smoothed by an exponential moving average. Players whose running score stays low are archived and restored when it recovers; Variant D additionally uses an LLM to rewrite the lowest-scoring prompt and propose new operators. Experiments on HumanEval, MBPP, GSM8K, MATH, HotpotQA, and DROP use three local 8-bit backbones (Qwen3.5-9B, Qwen3.8-27B, Gemma 4 31B-it) and compare with AFlow, ADAS, GPTSwarm, AgentSquare, DyLAN, MaAS, and HarnessX, concluding that Sprune is competitive.

Critical strengths
  1. Reversible, jointly attributed pruning. Archived players continue to be credited and can be restored, avoiding irreversible pruning, and crediting operators, prompts, and edges in one cooperative game with Owen-value unions (so like is compared with like) is a reasonable modeling choice. Code is released.
Critical weaknesses
  1. Hyperparameters are selected on the test set. For the 9B and 27B columns of Table 2, Sprune reports the best test score over 45–48 (Variant A) or 9–31 (Variant D) hyperparameter configurations, each run on a single seed, whereas each baseline runs one configuration. The authors acknowledge that "hyperparameter selection confounds the comparison in SPRUNE's favor; these cells are an upper bound", and the protocol-conformant number (a three-seed replay of the validation-selected configuration) is not reported. The only protocol-conformant column (31B, default configuration, three seeds) shows very large variance (HumanEval 79.1±15.7, HotpotQA 24.5±16.2), with GSM8K and MATH unfinished. Even with best-of-search selection, Sprune ranks first in only 4 of the 16 cells it finished, versus 7 of 18 for AgentSquare.
  2. Most baseline numbers are not produced by the baselines' own code, and the evaluation pipeline appears unreliable. According to Table 10 (Appendix H), AgentSquare and HarnessX natively support none of the six benchmarks, GPTSwarm supports only HumanEval, DyLAN only HumanEval and MATH, ADAS only DROP and GSM8K (via MGSM), and MaAS only HumanEval, GSM8K, and MATH. All other cells are "evaluated through the shared unified-task graph", which is never defined in the paper (no description of how each method is mapped, nor of the prompts or answer parsers). Every cell of AgentSquare—the strongest baseline—comes from this adapter, as do the implausibly low numbers such as HarnessX's GSM8K 3.9/4.3/4.0 and DyLAN's 5.7 on the 27B backbone. Even native cells look problematic: MaAS on HumanEval, run with its native runner at what Appendix H calls "the published budget", scores only 26.5±7.1 and 27.5±12.8, and AFlow finished only 3 of its 18 cells. All calls use a 2048-token output cap, Qwen3.5-9B has an 8192-token context, the thinking-mode setting is not stated, and MATH (Level 5, four categories) stays below 20 for every method without explanation.
Questions for authors
  1. (Additional experiment) What are the results of the protocol-conformant setting—configuration selected on the validation split, three seeds on the test split—for the 9B and 27B backbones?
  2. (Clarification) What exactly is the "shared unified-task graph"? Which prompts and answer parsers are used for each adapter-based cell?
  3. (Clarification) Is thinking mode disabled for the Qwen models? Under the 2048-token cap, what fraction of outputs are truncated or fail to parse for each method?
  4. (Clarification) Why does MaAS score 26.5 / 27.5 on HumanEval under its native runner, and why did AFlow finish only 3 of 18 cells?
3 comments
AuthorsOpenReview ↗

Response to Reviewer fRSr (1/2)

We thank the reviewer for the thorough review and for the close look at individual cells of Table 2 and of Table 10 in the submission (now Table 13). We traced every flagged cell; below, we give the cause of each and its revised value, taking the points in the order of the review.

W1. Hyperparameters selected on the test set. We agree with the reviewer that configurations should be selected without test scores. Every Sprune cell of Table 2 now follows the protocol requested in Q1 (Section 5.1, Appendix D.1). For each benchmark and LLM, we run 36 configurations: four variants, each with the nine configurations of an orthogonal array that includes the default. The configuration with the highest validation score is then rerun with three seeds, and Table 2 reports its mean test score. The test split plays no part in the selection.

Under this protocol, the variance is small. Across seeds, the standard deviation of the Sprune row is at most 3.3 points (GSM8K, Qwen3.5-9B) and below one point in 11 of 18 columns (Table 2). In the revised Table 2, the 31B cells cited by the reviewer are 98.0±0.9 on HumanEval and 62.5±0.1 on HotpotQA. With the default hyperparameters, they are 98.2±0.4 and 62.3±0.2 (Table 7). GSM8K and MATH are complete as well (91.5±3.2 and 94.4±1.6). The spread over configurations is also limited. Across the 36 configurations of a column, the interquartile range has a median of 1.4 points. It exceeds 5 points only on MATH with the two Qwen LLMs (Table 8).

AgentSquare remains the strongest baseline and ranks first in 14 of 18 columns. Sprune ranks among the top three in 9 columns and first in one (HotpotQA, Gemma 4 31B). Among methods with all six cells finished, Sprune has the second-highest six-benchmark average with each LLM (Figure 12). On this basis, the paper describes Sprune as competitive.

W2. Baseline numbers and the evaluation pipeline. We agree with the reviewer that the pipeline needed a closer look. While rerunning the experiments, we found two problems in the submitted runs, which the general response describes. Both are fixed, and all revised tables use the corrected setup.

  • Coverage. All AFlow cells use its original implementation, and in total 35 finished cells of Table 2 come from original implementations (marked *). Where no original runner exists for a benchmark, the method's own search cannot be run, so we rebuild its workflow instead (Appendix H; see Q2).
  • AgentSquare. As the reviewer points out, every AgentSquare cell uses the rebuilt workflow. This workflow runs the four module types of AgentSquare in a fixed order without its search. These cells therefore evaluate its structure rather than its optimizer, as Appendix H states. The rebuilt workflows do not appear to disadvantage the baselines, since the rebuilt AgentSquare workflow is the strongest entry in Table 2.
  • Low values. We traced these values to the evaluation, not to the methods. Under the standard scorer of Appendix I, which also executes program answers, HarnessX scores 91.2/93.6/95.7 on GSM8K (previously 3.9/4.3/4.0). DyLAN scores 83.8 on GSM8K with Qwen3.8-27B (previously 5.7). We discuss MaAS and AFlow under Q4, and the output cap, context, and thinking mode under Q3.
  • MATH below 20. These scores trace back to the 256-token cap described in the general response. Level 5 solutions are long; even at 2048 tokens, 14.6% of Sprune calls on MATH with the Qwen LLMs reach the cap. With the 2048-token cap, the Sprune and rebuilt cells on MATH score between 56 and 96. The lower original-implementation cells have separate causes (Appendix I). The initial AFlow workflow does not request the boxed format (44.9 and 53.1 with Qwen3.8-27B and Gemma 4 31B). MaAS with Qwen3.5-9B loses tasks to context overflows (Q3).
AuthorsOpenReview ↗

Response to Reviewer fRSr (2/2)

Q1. Protocol-conformant results for 9B and 27B. The validation-selected, three-seed results for these backbones are the main results of the revised paper (Table 2). With Qwen3.5-9B, Sprune ranks second on HumanEval (93.4±1.8), MATH (85.9±1.3), HotpotQA (54.5±0.4), and DROP (76.4±0.2). With Qwen3.8-27B, it ranks second on HotpotQA (64.9±0.6) and third on GSM8K (89.4±0.8) and MATH (85.0±2.5). Its weakest cells are MBPP with all three LLMs, and GSM8K and DROP with Qwen3.5-9B; on the latter two, it trails AgentSquare by 10.2 and 11.5 points. Averaged over the six benchmarks, Sprune reaches 76.6 against 82.9 for AgentSquare with Qwen3.5-9B, and 83.2 against 86.4 with Qwen3.8-27B (Section 5.2).

Q2. The "shared unified-task graph". We now call it the rebuilt workflow and define it in Appendix H. It runs the agents, operators, or modules published by the method in their published order (Table 14). AgentSquare, for example, runs planning, reasoning, tool use, and memory. Each agent makes one LLM call with a role instruction shared across all methods, and the role prompts are included in the released code. The workflow runs each validate task and then each test task once, with all operators active. The reference fields are removed before the first operator runs, and the answer is the output of the operator shown in bold in Table 14.

As for the answer parsers, a single standard scorer parses the outputs of every method (Appendix I). It takes the last number on GSM8K, and the last boxed expression, or otherwise the last number, on MATH. HotpotQA uses SQuAD-normalized F1, DROP the best F1 over the references, and HumanEval and MBPP the official tests. A final output that is a program is executed, and its printed result is taken as the answer. For original-implementation cells, the method's own scorer drives its search, and its test outputs are rescored with the standard scorer.

Q3. Thinking mode, truncation, and parse failures. Thinking is disabled for all three LLMs (Table 6); of 8.0M stored completions, 13 contain an empty think tag. Under the 2048-token cap, 14.6% of Sprune calls and 12.8% of baseline calls end at the cap on MATH with the two Qwen LLMs. The share is 0.6–0.8% on MATH with Gemma 4 31B, and 0.5% and 0.7% on the other benchmarks (Appendix I). MaAS keeps the 4096-token limit of its original implementation (Appendix H).

For parse failures, we measured from the released per-task outputs the share of test predictions from which the scorer extracts no answer. This is the case when there is no number, no boxed expression, an empty answer, or no compilable program; empty outputs count as failures. Pooled over the finished cells, the share is 0.17% for Sprune, 0.06–0.19% for AgentSquare, ADAS, DyLAN, HarnessX, and GPTSwarm, 0.75% for AFlow, and 2.94% for MaAS. The highest single-benchmark rates are MaAS on MATH, HumanEval, and GSM8K (12.9%, 9.8%, and 3.7%; Appendix H), GPTSwarm on HumanEval (3.1%), and AFlow on MATH (2.4%). ADAS DROP with Qwen3.5-9B is excluded, because its original implementation retains only its score.

Qwen3.5-9B runs with an 8192-token context. DyLAN receives 16384 tokens with this LLM, since its debate places all answers in one prompt (Appendix D.2, H). For MaAS with the same LLM, 142–160 of the 486 MATH test predictions per seed (48–56 of 1055 on GSM8K) are context-overflow errors scored as 0. Including them, its pooled share is 5.1%. We are rerunning MaAS MATH with a 32k context.

Q4. MaAS on HumanEval and AFlow coverage. The submitted MaAS runs on HumanEval, which scored 26.5 and 27.5, were affected by server timeouts. After rerunning, MaAS scores 77.4, 93.4, and 94.4, although with Qwen3.5-9B, 78 of its 393 predictions are still the output "Timeout". We show these cells but do not rank them. This is because MaAS, like AFlow, runs the docstring examples as public tests, and 295 of the 372 scoring asserts appear in them (Appendix I).

AFlow has 16 of 18 cells in the revision, against 3 in the submission. These cells run at its published budget of 20 rounds with five validation rounds (Table 13, Appendix H). We are completing the remaining two. Where AFlow scores low, the cause lies in the optimizer: with a local LLM as optimizer, many generated workflows did not execute (Appendix I). In 10 of 42 seeds, AFlow therefore kept its initial workflow. The published AFlow setup uses Claude-3.5-Sonnet as optimizer, whereas here all methods share one local LLM, so its position reflects this shared setup rather than the method itself (Appendix H).

We hope that these clarifications and the revised experiments address the reviewer's concerns, and we would be grateful if the reviewer would consider updating the score accordingly.

AuthorsOpenReview ↗

Follow-up: validation-selected results and baselines

We thank the reviewer again for the review. We have uploaded the revised PDF. Table 2 now reports the validation-selected configuration over three test seeds, and Appendix H describes how each baseline is run. We would be grateful to know whether our responses address the reviewer's concerns. We are happy to answer any further questions before the discussion period ends.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Response to all reviewers: summary of concerns and changes

We thank the reviewers for their careful and constructive reviews, and the area chair for handling our submission.

Our contributions. Many workflow-optimization methods tie their edits to per-component credit, but this coupling has not been named or studied as a design space. We call it credit-and-act (CrAct), with three parts: a credit estimator, an accumulation rule, and an action set. Sprune is a training-free CrAct method for graph-based workflows. It credits operators, prompts, and edges jointly with Owen values, accumulates the credit with an exponential moving average, and acts with a reversible action set: archive, restore, prompt rewrite, and operator expansion. Four variants share archive and restore: variant A uses only these two actions, B also rewrites prompts, C also adds operators, and D does both. To our knowledge, it is the first CrAct loop whose credit covers all three component types jointly. Because the running score that drives every action is also the reported attribution, Sprune is interpretable by construction. We evaluate it against seven published baselines on six benchmarks and three local LLMs, analyze how the credit is distributed, and release the code and raw results.

What we addressed

  • Selection on the validation split only (fRSr, quYM, cq5L).
  • Baselines run with their original implementations where they exist, and defined rebuilt workflows elsewhere (fRSr, cq5L).
  • The shared LLM setup, stated for each column (fRSr).
  • Analyses of Eq. 6, restoration, and the faithfulness of the credit, and token costs per method (cq5L).
  • Redrawn figures and a discussion of the benchmarks (quYM).

The replies below give the details and the locations in the revised paper.

Main results. Under the same LLM setup, Sprune is competitive with seven well-known workflow-optimization methods on six benchmarks and three local LLMs (Table 2). Within its graph-based family, it is ahead of GPTSwarm and MaAS in 11 of 18 columns, tied in one, and behind in six. Among methods with all six cells finished, its average is second with each LLM (Figure 12). Its frozen workflows are also cheaper at test time: they use fewer tokens per task than the median rebuilt baseline in all 18 columns (Table 11).

Fixes and code. While re-running the experiments, we found two problems in the submitted runs besides the test-split selection: an output cap of 256 tokens instead of the stated 2048 for Sprune and the baselines run in our evaluation code, and a shared baseline runner that passed the reference answer into some operator prompts. All revised tables use the corrected setup. A final check found that our harness gives each problem one generation try in the original implementations of AFlow and MaAS. Appendix H now lists the run-time guards. The anonymous repository holds the per-task outputs, raw runs, the selection record, and a script that recomputes Table 2. The remaining runs (Appendix M) will go into the final revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.