- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies whether the effect of replacing one agent in a multi-agent workflow can be predicted before running the modified workflow. In a controlled sequential system, the authors find that an agent’s historical performance provides a useful prediction baseline, while a small number of ordered pairwise relationships between agents add most of the remaining predictive value. They call this pattern low-order replaceability.
The authors validate the finding through more than 200,000 workflow executions, testing both predictions on unseen replacements and replacement choices on a separate evaluation set. Adding agent relations raises the correlation between predicted and observed effects from 0.552 to 0.643; nine selected relations retain nearly the performance of all 90. Using these signals to choose replacements increases average workflow success from 46.86% to 53.17%.
Critical strengths
-
The paper separates model fitting, replacement prediction, and actual replacement decisions across different task sets. This makes the improvement in workflow success a meaningful result on unseen tasks. It shows that patterns learned from historical workflows can guide useful changes to new workflows, which is the paper’s most convincing contribution.
-
The authors also take care to assess execution noise and check whether their findings hold under different analysis choices. These checks are particularly valuable for a paper built around statistical patterns. They make the results easier to assess and give a clear basis for addressing the remaining concerns.
Critical weaknesses
-
The claim that low-order information is sufficient has not been tested against a model that can use the full workflow sequence. The reported comparisons mainly use additive models with hand-crafted features; even the positional model cannot learn interactions between agents. A sequence-aware predictor might capture information these models miss. Please compare against one interaction-capable predictor using the same data and splits, and estimate the prediction ceiling set by execution noise. This would show how much unexplained signal remains beyond the reported correlation of 0.643.
-
The prediction results also lack uncertainty estimates. Several differences used to rank the models are small, including the correlations for richer task features, all pairwise relations, and the selected relations. With only 100 workflow seeds drawn from shared tasks, it is unclear whether those differences are reliable. Please report paired, task-clustered confidence intervals for differences in correlation and prediction error, especially between the selected-relation and full-relation models.
Questions for authors
The paper uses a single shared backbone in every setting, with one DeepSeek-V4-Flash row in Table 11 that has no confidence interval. Could you add one or two more backbones?
5 comments
Response to Reviewer cpsa (2/3): W1 (Interaction-Capable Predictors and Noise Ceiling)
W1: Comparison with predictors that can use the full sequence or interactions, and the prediction ceiling set by execution noise
On MATH, your expectation partly holds: predictors that model interactions do recover signal the additive model misses. This signal comes from two low-order inputs, agents and ordered pairs; adding the full sequence or positions brings no further information. All predictors use the same training tasks and test replacements as the original paper.
Predictor comparison. The table below matches Table 2 in the paper: r is reported on all replacements / the above-noise subset, and Δr is the paired difference from the additive model on all replacements.
| Predictor | Input | MATH r | MATH Δr | MMLU-Pro r | MMLU-Pro Δr |
|---|---|---|---|---|---|
| Additive (Full Relation) | agents + ordered pairs | 0.531 / 0.643 | — | 0.479 / 0.586 | — |
| XGBoost | agents + ordered pairs | 0.819 / 0.849 | +0.289 [+0.183, +0.326] | 0.422 / 0.572 | −0.057 [−0.158, +0.071] |
| MLP | agents + ordered pairs | 0.782 / 0.817 | +0.251 [+0.148, +0.294] | 0.531 / 0.639 | +0.052 [−0.039, +0.120] |
| MLP | full sequence | 0.687 / 0.743 | +0.156 [+0.063, +0.211] | 0.433 / 0.553 | −0.046 [−0.155, +0.102] |
(MATH: 200 replacements, 100 above noise; MMLU-Pro: 100 replacements, 41 above noise.)
- The full sequence adds no information in either setting: The same MLP performs worse with the full sequence than with agents and ordered pairs alone (MATH, all replacements: Δr −0.095 [−0.120, −0.047]; MMLU-Pro: 0.433 vs. 0.531).
- Positions and interactions: You noted that the positional model cannot learn interactions. Giving XGBoost positions together with counts and pairs yields r = 0.818 / 0.850, the same as without positions (Δr −0.001 [−0.012, +0.010]) (App. F.3).
- On MATH, the gain comes from a low-order solver saturation effect: The first solver-type agent matters most, and additional solvers add little. When interactions are allowed only among solver features, XGBoost retains 86% of its gain over the additive model; forbidding these interactions removes 59% of the gain (App. F.4). This structure can be expressed with agent counts and their pairwise combinations alone. Six of our nine stable relations are exactly the negative solver–solver relations, which approximate the same structure additively.
- On MMLU-Pro: Replacement effects are noisier here, and the training data contain little learnable interaction; the flexible predictors perform on par with the additive model (App. F.5).
Noise ceiling. We estimate the reliability ρ of the observed effects by variance decomposition over three repeated runs; the ceiling on r is √ρ. Parentheses give the share of reliable variance explained by each model, r²/ρ (App. F.6, Table 19):
| MATH all 200 | MATH above-noise | MMLU-Pro all 100 | MMLU-Pro above-noise | |
|---|---|---|---|---|
| Reliability ρ | 0.816 [0.69, 0.88] | 0.908 [0.84, 0.94] | 0.476 [0.20, 0.63] | 0.798 [0.69, 0.87] |
| Ceiling √ρ | 0.90 | 0.95 | 0.69 | 0.89 |
| Full Relation | 0.531 (35%) | 0.643 (45%) | 0.479 (48%) | 0.586 (43%) |
| Stable / Top-9 Relation | 0.540 (36%) | 0.641 (45%) | 0.467 (46%) | 0.637 (51%) |
| XGBoost (agents + pairs) | 0.819 (82%) | 0.849 (79%) | 0.422 (37%) | 0.572 (41%) |
| MLP (full sequence) | 0.687 (58%) | 0.743 (61%) | 0.433 (39%) | 0.553 (38%) |
- How much signal remains beyond 0.643: On the above-noise subset of MATH, 0.643 explains 45% of the reliable variance, leaving 55%. XGBoost, with the same low-order inputs, explains another 34 percentage points of it (45% → 79%), leaving 21%; the MLP on the full sequence reaches only 61%.
- MMLU-Pro: The relation models explain 43–51%, and the flexible predictors do not differ significantly from them (all Δr CIs in Table 2 include 0).
- Decisions still use the Stable Relation Model: Each of its terms corresponds to a specific agent or relation, so one can see directly how low-order signals drive replacement decisions. Its explained share is also more stable across the two settings (MATH 36%, MMLU-Pro 46%; XGBoost 82% and 37%).
(Answers to W2, Q1, and the presentation issues follow in the next comment.)
Response to Reviewer cpsa (3/3): W2, Q1, Scope, and Presentation
W2: Paired, task-clustered CIs for model differences
The 9 selected relations show no detectable difference from all 90 in r, MAE, or RMSE; the improvement of the relation model over Agent Performance is significant in r and RMSE.
Table 1 now reports paired Δr and CIs for each information type relative to Agent Performance, on both all 200 replacements and the above-noise subset. The bootstrap resamples tasks as clusters, which accounts for the correlation among workflows that share tasks. The differences you highlighted are below (MAE and RMSE are reductions in pp; positive is better):
| Comparison | Subset | Δr | MAE reduction | RMSE reduction |
|---|---|---|---|---|
| Stable vs. Full | above-noise | −0.001 [−0.023, +0.022] | −0.11 [−0.34, +0.12] | −0.07 [−0.26, +0.16] |
| Stable vs. Full | all 200 | +0.009 [−0.014, +0.031] | +0.25 [−0.03, +0.32] | +0.14 [−0.05, +0.27] |
| Stable vs. AP | above-noise | +0.089 [+0.030, +0.130] | +0.19 [−0.10, +0.64] | +0.89 [+0.36, +1.21] |
| Stable vs. AP | all 200 | +0.105 [+0.043, +0.140] | +0.00 [−0.24, +0.31] | +0.48 [+0.11, +0.69] |
| Function-Type Richness vs. AP | above-noise | +0.065 [+0.030, +0.091] | +0.07 [−0.04, +0.32] | +0.50 [+0.25, +0.65] |
| Stable vs. Function-Type Richness | above-noise | +0.024 [−0.037, +0.076] |
- The 9 selected relations show no detectable difference from all 90: For all three metrics, the CIs of the differences include 0 on both subsets.
- Relations significantly improve r and RMSE over Agent Performance. On all 200 replacements, MAE is unchanged: half of these replacements have effects within execution noise, and these near-zero replacements dominate the MAE.
- Richer task features: Among Position, MATH Type, Agent Richness, Stage, Functional Composition, and Function-Type Richness, only Function-Type Richness improves significantly over Agent Performance, and it does not differ significantly from the Stable Relation Model.
Q1: Could you add one or two more backbones, and a CI for the DeepSeek-V4-Flash row?
We added two backbones, gpt-6-luna and gpt-6-sol. As with DeepSeek-V4-Flash, each backbone is evaluated on 20 OMNI-MATH test tasks × 10 workflows, with test tasks disjoint from training tasks; the models are estimated from 500 (DeepSeek-V4-Flash), 2,000 (gpt-6-luna), and 1,000 (gpt-6-sol) historical executions (App. E.1). Success-rate gains (pp):
| Backbone | Agent − No Repl. | Relation − No Repl. | Relation − Agent |
|---|---|---|---|
| DeepSeek-V4-Flash | +3.5 [−2.5, +9.0] | +4.5 [−0.5, +9.5] | +1.00 [−4.50, +6.50] |
| gpt-6-luna | +6.5 [+2.0, +11.5] | +7.0 [+2.0, +12.0] | +0.50 [−3.50, +4.00] |
| gpt-6-sol | +6.0 [+1.0, +11.0] | +7.5 [+3.5, +13.0] | +1.50 [−2.00, +5.50] |
- On all three backbones, replacements chosen by both models raise the success rate; on the two new backbones, both gains have CIs that exclude 0. The Relation − Agent gain is positive on all three backbones.
- Each backbone has only 200 test executions, so the Relation − Agent CI of a single setting is wide. Table 11 now reports sample sizes and CIs for all 8 settings: the gain is positive in all 8, there is no detectable heterogeneity across settings (τ² = 0), and the pooled gain is +1.19 pp [+0.61, +1.76], and still +0.92 pp [+0.17, +1.67] without the main experiment.
Scope: The new backbones test whether the conclusions hold across models. Within a workflow, agents still share one backbone; workflows that mix different models are left for future work.
Presentation: We unified the terminology (above-noise subset, Full / Stable Relation Model) and replaced the original Figure 2 with main-text Table 1, which lists the values and CIs for each information type directly.
Thank you again for your review! We hope these additions address your concerns, and we would be glad to discuss any further questions.
Response to Reviewer cpsa (1/3): Summary of New Results and Revisions
Thank you very much for your careful and insightful review! You identified the decision gains on unseen tasks as the paper's most convincing contribution and noted our treatment of execution noise; this is very encouraging to us.
Your questions were a great inspiration to us, and we ran additional experiments and analyses. To save your time, we first summarize the most important new results:
Following your suggestion, we trained predictors that can model interactions and the full workflow sequence (XGBoost and MLP) on the same data and splits, and estimated the prediction ceiling set by execution noise. All hyper-parameters were selected by grouped cross-validation on the training tasks only. The results provide further evidence that replacement effects are low-order:
Your question about how much signal remains beyond 0.643 showed us the limits of the additive model, and also showed that the remaining signal lies in low-order information as well. Our central claim is that the predictable information lies in low-order signals, and the additive relation model is one simple, interpretable way to use them.
We answer your questions in turn in the following comments. All intervals are 95% task-cluster bootstrap CIs.
(Our answer to W1 follows in the next comment.)