- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper argues that agent selection in collaborative multi-agent systems should not be based solely on individual capabilities, as some weaker agents may achieve better performance when working together, depending on whether their capabilities are redundant. The paper trains a model to predict team performance by sampling different agent combinations and using their execution results as training data, hoping that the model can learn the underlying features that determine whether different agents collaborate more effectively.
Critical strengths
-
The problem is meaningful, as blindly selecting the best individual agents may lead to suboptimal team performance.
-
The experiments cover multiple benchmarks and include ablation studies.
-
The method is supported by theoretical analysis, with mathematical proofs provided.
Critical weaknesses
-
The paper compares fixed token budget limits rather than actual token consumption and does not account for the cost of offline learning.
-
Performance gains are marginal on most benchmarks, with unclear statistical significance, while the substantial improvement on DS-1000 lacks explanation.
-
The paper does not report prediction errors or ranking consistency between predicted and actual team performance, making the predictor's reliability unclear.
-
Although the paper emphasizes complementarity and redundancy, it relies on a black-box predictor without explaining the learned interactions or ruling out simple role diversity as the source of improvement.
Questions for authors
-
The paper compares token consumption using fixed token budget limits and does not account for the token consumption of offline learning. I did not find a comparison of the actual token consumption across different methods, which seems somewhat unfair.
-
The method shows very small improvements on most datasets, sometimes even comparable to statistical variance, which is something I find questionable. However, it performs very well on DS-1000. I think the authors should provide a detailed explanation of what causes such a significant improvement on this particular dataset.
-
Using a small model to predict team contributions seems quite fancy. However, the paper does not report the prediction accuracy or errors of this small model, nor the consistency between the predicted rankings and actual rankings of candidate teams. Therefore, it is difficult to determine whether the small model has learned the underlying logic for evaluating teams. Furthermore, as the paper suggests, the small model may have simply learned a very basic principle of making roles as different as possible, rather than capturing complementarity and redundancy. Further investigation may be needed to understand what actually causes the performance improvements.
We thank the reviewer for the careful assessment and constructive suggestions. Below, we address the four concerns using the existing analyses and additional measurements.
W1. Realized Resource Consumption and Offline Learning Cost
We agree that configured resource limits should be complemented by actual consumption measurements. In addition to the budget sweeps in Figure 5, we report the following measured per-query costs. Relative reductions are calculated against DMoA.
These measurements demonstrate lower average online resource consumption in the measured evaluation setting. TeamRoute's average routing time is approximately 0.27% of its average end-to-end latency, indicating small routing overhead.
We additionally report an offline model-training time of approximately 2-5 minutes per dataset. Once trained, the predictor remains fixed across queries and changes in runtime budgets or preferences within the same agent pool and execution protocol.
W2. Task-Dependent Gains
We agree that the gains vary across tasks . Relative to DMoA, the gains are 12.55 percentage points on DS-1000, 2.38 on AQuA, 2.35 on SVAMP, and 1.75 on HumanEval, while other benchmarks show smaller gains.
To investigate the DS-1000 result, we extended the Random and Best Fixed comparisons to this dataset:
TeamRoute exceeds Random and Best Fixed by 19.45 and 18.00 percentage points, respectively. Thus, the strong DS-1000 result also appears against these within-framework controls. This is consistent with the interpretation that different queries benefit from different specialist combinations.
W3. Predictor Reliability and Ranking Consistency
Our additional predictor evaluation reports prediction accuracy above 90% on every evaluated dataset. The reported accuracies for HumanEval, AQuA, and MultiArith are:
These are predictor-level measurements and are distinct from the accuracy of the final answers produced by selected teams. Section 5.4 provides complementary decision-level evidence: the compared predictors score the same held-out candidate combinations under shared selection parameters, independently of OT candidate generation. The stronger selected-answer performance of the full Brain supports the practical value of its predictions for team selection.
W4. Structured Interactions and Role Diversity
The Experience Brain uses an explicit structural decomposition. Equation (4) combines context-dependent role, pairwise, and cardinality factors in the latent score space before decoding the predicted outcomes:
zθ(x,S)=∑i∈Sri(x)+∑i,j∈S,;i<jpij(x)+g∣S∣(x),y^θ(x,S)=ψ(zθ(x,S)).
The factors are learned from team-execution observations. Proposition 1 motivates the additional representational capacity of pairwise terms, while the ablations evaluate their empirical contribution.
In Table 2, removing pairwise factors reduces AQuA accuracy from 88.98% to 76.38%. More directly, Section 5.4 and Appendix C.7 compare predictors trained on the same filtered data and evaluated on the same held-out candidate teams with shared selection parameters. Full Brain improves selected-answer accuracy over the variant without pairwise factors by 3.51, 1.18, and 1.40 percentage points on HumanEval, AQuA, and MultiArith, respectively. Because the available roles and candidate teams are held fixed, this comparison supports a benefit from interaction modeling beyond simply providing a diverse role pool.