- Rating
- 2
- Confidence
- 2(−1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TopoPAG, which learns task-specific preferences from the execution outcomes of candidate communication topologies and uses them to guide topology generation in LLM-based multi-agent systems. The method constructs a continuous topology representation from the learned preferences and updates it during generation to guide the prediction of agent roles and communication edges. It also uses preference-weighted training to train the topology generator. Experiments on six benchmarks show improvements over the compared methods.
Critical strengths
1.The paper addresses an interesting problem. Existing methods mainly use execution feedback to evaluate complete graphs, whereas this work uses the relative performance of different topologies to guide graph generation step by step.
2.The experimental results are consistent. TopoPAG outperforms the compared methods on all six benchmarks. The ablation studies also show the contributions of preference learning and adaptive updates. In addition, evaluations with different LLM backbones provide further evidence of the method's effectiveness.
Critical weaknesses
1.The need for the added complexity is not sufficiently established. TopoPAG improves average performance over ARG-Designer by 2.24 percentage points, with smaller gains on some benchmarks. Although the paper includes ablation studies, it lacks a direct comparison with a simpler topology selection method. As a result, it remains unclear whether the proposed generation mechanism is necessary.
2.The experimental evaluation has some gaps. The paper does not report statistical uncertainty for the main results or examine the stability of the learned topology preferences. In addition, the efficiency analysis mainly focuses on prompt-token usage and does not fully account for offline topology evaluation, training costs, or inference latency.
Questions for authors
1.How does TopoPAG compare with a simpler method that directly predicts and selects the most suitable candidate topology using the same training data? Could this method achieve similar results without continuous topology representations and adaptive generation?
2.Some improvements are relatively small, particularly on GSM8K (+0.51 percentage points) and MMLU (+1.31 percentage points). Could the authors report results across multiple runs, together with standard deviations or confidence intervals?
3.What is the total computational cost of TopoPAG, including the evaluation of 18 candidate topologies, training, and inference? How does it compare with ARG-Designer and OFA-MAS when all these costs are taken into account?
4.Are candidate topologies evaluated multiple times for each training task? Given the stochastic nature of LLM outputs, how stable are the resulting topology preferences across repeated executions?
4 comments
Author Response to Reviewer Z2d3
Q4. Topology Preference Stability
We thank the reviewer for highlighting the impact of stochastic LLM executions on preference reliability. In the original data collection, each query–anchor pair is evaluated once. To assess stability, we repeated all 18 anchor executions three times on 18 sampled development queries (3 per benchmark), totaling 972 executions with the original reward function.
We measure Pairwise Agreement (consistency of anchor-pair reward orderings across runs), Coverage (proportion of comparisons retained after excluding ties), and Top-1 Set Jaccard (overlap of highest-reward anchor sets across runs). The respective macro averages of 97.56%, 96.82%, and 94.44% indicate that both relative preferences and top-performing anchors remain largely consistent across repeated executions, supporting the reliability of the rollout-derived preference supervision. We have included the preference stability results and analysis in the new Section 4.5 (Topology Preference Analysis), with detailed metric definitions and formulas provided in Appendix E.
| Benchmark | Queries | Pairwise Agreement | Coverage | Top-1 Set Jaccard |
|---|---|---|---|---|
| MMLU | 3 | 91.68% | 97.24% | 88.89% |
| GSM8K | 3 | 100.00% | 96.73% | 100.00% |
| AQuA | 3 | 100.00% | 96.73% | 100.00% |
| MultiArith | 3 | 98.05% | 96.73% | 100.00% |
| SVAMP | 3 | 100.00% | 96.73% | 100.00% |
| HumanEval | 3 | 95.65% | 96.73% | 77.78% |
| Macro Average | 18 | 97.56% | 96.82% | 94.44% |
Thank you for your detailed responses and additional experiments. I appreciate the authors' efforts to address my concerns.
However, I still have substantial reservations about the paper's core contribution. The proposed method appears to mainly combine existing techniques, without introducing a clearly novel learning principle, architecture, or generalizable theoretical insight. Moreover, the performance improvements over strong baselines are relatively modest, while the method consumes more inference tokens than ARG-Designer. Without a comparison under matched token budgets, it remains unclear whether the gains come from the proposed mechanism or simply from additional computation. Overall, the contribution appears to be primarily engineering-oriented, and concerns remain regarding the attribution of the experimental gains and the broader scientific significance of the work. Therefore, I am still not fully convinced of the significance of the proposed approach beyond combining existing methods.
Nevertheless, I acknowledge the additional experimental evidence and the authors' efforts to address my concerns. But the novelty of this paper is insufficient, I keep score at 2. The current computational cost analysis is limited to MMLU. I would like to see cost evaluations on additional benchmarks before considering raising my score.
Author Response to Reviewer Z2d3
We sincerely thank the reviewer for their continued engagement and thoughtful feedback, and apologize for our delayed response. We hope that our additional clarifications and experimental results will help address the remaining concerns.
The distinction of TopoPAG lies in how topology design itself is conceptualized. Existing approaches primarily view topology design as finding, selecting, or optimizing effective structures. In contrast, TopoPAG takes a step further by treating topology design as a process of learning reusable, task-specific structural preferences from past execution experience and applying them to construct new topologies. Specifically, it uses the comparative outcomes of candidate topologies to learn these preferences, embeds them into a continuous topology space, and progressively updates the resulting structural guidance during graph generation. In this way, TopoPAG connects past execution experience with future topology construction, allowing learned preferences to guide the generation of new topologies. This shifts topology design from learning which structures perform well to learning how structural preferences can guide the construction of new topologies, and more broadly, from identifying the best existing topologies to learning how effective topologies can be constructed.
In addition, we have extended the cost evaluation from MMLU to all six benchmarks. The complete results are presented below.
Computational cost comparison across six benchmarks.
| Dataset | Method | Collection (M) ↓ | Inference (Tokens/Q) ↓ | Total Tokens (M) ↓ | Training (GPU-h) ↓ | Latency (s/Q) ↓ |
|---|---|---|---|---|---|---|
| MMLU | ARG-Designer | 2.55 | 2,561 | 2.94 | 0.36 | 11.20 |
| MMLU | OFA-MAS | 2.62 | 4,608 | 3.33 | 0.33 | 33.10 |
| MMLU | TopoPAG | 2.93 | 3,215 | 3.42 | 0.29 | 12.04 |
| GSM8K | ARG-Designer | 1.95 | 4,894 | 8.41 | 0.32 | 12.55 |
| GSM8K | OFA-MAS | 1.80 | 4,973 | 8.36 | 0.27 | 22.60 |
| GSM8K | TopoPAG | 2.22 | 4,146 | 7.69 | 0.34 | 11.02 |
| AQuA | ARG-Designer | 2.23 | 4,321 | 3.33 | 0.32 | 23.10 |
| AQuA | OFA-MAS | 1.82 | 4,305 | 2.91 | 0.26 | 43.14 |
| AQuA | TopoPAG | 2.64 | 5,019 | 3.92 | 0.35 | 22.49 |
| MultiArith | ARG-Designer | 1.73 | 4,418 | 4.38 | 0.33 | 10.81 |
| MultiArith | OFA-MAS | 1.54 | 4,912 | 4.49 | 0.31 | 22.21 |
| MultiArith | TopoPAG | 2.09 | 3,995 | 4.49 | 0.30 | 8.35 |
| SVAMP | ARG-Designer | 1.43 | 4,580 | 6.01 | 0.37 | 11.24 |
| SVAMP | OFA-MAS | 1.39 | 4,340 | 5.73 | 0.32 | 19.60 |
| SVAMP | TopoPAG | 1.62 | 3,991 | 5.61 | 0.33 | 7.69 |
| HumanEval | ARG-Designer | 2.49 | 2,533 | 2.91 | 0.37 | 9.75 |
| HumanEval | OFA-MAS | 3.03 | 4,898 | 3.83 | 0.33 | 26.20 |
| HumanEval | TopoPAG | 2.73 | 2,556 | 3.15 | 0.30 | 10.26 |
| Average | ARG-Designer | 2.06 | 3,884 | 4.66 | 0.34 | 13.11 |
| Average | OFA-MAS | 2.03 | 4,673 | 4.77 | 0.30 | 27.81 |
| Average | TopoPAG | 2.37 | 3,821 | 4.71 | 0.32 | 11.98 |
The results show that TopoPAG incurs moderately higher offline collection costs (2.37M vs. 2.06M tokens for ARG-Designer). However, its average inference token consumption is comparable to ARG-Designer (3,821 vs. 3,884 tokens/query) and lower than OFA-MAS (4,673 tokens/query). Consequently, the overall token consumption also remains comparable (4.71M for TopoPAG, 4.66M for ARG-Designer, and 4.77M for OFA-MAS). TopoPAG also achieves lower average inference latency than both baselines (11.98 vs. 13.11 and 27.81 s/query, respectively). We have added these results to Section 4.2 and Appendix E of the updated paper.
Author Response to Reviewer Z2d3
We thank the reviewer for recognizing the novelty of our approach and its consistent empirical improvements. We have strengthened the evidence for the proposed mechanism and addressed the concerns regarding statistical reliability, computational costs, and preference stability. All corresponding updates are included in the revised manuscript and highlighted in blue.
Q1. Comparison with Simpler Alternatives
We thank the reviewer for raising this important concern. To isolate the benefits of the proposed generation mechanism, we evaluated two simpler alternatives: TopoPAG-Top1, which directly executes the highest-preference anchor, and TopoPAG-NoState, which retains the same preference-weighted training data but removes the PCA-based topology state. Under comparable execution settings, TopoPAG consistently outperforms both alternatives across all six benchmarks, achieving 93.03% average accuracy versus 90.67% and 89.89%, respectively. These improvements (+2.36 and +3.14 percentage points) provide direct evidence that continuous preference-guided generation offers benefits beyond simpler topology selection and generation. These comparisons have been incorporated into Table 1 and the discussion in Section 4.2, with implementation details provided in Appendix D.
Q2. Statistical Reliability
We thank the reviewer for emphasizing the importance of statistical reliability. We show 10 repeated runs across all six benchmarks and additionally computed 95% paired bootstrap confidence intervals using 10,000 query-level resamples against the strongest baseline on each benchmark. As shown below, TopoPAG maintains consistent performance across repeated runs, with particularly low variability on GSM8K (93.84 ± 0.15%). Moreover, the paired confidence intervals have positive lower bounds on five of the six benchmarks, providing further support for the reported performance gains. The results are presented in the new Statistical Reliability analysis and table in Appendix E.
Mean ± Std is computed across 10 repeated inference runs. Confidence intervals are obtained from 10,000 paired bootstrap resamples of matched query-level outcomes.
Q3. Comprehensive Computational Cost
We thank the reviewer for emphasizing the need for a comprehensive cost evaluation. We quantified the offline evaluation of 18 candidate topologies, model training, and end-to-end inference, comparing TopoPAG with ARG-Designer and OFA-MAS on MMLU. As shown below, TopoPAG incurs higher offline token consumption (2.93M), but achieves the lowest training cost (0.29 GPU-h) and competitive inference latency (12.04 s/query), close to ARG-Designer (11.20 s/query) and substantially lower than OFA-MAS (33.10 s/query). These results provide a more complete picture of the computational trade-offs underlying TopoPAG's performance improvements. We have added the comprehensive cost comparison and corresponding discussion in Section 4.2, with measurement details provided in Appendix E.