- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes POET, a prior-aware preference optimization framework for generating task-adaptive communication topologies in LLM-based multi-agent systems. It constructs success–failure graph preferences from recorded executions and uses task difficulty, graph complexity, and role diversity to guide pair sampling and loss weighting. The approach trains an autoregressive topology generator without additional LLM calls during preference optimization. Experiments on six benchmarks report an average accuracy of 91.22%.
Critical strengths
-
POET learns from both successful and failed topologies through a clear preference-learning formulation, reusing existing execution records without additional LLM calls during preference optimization.
-
The paper compares several topology-design methods and includes plain-DPO ablations, query-conditioning tests, fixed-topology controls, and cross-model transfer evaluations, which help assess different aspects of the approach.
Critical weaknesses
-
POET relies on relatively small preference datasets, with only five MMLU training queries containing both successful and failed executions. The paper also does not report variability across multiple runs, making it difficult to assess the stability of the reported improvements.
-
On GSM8K, the fixed Tree-4 topology achieves 93.98% accuracy compared with POET's 94.06%, while consuming considerably fewer tokens. This raises concerns about whether the additional complexity of adaptive topology generation is justified by its performance gains.
-
The component ablations are mainly conducted on MMLU and GSM8K, while the other benchmarks provide limited evidence about the individual contributions of task difficulty, graph complexity, and role diversity.
Questions for authors
-
Could the authors report results across multiple random seeds to demonstrate the stability of POET's improvements, particularly given the limited number of training queries with mixed execution outcomes?
-
Given that Tree-4 achieves nearly identical accuracy with fewer tokens on GSM8K, does POET retain an advantage when compared under similar inference budgets?
-
Since POET generates larger teams after preference optimization, how much of the improvement comes from task-specific role and communication structures rather than simply using more agents?