- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes PRISM, a post-training method for multi-agent topology generation. It trains a teacher conditioned on another successful topology of the same query as privileged information, and then performs on-policy distillation (OPD) to improve the student topology generator. Experiments show improvements over SFT and existing topology design baselines.
Critical strengths
- The paper studies an interesting problem of learning multi-agent topology generators from execution-verified graphs.
- The experiments cover six benchmarks and include useful controls to distinguish the contributions of OPD and privileged information.
Critical weaknesses
- Mismatch between the claimed challenges and the proposed solution. The paper motivates the method by the lack of preferences among successful topologies, distribution bias in SFT, and exposure bias. However, another successful topology does not provide any preference or quality signal over the training target. OPD addresses exposure bias, but it is unclear how the proposed method resolves the first two challenges.
- Limited novelty in methodology design. The proposed approach mainly combines existing on-policy distillation with privileged information learning. It is fine, but during doing so, the motivation for using another successful topology as privileged information is not sufficiently justified.
- Insufficient evidence for improved query-specific topology design. The topology intervention experiments in Appendix H show that PRISM's role assignments perform similarly to random or query-permuted assignments. Moreover, on AQuA, the method mainly recovers the performance degradation introduced by SFT, with only a marginal improvement over the original ARG-Designer.
Questions for authors
Why should conditioning the teacher on another successful topology provide better supervision? Since both topologies are successful, what information allows the teacher to distinguish better and worse topology decisions?
2 comments
Dear Reviewer tEH3,
We sincerely appreciate the time and effort you devoted to reviewing our work, as well as your constructive and encouraging comments. We have addressed each of your concerns in our rebuttal and hope our responses are clear and satisfactory. Should you have any further questions or suggestions, we would be glad to discuss them in more detail.
Thank you for your thoughtful engagement with our paper.
Sincerely,
The Authors of Submission 12333
Official response for Reviewer tEH3
Thank you to Reviewer tEH3 for the thoughtful comments. We address the questions and concerns below.
[Q1] What does the sibling give the teacher, if both topologies succeeded?
The sibling does not let the teacher rank topologies. The teacher is fit by likelihood on execution-verified graphs, conditioned on a different verified graph for the same query. It therefore models the family of topologies that worked for that query, and the student is pulled toward this conditional distribution rather than the marginal over all queries. Two controls show that the content is used and survives distillation:
[W1] The claimed challenges do not match the solution.
We agree and will revise the framing. On-policy distillation addresses exposure bias (the constant-input teacher improves AQuA by +9.05 over SFT). Conditioning on a query-specific sibling addresses SFT's distribution collapse: the top-1 topology share is 0.346 for SFT, against 0.171 for the starting checkpoint and 0.157 for PRISM (Section 5.3). The lack of a preference among successful graphs is not resolved by PRISM, and the Introduction will say so.
[W2] Novelty
We agree that on-policy distillation and privileged information are established ideas, and we will describe PRISM as an adaptation of them rather than a new learning principle. Our contribution is their combination in a setting where neither transfers directly: graph-valued actions, a verified graph for the same query as the privileged signal, and no reward model or preference data. To the best of our knowledge, this is the first work to study post-training methods for multi-agent system topology design. Prior topology generators are trained by supervised learning or reinforcement learning on the generator itself, and none addresses the post-training problem that PRISM targets (exposure bias and distribution collapse in graph generation). We will add a related-work paragraph that states this positioning and the differences from existing approaches.
[W3] Evidence for query-specific design, and AQuA
We do not claim query-specific role assignment, and Appendix H already states that the intervention "does not establish that PRISM's particular crews, or their assignment to particular queries, carry benefit." Consistent with this, PRISM's crews permuted across questions score 78.39 against 77.66 for the original assignment (p = 1.00), and a size-matched random crew scores 75.46 (p = 0.61). What the intervention does show is that SFT's own crew is worse (61.54 against 77.66; 4 vs 21 discordant, p = 0.0009), so PRISM's gain comes from avoiding SFT's collapse.
On AQuA we agree that the gain over ARG-Designer is small (76.51 vs 75.33). The larger gains over ARG-Designer are on GSM8K (87.77 vs 80.21) and in the four-benchmark average (89.16 vs 86.70).
We hope our responses address your concern, and we will add these discussions to the revision. If any concerns remain, we would be happy to discuss them further.