- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper argues that existing multi-agent collaboration topology optimization methods are primarily developed independently within individual clients, making it difficult for clients to reuse collaboration experience from one another. To address this issue, the paper proposes a personalized federated topology optimization framework that represents collaboration experience as probabilistic parameters of communication edges, thereby constructing a general shared probabilistic graph. When transferring this shared knowledge to individual clients, the framework combines the shared graph with local parameters to generate personalized topologies. Additionally, the paper trains a performance predictor to evaluate candidate topologies.
Critical strengths
-
The experiments are relatively comprehensive, covering different task types and ablation studies.
-
The paper addresses a meaningful problem: the difficulty of reusing collaboration experience across different clients, which has practical value.
-
The paper considers practical costs and designs a performance predictor to reduce the cost of topology optimization.
Critical weaknesses
-
The transferability of the experience summarized in the shared probabilistic graph across different clients is not sufficiently justified. For example, in tasks requiring a definite decision, the topology involving decision-making roles may be important, whereas in open-ended tasks, such roles may be less necessary or even dispensable.
-
The final performance improvements appear to be relatively small, and it is unclear whether these gains could be partly due to randomness.
-
The precise definition of a “client” is not sufficiently clear, particularly regarding whether the proposed framework still applies when different clients use different large language models.
-
The unique contribution of this work compared with existing methods is not sufficiently highlighted, making it difficult to identify the most distinctive insight of the paper.
Questions for authors
Thanks for your paper but I have some little problem
- Why should the experience summarized in the shared probabilistic graph be transferable across different clients? For example, in tasks requiring a definite decision, the topology involving decision-making roles may be very important, whereas in open-ended tasks, such roles may be less necessary or even dispensable.
- The final performance improvements appear to be relatively small. Could these gains be partly due to randomness?
- What is the precise definition of a “client” in this paper? Does the proposed definition and framework also apply when different clients use different large language models?
- Compared with existing methods, what is the unique contribution of this work? Which part of the paper best highlights its most distinctive insight?
2 comments
Thank you for your detailed response and additional experiments. I appreciate your efforts, particularly the additional statistical analysis addressing the concern about randomness.
However, I feel that my two main concerns have not been fully resolved.
-
Transferability. I remain unconvinced that the collaboration experience summarized in the shared probabilistic graph is generally transferable. First, different task types may require substantially different collaboration structures. Second, different LLMs may exhibit different collaboration behaviors. Although your additional experiments, including the Qwen3-8B predictor-transfer experiment, provide some supporting evidence, they do not fully address these concerns or establish the conditions under which such experience can be effectively transferred.
-
Distinctive contribution. Since the three main technical components have foundations in existing research, I see the paper's primary contribution as a system-level integration and exploration of these methods. I recognize the value of this combination and its practical design contribution. However, I am still not convinced that the paper has sufficiently demonstrated the distinctive advantages of sharing aligned edge parameters over closely related federated methods such as DAWN.
Given these remaining concerns, I would prefer to maintain my score and consider the perspectives of the other reviewers before deciding whether to raise my rating.
Response to Reviewer Xq9E
We thank Reviewer Xq9E for raising the questions about transferability, randomness, the client definition, and the distinctive contribution.
Q1. Transferability. The shared object is routing structure rather than task content. The two suites use three non-IID clients covering coding, mathematics, and reasoning under a common role schema. Shared logits encode reusable entry and decomposition patterns; the personalized component adjusts edges for each client's task distribution, and the task-conditioned predictor ranks candidates per task. In Table 2, the complete configuration has the highest mean in both suites and improves over Shared + Predictor on five of six clients. The client-level variation is handled by sharing reusable topology structure while retaining client-specific edges.
Q2. Randomness. We evaluate this concern using matched outcomes rather than independent averages. Complete and Local + Predictor use the same pretrained predictor initialization, local-update rule, feedback schedule, held-out items, and fixed seed. On the same 309 items per suite, Complete wins in 19 vs. 6, 24 vs. 9, 16 vs. 5, and 10 vs. 2 item pairs for AF-3, GD-3, AF-20, and GD-20, respectively; the exact two-sided McNemar p-values are 0.014, 0.014, 0.027, and 0.039. The corresponding 95% paired intervals are all above zero: [+0.97, +7.45], [+1.29, +8.41], [+0.65, +6.47], and [+0.65, +4.85] points. The same direction appears in both workflow suites and at both local-data sizes. The 0.65/0.97-point margins in Table 2 compare with the strong CoT reference; the matched federated comparison is 2.59–4.86 points. Thus, the reported federated margin is a consistent paired contrast across the two suites and both local-data settings, rather than an isolated mean from one client or one test set.
Q3. Client. A client is a unit with its own task distribution, local history, and executor configuration. The main experiments use three non-IID clients per suite with GPT-4o-mini as the suite executor. Appendix G additionally transfers the predictor from GPT-4o-mini to Qwen3-8B using 240 target-executor outcomes while retaining the task splits, agent pool, topology policy, and role schema. Eqs. (5)–(6) aggregate aligned topology parameters rather than model weights, so executor heterogeneity is compatible when clients share the role schema and edge coordinates.
Q4. Distinctive contribution. The core contribution is to use aligned communication-edge logits as the federated experience unit and decompose them into a shared tensor and a client-specific tensor in Eqs. (5)–(6). This lets clients reuse executable routing structure through mean aggregation while preserving local preferences. Predictor guidance is the efficiency component that reduces the cost of searching these distributions. Figure 2 and Section 4.2 highlight this shared/personalized topology decomposition; compared with DAWN, the exchanged object is aligned edge logits rather than a workflow generator.