- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper addresses the "knowledge silo" and negative transfer issues in LLM-based Multi-Agent Systems (MAS) under federated learning settings, where topology learning is tightly coupled with task-specific supervision. To this end, the authors propose FedGOTA. On the client side, it employs multi-view graph self-supervised contrastive learning to decouple topology generation from task labels. On the server side, it probes client generators with shared semantic anchors to construct graph codebooks and introduces Optimal Transport (OT) alignment to quantify structural affinity among clients, thereby enabling personalized parameter aggregation. Experiments on six cross-domain reasoning benchmarks demonstrate that FedGOTA outperforms existing baselines in both reasoning accuracy and token efficiency. Ablation studies and other analytical experiments further validate the rationale behind the FedGOTA design.
Critical strengths
- Clear Presentation and Motivation. The paper is well-written, and the motivation for addressing the coupled topology learning and task supervision is logically sound and easy to follow. The transition from the identified bottlenecks to the proposed methodology is smooth.
- Solid Theoretical Formulation. The theoretical foundation is rigorous. The formalization of self-supervised decoupling and the structural affinity measurement based on Optimal Transport are well-designed and theoretically grounded.
- Comprehensive Empirical Evaluation. The experiments cover six authoritative benchmarks across four domains and compare against 14 baselines, including recent adaptive MAS techniques. Ablation studies, token consumption analysis, aggregation weight visualization, and case studies mutually corroborate the effectiveness of the proposed method.
Critical weaknesses
- Lack of Direct Empirical Validation for Core Claims. The central premise of the paper is that self-supervised decoupling enables the reuse and transfer of cross-task collaboration structures, and OT alignment facilitates personalized cross-client knowledge transfer. However, the current experiments do not directly validate these claims. For instance, it remains unclear how a generator trained on Task A performs when directly applied to generate topologies for Task B. Although the paper repeatedly emphasizes "reusability," it fails to introduce any unseen tasks during training to verify the generalization ability of the generator.
- Insufficient Justification for the Optimal Transport (OT) Module. As a core innovation highlighted in the title, OT is not compared against simpler structural distance metrics, such as Cosine Similarity between adjacency matrices, Frobenius norm, or Maximum Mean Discrepancy (MMD). If these computationally cheaper metrics can yield similar aggregation weight distributions and final performance, the necessity of introducing the heavy OT machinery becomes questionable.
- Limited Efficiency and Scalability Analysis. The efficiency analysis is currently restricted to theoretical complexity in the appendix, lacking empirical data on actual wall-clock time and memory footprint. There is no comparison of the actual training/inference time between FedGOTA and baseline methods. Furthermore, to justify the use of OT, it is crucial to compare the time and memory consumption of the OT module against simpler distance metrics. Without this, the practical advantage of OT is not fully supported.
Questions for authors
- Generalization Experiments: Regarding Weakness 1, please provide zero-shot cross-task transfer experiments (e.g., applying a generator trained on Task A directly to Task B) or evaluations on completely unseen tasks. This is critical to substantiating the paper's core claims regarding "reusability."
- OT vs. Simpler Metrics: Regarding Weakness 2, please provide ablation studies comparing the OT alignment with simpler structural distance metrics (e.g., Frobenius norm, Cosine similarity, MMD) in terms of both final performance and the resulting aggregation weight distributions.
- Empirical Efficiency Analysis: Regarding Weakness 3, please provide quantitative analyses of wall-clock time consumption and peak memory footprint for both FedGOTA and representative baselines. Specifically, how much overhead does the OT module add compared to simpler distance metrics?
- Performance Gap on MMLU-Pro: Could the authors elaborate on why FedGOTA underperforms compared to specific baselines on the MMLU series, particularly the >1% gap on MMLU-Pro? Is this related to the shared structural prior or the nature of general reasoning tasks?
3 comments
Most of my concerns have been resolved. While the rebuttal slightly conflates "model generalization" with "federated marginal gains" (W1), and leaves the scalability issue for larger agent pools unaddressed. Overall, I recommend upgrading the score to "Weak accept".
Thank you very much for your positive feedback and for recognizing the value of our work.
Thank you very much for your positive feedback and for recognizing the value of our work. In our rebuttal, Tables 2 and 3 focus on demonstrating cross-domain generalization, whereas Table 4 highlights that our method achieves results comparable to centralized joint training through federated collaboration, aiming to emphasize the effectiveness of federated learning. It is worth noting that multi-domain joint training indeed requires greater agent diversity and a larger agent pool compared to single-domain training methods such as G-Designer. However, compared to OFA-MAS, which relies on 21 candidate agents, our method remains highly competitive, utilizing a candidate pool of only 10 experts to achieve effective collaboration in a distributed setting without sharing raw data.
For W1, in our federated setting, clients keep their data local and learn topology generators without task-specific answer supervision. During inference, each client directly employs its personalized generator without further fine-tuning, enabling zero-shot inference under its local task distribution. Moreover, the local topology generator introduced in Section 4.1 (hereafter referred to as FedGOTA-L) enables direct cross-task reuse without task-specific supervision or further optimization. As shown in Table 2 and Table 3, FedGOTA-L trained on MMLU and MMLU-Pro or GSM-H and AQuA can be directly transferred to unseen tasks without further fine-tuning, consistently outperforming Vanilla and OFA-MAS. Furthermore, as shown in Table 4, a single FedGOTA-L foundation model jointly trained across all pooled datasets without the federated constraint achieves a 71.13% average accuracy when reused across all tasks, surpassing all baselines on average.
|Table 2 (Source: MMLU + MMLU-Pro) | AQuA | GSM-H | HumanEval | | -- | - -| -- | -- | | Vanilla | 81.28% | 47.95% | 82.27% | | OFA-MAS | 79.00% | 48.85% | 87.94% | | FedGOTA-L | 84.02% | 61.64% | 88.65% |
For W2, we have conducted additional ablation experiments comparing OT alignment with simpler affinity metrics, including Graph Cosine, Weight Cosine, Frobenius norm, and MMD. As shown in Table 5, OT consistently achieves competitive performance across all benchmarks.
For W3, we quantitatively compare the alignment runtime and memory usage of different server-side affinity measures in Table 6. Although iterative OT takes longer than simpler affinity metrics, its server-side alignment requires only 3.44 seconds, with an average memory usage of 617.78 MiB.
For W4, the performance gaps on MMLU and MMLU-Pro in Table 1 should be considered in light of the different training objectives, as several adaptive baselines rely on task-specific supervision or LLM feedback to optimize collaboration structures. In contrast, FedGOTA learns collaboration topologies without task-specific supervision while keeping client data local. Through personalized federated learning across heterogeneous tasks, FedGOTA learns reusable collaboration topology generators that can be directly applied to downstream questions without further fine-tuning. Regarding the shared structural prior and general reasoning tasks, the performance gap on MMLU-Pro may partly reflect the trade-off between cross-task transferability and task-specific specialization. Specifically, FedGOTA learns transferable collaboration patterns across heterogeneous tasks, which may be less tailored to the complex reasoning demands of MMLU-Pro than methods optimized using feedback specific to each task. While supervised baselines may benefit from task-specific optimization on challenging reasoning tasks, FedGOTA emphasizes reusable collaboration patterns across heterogeneous tasks without relying on task-specific supervision.