- Rating
- 2
- Confidence
- 2
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper targets to provide a method for different clients, with its own agent system working on different private task and dataset, to train a agent communication controller together via federated learning, to further improve the cooperation between agents and get a better task performance. The scope of this paper is to optimize the controller with LLM or agent untrained. During each round, on the client side, the GLV-inspired controller is used to model the hidden state evolution for generating the communication topology of agents, and the controller then is updated by the final answer for the local task. Then the server will gather all the controller weights from local clients and carry out the weight aggregation and the knowledge distillation by probe and LOO to update the server-side controller. On the evaluation, 6 clients were evaluated on several LLM benchmarks includes Q&A, math and HumanEval and so on. What's more, ablation study on different components as well as unseen task generalization were also evaluated.
Critical strengths
-
This paper designed a unified agent communication controller for even heterogenous clients: (1) the controller can handle different tasks, agent configuration as well as the communication topologies and steps. (2) It did provide a complete Federated learning method to train the controller for each clients without sharing the private dataset and agent configuration and so on.
-
Evaluation including ablation on the server side as well as the unseen task generalization provided more evidence: (1) ablation study for different components including GLV, probe, LOO and so on. (2) On unseen task, proposed method has more uplift compared to the baseline, which provided important evidence.
Critical weaknesses
-
Currently the cross client-sharing training benefit was not directly demonstrated: (1 ) there should be a ablation study on GLV controller training result on solo client compared with federated GLV controller training, under the same local data, local update as well as the agent configuration and so on. (2) Currently the main results include the ablation on different controller model and training method, but it didn't isolate the contribution of federated learning. (3) Since the federated collaborative learning is the core part of this paper, lacking of the demonstration of benefit of federated learning reduced the technical soundness.
-
Currently the ablation study can not demonstrate the necessary of GLV:(1) current result can only demonstrate the benefit of dynamic state evolution by comparing to other baselines, however, it lacks of comparation with other simple dynamic controller such as simple RNN controller. (2) This is important because the GLV is the core design of the FedGLV. Lack of ablation study on other dynamic baselines also reduce the soundness.
Questions for authors
My questions are closely related to the critical weakness: author should provide (1) more evidence of the isolated federated collaborated learning benefit and (2) more evidence of the necessary of GLV instead of other simple dynamic controller algorithm such as simple RNN , MLP with step embedding and so on.