- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces an approach called FedSocratic, which is basically a framework for knowledge transfer between heterogeneous LLM models. The main motivation is that sharing model parameters is expensive, especially in LLMs, so the authors proposed using structured Socratic questioning to identify client capabilities, which is done by the server and then the server provides this verified supervision to weaker clients, which then fine tune their models locally using LoRA.
Critical strengths
S1: The paper tries to address an important problem in federated fine-tuning of LLMs where traditional parameter aggregation approaches fail due to model heterogeneity.
S2: The pipeline of FedSocratic is easy to follow.
S3: FedSocratic achieves strong performance compared to the selected baselines.
Critical weaknesses
W1: There is a misunderstanding of how knowledge transfer works. In the main answer only configuration and based on the proposed approach, what FedSocratic is actually doing is not transferring knowledge between other clients but actually only between the generated server data and the selected clients. I would argue that client X1 doesn't learn anything about Client X2 so knowledge is not shared between clients at all. That sounds like we get a generator to generate some questions and answers that you basically forward it to a specific client to increase his expertise. In other words, if the server generates the questions and already has their correct answers and basically sends those answers to a selected client as fine tuned data what unique knowledge does the teacher client contribute?
W2: FedSocratic approach assumes that the server is already aware of the underlying rules. I believe this is a strong assumption that could limit its practicality also as far as i understand from Table 8 ( Appendix E) the performance of FedSocratic is reduced when testing on standard public benchmarks.
W3: The selected baselines are a bit limited. For example, there are multiple methods that tackle the same issue of federated fine-tuned LLMs some of them don't operate on model parameters, similar to FedSocratic, such as FedCoFiT [1] and some solve the same issue but operate on model parameters, such as FLoRA [2] and FlexLoRA [3]
W4: The main experiments contain only 4 clients. I understand there may be a computional resources constrains espcaiily with LLMs but one could try a larger number of clients, variable client participation or limited client availability to show scalibility with small LLMs as in the current setup it's hard to verifie the scalibilty.
W5: Excluding acquisition, calibration, and candidate selection costs from the computational budget doesn't make sense to me as FedSocratic can train and evaluate multiple questions and replay mixtures where other baslines doesn't need that.
W6: As far as I understand, the verifier only checks the final answer right ? so how does FedSocratic make sure that the client explanation doesn't have incorrect reasoning during local fine tuning ?
W7: FedSocratic avoids sharing model parameters, but as far as i understand, clients share generated answers, correctness, capability information and generated explanations which is basically a lot of information shared between the clients and the server, which raises privacy concerns for the federated learning approach since the paper doesn't provide any privacy analysis.
[1] Abourayya, Amr, Jens Kleesiek, and Michael Kamp. "Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs."
[2] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513–22533, 2024.
[3] Bai, Jiamu, et al. "Federated fine-tuning of large language models under heterogeneous tasks and client resources." Advances in Neural Information Processing Systems 37 (2024): 14457-14483.
Questions for authors
Check Weaknesses