acceptodds
Under review as a conference paper at ICLR 2027

FedSocratic: Exploring Knowledge Transfer through Socratic Questioning in Heterogeneous Federated LLMs

Abstract

Federated learning enables collaborative improvement of large language models while keeping training data local. However, architectural differences hinder parameter aggregation, while output-based knowledge exchange often relies on shared public training corpora. We study how heterogeneous LLMs can identify and transfer complementary knowledge without a shared public training corpus or parameter alignment. Inspired by Socratic inquiry, we propose FedSocratic, which combines structured questioning, answer verification, and local adaptation. Base questions and answer-preserving and answer-changing follow-ups probe client capabilities; executable rules verify final answers, and receivers learn from accepted supervision while retaining their architectures. Experiments with four heterogeneous clients across four verifiable task domains demonstrate improved performance on unseen procedural states. Controlled studies show that complete question coverage improves pooled weak-domain accuracy over omitting question types at matched record and update budgets. With response text retained, capability-selected peers outperform verified self-training and random peers in mean cross-domain accuracy on matched accepted questions and record counts. These findings support knowledge transfer through structured questioning within verifiable task families and highlight question coverage and supervision-source selection as key design choices.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Heterogeneous Federated Learning, Large Language Models, Knowledge Transfer, Socratic Questioning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_iM5vOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper introduces an approach called FedSocratic, which is basically a framework for knowledge transfer between heterogeneous LLM models. The main motivation is that sharing model parameters is expensive, especially in LLMs, so the authors proposed using structured Socratic questioning to identify client capabilities, which is done by the server and then the server provides this verified supervision to weaker clients, which then fine tune their models locally using LoRA.

Critical strengths

S1: The paper tries to address an important problem in federated fine-tuning of LLMs where traditional parameter aggregation approaches fail due to model heterogeneity.

S2: The pipeline of FedSocratic is easy to follow.

S3: FedSocratic achieves strong performance compared to the selected baselines.

Critical weaknesses

W1: There is a misunderstanding of how knowledge transfer works. In the main answer only configuration and based on the proposed approach, what FedSocratic is actually doing is not transferring knowledge between other clients but actually only between the generated server data and the selected clients. I would argue that client X1 doesn't learn anything about Client X2 so knowledge is not shared between clients at all. That sounds like we get a generator to generate some questions and answers that you basically forward it to a specific client to increase his expertise. In other words, if the server generates the questions and already has their correct answers and basically sends those answers to a selected client as fine tuned data what unique knowledge does the teacher client contribute?

W2: FedSocratic approach assumes that the server is already aware of the underlying rules. I believe this is a strong assumption that could limit its practicality also as far as i understand from Table 8 ( Appendix E) the performance of FedSocratic is reduced when testing on standard public benchmarks.

W3: The selected baselines are a bit limited. For example, there are multiple methods that tackle the same issue of federated fine-tuned LLMs some of them don't operate on model parameters, similar to FedSocratic, such as FedCoFiT [1] and some solve the same issue but operate on model parameters, such as FLoRA [2] and FlexLoRA [3]

W4: The main experiments contain only 4 clients. I understand there may be a computional resources constrains espcaiily with LLMs but one could try a larger number of clients, variable client participation or limited client availability to show scalibility with small LLMs as in the current setup it's hard to verifie the scalibilty.

W5: Excluding acquisition, calibration, and candidate selection costs from the computational budget doesn't make sense to me as FedSocratic can train and evaluate multiple questions and replay mixtures where other baslines doesn't need that.

W6: As far as I understand, the verifier only checks the final answer right ? so how does FedSocratic make sure that the client explanation doesn't have incorrect reasoning during local fine tuning ?

W7: FedSocratic avoids sharing model parameters, but as far as i understand, clients share generated answers, correctness, capability information and generated explanations which is basically a lot of information shared between the clients and the server, which raises privacy concerns for the federated learning approach since the paper doesn't provide any privacy analysis.

[1] Abourayya, Amr, Jens Kleesiek, and Michael Kamp. "Beyond Parameter Aggregation: Semantic Consensus for Federated Fine-Tuning of LLMs."

[2] Ziyao Wang, Zheyu Shen, Yexiao He, Guoheng Sun, Hongyi Wang, Lingjuan Lyu, and Ang Li. Flora: Federated fine-tuning large language models with heterogeneous low-rank adaptations. Advances in Neural Information Processing Systems, 37:22513–22533, 2024.

[3] Bai, Jiamu, et al. "Federated fine-tuning of large language models under heterogeneous tasks and client resources." Advances in Neural Information Processing Systems 37 (2024): 14457-14483.

Questions for authors

Check Weaknesses

Reviewer_saAKOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

FedSocratic addresses federated learning across clients that differ in architecture and tokenizer — so parameter aggregation is unavailable — and hold different domain expertise. The goal is to exchange knowledge without a shared public corpus and without parameter alignment, using natural language as the medium. A server-side coordinator samples a procedural state z and emits a three-phase knowledge unit: a base question, an answer-preserving "invariant" follow-up, and an answer-changing "flip" follow-up, together with an executable certificate that verifies final answers. Clients answer, the certificate defines the trusted set, a passing non-receiver peer is chosen by prior verified domain accuracy, and each receiver fine-tunes a local LoRA adapter on the accepted records mixed with verified replay, behind a calibration gate. Four heterogeneous 7–8B clients (code, math, finance, medical) are evaluated on code execution, mathematics, financial calculation and deterministic medical rule application. Functionally, the server collects and filters answers across models and redistributes them as local fine-tuning data.

Critical strengths

1. The framework itself is a genuinely different construction. Using server-generated natural-language questions as the exchange medium departs from both established options: parameter-space aggregation, which heterogeneous architectures and tokenizers rule out, and output exchange over a shared public corpus, which licensing and coverage constraints often rule out. Replacing that corpus with procedurally generated states plus executable verification is a clean idea, and the reason I engaged with the setting rather than dismissing it.

2. The experiments are deliberately controlled, and the paper reports results that cut against it. The phase-coverage study is budget-matched (24 states, 72 records, 720 optimizer steps, matched skill composition, deletions repeating retained questions), and the supervision-source study matches accepted questions and record counts across sources — the right controls for separating the method's effect from that of additional training. The paper also publishes its negative findings: public-benchmark losses, a predeclared null confirmation, native-accuracy regressions.

Critical weaknesses

1. The paper's own controls undercut its thesis, and the decisive baseline is missing. The coordinator generates every state and holds the certificate, so it already knows every answer, and the paper's numbers suggest the peers add little. Fixed server-certified acquisition reaches 65.68% overall against peer-informed acquisition's 55.09% (Fig. 4); with the 14B composition a fixed certificate plus matched replay reaches 53.99% against FedSocratic's 49.65%, and peer-only without completion (44.62%) falls below no transfer (46.88%) (Table 7); and in Table 2, 10.34 of the 15.21-point gain comes from the non-adaptive certificate alone. The method is then essentially data selection for local LoRA training — yet it is never compared against fine-tuning on well-chosen examples from existing public datasets, since those benchmarks are reserved for evaluation (§4.1) and FedMD/Fed-ET/FedMKT are re-adapted to the paper's own question interface. On those benchmarks every configuration loses ground: the portfolio falls from 49.22% to 44.56% (Table 8), and the predeclared confirmation in Appendix F.2 averaged −0.28 points.

2. There is no threat model, in a setting whose premise is privacy. The paper never states what the coordinator is trusted to do, while the architecture gives it full control of the query stream and every client's raw free-text answer. Adaptive acquisition (§3.1, B.2) tunes difficulty and skill per target after each certified outcome — an active query-selection loop, exactly the primitive a curious or adversarial server would use to probe a client's private data, and the response-retaining configuration returns free text that can surface memorized content. Injected questions, membership inference, differential privacy and secure aggregation all go undiscussed. For a method selling the premise that training data stays local, that is a material gap, not future work.

3. Costs are excluded rather than measured. No communication volume, token count or round complexity is reported for the language-space interface, and acquisition and candidate-search costs sit explicitly outside the matched budget (§4.2, Table 4), so the comparisons never price what the paper proposes.

Minor, but it drives my Presentation score. Figure 1 fails as a teaser: "peer" is undefined there, only the leftmost panel registers, and even its intended claim — that the boxed best source is never the nominal one — did not reach me on first reading. Figure 2 compresses the pipeline into unreadably small type with no clear reading order, and the results figures are large relative to the few numbers they carry, several duplicating tables.

Questions for authors

Clarifications that could change my assessment:

  1. Given Fig. 4 and Table 7, what does peer supervision add over fixed server-certified acquisition, and when should a deployment prefer peers?
  2. What is the threat model for the coordinator? Can an adaptive question schedule be used to extract client-private information, and what prevents it?
  3. Has a baseline been run that fine-tunes the same LoRA on curated examples from existing public datasets? If not, what comparison do you expect?
  4. What reconciles the roughly +15-point internal gain with −4.66 points on the public tasks (Table 8), and which should a reader carry away?
  5. Can you give confidence intervals for Fig. 6, the Table 3 contrasts and Fig. 7? Those margins correspond to 1–5 test questions, and intervals are reported elsewhere in the paper.
  6. What are the per-client communication volumes, versus FedMD, Fed-ET and FedMKT?
Reviewer_GNTeOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper focuses on collaboratively tuning large language models with different domain clients. It proposes FedSocratic, in which a server generates structured questions, verifies client answers, and selects peer supervision for local updates of heterogeneous language models.

Critical strengths

The paper includes useful controlled comparisons.

The paper transparently reports negative results, including stronger server-only performance and degraded public-benchmark performance.

Critical weaknesses

Firstly, the main concern is the motivation of the federated setting. Different client expertise could make collaboration useful, but the server already knows the task-generation rules and correct answers. The author needs to be clear about what the participated client contribute to the system.

Secondly, as different clients have different domain knowledge and work well in different areas, the author does not explain why there needs a federated system.

Thirdly, it is unclear whether the baselines are well suited to this setting. FedMD, Fed-ET, and FedMKT are adapted to the proposed question-answer interface, but the paper does not sufficiently justify these adaptations

Questions for authors

In what practical scenarios do clients possess critical knowledge that servers cannot provide?

How were FedMD, Fed-ET, and FedMKT adapted to this setting, and were their key mechanisms preserved?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.