acceptodds
Under review as a conference paper at ICLR 2027

Equitable Unified Learning for LLM Approximation

Abstract

Fine-tuning large language models (LLMs) is resource-intensive: it requires a large centralized corpus and substantial compute and memory. This raises three problems. First, such fine-tuning is out of reach for many institutions with limited resources. Second, architecture heterogeneity: institutions choose different architectures suited to their tasks, so models cannot be averaged. Third, data heterogeneity: each institution holds private data that is not independently and identically distributed (non-IID) and must remain local. To address these problems, we propose Unified Learning with Language Models (UniLearnLM), an equitable approach to LLM approximation that fine-tunes small language models (SLMs) through GNN-based unified federated learning. UniLearnLM represents each client's architecture as a typed graph and federates one shared graph neural network (GNN) whose output modulates each frozen backbone through a private adapter, so neither private data nor backbone weights leave a client, client architectures need not match, and under 0.07% of any client's parameters is communicated. Experiments on five language tasks (named entity recognition, question answering, topic classification and two summarization tasks) with four SLM families, and on educational question and answer generation, including a live federation in Tunisia, Germany, Colombia and Peru, show that UniLearnLM is on par with the strongest published heterogeneous federation on single-task federations and yields deeper questions than text-consensus distillation on two testbeds, with depth comparable to local training. Code will be released upon acceptance.

Primary area:
foundation or frontier models, including LLMs
Keywords:
federated learning, model heterogeneity, small language models, graph neural networks, FiLM modulation, parameter-efficient fine-tuning, question answer generation, AI in education

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_u9rKOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Unified Learning with Language Models (UniLearnLM), an equitable approach to LLM approximation that fine-tunes small language models (SLMs) through GNN-based unified federated learning. Four hypotheses anchor the evaluation.

Critical strengths
  1. This paper describes a federation across institutions in four countries. I guess this means real-world multi-institution deployment.
  2. This paper has a broad evaluation and controls.
  3. The paper also honestly reports negative and fragile results.
Critical weaknesses
  1. This paper looks like a very comprehensive combination of SLM, GNN, and federated learning that builds a pretty comprehensive system, making it more of an engineering effort than a methodological contribution.
  2. The paper seems to show that federated learning doesn't help much? Though negative results are appreciated. More detailed analyses or theoretical research would be helpful in explaining this.
  3. The LLM-approximation claim does not seem to be largely supported. The SLM mixtures score way lower than LLMs.
  4. This paper is surprisingly long, with 72 pages. But much of the appendix reads like an accumulated rebuttal log rather than a paper.
Questions for authors
  1. Code will be released upon acceptance. It would be nice if the code were actually released.
  2. The main results are obtained with a single seed. Better to run over 3 seeds.
6 comments

Thank you for a review that credits the paper with "a broad evaluation and controls", a federation "across institutions in four countries", and for the fact that it "honestly reports negative and fragile results". This response spans 2 comments.

How to read the tables:

  • Bold marks the best value per column; cells are mean ± sd over the seeds or folds named in the header.
  • Bcls is our Bloom classifier's score of question depth (one level = 0.2); FiLM-aware round selection scores the deployed model.
  • New tables are in Appendix AG of the revised PDF (page 73).
  • Edits in the revised PDF appear in blue text, so that each change is easy to find.

"more of an engineering effort than a methodological contribution."

The method is one idea: share a model of each client's architecture instead of weights, outputs or text. That idea lets models that cannot be averaged learn together without any public corpus. The interface it gives is new among the methods we compare:

Methodwhat clients exchangepublic text needed
FedHeNN (Makhija et al., 2022), FedMKT (Fan et al., 2025), TC-Best/TC-Ens (Li and Wang, 2019)representations, logits or text on a public poolyes
pFedHN (Shamsian et al., 2021)weights generated by a server hypernetworkno
UniLearnLMone encoder of 87,744 parametersno

With it, UniLearnLM reaches the same performance as FedHeNN over three seeds (difference between FedHeNN and UniLearnLM: -0.0003 ± 0.0074 on CoNLL-2003, +0.0020 ± 0.0068 on AG News, neither significant; Table 20). It serves four model families with different tokenizers, and it ran live at four institutions on three continents (p. 9).

"The paper seems to show that federated learning doesn't help much?"

UniLearnLM is ahead of local training on CoNLL-2003 over three seeds (0.7668 against 0.7323, p = 0.016; Table 17) and above it on four of five single tasks (next comment). Whether the averaging step adds accuracy over the same model trained alone is the no-FedAvg control experiment. On the public testbed it is Table 37. On CoNLL-2003 the control experiments are in progress; their table follows in our answer to Reviewer R7sP.

On question depth, our Bloom classifier Bcls scores UniLearnLM above both federations that share text under FiLM-aware round selection (+0.0405 against TC-Best and +0.0373 against TC-Ens) and below local training; all three intervals exclude zero (Table 36 in Appendix Z, page 63). The holistic LLM judge of the paper (LLM-J) puts it above local training in all three folds (0.6944 against 0.6831), without significance (p = 0.078). On 500 passage-paired test questions, two LLM judges blind to the system (GPT-4o and Claude Opus 5.5) do not separate UniLearnLM from TC-Best or from TC-Ens: all four of their intervals include zero, while Bcls separates it from TC-Best only (Table 47 in Appendix AG, page 73). So we present the depth result as a measurement by our classifier Bcls, not as one that independent judges confirm.

This comment continues our answer above.

"The LLM-approximation claim does not seem to be largely supported. The SLM mixtures score way lower than LLMs."

On grounding, UniLearnLM's small models are 1st of all 7 systems on sentence-level faithfulness of the full answers, under both ways of combining the three institutions' models:

T global, 916 questions, all seven systems scored by one code, with the difference to the best LLM [95% interval over the 179 test contexts]Faith, sentence levelQAFactEval
UniLearnLM, mean over the three institutions' models0.1903 (+0.029 [+0.005, +0.053])0.1752 (+0.019 [-0.008, +0.045])
UniLearnLM, each question by its own institution's model0.1875 (+0.026 [+0.002, +0.051])0.1690 (+0.012 [-0.017, +0.042])
best of six LLMs0.1610, Gemini-2.5-Flash (Comanici et al., 2025)0.1567, Llama-3.1-8B (Grattafiori et al., 2024)

As the table shows, the faithfulness interval excludes zero in both forms. On QAFactEval UniLearnLM has the highest mean, but the interval includes zero, so it is at the same performance as the best of the six LLMs. Of the nine metrics of Table 34, these are the two on which it is 1st. With every answer cut to 15 words it again has the highest mean on both measures: the mean over the models reaches 0.1873 against 0.1714 for Mistral-7B (Jiang et al., 2023) on faithfulness, and 0.1583 against 0.1376 for Llama-3.1-8B on QAFactEval (Table 23). There too it is at the same performance as the best LLM, since every interval includes zero. So small models that stay inside the institutions ground their answers at least as well as the six LLMs, and no passage leaves an institution. Table 34 uses the first form, and the second is the system as deployed. Sentence-level faithfulness checks each sentence of an answer against the passage, and the revised Tables 23 and 34 use it for all seven systems. On the holistic LLM judge the small models trail all six LLMs (Table 34). The submitted paper states that Hypothesis 2 "holds for grounding and not for lexical overlap or the holistic judge". The revision narrows Hypothesis 2 to sentence-level faithfulness of full answers (Section 5.3, page 8).

"The main results are obtained with a single seed. Better to run over 3 seeds."

The paper already runs its central comparison, UniLearnLM against local training, over seeds 42, 1 and 2 (Table 17):

five single-task federations, 3 seeds (Table 17)CoNLLSQuADAG NewsXSumSAMSum
local training0.7323 ± 0.00990.7614 ± 0.01850.8118 ± 0.04460.2525 ± 0.00150.3897 ± 0.0021
UniLearnLM0.7668 ± 0.00250.7697 ± 0.00700.8664 ± 0.00650.2541 ± 0.00110.3873 ± 0.0060

As the table shows, UniLearnLM is above local training on four of five single tasks, significantly on CoNLL (paired t-test, p = 0.016), with a smaller spread across seeds on four of five. Three seeds also exist for FedHeNN (Makhija et al., 2022) and both graph control experiments (Table 20) and for the partition control experiments (Table 19). The educational tables are the ones at one seed.

"much of the appendix reads like an accumulated rebuttal log rather than a paper."

The revised PDF names one checkpoint rule for the main text (Section 5.2, pages 7-8) and labels every appendix table selected under the earlier rule. The paragraphs that narrated reruns and corrections are removed from the appendix. Every table keeps its number, so every table number cited here stays valid, and each table states which selection rule it uses (the two rules are defined in Appendix C.3, page 24).

"It would be nice if the code were actually released."

The anonymized code is available at https://figshare.com/s/f643420b029b2ed607d9. The revised Reproducibility statement (page 10) now says, as the abstract does, that the code will be released upon acceptance.

If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive.

You asked for the main results over three seeds, and our earlier comment said that the educational tables were the ones at one seed. We ran it. This comment gives the result as it came out.

Three seeds on the educational public testbed. UniLearnLM under FiLM-aware round selection and local training of the same three institutions, with seeds 1 and 2 added to the seed of the submitted tables, three folds each. Bcls on the global test, mean of the three institutions and the three folds:

SeedUniLearnLMLocal trainingDifference between UniLearnLM and local training
420.20280.2238-0.0210
10.20930.1779+0.0314
20.20590.1815+0.0245
Mean ± sd over 3 seeds0.2060 ± 0.00330.1944 ± 0.0255+0.0116 (paired t-test, p = 0.5527)

As the table shows:

  • UniLearnLM is above local training in 2 of 3 seeds (0.2060 against 0.1944 on average). With three seeds the paired t-test does not reach significance, so we claim none.
  • UniLearnLM varies less across seeds than local training (sd 0.0033 against 0.0255).
  • In the two added seeds both methods ran on the same card type in the same days, and UniLearnLM is above local training in 6 of 6 folds. Seed 42 is the row of the submitted tables, whose two methods were run months apart.
  • TC-Ens (Li and Wang, 2019) was run at seed 42 only. UniLearnLM is above it at that seed (0.2028 against 0.1655).

The other three lightweight scores of Table 2, from the same runs. Mean ± sd over the three seeds:

ScoreUniLearnLMLocal trainingDifference between UniLearnLM and local trainingSeeds where UniLearnLM is above
ROUGE-L (Lin, 2004)0.4405 ± 0.00250.4329 ± 0.0033+0.0075 (paired t-test, p = 0.0555)3 of 3
BLEU-4 (Papineni et al., 2002)0.1723 ± 0.00250.1618 ± 0.0024+0.0105 (p = 0.0159)3 of 3
BERTScore (Zhang et al., 2020)0.8644 ± 0.00080.8603 ± 0.0009+0.0041 (p = 0.0030)3 of 3

As the table shows, UniLearnLM is above local training on all three scores in 3 of 3 seeds, and the paired t-test is significant on BLEU-4 and BERTScore. On ROUGE-L it does not reach significance.

Changes in the next revision: Table 2 reports UniLearnLM and local training as mean ± sd over three seeds on these four scores, and the text states that their difference in Bcls is not significant.

If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive. Thanks.

Reviewer_KQmnOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes UniLearnLM, a heterogeneous federated learning framework that allows institutions to collaboratively fine-tune small language models without sharing private training data or requiring identical model architectures. Each client retains its own frozen backbone and private LoRA and FiLM adapters, while a shared GNN encodes model architecture information and conditions local feature modulation. Only the GNN parameters are aggregated across clients.

The framework is evaluated on five standard NLP tasks and educational question-answer generation, including public datasets, a live multi-institution deployment, and a controlled private federation. The results show competitive performance against several heterogeneous federated learning baselines, improvements on selected tasks, and better pedagogical-depth scores than text-consensus distillation in the educational setting.

I find the motivation of this work meaningful: enabling institutions with heterogeneous models and private datasets to collaborate through a small shared module. My main concern is whether this module provides a clear benefit beyond local adaptation. In particular, does it transfer knowledge from other institutions, or mainly improve general task adaptation? The current ablations do not sufficiently isolate the contributions of federated sharing, architecture information, and additional local adaptation capacity. This makes the methodological novelty and the source of the reported improvements unclear.

Critical strengths
  1. Practical framework for heterogeneous federated learning.

The paper addresses a relevant problem: enabling institutions with different language model architectures and limited resources to collaborate without exchanging raw training data. The proposed framework provides a lightweight communication interface by aggregating only a shared GNN with 87,744 parameters, while keeping the pretrained backbones and local adapters private.

  1. Broad evaluation with encouraging empirical results.

The paper evaluates the method across multiple architectures, standard NLP tasks, and educational QA generation, including a real multi-institution deployment. The three-seed CoNLL results improve from 0.7323 to 0.7668 (Table 17), and the controlled private QA experiment improves the Bloom-depth score from 0.1867 to 0.2218 (Table 45). These results suggest that the framework can be useful in practical heterogeneous learning settings.

Critical weaknesses
  1. The contribution of the shared architecture encoder is not sufficiently established.

The central methodological claim is that sharing a GNN conditioned on model architectures enables effective collaboration across heterogeneous language models. However, the capacity-matched no-FedAvg ablation in Table 37 achieves nearly identical performance to UniLearnLM (0.2010 vs. 0.2028 on Bloom depth), while the client-index control also performs similarly (Table 21). The three-seed graph ablations in Table 20 do not demonstrate a consistent advantage from architecture information.

This makes it difficult to determine whether the improvements come from federated knowledge sharing, architecture-aware conditioning, or simply the additional local adaptation capacity. Moreover, graph-conditioned federated hypernetworks and client-customized adapter generation have already been studied. The specific benefit of generating FiLM modulation from architecture representations, rather than using alternative adapter-generation mechanisms, remains unclear.

  1. The experimental evidence does not fully support the interpretation of knowledge transfer and educational improvement.

The Knowledge Transfer Index (KTI) is interpreted as evidence that knowledge from one institution is transferred to another. However, the QA evaluation provides the relevant course passages at inference time. Improved performance on previously unseen topics may therefore reflect better general reading and QA-generation abilities rather than acquisition of another institution's specific knowledge. The current experiments do not distinguish these explanations.

The educational-depth results are also sensitive to model selection. In Appendix U and Table 36, using FiLM-aware checkpoint selection changes the public QA Bloom-depth score from 0.2313 to 0.2028, below local training at 0.2238. Furthermore, the main Bloom classifier has not been validated against human judgments in this domain. These issues weaken the evidence for the paper's central claims about knowledge transfer and pedagogical improvement.

Questions for authors
  1. Can the authors evaluate the capacity-matched no-FedAvg and client-index controls on CoNLL or the controlled private QA setting, where UniLearnLM shows clearer improvements? This would help establish whether the gains come from federated sharing and architecture conditioning rather than additional local adaptation capacity.

  2. Can the authors distinguish curriculum-specific knowledge transfer from general task adaptation? For example, keeping the receiving client, evaluation passages, and training budget fixed, how would performance change if the source client's curriculum were replaced with different-topic material matched in size and difficulty?

  3. Which checkpoint-selection protocol should define the primary educational QA results? Can the authors provide the main comparisons using consistent FiLM-aware model selection and clarify whether the conclusions about pedagogical depth remain valid?

6 comments

Thank you for a review that found "the motivation of this work meaningful: enabling institutions with heterogeneous models and private datasets to collaborate through a small shared module", and the framework "a lightweight communication interface". This response spans 3 comments; a fourth follows with the control experiments that are still running.

How to read the tables:

  • Bold marks the best value per column; cells are mean ± sd over the seeds or folds named in the header.
  • Bcls is our Bloom classifier's score of question depth on the revised taxonomy (one level = 0.2).
  • FiLM-aware round selection scores the deployed model, with its modulation active, and defines every educational result here.
  • New tables are in Appendix AG of the revised PDF (page 73).
  • Edits in the revised PDF appear in blue text, so that each change is easy to find.

"The contribution of the shared architecture encoder is not sufficiently established. [...] whether the improvements come from federated knowledge sharing, architecture-aware conditioning, or simply the additional local adaptation capacity."

The interface is what the paper establishes, and the tables show it. Of the methods we compare, UniLearnLM is the only one that reaches FedHeNN's accuracy without any public text:

Method (Appendix U)what clients exchangepublic text needed
FedHeNN (Makhija et al., 2022)representation Gram matrices on a public poolyes
FedMKT (Fan et al., 2025)token logits with a server LLMyes
TC-Best, TC-Ens (Li and Wang, 2019)generated text on a public poolyes
pFedHN (Shamsian et al., 2021)weights generated by a server hypernetworkno
UniLearnLMone encoder, 87,744 parametersno

Over three seeds, the difference between FedHeNN and UniLearnLM is -0.0003 ± 0.0074 on CoNLL-2003 and +0.0020 ± 0.0068 on AG News, and a paired test finds neither significant (Table 20). UniLearnLM is above pFedHN, the other method without public text, on four of five tasks (Table 1). It does this across four families with different tokenizers and hidden sizes.

On the mechanism, the paper states itself that "neither a learned client index nor an untyped graph differs significantly from UniLearnLM in any of the four paired tests over three seeds" (p. 6). So the claim that holds is the interface, not architecture conditioning. A plausible reason, which we have not yet tested: in a fixed federation each client's graph identifies it, so an index carries the same information.

On prior generators: pFedHN and graph hypernetworks (Knyazev et al., 2021; Litany et al., 2022; Lim et al., 2024) generate weights that replace a pretrained backbone, and C2A (Kim et al., 2023) serves one shared model (pp. 1-3). UniLearnLM modulates frozen backbones of different families, and its Bcls is above pFedHN's under FiLM-aware round selection (+0.0214, interval excluding zero; Table 36).

This comment continues our answer above.

"Can the authors evaluate the capacity-matched no-FedAvg and client-index controls on CoNLL or the controlled private QA setting, where UniLearnLM shows clearer improvements?"

We run both control experiments exactly as you ask, with the arms of each seed on the same card type. They are in progress on CoNLL-2003, and we post their table in a follow-up comment as soon as the cells land, within the discussion period.

In the submitted three-seed series the client-index control experiment is already at the same performance as UniLearnLM on CoNLL: the paired test over seeds finds no significant difference between the two (Table 20).

"Which checkpoint-selection protocol should define the primary educational QA results?"

FiLM-aware round selection, because it scores the model that is deployed. Under it the paper already reports every comparison (Tables 21 and 36, public testbed, three folds):

difference in Bcls between UniLearnLM andvalue95% interval
TC-Best (Li and Wang, 2019)+0.0405excludes zero
TC-Ens (Li and Wang, 2019)+0.0373excludes zero
pFedHN (Shamsian et al., 2021)+0.0214excludes zero
FedHeNN (Makhija et al., 2022)-0.0034includes zero
FedMKT (Fan et al., 2025)-0.0215excludes zero
local training-0.0210excludes zero

As the table shows, under one rule Bcls scores UniLearnLM above three of the five federations (both text-consensus federations and pFedHN), at the same performance as FedHeNN, and below FedMKT and local training. On the paper's holistic LLM judge (LLM-J), UniLearnLM and UniLearnLM+ are above local training in all three folds (0.6944 and 0.7000 against 0.6831; Table 2, page 7), although with three folds the paired test gives p = 0.078 and 0.088, so we claim no significance. Whether that ranking holds beyond our classifier is your Bloom point. We answer it in our next comment with a blind LLM judge check, which does not confirm the gap to text-consensus distillation. We therefore remove the depth clause from the abstract (page 1) and state every depth result as a Bcls measurement (Section 5.2, pages 7-8). We rerun the private testbed under the FiLM-aware rule, with UniLearnLM and local training on the same card, and post the result in the same follow-up comment.

This comment continues our answer above.

"Improved performance on previously unseen topics may therefore reflect better general reading and QA-generation abilities rather than acquisition of another institution's specific knowledge."

The index compares the federated model with local training on topics that only the partner teaches (Eq. 17), with the passage in the input, so it cannot separate the two, and we rename it "improvement on partner topics" (Appendix R, pages 39-40). The paper itself reads its mean "as a direction rather than as a measured effect" (p. 70).

We have prepared your curriculum control experiment; it runs on the same cards after the private-testbed rerun. The partner's own passages are replaced, one for one, by sections of two open textbooks on other subjects (OpenStax Psychology 2e and Introduction to Sociology 3e), matched in length within 6 words and as closely as the books allow in reading level (Flesch-Kincaid grade 11.6 against 11.2). If the improvement on partner topics remains when the partner's curriculum is gone, it is general skill; if it falls, it is the partner's knowledge. Its table follows in the same follow-up comment.

"the main Bloom classifier has not been validated against human judgments in this domain."

The paper says so (p. 54), and every depth claim here is stated for Bcls as an automatic proxy. We ran the two checks that need no human labels.

First, the classifier against an LLM depth judge, question by question, on the questions generated in the submitted runs:

Questions scored by both estimatorsquestionsSpearmanKendall tau-b, per system
public testbed, four systems pooled32,9750.4680.404 to 0.420
private federation, four systems pooled59,6580.5910.514 to 0.565

Second, the ranking of the systems. On 500 test passages, each with one question from UniLearnLM, TC-Best and TC-Ens (Li and Wang, 2019), two LLM judges of different families scored all 1,500 questions under the Bloom rubric, blind to the system:

500 passage-paired questions, difference in Bloom levelsagainst TC-Bestagainst TC-Ens
Bcls+0.14 [+0.01, +0.26]+0.06 [-0.08, +0.20]
GPT-4o judge+0.02 [-0.09, +0.13]-0.05 [-0.15, +0.06]
Claude judge-0.01 [-0.06, +0.04]-0.03 [-0.09, +0.03]

As the tables show, the classifier and an LLM judge agree moderately question by question. But the two LLM judges (GPT-4o and Claude Opus 5.5) do not separate UniLearnLM from either federation: all four of their intervals include zero, while Bcls separates it from TC-Best only. So we present the depth result as a measurement by our classifier Bcls, not as one that independent judges confirm. On these questions the judges agree with each other more than either agrees with Bcls (Table 47 in Appendix AG, page 73). Neither check is a human validation.

Changes in the revision:

  • The public-testbed results of the main text are FiLM-aware, and every table or panel selected under the earlier rule is labelled so (Section 5.2, pages 7-8).
  • The depth clause removed from the abstract (page 1), and every depth claim stated as a Bcls measurement (Section 5.2, pages 7-8, and Limitations, page 9).
  • The LLM judge check is in Appendix AG (page 73), and the control experiments above are added there as they land.
  • The index renamed "improvement on partner topics" (Appendix R, pages 39-40).
  • The contribution stated as the interface (Section 1, page 2), with Hypothesis 4 not supported.

If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive.

Reviewer_R7sPOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

UniLearnLM enables institutions using heterogeneous small language models to train jointly. Each client retains a frozen pretrained backbone, private LoRA parameters, and a private FiLM adapter. A shared GNN reads the local model’s architecture graph and produces representations that FiLM converts into hidden-state modulation. Only the 87,744-parameter GNN is averaged across clients, avoiding direct alignment of language-model tensors or a shared text corpus within the proposed training protocol. The evaluation covers standard NLP tasks, educational question-answer generation, and a live deployment involving four participating institutions, with analyses of client-level gains, question depth, and comparison with zero-shot LLMs.

Critical strengths

1. A clear interface for heterogeneous models with low communication volume. The fixed-size shared GNN and private, dimension-specific FiLM adapters accommodate different backbones and tokenizers while retaining pretrained weights. Each round exchanges only 87,744 GNN parameters rather than the language models or private adapters (Section 3; Appendix K). This addresses a useful setting in which institutions cannot pool their curricula or adopt one common model. The concrete advantage is communication efficiency, rather than the absence of additional training cost.

2. Useful positive findings and transparent reporting of limitations. The three-seed CoNLL result improves over local training, with F1 of 0.7668 versus 0.7323 (Table 17). The automatic depth advantage over the evaluated text-consensus distillation methods is also relatively consistent across the public and private educational testbeds. The live deployment, relevant controls such as no-FedAvg, corrected reruns, and explicit reporting of negative transfer and limitations of the LLM comparison make the empirical study informative.

Critical weaknesses

The additional benefit of collaborative training needs clearer support. Table 37 keeps the same local models and adaptation modules and compares whether GNN parameters are exchanged. The automatic question-depth scores are 0.2028 for the full method and 0.2010 for independent training (no-FedAvg), with different data splits favoring different groups. This comparison alone leaves the benefit of collaboration uncertain. It would help to explain more clearly what collaboration contributes, drawing on the existing experiments and corrected results.

Questions for authors

Which existing result most clearly demonstrates a benefit from collaboration between institutions? Please briefly explain what exchanging GNN parameters adds over independent training, taking the corrected results and the no-FedAvg control into account.

4 comments

Thank you for a review that read our control experiments closely and found UniLearnLM "a clear interface for heterogeneous models with low communication volume", with "relevant controls such as no-FedAvg, corrected reruns, and explicit reporting of negative transfer". This response spans 2 comments; a third follows with the control experiments that are still running.

How to read the tables:

  • Bold marks the best value per column; cells are mean ± sd over the seeds or folds named in the header.
  • Bcls is our Bloom classifier's score of question depth on the revised taxonomy; one level is 0.2.
  • FiLM-aware round selection scores the deployed model, with its modulation active, and is the rule for every educational result here.
  • The new tables are in Appendix AG of the revised PDF (page 73).
  • Edits in the revised PDF appear in blue text, so that each change is easy to find.

"Which existing result most clearly demonstrates a benefit from collaboration between institutions? Please briefly explain what exchanging GNN parameters adds over independent training, taking the corrected results and the no-FedAvg control into account."

First, where UniLearnLM stands in the submitted results. Against independent training, it is ahead where you also see its clearer improvement. Over three seeds it is above local training on four of five single tasks, and on CoNLL-2003 it reaches 0.7668 ± 0.0025 against 0.7323 ± 0.0099 (paired t-test, p = 0.016; Table 17).

UniLearnLM's only exchange is an encoder of 87,744 parameters, and with it UniLearnLM reaches the same performance as FedHeNN (Makhija et al., 2022). FedHeNN is above the other two published heterogeneous federations, pFedHN (Shamsian et al., 2021) and FedMKT (Fan et al., 2025), on four of five tasks (Table 1), and it aligns representations on a shared public pool:

3 seeds, paired by seed (Table 20)CoNLL-2003 F1AG News acc.
FedHeNN (Makhija et al., 2022)0.7693 ± 0.00870.8679 ± 0.0113
UniLearnLM0.7695 ± 0.00590.8659 ± 0.0057
difference between FedHeNN and UniLearnLM-0.0003 ± 0.0074+0.0020 ± 0.0068

As the table shows:

  • UniLearnLM is at the same performance as FedHeNN on both tasks, so it reaches the accuracy of a federation that needs public data, without any, while exchanging only 87,744 parameters.
  • It serves four model families with different tokenizers, and it ran live at four institutions on three continents, with uneven participation (Section 5.2, page 8).

This comment continues our answer above.

"The automatic question-depth scores are 0.2028 for the full method and 0.2010 for independent training (no-FedAvg) [...] This comparison alone leaves the benefit of collaboration uncertain."

The paper reads Table 37 the same way: it "does not show that the averaging step produces the depth result, and we do not claim it does" (p. 63). That control experiment ran on the public testbed, where Bcls shows no improvement for UniLearnLM over local training. We therefore run it where UniLearnLM is ahead of local training, with the arms of each seed on the same card type. The control experiments are in progress on CoNLL-2003, and we post their table in a follow-up comment as soon as the cells land, within the discussion period.

On CoNLL the client-index control experiment is already at the same performance as UniLearnLM over three seeds (Table 20), so these cells separate the averaging step from the adapters UniLearnLM adds at every client.

On depth against the federations that share text, TC-Best and TC-Ens (Li and Wang, 2019), our Bloom classifier Bcls scores UniLearnLM above both under FiLM-aware round selection (+0.0405 against TC-Best and +0.0373 against TC-Ens, each with an interval that excludes zero; Table 36 in Appendix Z, page 63). On 500 test passages, each with one question from each system, two LLM judges of different families scored all 1,500 questions blind to the system:

500 passage-paired questions, difference in Bloom levelsagainst TC-Bestagainst TC-Ens
Bcls+0.14 [+0.01, +0.26]+0.06 [-0.08, +0.20]
GPT-4o judge+0.02 [-0.09, +0.13]-0.05 [-0.15, +0.06]
Claude judge-0.01 [-0.06, +0.04]-0.03 [-0.09, +0.03]

As the table shows, the two LLM judges (GPT-4o and Claude Opus 5.5) do not separate UniLearnLM from either federation: all four of their intervals include zero, while Bcls separates it from TC-Best only. So we present the depth result as a measurement by our classifier Bcls, not as one that independent judges confirm, and we do not count it as a benefit of collaboration.

Changes in the revision:

  • The LLM judge check is added to Appendix AG (page 73), and the CoNLL control experiments follow there as they land.
  • The abstract (page 1) states parity with the strongest published heterogeneous federation without any public data, and the introduction (Section 1, page 2) states the contribution as the interface, one exchanged encoder of 87,744 parameters. Section 5.2 (pages 7-8) states the depth result as a Bcls measurement, with the LLM judge check beside it.
  • The paragraph on transferring knowledge (Section 6, page 9) is removed from the main text until the curriculum control experiment that Reviewer KQmn asks for is in.

If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive.

This comment completes our answer on the no-FedAvg control experiment with the runs that were still going when we posted. It holds three tables, and we report each result as it came out.

1. Where UniLearnLM is ahead of independent training: the private testbed, under one checkpoint rule. UniLearnLM under FiLM-aware round selection against local training of the same five institutions, and against the no-FedAvg control experiment of point 2 on the same folds. All three ran on the same card type. Bcls on the global private test, mean of the five institutions:

FoldUniLearnLMno-FedAvgLocal training
10.19450.20260.1830
20.21640.19740.1714
30.23330.23860.1825
Mean ± sd over 3 folds0.2147 ± 0.01950.2129 ± 0.02240.1790 ± 0.0065

As the table shows:

  • UniLearnLM is above local training in 3 of 3 folds, 4 of 5 institutions and 12 of 15 institution-fold cells (0.2147 against 0.1790). With three folds the paired t-test does not reach significance (p = 0.1004), so we claim none.
  • UniLearnLM is at the same performance as the model without averaging (0.2147 against 0.2129; above in 1 of 3 folds, p = 0.8470).

Bcls is our classifier's measurement, as our first answer states.

2. What exchanging the encoder adds on CoNLL-2003: the no-FedAvg control experiment. It trains the same architecture, schedule and seed as UniLearnLM, with the encoder never averaged by FedAvg (McMahan et al., 2017). Both arms of each seed ran on the same card type. Entity F1 on the global test set, mean of the four clients:

SeedUniLearnLMno-FedAvgDifference between UniLearnLM and no-FedAvg
420.76290.7731-0.0102
10.76170.7683-0.0065
20.76090.7704-0.0096
Mean ± sd over 3 seeds0.7618 ± 0.00100.7706 ± 0.0024-0.0088 (paired t-test, p = 0.0163)

As the table shows, the model without averaging is above UniLearnLM in 3 of 3 seeds, and the paired test is significant. So on CoNLL-2003 the improvement of UniLearnLM over local training (Table 17) comes from the adapters that every client adds, and not from averaging the encoder. Two seeds on a second card type and one seed on a third give the same reading.

3. The setting where exchange should help most: a client with little data. The same control experiment with one change: client 0 trains on 10% of its training examples. Entity F1 of that client on the global test set:

SeedUniLearnLMno-FedAvgDifference between UniLearnLM and no-FedAvg
420.63990.6392+0.0007
10.67540.7081-0.0327
20.61070.6273-0.0166
Mean ± sd over 3 seeds0.6420 ± 0.03240.6582 ± 0.0436-0.0162 (paired t-test, p = 0.2350)

As the table shows, UniLearnLM is above the model without averaging in 1 of 3 seeds, and the difference is not significant. Exchange does not help the client with little data on this task either. Node failures interrupted the scoring of these runs, so the score files are assembled from the scoring passes, every file identical across the passes that wrote it.

Our answer to your question, with these results in:

  • The result that most clearly shows UniLearnLM ahead of independent training is the private testbed (first table), where the institutions hold different curricula.
  • On that testbed the same control experiment places UniLearnLM at the same performance as the model without averaging. So there too the improvement over local training comes from the adapters that every institution adds, and not from averaging the encoder.
  • On CoNLL-2003, where every client holds the same task, exchanging the encoder adds no accuracy. The accuracy that reaches FedHeNN (Makhija et al., 2022) without any public data is produced by the adapters, and the exchange costs 87,744 parameters per round.

Changes in the next revision: the three tables go into Appendix AG, and the main text states that on CoNLL-2003 and on the private testbed the improvement over local training comes from the adapters.

If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive. Thanks.

Dear Reviewer R7sP,

Thank you again for your review. As the discussion period closes, we want to make sure that our responses reached you.

Our third comment completes the answer to your main question, what exchanging the encoder adds, with the no-FedAvg control experiment on CoNLL-2003 over three seeds, the private testbed under one checkpoint rule, and a client with little data. On the private testbed, UniLearnLM is above local training in 3 of 3 folds, 4 of 5 institutions and 12 of 15 institution-fold cells (0.2147 against 0.1790).

If any point remains open, or if a further analysis would be decisive for you, please let us know and we will answer at once. If our responses address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.