- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Unified Learning with Language Models (UniLearnLM), an equitable approach to LLM approximation that fine-tunes small language models (SLMs) through GNN-based unified federated learning. Four hypotheses anchor the evaluation.
Critical strengths
- This paper describes a federation across institutions in four countries. I guess this means real-world multi-institution deployment.
- This paper has a broad evaluation and controls.
- The paper also honestly reports negative and fragile results.
Critical weaknesses
- This paper looks like a very comprehensive combination of SLM, GNN, and federated learning that builds a pretty comprehensive system, making it more of an engineering effort than a methodological contribution.
- The paper seems to show that federated learning doesn't help much? Though negative results are appreciated. More detailed analyses or theoretical research would be helpful in explaining this.
- The LLM-approximation claim does not seem to be largely supported. The SLM mixtures score way lower than LLMs.
- This paper is surprisingly long, with 72 pages. But much of the appendix reads like an accumulated rebuttal log rather than a paper.
Questions for authors
- Code will be released upon acceptance. It would be nice if the code were actually released.
- The main results are obtained with a single seed. Better to run over 3 seeds.
6 comments
This comment continues our answer above.
"The LLM-approximation claim does not seem to be largely supported. The SLM mixtures score way lower than LLMs."
On grounding, UniLearnLM's small models are 1st of all 7 systems on sentence-level faithfulness of the full answers, under both ways of combining the three institutions' models:
| T global, 916 questions, all seven systems scored by one code, with the difference to the best LLM [95% interval over the 179 test contexts] | Faith, sentence level | QAFactEval |
|---|---|---|
| UniLearnLM, mean over the three institutions' models | 0.1903 (+0.029 [+0.005, +0.053]) | 0.1752 (+0.019 [-0.008, +0.045]) |
| UniLearnLM, each question by its own institution's model | 0.1875 (+0.026 [+0.002, +0.051]) | 0.1690 (+0.012 [-0.017, +0.042]) |
| best of six LLMs | 0.1610, Gemini-2.5-Flash (Comanici et al., 2025) | 0.1567, Llama-3.1-8B (Grattafiori et al., 2024) |
As the table shows, the faithfulness interval excludes zero in both forms. On QAFactEval UniLearnLM has the highest mean, but the interval includes zero, so it is at the same performance as the best of the six LLMs. Of the nine metrics of Table 34, these are the two on which it is 1st. With every answer cut to 15 words it again has the highest mean on both measures: the mean over the models reaches 0.1873 against 0.1714 for Mistral-7B (Jiang et al., 2023) on faithfulness, and 0.1583 against 0.1376 for Llama-3.1-8B on QAFactEval (Table 23). There too it is at the same performance as the best LLM, since every interval includes zero. So small models that stay inside the institutions ground their answers at least as well as the six LLMs, and no passage leaves an institution. Table 34 uses the first form, and the second is the system as deployed. Sentence-level faithfulness checks each sentence of an answer against the passage, and the revised Tables 23 and 34 use it for all seven systems. On the holistic LLM judge the small models trail all six LLMs (Table 34). The submitted paper states that Hypothesis 2 "holds for grounding and not for lexical overlap or the holistic judge". The revision narrows Hypothesis 2 to sentence-level faithfulness of full answers (Section 5.3, page 8).
"The main results are obtained with a single seed. Better to run over 3 seeds."
The paper already runs its central comparison, UniLearnLM against local training, over seeds 42, 1 and 2 (Table 17):
| five single-task federations, 3 seeds (Table 17) | CoNLL | SQuAD | AG News | XSum | SAMSum |
|---|---|---|---|---|---|
| local training | 0.7323 ± 0.0099 | 0.7614 ± 0.0185 | 0.8118 ± 0.0446 | 0.2525 ± 0.0015 | 0.3897 ± 0.0021 |
| UniLearnLM | 0.7668 ± 0.0025 | 0.7697 ± 0.0070 | 0.8664 ± 0.0065 | 0.2541 ± 0.0011 | 0.3873 ± 0.0060 |
As the table shows, UniLearnLM is above local training on four of five single tasks, significantly on CoNLL (paired t-test, p = 0.016), with a smaller spread across seeds on four of five. Three seeds also exist for FedHeNN (Makhija et al., 2022) and both graph control experiments (Table 20) and for the partition control experiments (Table 19). The educational tables are the ones at one seed.
"much of the appendix reads like an accumulated rebuttal log rather than a paper."
The revised PDF names one checkpoint rule for the main text (Section 5.2, pages 7-8) and labels every appendix table selected under the earlier rule. The paragraphs that narrated reruns and corrections are removed from the appendix. Every table keeps its number, so every table number cited here stays valid, and each table states which selection rule it uses (the two rules are defined in Appendix C.3, page 24).
"It would be nice if the code were actually released."
The anonymized code is available at https://figshare.com/s/f643420b029b2ed607d9. The revised Reproducibility statement (page 10) now says, as the abstract does, that the code will be released upon acceptance.
If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive.
You asked for the main results over three seeds, and our earlier comment said that the educational tables were the ones at one seed. We ran it. This comment gives the result as it came out.
Three seeds on the educational public testbed. UniLearnLM under FiLM-aware round selection and local training of the same three institutions, with seeds 1 and 2 added to the seed of the submitted tables, three folds each. Bcls on the global test, mean of the three institutions and the three folds:
| Seed | UniLearnLM | Local training | Difference between UniLearnLM and local training |
|---|---|---|---|
| 42 | 0.2028 | 0.2238 | -0.0210 |
| 1 | 0.2093 | 0.1779 | +0.0314 |
| 2 | 0.2059 | 0.1815 | +0.0245 |
| Mean ± sd over 3 seeds | 0.2060 ± 0.0033 | 0.1944 ± 0.0255 | +0.0116 (paired t-test, p = 0.5527) |
As the table shows:
- UniLearnLM is above local training in 2 of 3 seeds (0.2060 against 0.1944 on average). With three seeds the paired t-test does not reach significance, so we claim none.
- UniLearnLM varies less across seeds than local training (sd 0.0033 against 0.0255).
- In the two added seeds both methods ran on the same card type in the same days, and UniLearnLM is above local training in 6 of 6 folds. Seed 42 is the row of the submitted tables, whose two methods were run months apart.
- TC-Ens (Li and Wang, 2019) was run at seed 42 only. UniLearnLM is above it at that seed (0.2028 against 0.1655).
The other three lightweight scores of Table 2, from the same runs. Mean ± sd over the three seeds:
| Score | UniLearnLM | Local training | Difference between UniLearnLM and local training | Seeds where UniLearnLM is above |
|---|---|---|---|---|
| ROUGE-L (Lin, 2004) | 0.4405 ± 0.0025 | 0.4329 ± 0.0033 | +0.0075 (paired t-test, p = 0.0555) | 3 of 3 |
| BLEU-4 (Papineni et al., 2002) | 0.1723 ± 0.0025 | 0.1618 ± 0.0024 | +0.0105 (p = 0.0159) | 3 of 3 |
| BERTScore (Zhang et al., 2020) | 0.8644 ± 0.0008 | 0.8603 ± 0.0009 | +0.0041 (p = 0.0030) | 3 of 3 |
As the table shows, UniLearnLM is above local training on all three scores in 3 of 3 seeds, and the paired t-test is significant on BLEU-4 and BERTScore. On ROUGE-L it does not reach significance.
Changes in the next revision: Table 2 reports UniLearnLM and local training as mean ± sd over three seeds on these four scores, and the text states that their difference in Bcls is not significant.
If these additions address the concerns behind your assessment, we would be grateful if you would consider updating your evaluation, and we are happy to run any further analysis you consider decisive. Thanks.
Thank you for a review that credits the paper with "a broad evaluation and controls", a federation "across institutions in four countries", and for the fact that it "honestly reports negative and fragile results". This response spans 2 comments.
How to read the tables:
The method is one idea: share a model of each client's architecture instead of weights, outputs or text. That idea lets models that cannot be averaged learn together without any public corpus. The interface it gives is new among the methods we compare:
With it, UniLearnLM reaches the same performance as FedHeNN over three seeds (difference between FedHeNN and UniLearnLM: -0.0003 ± 0.0074 on CoNLL-2003, +0.0020 ± 0.0068 on AG News, neither significant; Table 20). It serves four model families with different tokenizers, and it ran live at four institutions on three continents (p. 9).
UniLearnLM is ahead of local training on CoNLL-2003 over three seeds (0.7668 against 0.7323, p = 0.016; Table 17) and above it on four of five single tasks (next comment). Whether the averaging step adds accuracy over the same model trained alone is the no-FedAvg control experiment. On the public testbed it is Table 37. On CoNLL-2003 the control experiments are in progress; their table follows in our answer to Reviewer R7sP.
On question depth, our Bloom classifier Bcls scores UniLearnLM above both federations that share text under FiLM-aware round selection (+0.0405 against TC-Best and +0.0373 against TC-Ens) and below local training; all three intervals exclude zero (Table 36 in Appendix Z, page 63). The holistic LLM judge of the paper (LLM-J) puts it above local training in all three folds (0.6944 against 0.6831), without significance (p = 0.078). On 500 passage-paired test questions, two LLM judges blind to the system (GPT-4o and Claude Opus 5.5) do not separate UniLearnLM from TC-Best or from TC-Ens: all four of their intervals include zero, while Bcls separates it from TC-Best only (Table 47 in Appendix AG, page 73). So we present the depth result as a measurement by our classifier Bcls, not as one that independent judges confirm.