- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper addresses accuracy and calibration in personalized federated LoRA fine-tuning and proposes FLAME, which has three parts:
After clustering clients by the similarity of their LoRA B matrices, a router combines a trainable individual LoRA, a frozen intra-cluster aggregate and frozen cross-cluster aggregates. On the server, A is aggregated after alignment on a shared subspace (the Grassmannian aggregator), while B is averaged directly. After training, each client fits a local KFAC Laplace approximation to obtain calibrated predictions. On 12-client classification and 8-client generation benchmarks, FLAME improves accuracy, ECE and NLL over the baselines.
Critical strengths
- The Laplace method is cheap.
- The experiments are very comprehensive; multiple methods were compared, and ablation studies were also conducted.
- Convergence analysis is provided.
Critical weaknesses
- Figure 2 looks very much like it was generated by AI; the resolution is very low. However, the AI statement doesn't mention it.
- Laplace can be applied to other baselines and bring significant improvement in calibration. The calibration gain attributable to FLAME's own design is small.
- The theory contributes little. The convergence theorem is a standard SGD result for a single client on a slowly changing objective and does not analyze interaction between clients.
- The improvement in accuracy is very small. Accuracy gains are within seed variation. Only three seeds are used and no significance tests are reported.
- The models used in this paper are old. (Llama-2-7B and Llama-3.2-1B),
Questions for authors
- Is the A aggregate invariant when clients' (A, B) pairs differ by different R? Why not align A and B jointly or aggregate the products BA? Do clients start from the same initialization?
- Can you add more seeds and paired significance tests? Why are some standard deviations zero in the generation results?(0.065±0.000, 0.727±0.000)
- Why Table 2 reports identical accuracy with and without Laplace (80.86±0.35)?
- Can you use advanced models to evaluate the performance of FLAME?
2 comments
Gentle Follow-up for Reviewer egpy on Our Rebuttal
Dear Reviewer egpy,
Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.
- Calibration (W2). FLAME keeps the lowest ECE and NLL when every baseline uses the same Laplace stage, and FLAME without the Grassmannian aggregator also stays behind FLAME under this stage, so the remaining gap is attributable to FLAME's design.
- Theory (W3). Theorem 1 couples each client to the others through its frozen experts, whose round-to-round drift enters the bound, and Lemma 6 shows that the Grassmannian aggregate drifts no more than the Euclidean mean.
- Significance (W4 + Q2). With seven seeds, FLAME's accuracy gain over FedAvg, FedDPA, FedALT, and FedLEASE is significant under both paired tests.
- Newer model (W5 + Q4). On Qwen3-1.7B-Base, FLAME again achieves the highest accuracy and the lowest ECE.
- Questions (Q1, Q3). The subspace of the aggregate is invariant to client-specific factorizations, all clients share the same initialization, and the Laplace stage leaves the MAP accuracy unchanged, which is why the two rows of Table 2 share the same accuracy.
We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.
Rebuttal to Reviewer egpy
We thank the reviewer for noting that the Laplace method is cheap, that the experiments are very comprehensive, with multiple compared methods and ablation studies, and that a convergence analysis is provided.
W1
Figure 2 is a schematic illustration only. FLAME is fully specified by Eqs. (3)–(12) and Algorithm 1, which define every component shown in the figure. We will replace it with a high-resolution figure.
W2
FLAME stays best when every baseline uses the same Laplace stage. Table 10 applies the same post-hoc Laplace stage to every baseline, and FLAME keeps the lowest ECE and NLL. FLAME without the Grassmannian aggregator also stays behind FLAME under the same stage. The remaining gap is therefore attributable to FLAME's design.
W3
Theorem 1 couples each client to the others through its frozen experts. Every frozen expert of client n is aggregated from other clients' uploaded factors (Eqs. (6)–(9)), and the round-to-round change of these experts, ΔT, enters Eq. (13) through the drift term 2CSΔT/(ηK). The bound therefore depends on how the other clients' updates move the frozen experts. The aggregator tightens this coupling term. Lemma 6 (Appendix C.4) shows that the Grassmannian aggregate drifts no more than the Euclidean mean (Eq. (28)).
W4 + Q2
More seeds and paired tests. We add four seeds to the three of Table 1 on MCQA-12c and test FLAME's accuracy against FedAvg, FedDPA, FedALT, and FedLEASE with paired tests over all seven seeds. Both tests are one-sided, in the direction of FLAME improving on the baseline.
FLAME's accuracy gain over all four baselines is significant under both tests.
Zero standard deviations. The three seeds differ by less than 0.001, so the standard deviation rounds to 0.000 at three decimals.
W5 + Q4
FLAME's gains carry over to a newer model family. We add Qwen3-1.7B-Base on MCQA-12c under the same protocol, reporting mean ± standard deviation over three seeds:
On the newer backbone, FLAME again achieves the highest accuracy and the lowest ECE.
Q1
Q3
Accuracy is that of the MAP prediction. The post-hoc Laplace stage leaves the trained weights unchanged and provides the predictive distribution of Eq. (12), which ECE and NLL evaluate. The two rows of Table 2 therefore share the same accuracy.