acceptodds
Under review as a conference paper at ICLR 2027

FLAME: Federated Laplace-Aligned Mixture of Experts for Personalized LLM Fine-Tuning

Abstract

Personalized federated fine-tuning of large language models calls for adaptation to heterogeneous client tasks together with reliable uncertainty estimates under limited local data. Existing personalized federated low-rank adaptation (LoRA) methods expose only a single level of cross-client locality, average the LoRA factors and identically despite their structural asymmetry, and return *point* estimate adapters whose predictions remain poorly calibrated. To address these limitations, we propose **FLAME**, a **F**ederated **L**aplace-**A**ligned **M**ixture of **E**xperts whose three design choices are each grounded in the geometry of the learned parameter space. To capture hierarchical locality across client adaptations, FLAME jointly routes among self, intra-cluster, and cross-cluster experts, with the trainable individual expert active in every forward pass. To resolve the LoRA basis ambiguity that Euclidean aggregation introduces, FLAME aligns the input-side factor on the Grassmann manifold and keeps the output-side factor in Euclidean space, where the entry-level task-specific signals live. For *calibrated* per-client predictive distributions, FLAME fits a local post-hoc Kronecker-factored Laplace posterior over each client’s individual expert, leaving training and communication untouched. We further establish a non-asymptotic convergence guarantee for FLAME and show that the Grassmannian aggregator tightens the aggregation-drift more than Euclidean averaging. Experimental results across -client MCQA classification benchmark and -client FLAN NLG benchmarks spanning IID, cross-dataset, and label-skew heterogeneity show that FLAME consistently improves both accuracy and calibration (ECE, NLL) over all baselines.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Personalized federated learning, Parameter-efficient fine-tuning, Mixture of experts, Grassmannian aggregation, Laplace approximation, Uncertainty quantification

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_egpyOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses accuracy and calibration in personalized federated LoRA fine-tuning and proposes FLAME, which has three parts:

After clustering clients by the similarity of their LoRA B matrices, a router combines a trainable individual LoRA, a frozen intra-cluster aggregate and frozen cross-cluster aggregates. On the server, A is aggregated after alignment on a shared subspace (the Grassmannian aggregator), while B is averaged directly. After training, each client fits a local KFAC Laplace approximation to obtain calibrated predictions. On 12-client classification and 8-client generation benchmarks, FLAME improves accuracy, ECE and NLL over the baselines.

Critical strengths
  1. The Laplace method is cheap.
  2. The experiments are very comprehensive; multiple methods were compared, and ablation studies were also conducted.
  3. Convergence analysis is provided.
Critical weaknesses
  1. Figure 2 looks very much like it was generated by AI; the resolution is very low. However, the AI ​​statement doesn't mention it.
  2. Laplace can be applied to other baselines and bring significant improvement in calibration. The calibration gain attributable to FLAME's own design is small.
  3. The theory contributes little. The convergence theorem is a standard SGD result for a single client on a slowly changing objective and does not analyze interaction between clients.
  4. The improvement in accuracy is very small. Accuracy gains are within seed variation. Only three seeds are used and no significance tests are reported.
  5. The models used in this paper are old. (Llama-2-7B and Llama-3.2-1B),
Questions for authors
  1. Is the A aggregate invariant when clients' (A, B) pairs differ by different R? Why not align A and B jointly or aggregate the products BA? Do clients start from the same initialization?
  2. Can you add more seeds and paired significance tests? Why are some standard deviations zero in the generation results?(0.065±0.000, 0.727±0.000)
  3. Why Table 2 reports identical accuracy with and without Laplace (80.86±0.35)?
  4. Can you use advanced models to evaluate the performance of FLAME?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer egpy

We thank the reviewer for noting that the Laplace method is cheap, that the experiments are very comprehensive, with multiple compared methods and ablation studies, and that a convergence analysis is provided.

W1

Figure 2 is a schematic illustration only. FLAME is fully specified by Eqs. (3)–(12) and Algorithm 1, which define every component shown in the figure. We will replace it with a high-resolution figure.

W2

FLAME stays best when every baseline uses the same Laplace stage. Table 10 applies the same post-hoc Laplace stage to every baseline, and FLAME keeps the lowest ECE and NLL. FLAME without the Grassmannian aggregator also stays behind FLAME under the same stage. The remaining gap is therefore attributable to FLAME's design.

W3

Theorem 1 couples each client to the others through its frozen experts. Every frozen expert of client nn is aggregated from other clients' uploaded factors (Eqs. (6)–(9)), and the round-to-round change of these experts, ΔT\Delta_T, enters Eq. (13) through the drift term 2CSΔT/(ηK)2C_S\Delta_T/(\eta K). The bound therefore depends on how the other clients' updates move the frozen experts. The aggregator tightens this coupling term. Lemma 6 (Appendix C.4) shows that the Grassmannian aggregate drifts no more than the Euclidean mean (Eq. (28)).

W4 + Q2

More seeds and paired tests. We add four seeds to the three of Table 1 on MCQA-12c and test FLAME's accuracy against FedAvg, FedDPA, FedALT, and FedLEASE with paired tests over all seven seeds. Both tests are one-sided, in the direction of FLAME improving on the baseline.

FLAME vs.Wilcoxon signed-rank p (84 client–seed pairs)Paired t-test p (7 seeds)
FedAvg2.8e-106.4e-5
FedDPA0.00470.027
FedALT0.0110.010
FedLEASE0.0280.041

FLAME's accuracy gain over all four baselines is significant under both tests.

Zero standard deviations. The three seeds differ by less than 0.001, so the standard deviation rounds to 0.000 at three decimals.

W5 + Q4

FLAME's gains carry over to a newer model family. We add Qwen3-1.7B-Base on MCQA-12c under the same protocol, reporting mean ± standard deviation over three seeds:

MethodAcc ↑ECE ↓NLL ↓
FLAME87.22 ± 1.280.086 ± 0.0040.87 ± 0.11
FedAvg85.83 ± 2.050.118 ± 0.0170.69 ± 0.08
FedALT86.33 ± 0.800.135 ± 0.0071.77 ± 0.10
FedLEASE86.75 ± 0.720.133 ± 0.0061.75 ± 0.16

On the newer backbone, FLAME again achieves the highest accuracy and the lowest ECE.

Q1

  • Invariance. The subspace of the AA aggregate is invariant: by Lemma 5 (Appendix C.4), the consensus projector used in Eq. (8) is unchanged under client-specific Am↦RmAmA_m\mapsto R_mA_m, Rm∈GL(r)R_m\in\mathrm{GL}(r).
  • Aligning AA and BB jointly. Aligning BB as well as AA (Grass–Grass in Table 8b) lowers accuracy relative to FLAME's Grass–Eucl, consistent with Observation 2 that BB carries task identity in its entries rather than in its subspace.
  • Aggregating BABA. Averaging the products BmAmB_mA_m yields an update of rank up to ∣P∣ r|\mathcal P|\,r, which no longer has the rank-rr form of an expert in Eq. (4).
  • Initialization. Yes. All clients start from the same initialization, Kaiming for AA and zero for BB (Sections 2.1 and 3.1).

Q3

Accuracy is that of the MAP prediction. The post-hoc Laplace stage leaves the trained weights unchanged and provides the predictive distribution of Eq. (12), which ECE and NLL evaluate. The two rows of Table 2 therefore share the same accuracy.

AuthorsOpenReview ↗

Gentle Follow-up for Reviewer egpy on Our Rebuttal

Dear Reviewer egpy,

Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.

  • Calibration (W2). FLAME keeps the lowest ECE and NLL when every baseline uses the same Laplace stage, and FLAME without the Grassmannian aggregator also stays behind FLAME under this stage, so the remaining gap is attributable to FLAME's design.
  • Theory (W3). Theorem 1 couples each client to the others through its frozen experts, whose round-to-round drift enters the bound, and Lemma 6 shows that the Grassmannian aggregate drifts no more than the Euclidean mean.
  • Significance (W4 + Q2). With seven seeds, FLAME's accuracy gain over FedAvg, FedDPA, FedALT, and FedLEASE is significant under both paired tests.
  • Newer model (W5 + Q4). On Qwen3-1.7B-Base, FLAME again achieves the highest accuracy and the lowest ECE.
  • Questions (Q1, Q3). The subspace of the AA aggregate is invariant to client-specific factorizations, all clients share the same initialization, and the Laplace stage leaves the MAP accuracy unchanged, which is why the two rows of Table 2 share the same accuracy.

We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.

Reviewer_Mzy8OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposed FLAME, a scheme for personalized federated fine-tuning with LoRA. The paper first presents empirical observations about how LoRA factor B and A behave in optimization, and suggest using B as an indication for client’s similarity and use Grassmann aggregation for A. To further enhance client downstream calibration, FLAME use post-hoc KFAC Laplace approximation on its expert and derive uncertainty.

Critical strengths

The paper presents an interesting intuition that identifies the roles for B and A in LoRA(A commonly carries shared information across subspace compared with B).

Critical weaknesses
  1. In Table 2’s ablation study, it seems that the main source of performance contribution of FLAME is centered on Laplace approximation. In other cases(w/o Lap or w/o both grass and lap), FLAME is having ECE of around 0.204, which is even larger compared with other baselines(e.g. FedDPA, 0.199). However, in table 10, it seems FedDPA+Laplace approximation yielding larger ECE compared with FLAME. Could the author explain why this is the case?
  2. The author put related work into the appendix. It is highly suggested for author to put related work in the main text to ensure fairness in terms of page limit.
  3. Some writing inconsistencies: a. It seems section B2 in appendix has no words and not complete? b. Equation 13 for theorem 1 contains some undefined notations (e.g. C_S not defined in main text).
Questions for authors

See weakness

2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer Mzy8

We thank the reviewer for finding the intuition that identifies the distinct roles of AA and BB in LoRA interesting, with AA carrying shared information at the subspace level compared with BB.

W1

The accuracy gain comes from the architecture. The Laplace stage leaves accuracy unchanged (Table 2), so FLAME's accuracy gain over the baselines in Table 1 comes entirely from its routing and aggregation.

Before Laplace, all point-estimate methods lie in the same ECE range. This is Observation 3: FLAME without Laplace and the baselines each return a single MAP adapter, and their ECE values are close regardless of the routing design (Tables 1 and 2).

After Laplace, the architecture determines how much the same stage helps. In FLAME, the posterior is placed on the individual expert, and the router keeps this expert active at every token with a gate weight bounded away from zero (Eqs. (4)–(5)), so every posterior sample enters the prediction (Remark in Section 3.3). The Grassmannian aggregator also matters after Laplace: FLAME without it stays behind FLAME under the same stage (Table 10). The ECE of the MAP adapter therefore does not determine the ECE after Laplace, and FLAME's architecture is the one that benefits most from the same posterior.

W2

The closest methods are discussed in the main text. Section 1 compares FLAME with the closest personalized federated LoRA methods (FedDPA, FedALT, FedLEASE, and FLoRG) and with Bayesian LoRA methods, and Figure 3 compares their designs along locality, aggregation geometry, and uncertainty treatment. Appendix A provides the full related work. We will move a condensed related-work section into the main text.

W3

Appendix B.2 contains Table 2, which is discussed in Section 5.3 (Ablations).

CSC_S is the constant of Assumption 4 (Appendix C.2), the Lipschitz constant of the local loss with respect to the frozen-expert configuration: ∣Fn(θ;ξ;Eˉ,Gn)−Fn(θ;ξ;Eˉ′,Gn)∣≤CS∥Eˉ−Eˉ′∥|F_n(\theta;\xi;\bar{\mathbf E},G_n)-F_n(\theta;\xi;\bar{\mathbf E}',G_n)|\le C_S\|\bar{\mathbf E}-\bar{\mathbf E}'\|. It converts the cumulative frozen-expert drift ΔT\Delta_T into the term 2CSΔT/(ηK)2C_S\Delta_T/(\eta K) of Eq. (13). We will define CSC_S in the main text before Theorem 1.

AuthorsOpenReview ↗

Gentle Follow-up for Reviewer Mzy8 on Our Rebuttal

Dear Reviewer Mzy8,

Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.

  • Source of the gains (W1). The Laplace stage leaves accuracy unchanged, so FLAME's accuracy gain over the baselines comes entirely from its routing and aggregation. Before Laplace, all point-estimate methods lie in the same ECE range (Observation 3). After Laplace, FLAME benefits most from the same stage, because its router keeps the individual expert, on which the posterior is placed, active at every token, and FLAME without the Grassmannian aggregator stays behind FLAME under this stage.
  • Related work (W2). Section 1 and Figure 3 compare FLAME with the closest personalized federated LoRA and Bayesian LoRA methods, and we will move a condensed related-work section into the main text.
  • Writing (W3). Appendix B.2 contains Table 2, and CSC_S is the Lipschitz constant of Assumption 4, which we will define in the main text before Theorem 1.

We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.

Reviewer_qK2COpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FLAME, a framework for personalized federated LoRA fine-tuning that combines hierarchical expert sharing, factor-specific aggregation, and local post-hoc uncertainty estimation. Each client routes among an individual adapter, an aggregate of same-cluster peers, and aggregates from other clusters. The server aggregates the input-side LoRA factor using subspace geometry while averaging the output-side factor in Euclidean space. After training, each client applies a KFAC Laplace approximation to improve predictive calibration. Experiments on classification and generation benchmarks show improvements over the evaluated baselines.

Critical strengths
  1. The paper addresses a practically relevant objective: improving both personalized task performance and prediction calibration under heterogeneous client data.
  2. The empirical observations concerning the different structures of the two LoRA factors provide a concrete motivation for asymmetric aggregation.
  3. The evaluation includes useful component ablations, local-only comparisons, temperature-scaling controls, Laplace-augmented baselines, and system measurements. These help distinguish the contributions of the architecture, aggregation, and uncertainty estimation.
Critical weaknesses
  1. The methodological novelty over existing work is not sufficiently established. Task heterogeneity, LoRA aggregation mismatch, and overconfidence are established problems. Existing methods already provide local/global personalization, rest-of-world adapters, clustered experts, and geometry-aware aggregation. The paper should more precisely explain what limitation of the closest methods requires these particular changes, and why their combination constitutes a substantial methodological advance.

  2. The proposed aggregation does not clearly resolve the basis ambiguity used to motivate it. Although the consensus subspace is invariant to a change of basis, the subsequent coefficient averaging remains dependent on the clients’ factor representations, and the output-side factors are averaged independently. For example, two clients with simultaneously sign-flipped factors represent identical effective updates, but their aggregated factors can still cancel. This does not invalidate the empirical gains, but it challenges the stronger claim that the complete aggregation operator resolves LoRA’s basis ambiguity.

  3. The uncertainty formulation needs clarification. The method is described as fitting a posterior over the individual LoRA factors, whereas the sampling procedure is formulated over their effective weight matrix. These are different parameterizations, and samples of the effective matrix need not preserve the adapter’s low rank. The paper should clarify which parameter space is used in implementation and how the curvature approximation and sampling procedure correspond to it.

  4. The calibration advantage should be interpreted relative to calibrated baselines. The paper already shows that applying the same Laplace stage to existing methods substantially narrows the gap: FedALT reaches ECE 0.081 compared with FLAME’s 0.064. This suggests that much of the headline calibration improvement comes from the established post-hoc technique. Moreover, Table 10 reports per-client best results between MAP and Laplace; the selection protocol and whether it uses evaluation labels should be specified.

Questions for authors
  1. What is the main technical advance over combining an existing clustered personalization method, a geometry-aware aggregator, and local Laplace calibration?
  2. Is the complete aggregation operator invariant to equivalent LoRA factorizations, or is only the estimated consensus subspace invariant?
2 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer qK2C

We thank the reviewer for recognizing that the paper addresses a practically relevant objective of improving both personalized task performance and calibration under heterogeneous client data, that the empirical observations on the two LoRA factors provide a concrete motivation for asymmetric aggregation, and that the component ablations, local-only comparisons, temperature-scaling controls, Laplace-augmented baselines, and system measurements help distinguish the contributions of the architecture, aggregation, and uncertainty estimation.

W1 + Q1

Each existing component falls short of its counterpart in FLAME.

  • Clustered personalization: FedLEASE, the clustered mixture-of-experts method, stays behind FLAME even when both use the same Laplace stage (Table 10).
  • Individual-plus-rest-of-world personalization: FedALT also stays behind FLAME under the same Laplace stage (Table 10).
  • Geometry-aware aggregation: applying the Grassmannian aggregator to both AA and BB lowers accuracy relative to FLAME's selective aggregator (Table 8b), and replacing it with Euclidean averaging leaves FLAME worse calibrated under the same Laplace stage (Table 10).

The advance is the design that makes these parts work together.

  • Locality. Existing personalized routers expose a single level of cross-client locality (Section 1). FLAME routes jointly over the individual, intra-cluster, and cross-cluster levels, and each level contributes (Table 8a).
  • Aggregation. FLAME aggregates the two LoRA factors differently, following the measured asymmetry: AA shares structure only in its row space and only within clusters (Observation 1), whereas BB carries task identity in its entries (Observation 2).
  • Calibration. The router keeps the individual expert, on which the posterior is placed, active at every token with a gate weight bounded away from zero (Eqs. (4)–(5)), so every posterior sample enters the prediction (Remark in Section 3.3).

W2 + Q2

The consensus subspace is invariant. By Lemma 5 (Appendix C.4), the projectors Qm⊤QmQ_m^\top Q_m, their average PˉP\bar P_{\mathcal P} (Eq. (6)), and the consensus projector are unchanged under any client-specific Am↦RmAmA_m\mapsto R_mA_m with Rm∈GL(r)R_m\in\mathrm{GL}(r). The row space onto which Eq. (8) projects the aggregate therefore does not depend on how each client factorizes its update. This is the ambiguity the aggregator targets: within a cluster, clients' AA factors share a row space (Observation 1, Figure 1(a)), and the projector identifies this row space regardless of each client's basis.

Peers start from a common frame. All clients start from the same initialization of AA and BB (Section 3.1), and each peer set lies within a single cluster (Section 3.2), so the factors averaged in Eqs. (8)–(9) start from a common frame rather than from arbitrary gauges such as the sign-flipped pair.

W3

The posterior is over the individual-expert factors (An,Bn)(A_n,B_n), as described in Section 3.3, and the implementation uses the same parameter space. As in Laplace-LoRA (Yang et al., 2024), AnA_n and BnB_n are treated as two separate linear layers, and both the KFAC curvature and the posterior samples are in the coordinates of (An,Bn)(A_n,B_n), so each sample remains a rank-rr update Bn,sAn,sB_{n,s}A_{n,s}. Details are in our response to Reviewer uPki, Q2. We will clarify this in Eqs. (10)–(12).

W4

FLAME stays best when every baseline uses the same Laplace stage. In Table 10, FLAME has the lowest ECE and NLL. The gain is not reproduced by Laplace alone: FLAME without the Grassmannian aggregator also stays behind FLAME under the same stage.

The same selection rule is applied to every method. For each method and client, Table 10 takes the better of the MAP and Laplace predictions on the evaluation labels, with the identical rule for FLAME and for every baseline, so all methods are compared under the same protocol.

AuthorsOpenReview ↗

Gentle Follow-up for Reviewer qK2C on Our Rebuttal

Dear Reviewer qK2C,

Thank you again for your review. As the discussion period closes today, we would like to briefly summarize our response.

  • Technical advance (W1 + Q1). Each existing component falls short of its counterpart in FLAME: FedLEASE and FedALT stay behind FLAME under the same Laplace stage, and applying the Grassmannian aggregator to both factors lowers accuracy relative to FLAME's selective aggregator. The advance is the design that makes these parts work together: three levels of locality that each contribute, aggregation that follows the measured asymmetry of the two factors, and a router that keeps the individual expert, on which the posterior is placed, active at every token.
  • Invariance (W2 + Q2). The consensus subspace onto which the aggregate is projected is invariant to each client's factorization (Lemma 5), and all clients start from the same initialization, so the factors averaged within a cluster share a common frame.
  • Posterior (W3). The posterior is over the individual-expert factors, and both the KFAC curvature and the samples are in these coordinates, so each sample remains a rank-rr update.
  • Calibration (W4). FLAME keeps the lowest ECE and NLL when every baseline uses the same Laplace stage, and FLAME without the Grassmannian aggregator also stays behind FLAME under this stage.

We sincerely appreciate your time and would be grateful for any feedback on whether our response addresses your concerns. If it does, we would be grateful if you would kindly consider updating your rating.

Reviewer_uPkiOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies personalized federated LLM fine-tuning problems and proposes an FLAME method. FLAMEintegrates multiple LoRA experts (individual, intra-cluster, and cross-cluster). Compared to all baseline methods, the results demonstrate that FLAME consistently achieves improvements in both accuracy and predictive calibration.

Critical strengths

This paper addresses a significant challenge: how to balance personalization, knowledge sharing, and prediction calibration in scenarios where client tasks vary and data is limited. This approach performs posterior fitting locally without requiring additional communication. Furthermore, the model architecture is intuitive, easy to understand, and holds practical value. The experimental comparisons are sufficiently comprehensive, and the performance gains demonstrated by the results show great promise.

Critical weaknesses

I have some concerns regarding the theoretical rationale behind the core method, the rigor of the theoretical analysis, and the persuasiveness of the experimental results. Furthermore, it is difficult to determine whether the improvements in calibration performance stem from the architecture itself or from the post-processing steps. Finally, it is also questionable whether an appropriate balance has been struck between performance gains and computational overhead.

Questions for authors

[1] The sum may be zero when B1​​=aB_1 ​​= a and B2=−aB_2 = -a in Eqs. (8)–(9); then how do we deal with the boundary?

[2] The posterior parameterization requires further clarification (see Equations (10)–(12)). While the text describes the posterior distribution in terms of (A,B)(A, B), Equation (11) actually involves sampling a dense effective weight matrix W=BAW = BA, which typically possesses a higher rank. Please present the random variables, curvature approximation, prior and likelihood normalization, and explain how the MAP argument applies in this context carefully. Moreover, I want to know the posterior-state storage and sampling implementation on the 7B model.

[3] Existing results suggest that this combination of methods holds some potential under specific cases. However, it remains unclear why it works or what the specific contribution of each module is. Thus, I have some concerns regarding whether it can operate reliably in real-world federated learning scenarios.

[4] How the proposed method operates and performs in the following scenarios: 1) when a client has multiple tasks; 2) when data scales vary across clients or some clients are missing; and 3) when a cluster contains only a single client.

[5] Performance gains and overheads. How much additional memory and inference time does the proposed method require compared to the strongest baseline? The proposed method requires loading multiple models and extracting multiple samples, so, my concern is whether the performance gains are sufficient to cover these costs.

4 comments
AuthorsOpenReview ↗

Rebuttal to Reviewer uPki (1/2)

We thank the reviewer for recognizing that the paper addresses a meaningful problem of balancing personalization, knowledge sharing, and predictive calibration, that fitting the posterior locally without additional communication is practically attractive, that the architecture is intuitively clear and holds practical application value, and that the experimental comparison is sufficient, with promising improvements across both classification and generation.

W1 + Q3

FLAME stays best when every baseline uses the same Laplace stage. Table 10 applies FLAME's post-hoc Laplace stage to every baseline. FLAME keeps the lowest ECE and NLL, including against FLAME without the Grassmannian aggregator under the same stage. The calibration gain therefore does not come from the post-processing alone.

The Laplace stage is effective because of FLAME's architecture. The posterior is placed on the individual expert, and the router keeps this expert active at every token with a gate weight bounded away from zero (Eqs. (4)–(5)), so every posterior sample enters the prediction (Remark in Section 3.3).

Every module contributes. Table 2 removes the Laplace stage and the Grassmannian aggregator, Table 8a removes each expert level, Table 8b crosses the aggregators for AA and BB, and Table 9 replaces the frozen experts with random weights. Every removal or replacement degrades FLAME.

Why it works. Each module matches a property measured on the trained LoRA factors (Section 2.2): AA shares its row space only within clusters (Observation 1), BB carries task identity in its entries (Observation 2), and the calibration gap of point estimates persists across routing designs (Observation 3).

Real-world federated conditions are addressed in Q4 below.

W2

Scale. Table 7 increases the number of clients to 16 under the same protocol, and FLAME keeps the highest accuracy and the lowest ECE and NLL.

Heterogeneity. Heterogeneity varies across the benchmarks of Table 1, from IID replicas (FLAN) to different source datasets for the same task (FLAN-XD) and disjoint label partitions within a task (FLAN-LS), and FLAME is best on every metric in each.

W3 + Q5

The MAP path costs about the same as the strongest baseline. On MCQA-12c, FLAME has the same uplink and downlink as FedLEASE and a nearly identical model state (Table 5), and its MAP latency is close to that of FedLEASE (Table 6). The same holds on Llama-2-7B:

MAP inference on Llama-2-7BLatency (ms per sequence)Peak GPU memory (MB)
Single adapter22.868,108
FedALT24.478,108
FedLEASE27.088,127
FLAME28.358,182

The posterior adds a one-time local fit and SS forward passes. The fit is a single forward–backward sweep without parameter updates and requires no communication, and posterior-predictive latency grows linearly with SS (Appendix B.6; Llama-2-7B figures in Q2 below). For serving, the cached posterior samples add SS adapter-sized copies per client to the MAP model state.

The calibration gain justifies this cost. At this cost, FLAME lowers ECE and NLL well below every baseline, including baselines given the same Laplace stage (Tables 1 and 10).

Q1

Eqs. (8)–(9) remain well defined when the sum is zero: the corresponding frozen expert then has a zero update and contributes nothing to Eq. (4). The individual expert is selected in every forward pass by the enlarged top-(M+1)(M{+}1) router (Eqs. (4)–(5)), so every client's prediction remains well defined.

AuthorsOpenReview ↗

Rebuttal to Reviewer uPki (2/2)

Q2

  • Random variables. As described in Section 3.3, the posterior is over the individual-expert factors (An,Bn)(A_n,B_n). As in Laplace-LoRA (Yang et al., 2024), AnA_n and BnB_n are treated as two separate linear layers, each with its own pair of Kronecker factors, so each posterior sample is a rank-rr update Bn,sAn,sB_{n,s}A_{n,s}.
  • Curvature. For each of AnA_n and BnB_n, the curvature is approximated by KFAC, whose two Kronecker factors are the Gram matrices of that layer's inputs and output gradients on Dn\mathcal{D}_n.
  • Prior and normalization. The likelihood curvature is summed over the TnT_n local samples of client nn, as in Ritter et al. (2018) and Laplace-LoRA (Yang et al., 2024), and the prior is an isotropic Gaussian whose precision αn\alpha_n is selected per client by maximizing the KFAC log marginal likelihood on Dn\mathcal{D}_n.
  • MAP justification. The expansion is taken in the same coordinates (An,Bn)(A_n,B_n) in which local training optimizes the objective and is centered at the trained factors. The individual expert is trained with the frozen experts fixed in the forward pass, which provides the local optimum required by the expansion (Remark in Section 3.3).
  • Posterior state and sampling on Llama-2-7B.
Llama-2-7B, per client
One-time posterior fit997 s
Drawing one posterior sample12.85 ms
Cached posterior samples for S=10S=10 (fp16)80 MB

We will clarify the parameterization in Eqs. (10)–(12).

Q4

  • Multiple tasks on one client. The individual expert and the router are trained on the client's own data, and the router weights the experts at every token (Eq. (4)), so tokens from different local tasks can draw on different experts, including the cross-cluster experts built from the other clusters.
  • Unequal data sizes. Each client trains and fits its posterior on its own data. The likelihood curvature is summed over the client's TnT_n samples (Q2), so a client with fewer samples receives a correspondingly wider posterior.
  • Absent clients. The server builds each frozen expert from the peers that upload in the round (Eqs. (6)–(9)). An absent client keeps its individual expert and the frozen experts it last received, and receives the current ones when it returns.
  • Single-client clusters. The intra-cluster peer set is empty, so the client routes over its individual expert and the cross-cluster experts, and the individual expert remains selected in every forward pass (Eqs. (4)–(5)).

References

Ritter, H., Botev, A., and Barber, D. A scalable Laplace approximation for neural networks. In International Conference on Learning Representations (ICLR), 2018.

Yang, A. X., Robeyns, M., Wang, X., and Aitchison, L. Bayesian low-rank adaptation for large language models. In International Conference on Learning Representations (ICLR), 2024.

I appreciate the authors' responses and the additional experimental results. However, I still have some concerns regarding Q2, and therefore I am inclined to maintain my original rating.

AuthorsOpenReview ↗

Clarification on the Remaining Concern in Q2

Dear Reviewer uPki,

Thank you for reading our response and the additional results. We address the remaining concern on Q2 point by point.

The posterior is not over a dense effective weight. The random variables are the individual-expert factors AA and BB. Each posterior sample is a pair (A~,B~)(\tilde{A}, \tilde{B}), and the corresponding update B~A~\tilde{B}\tilde{A} has rank at most rr, as in Laplace-LoRA (Yang et al., 2024). No Gaussian is placed on the effective weight, so no higher-rank update is sampled.

Curvature and sampling use the same coordinates. AA and BB are treated as two separate linear layers, each with its own pair of KFAC factors computed on the client's data, and the samples are drawn from this KFAC posterior. The curvature is therefore that of the factor coordinates; the curvature in effective-weight coordinates is not used.

The MAP justification requires stationarity only in these coordinates. The Laplace expansion is taken in (A,B)(A, B) and centered at the trained factors, which are the coordinates in which local training optimizes the objective with the frozen experts fixed. Stationarity in the unrestricted effective-weight space is therefore not required.

Prior and normalization. The likelihood curvature is summed over the local samples of each client, as in Ritter et al. (2018) and Laplace-LoRA, and the precision of the isotropic Gaussian prior is selected per client by maximizing the KFAC log marginal likelihood.

We will state these random variables explicitly in Eqs. (10)–(12). We hope this resolves the remaining concern, and we would be grateful if you would kindly consider updating your rating.

Comments on the paper

1 comment
AuthorsOpenReview ↗

AC Rebuttal Summary: Matched-Laplace Comparisons, Seven-Seed Significance Tests, and a Newer Backbone

Dear Area Chair,

We thank you and the reviewers for the discussion. Below, each item refers to the corresponding W/Q in our response to that reviewer.

Reviewer uPki (Rating 2: Weak rejection)

Reviewer uPki finds that the paper addresses "a meaningful problem: balancing personalization, knowledge sharing, and predictive calibration when clients have different tasks and limited data", that "performing posterior fitting locally without additional communication is practically attractive", that the architecture "is intuitively clear, easy to understand, and holds practical application value", and that "the reported improvements are promising across both classification and generation settings." After the discussion, the reviewer wrote: "I appreciate the authors' responses and the additional experimental results."

Source of the calibration gain (W1 + Q3). When every baseline receives the same Laplace stage, FLAME keeps the lowest ECE and NLL (Table 10), and every module contributes (Tables 2, 8, and 9).

Scale and heterogeneity (W2). With 16 clients, FLAME keeps the highest accuracy and the lowest ECE and NLL (Table 7), and it is best on every metric across the heterogeneity settings of Table 1.

Cost (W3 + Q5). FLAME's MAP latency and memory are close to FedLEASE on both Llama-3.2-1B and Llama-2-7B (Table 6 and Table 6 (Llama-2-7B MAP)), and the posterior adds a one-time local fit without communication (Table 6 (Llama-2-7B posterior)).

Reviewer qK2C (Rating 2: Weak rejection)

Reviewer qK2C finds that the paper addresses "a practically relevant objective: improving both personalized task performance and prediction calibration under heterogeneous client data", that "the empirical observations concerning the different structures of the two LoRA factors provide a concrete motivation for asymmetric aggregation", and that the evaluation's ablations and controls "help distinguish the contributions of the architecture, aggregation, and uncertainty estimation." We have done our best to engage with the reviewer, addressing every point and following up to invite feedback.

Novelty (W1 + Q1). FedLEASE and FedALT stay behind FLAME under the same Laplace stage (Table 10), and aligning both factors lowers accuracy relative to FLAME's selective aggregator (Table 8b). FLAME routes jointly over three levels of locality, each of which contributes (Table 8a).

Consensus subspace (W2 + Q2). The consensus projector is invariant under any client-specific Am↦RmAmA_m\mapsto R_mA_m (Lemma 5), and all clients start from a common initialization (Section 3.1).

Posterior and selection rule (W3, W4). The posterior is over the individual-expert factors (An,Bn)(A_n,B_n), as in Laplace-LoRA (Eqs. (10)–(12)), and the same selection rule is applied to every method (Table 10).

Reviewer Mzy8 (Rating 2: Weak rejection)

Reviewer Mzy8 finds "an interesting intuition that identifies the roles for B and A in LoRA(A commonly carries shared information across subspace compared with B)." We have done our best to engage with the reviewer, addressing every point and following up to invite feedback.

Source of the gains (W1). The Laplace stage leaves accuracy unchanged, so FLAME's accuracy gain comes from its routing and aggregation (Table 2).

Presentation (W2, W3). A condensed related-work section is now in the main text (Section 6), and CSC_S is defined before Theorem 1 (Section 4).

Reviewer egpy (Rating 1: Clear rejection)

Reviewer egpy notes that "the Laplace method is cheap", that "the experiments are very comprehensive; multiple methods were compared, and ablation studies were also conducted", and that a "convergence analysis is provided." We have done our best to engage with the reviewer, addressing every point with new results and following up to invite feedback.

Figure 2 (W1). Figure 2 is replaced with a high-resolution version.

Calibration (W2). FLAME keeps the lowest ECE and NLL when every baseline receives the same Laplace stage (Table 10).

Theory (W3). Theorem 1 couples each client to the others through the drift of its frozen experts, and the Grassmannian aggregate drifts no more than the Euclidean mean (Lemma 6).

Significance (W4 + Q2). Over seven seeds, FLAME's accuracy gain over FedAvg, FedDPA, FedALT, and FedLEASE is significant under both paired tests (Table 1 (Significance)).

Newer backbone (W5 + Q4). With Qwen3-1.7B-Base, FLAME again achieves the highest accuracy and the lowest ECE (Table 1 (Qwen3)).

We have revised the PDF accordingly, with all changes highlighted in blue. To keep the table numbers cited in the discussion unchanged, new tables are numbered as variants of existing ones, e.g., Table 1 (Qwen3).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.