FLAME: Federated Laplace-Aligned Mixture of Experts for Personalized LLM Fine-Tuning

Personalized federated fine-tuning of large language models calls for adaptation to heterogeneous client tasks together with reliable uncertainty estimates under limited local data. Existing personalized federated low-rank adaptation (LoRA) methods expose only a single level of cross-client locality, average the LoRA factors $A$ and $B$ identically despite their structural asymmetry, and return *point* estimate adapters whose predictions remain poorly calibrated. To address these limitations, we propose **FLAME**, a **F**ederated **L**aplace-**A**ligned **M**ixture of **E**xperts whose three design choices are each grounded in the geometry of the learned parameter space. To capture hierarchical locality across client adaptations, FLAME jointly routes among self, intra-cluster, and cross-cluster experts, with the trainable individual expert active in every forward pass. To resolve the LoRA basis ambiguity that Euclidean aggregation introduces, FLAME aligns the input-side factor $A$ on the Grassmann manifold and keeps the output-side factor $B$ in Euclidean space, where the entry-level task-specific signals live. For *calibrated* per-client predictive distributions, FLAME fits a local post-hoc Kronecker-factored Laplace posterior over each client’s individual expert, leaving training and communication untouched. We further establish a non-asymptotic convergence guarantee for FLAME and show that the Grassmannian aggregator tightens the aggregation-drift more than Euclidean averaging. Experimental results across $12$-client MCQA classification benchmark and $8$-client FLAN NLG benchmarks spanning IID, cross-dataset, and label-skew heterogeneity show that FLAME consistently improves both accuracy and calibration (ECE, NLL) over all baselines.