acceptodds
Withdrawn from ICLR 2027

Alignment Paradigm of Intrinsic Subspace Experts for Federated Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

The rapid scaling of foundation models has made full-parameter training increasingly impractical in federated scenarios due to prohibitive computational and communication costs, motivating the development of federated parameter-efficient fine-tuning (PEFT). Meanwhile, mixture-of-experts (MoE) based adaptation has shown promising capability in improving model specialization and scalability. However, directly applying MoE-based LoRA adaptation to federated fine-tuning remains challenging. The fundamental difficulty lies in various forms of misalignment introduced by decentralized optimization. Furthermore, MoE architectures introduce additional challenges under heterogeneous clients, where experts with identical positions may capture inconsistent knowledge and independently optimized routers may become difficult to aggregate. To address these challenges, we propose A-SMoE for federated fine-tuning. Instead of allowing local experts to freely learn independent adaptation spaces, A-SMoE exploits the intrinsic subspaces of pretrained models through singular value decomposition and freezes the corresponding input and output bases. This design provides a shared semantic coordinate system for all clients, thereby improving the consistency of expert representations and enabling more reliable federated aggregation. Moreover, we introduce a parameter-free routing mechanism based on subspace projection energy, eliminating additional router parameters and avoiding auxiliary objectives required for router balancing. Extensive experiments on multiple benchmark datasets demonstrate that A-SMoE consistently outperforms existing federated fine-tuning methods.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Federated Learning, Mixture of Experts

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_413QOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

A-SMoE addresses cross-client expert alignment in federated LoRA-MoE fine-tuning. It assigns disjoint pretrained singular-vector directions to experts and freezes their input and output bases, while learning structured internal transformations. Expert weights are determined from projection energy without a trainable router. A geometry-aware procedure aligns and aggregates the trainable factors on the server. The evaluation covers GLUE and FLAN, with additional vision, multimodal, and robustness experiments.

Critical strengths
  1. A coherent structural approach to expert alignment. Shared pretrained subspaces provide common coordinates for expert construction, routing, and aggregation across clients. The proposed aggregation rule is designed for the resulting constrained factorization, giving the principal components of the method a clear technical relationship.

  2. Broad and relevant empirical comparisons. The paper compares against multiple federated LoRA baselines and investigates the number of experts, singular-direction allocation, routing strategies, and aggregation components. The additional language generation and cross-modal experiments extend the evaluation beyond a single model or task family. The reported ablations also provide evidence that the aggregation components contribute to performance under the tested settings.

Critical weaknesses

1. The benefit of pretrained singular directions is not sufficiently separated from that of fixed shared coordinates.

A-SMoE builds on a parameter-efficient fine-tuning design with frozen SVD-derived bases, for which LoRA-XS provides a relevant precedent. Representation-aware routing also has precedents outside the federated setting, including ERMoE. The federated combination and structured aggregation are meaningful contributions, but freezing the bases simultaneously introduces a shared coordinate system, restricts the adaptation space, and incorporates pretrained singular directions. The current experiments do not fully distinguish the effects of these choices.

Table 5 compares allocations of pretrained singular directions, but a more direct control would replace them with fixed random orthogonal bases while matching expert count, trainable parameter budget, router, and aggregation procedure. A matched variant with greater flexibility in the internal transformation would further help determine whether improved optimization follows from the shared structural constraint. These comparisons would make the role of the proposed intrinsic-subspace construction more precise.

2. The aggregation guarantee and representation-alignment measurements are not yet connected directly to the claimed optimization benefit.

Theorem 1 establishes a reconstruction-error advantage relative to a particular factor-wise averaging comparator that uses the same projected orthogonal factor, under stated feasibility conditions. This is a useful property of the reconstruction procedure. Its scope, however, is more limited than a guarantee of improved federated optimization or functional expert alignment.

Table 3 indicates that removing aggregation alignment decreases GLUE G-Avg from 85.97% to 85.21%, which supports the practical value of this component. The paper does not directly report the discrepancy between the ideal weighted average of client expert updates and the reconstructed server update across communication rounds. Such measurements, compared with the stated factor-wise baseline and considered alongside task accuracy, would show how the theoretical result manifests during training. The CKA analysis measures representation similarity, which is related to the alignment motivation but is not a direct measure of aggregation error. Since local orthogonality is encouraged by a penalty, reporting ∥Re⊤Re−I∥F\|R_e^\top R_e-I\|_F before server projection would also help characterize the regime in which aggregation operates.

Additional result-reporting, efficiency, and presentation observations. A-SMoE reports the highest GLUE and FLAN averages, but its GLUE G-Avg advantage over FedRot (85.97% versus 84.87%) is largely attributable to QNLI: the A-SMoE G-ACC scores on SST-2, MRPC, and QQP are slightly lower than FedRot's. Although three seeds are mentioned, the main tables do not provide variability estimates. The parameter-free router also uses dense Softmax weighting by default, so the removal of trainable routing parameters does not itself imply sparse expert computation; an inference-latency comparison with Top-2 routing would be informative.

Table 1 contains a numerical inconsistency that should be verified: the displayed FedALT G-ACC values (73.05, 81.88, 89.01, and 88.77) average to approximately 83.18, whereas the reported G-Avg is 80.66. Figure 3 compresses six plots and their annotations into a single row; Figure 2 and Appendix Figures 5–6 contain small labels. The “LoRA-MoE+CKA” method name wraps awkwardly in Tables 1–2, and the citations to Meng et al. contain an unexpected question mark in several locations, including pages 2 and 4.

Questions for authors
  1. Role of pretrained subspaces. How does the method perform with fixed random orthogonal bases instead of dominant pretrained singular directions, with the expert count, trainable parameter budget, routing mechanism, and aggregation rule held fixed?

  2. Aggregation in practice. Could the authors report the reconstruction residual between the ideal average of client expert updates and the server's aggregated expert, compared with the factor-wise baseline across rounds? How large are the local orthogonality residuals before projection?

  3. Result verification and variability. Could the authors verify the FedALT G-Avg entry in Table 1 and report per-seed variability or paired differences for the main GLUE comparisons, particularly the task-dependent changes relative to FedRot?

Reviewer_SA3VOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes A-SMoE for federated fine-tuning. Directly applying MoE-based LoRA adaptation to federated learning faces various forms of misalignment problems introduced by decentralized optimization. Furthermore, in the case of heterogeneous clients, the MoE architecture presents additional challenges, such as experts in the same location potentially capturing inconsistent knowledge, and independently optimized routers struggling to aggregate. To address these challenges, A-SMoE utilizes singular value decomposition to extract the intrinsic subspace of the pre-trained model and freezes the corresponding input-output basis, thus providing a shared semantic coordinate system for all clients. This improves the consistency of expert representations and enables more reliable federated aggregation. Moreover, it introduces a parameter-free routing mechanism based on subspace projection energy. Extensive experiments validate the effectiveness of the proposed method.

Critical strengths
  1. This paper explores a very interesting direction. Federated LoRA aggregation itself can cause aggregation mismatches, and these mismatches are further amplified when adapted to the MoE architecture. Therefore, solving this problem is very meaningful for the scalability of federated fine-tuning.
  2. The solution proposed in this paper is interesting: all clients use the same frozen subspace for experts with the same ID. This provides a common structural reference before local training, preventing clients from learning completely independent expert bases. The separation of output subspaces for different experts also gives this design a clear mathematical interpretation.
  3. The overall logic of the paper is rigorous, and there is a certain theoretical basis to prove the effectiveness of the method. At the same time, the experimental evaluation is relatively comprehensive, including Language Understanding, Natural Language Generation, and vision-related experiments.
Critical weaknesses
  1. Firstly, regarding the experimental evaluation, although it is relatively comprehensive, the models used are either too small, such as RoBERTa-Large, or outdated, such as LLaMA2-7B. Therefore, some newer models and tasks can be used to demonstrate the generalization ability of the method. Also, for Tables 1 and 2, A-SMoE does not perform optimally on many tasks; it only has a certain advantage in the final average performance. Furthermore, I observed that even a simple FedSA can achieve similar performance.
  2. Section 3.3 claims that a globally shared input basis naturally guarantees routing consistency. I believe that using the same routing function for the same hidden vector is fine. However, adapters trained locally in previous layers may produce different hidden vectors for the same sample, resulting in different projected energies and routing decisions. Round-robin assignment also cannot universally guarantee expert usage balance under arbitrary client data distributions. Even though Figure 3 provides evidence of improved balance in the evaluated setting, it does not directly measure routing divergence among different clients on common samples.
  3. This paper also has many formatting issues: the title "ACKNOWLEDGMENTS" should be deleted; the font size of "legend" in Figure 1 (a) and (b) is too small; there are question marks in the citations on line 93 and line 183; the writing is inconsistent, some formulas have symbols (e.g., formula 6), and some formulas are missing symbols (e.g., formula 3). Therefore, there is still much room for improvement in this paper.
Questions for authors

Please see Weaknesses.

Reviewer_HcJfOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes A-SMoE, a method for MoE-style LoRA fine-tuning in federated learning. The authors argue that federated LoRA-MoE suffers from several kinds of misalignment across clients: aggregation, scale, orthogonal-basis, expert, and router misalignment. To address this, each expert’s input and output bases are frozen to disjoint singular-vector subspaces of the pretrained weight, assigned round-robin among the top singular directions. Only a small “diagonal–orthogonal–diagonal” core is trained locally. Routing is parameter-free and based on how much energy the input projects onto each expert’s subspace. On the server side, a geometry-aware aggregation scheme normalizes scales, projects the averaged rotation back onto the orthogonal group, and solves for the output scales in closed form. The authors prove this gives lower aggregation error than naive factor averaging. Experiments on GLUE (RoBERTa-large), FLAN (LLaMA2-7B), CIFAR-100 (ViT), and VQA (ViLT) show modest gains over 11 federated PEFT baselines.

Critical strengths
  1. The paper is clearly written.
  2. Providing sufficient observations about the misalignment of the heterogeneous clients.
  3. The experiments are broad, covering NLU, NLG, vision, and multimodal tasks against 11 baselines.
Critical weaknesses
  1. In Figure 1, "LoRA-MoE" and "LoRA-MoE + hungarian" are highly overlapped. Why? Due to the algorithm of CKA-based Hungarian, I think the misalignment of Non-iid data can be relieved. But figure shows no improvement on it, I think it should be explained and author should show the detailed achivement of "LoRA-MoE + hungarian".
  2. As for the provided second challenge: inconsistent knowledge in identical positions, I think CKA-based Hungarian also can address this challenge. So what's the insight point for this second challenge?
  3. What's the difference between A-SMoE and FedPissa [1], which also uses intrinsic subspaces to achieve reliable personalization?
  4. A routing mechanism with the parameter can provide a flexible way for domain data and heterogeneous clients, like FedLease [2]. I want to know why the author chose a parameter-free way, and how the author balances flexibility and efficiency.
  5. Some problems with the author's claim: "the singular vectors assumption has not been validated" in Line 94. For example, MiLoRA [3] claims that minor singular vectors would be better than the top singular vectors.
  6. The evidence in Appendix B doesn’t really back up the claim. S2, S4 and S6 all multiply by singular values, which drop off fast with rank. So even random matching would show a peak at the head, and those three metrics basically can’t tell anything. That leaves S1/S3/S5, and you say in B.4 that S1 and S3 are also high at the tail, which reads more like a U-shape than “head dominates.” There’s also no random baseline. A random ΔW would match each of the 1024 indices about equally often, so we can’t tell whether the head peak is real. The top-1 cosine values aren’t reported either. Could you add a random-ΔW baseline for S1/S3/S5 and report the cosine values?
  7. Line 094 lacks citation.
Questions for authors
  1. Thinking about the mechanism of CKA-based Hungarian, what's the insight point for this second challenge?
  2. What's the difference between A-SMoE and FedPissa [1], which also uses intrinsic subspaces to achieve reliable personalization?
  3. The author chose the head singular vector to construct experts but claims the singular vectors assumption has not been validated. I want to know the motivation for the author's SVD mechanism.

[1] FedPissa: Towards Federated Personalized Adaptation of Foundation Models via LoRA Subspace Mapping. ICML'26 [2] Adaptive LoRA Experts Allocation and Selection for Federated Fine-Tuning. Neurips'25 [3] MiLoRA: Harnessing Minor Singular Components for Parameter-Efficient LLM Fine-Tuning. NAACL'25

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.