- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
A-SMoE addresses cross-client expert alignment in federated LoRA-MoE fine-tuning. It assigns disjoint pretrained singular-vector directions to experts and freezes their input and output bases, while learning structured internal transformations. Expert weights are determined from projection energy without a trainable router. A geometry-aware procedure aligns and aggregates the trainable factors on the server. The evaluation covers GLUE and FLAN, with additional vision, multimodal, and robustness experiments.
Critical strengths
-
A coherent structural approach to expert alignment. Shared pretrained subspaces provide common coordinates for expert construction, routing, and aggregation across clients. The proposed aggregation rule is designed for the resulting constrained factorization, giving the principal components of the method a clear technical relationship.
-
Broad and relevant empirical comparisons. The paper compares against multiple federated LoRA baselines and investigates the number of experts, singular-direction allocation, routing strategies, and aggregation components. The additional language generation and cross-modal experiments extend the evaluation beyond a single model or task family. The reported ablations also provide evidence that the aggregation components contribute to performance under the tested settings.
Critical weaknesses
1. The benefit of pretrained singular directions is not sufficiently separated from that of fixed shared coordinates.
A-SMoE builds on a parameter-efficient fine-tuning design with frozen SVD-derived bases, for which LoRA-XS provides a relevant precedent. Representation-aware routing also has precedents outside the federated setting, including ERMoE. The federated combination and structured aggregation are meaningful contributions, but freezing the bases simultaneously introduces a shared coordinate system, restricts the adaptation space, and incorporates pretrained singular directions. The current experiments do not fully distinguish the effects of these choices.
Table 5 compares allocations of pretrained singular directions, but a more direct control would replace them with fixed random orthogonal bases while matching expert count, trainable parameter budget, router, and aggregation procedure. A matched variant with greater flexibility in the internal transformation would further help determine whether improved optimization follows from the shared structural constraint. These comparisons would make the role of the proposed intrinsic-subspace construction more precise.
2. The aggregation guarantee and representation-alignment measurements are not yet connected directly to the claimed optimization benefit.
Theorem 1 establishes a reconstruction-error advantage relative to a particular factor-wise averaging comparator that uses the same projected orthogonal factor, under stated feasibility conditions. This is a useful property of the reconstruction procedure. Its scope, however, is more limited than a guarantee of improved federated optimization or functional expert alignment.
Table 3 indicates that removing aggregation alignment decreases GLUE G-Avg from 85.97% to 85.21%, which supports the practical value of this component. The paper does not directly report the discrepancy between the ideal weighted average of client expert updates and the reconstructed server update across communication rounds. Such measurements, compared with the stated factor-wise baseline and considered alongside task accuracy, would show how the theoretical result manifests during training. The CKA analysis measures representation similarity, which is related to the alignment motivation but is not a direct measure of aggregation error. Since local orthogonality is encouraged by a penalty, reporting before server projection would also help characterize the regime in which aggregation operates.
Additional result-reporting, efficiency, and presentation observations. A-SMoE reports the highest GLUE and FLAN averages, but its GLUE G-Avg advantage over FedRot (85.97% versus 84.87%) is largely attributable to QNLI: the A-SMoE G-ACC scores on SST-2, MRPC, and QQP are slightly lower than FedRot's. Although three seeds are mentioned, the main tables do not provide variability estimates. The parameter-free router also uses dense Softmax weighting by default, so the removal of trainable routing parameters does not itself imply sparse expert computation; an inference-latency comparison with Top-2 routing would be informative.
Table 1 contains a numerical inconsistency that should be verified: the displayed FedALT G-ACC values (73.05, 81.88, 89.01, and 88.77) average to approximately 83.18, whereas the reported G-Avg is 80.66. Figure 3 compresses six plots and their annotations into a single row; Figure 2 and Appendix Figures 5–6 contain small labels. The “LoRA-MoE+CKA” method name wraps awkwardly in Tables 1–2, and the citations to Meng et al. contain an unexpected question mark in several locations, including pages 2 and 4.
Questions for authors
-
Role of pretrained subspaces. How does the method perform with fixed random orthogonal bases instead of dominant pretrained singular directions, with the expert count, trainable parameter budget, routing mechanism, and aggregation rule held fixed?
-
Aggregation in practice. Could the authors report the reconstruction residual between the ideal average of client expert updates and the server's aggregated expert, compared with the factor-wise baseline across rounds? How large are the local orthogonality residuals before projection?
-
Result verification and variability. Could the authors verify the FedALT G-Avg entry in Table 1 and report per-seed variability or paired differences for the main GLUE comparisons, particularly the task-dependent changes relative to FedRot?