acceptodds
Under review as a conference paper at ICLR 2027

FLoSER: Spectral Experts and Response-Aware Routing for Federated Sparse LoRA Adaptation

Abstract

Federated fine-tuning adapts large language models to distributed private data, but training and communicating task-specific parameters remain costly for resource-constrained clients. A recent resource-efficient paradigm reuses pretrained LoRA adapters as sparse rank-wise experts by freezing their rank-one components and training and aggregating only a lightweight token router. Although this enables token-level composition with few trainable and communicated parameters, it assumes that LoRA factor coordinates define stable expert identities. We show that equivalent factorizations preserve the same dense update but can induce different rank-one expert banks and sparse-routing families. Moreover, a stable expert bank alone does not ensure effective token-wise selection. We therefore propose Federated LoRA with Spectral Experts and Response-Aware Routing (FLoSER). FLoSER constructs an ordered spectral expert bank from the complete LoRA update—rank-one canonical for distinct singular values and block-canonical for repeated values—and augments learned routing with a signed, token-dependent correction derived from each expert's instantaneous response magnitude. This correction provides a learnable salience signal without changing the active expert budget; during federation, the backbone and expert bank remain frozen, and only router parameters are optimized and communicated. Mechanism analyses show that spectral construction removes factor-coordinate dependence and admits better sparse solutions, although learned routing does not automatically recover them. Experimental results show that FLoSER-S improves matched Raw rank-wise routing by up to 16.40 percentage points, while full FLoSER yields additional setting-dependent gains. Additional transfer and cross-task studies delineate backbone- and scale-sensitivity boundaries, and grouped controls contextualize the benefit of fine-grained learned routing under the primary heterogeneous protocol.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Federated Learning, Parameter-Efficient Fine-Tuning, Low-Rank Adaptation, Sparse Routing, Mixture of Experts, Large Language Models

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_pzrKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how to decompose an existing LoRA into experts and select a sparse subset of experts for each token in federated fine-tuning. The authors first formalize the coordinate dependence created when LoRA factor components are materialized as independently routable experts, and show that dense equivalence does not imply sparse-routing equivalence. They then propose FLoSER, combining a function-defined spectral expert bank with Response-Aware routing as a complementary, setting-dependent selection refinement that does not increase the active expert budget. Experimental results show that FLoSER-S improves matched Raw rank-wise routing by up to 16.40 percentage points, while full FLoSER yields additional setting-dependent gains.

Critical strengths

1)This paper investigates a very valuable question: how to effectively utilize pretrained LoRA knowledge for task adaptation. Furthermore, they discovered that while equivalent LoRA factorization preserve the same dense update, treating each rank-one component as an independent expert for Top-K routing can result in different sets of sparse functions.

2)A spectral expert construction method is proposed. For distinct non-zero singular values, the spectral expert is determined by a complete LoRA update, which can eliminate the ambiguity caused by arbitrary factor coordinates. The paper also accurately distinguishes between distinct singular values and repeated singular values. In addition, the compact SVD construction method can avoid constructing a large dense update matrix.

3)The authors demonstrated the effectiveness of the method through extensive experiments. In Table 2, FLoSER-S improved performance relative to Raw by 15.62 and 6.51 percentage points in IID and strong Non-IID federated settings, respectively. Meanwhile, Table 3 shows that even with EEQA disabled and expert quotas fixed for each matrix, the average improvement remained at 5.34 percentage points.

Critical weaknesses

1)The method has limited model generalization. In Table 2, the method effectively improves model performance for Qwen2.5-7B, but the performance improvement on Mistral-7B is very limited, with a maximum improvement of only 1.04. Therefore, the generality of the method needs to be verified on more model architectures and different model sizes.

2)The federation's experimental setup has certain limitations. The experiment was set up with 20 clients and two participants per round. The effectiveness of this method in large-scale federated training could not be evaluated. The performance of this method can be tested with 100 or more clients.

3)The experimental section only compared the performance of SmartFed, but did not compare other metrics such as resource overhead and convergence speed. It is recommended to add these additional overhead analyses as well.

4)The effectiveness of the response-aware routing module has not been fully analyzed. Table 2 shows that the average gain of the response module is zero under the Qwen strong Non-IID setting, but decreases by 0.78 percentage points under the Mistral IID setting. Therefore, the current experimental analysis more fully demonstrates the effectiveness of FLoSER-S than the complete FLoSER.

Questions for authors

See Weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer pzrK: Cross-Model Generalization, Client-Pool Expansion, Computational Cost, and Response-Aware Routing

Thank you for your comments. We address your four concerns below.

1. Generalization across model families and sizes (Weakness 1). We added Gemma-2-2B-IT, a smaller model from a third family, and a controlled Mistral K12/K24 study.

The original Qwen/Mistral comparison also changes adapter sources, ranks, and sparse budgets: K12 of 24 experts versus K12 of 48. We therefore compared K12/K24 within Mistral, fixing the banks and pairing Raw/SC initialization, data, and client schedules, with RA/EEQA off. At fixed R18, SC's mean generation-accuracy advantage is +0.13pp at K12 and +4.17pp at K24, with all three K24 pairs positive (+5.08/+5.86/+1.56pp). This shows that the limited original gain is not a fixed backbone limitation. K changes gate mass and optimization alongside support size, so the result concerns the routing configuration as a whole.

On Gemma-2-2B-IT with a different pretrained adapter bank, SC lowers R18 validation CE in all three seeds (mean 0.50948 vs. 0.51481). Mean generation accuracy is 43.23% vs. 42.58%, with SC-Raw differences +5.08/-2.73/-0.39pp. Thus, the third-family evidence shows consistent validation-loss improvements but mixed generation gains. Both studies use 256 shared GSM8K questions per seed, greedy 512-token decoding, and the Original Parser.

2. Extension to 100 clients (Weakness 2). Following your suggestion, we expanded to 100 clients, completing Raw/SC training and generation evaluation at 20/100 clients across three seeds.

Both settings use the same 2,560-example training pool, with two participating clients per round, 20 rounds, and 400 client optimizer updates per run. This matched design evaluates client-pool expansion under fixed participation and optimization budgets. Within each size, Raw/SC share partitions, schedules, and initialization; RA/EEQA are off.

At fixed R18 on the same 256 test questions, mean Raw/SC accuracy is 31.12%/42.58% at 20 clients and 24.22%/42.58% at 100 clients. Paired SC-Raw gains are +11.46 +/- 3.32pp and +18.36 +/- 2.03pp (three-seed sample SD); all three seeds favor SC at each size. The frozen evaluation uses greedy 512-token, dual-EOS decoding and the Original Parser.

SC's relative task benefit persists after a fivefold client-pool expansion, beyond the original 20-client configuration. Two clients still participate per round: participation falls from 10% to 2%. Over 20 rounds, the schedules covered 17/20 and 31/100 distinct clients, respectively, so realized client-data coverage also differs.

3. Communication, computation, and convergence (Weakness 3).

Communication. Qwen Raw/SC have identical FP32 router payloads: 18.38 MiB per state, or 73.52 MiB per round for two clients each downloading/uploading once. RA adds 0.0279% model payload, excluding initial distribution and network overhead.

Computation. On one RTX 4090, with matched batch-1 inputs padded to 384 tokens and three interleaved blocks, pooled median training-update times are 2.896/2.862/3.219 s for Raw/SC/RA (16 microbatches). RA's paired overhead over SC is 1.04%-14.28% across blocks. Despite its small parameter overhead, RA incurs measurable runtime cost. One-time Qwen CPU bank construction takes 0.135/0.050 s for SC/Raw. These are fixed-input, not generation-latency, measurements.

Convergence. At matched update and communication budgets, R18 Raw/SC validation CE is 0.3891/0.3344 at 20 clients and 0.4012/0.3562 at 100 clients. Full curves will accompany these fixed-budget comparisons.

Communication is reported as protocol-based payload accounting; the sequential simulation does not measure real network transfer or distributed aggregation latency.

4. RA mechanisms and conditional gains (Weakness 4). Our additional analyses show that RA changes expert supports and weights but does not consistently improve downstream accuracy. Response magnitude reflects response strength, but is insufficient to reliably indicate task-benefit direction.

On 16 questions, the three-seed mean Spearman correlation between response magnitude and selected experts' local first-order utility estimates is 0.015 for signed utility versus 0.461 for absolute utility. This suggests a stronger association with influence strength than beneficial direction.

Teacher-forced CE evaluates gold-prefix predictions; generation accuracy evaluates model-produced answers. In a separate paired Qwen seed-1 R18 diagnostic, SC+RA lowers validation CE 0.358938 to 0.356513, yet both score 105/256 under the same greedy 192-token Original-Parser protocol.

With theta and experts fixed and only alpha trained, all-token CE versus alpha0 improves in one seed and worsens in two at R18; it worsens in all three at R20. SC remains our main structural contribution; RA is an auxiliary, input-dependent correction with observable routing effects and conditional task benefits. We retain Table 2's zero and negative RA results.

Reviewer_vQEZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes FLoSER, a federated sparse LoRA framework that addresses a limitation of SmartFed: treating LoRA factor components as experts makes expert identities factorization-dependent, so equivalent factorizations can induce different sparse-routing behavior. FLoSER instead builds the expert bank from the SVD of the full LoRA update and adds a response-aware correction to the router logits. The authors report gains over a SmartFed-style baseline on GSM8K.

Critical strengths

formulation is mathematically sound. the counterexample is correct, and the scope (subspace-level invariance for repeated singular values) is stated carefully.

method is simple and cheap. The spectral bank is computed offline from the LoRA factors, and the expert budget and communication cost are unchanged.

timely and sound setting. reusing pretrained LoRAs in federated fine-tuning instead of training adapters from scratch.

reporting is fairly transparent. including multi-seed intervals and negative results (e.g., Mistral, MBPP).

Critical weaknesses

Missing causal link between mechanism and gains. The paper does not show that factor-coordinate dependence causes failures in practice. The proposed mechanism does not explain the gains: learned routers perform worse on the spectral bank than on the Raw bank (Table 15), yet downstream accuracy improves. Response-aware routing is claimed as a main contribution, but its effect is not significant.

Weak baselines. The primary baseline is an unofficial SmartFed reimplementation and is not evaluated on SmartFed’s benchmarks, so its fidelity cannot be verified. Closely related methods (official MoLE/LoRA-Flow, LoRAHub, SMoRA, SVD-based merging) are not compared, and the external baselines appear in only one setting.

Concerns on benchmarks and setup. Absolute accuracies are far below standard results (base Qwen2.5-7B: 5.47% on GSM8K), which raises concerns about the evaluation protocol. The Non-IID setting does not appear challenging: Raw scores higher under Non-IID than under IID. The scale is small (512 training samples, about 25 per client, 3 seeds), the headline gain is from a single seed, and two runs of the same configuration differ by 4.5 points.

Questions for authors
  1. Does the gain hold with norm-matched controls (e.g., σ-normalized spectral experts) or with a tuned Λ for Raw?

  2. Does the Raw reimplementation reproduce SmartFed’s reported results?

  3. Is there a realistic case, other than artificial rotation, where factor-coordinate dependence hurts SmartFed?

3 comments
AuthorsOpenReview ↗

Response to Reviewer vQEZ: Mechanism Evidence, Baseline Controls, and Evaluation Reliability

Thank you for the questions.

1. Mechanism & Q3: Does coordinate dependence affect practical execution? In a new control, we tested real task hidden states and a trained router, preserving the dense update via B′=BQB'=BQ, A′=Q⊤AA'=Q^{\top}A with orthogonal QQ and fixing activations, logits, supports, and gates. At the query projection (q_proj) in Qwen layer 14, across 64 inputs and three transformations, the median relative change in sparse output was 1.7024 for Raw versus 1.51 × 10⁻¹⁵ for SC. Dense-update discrepancies were below 7.31 × 10⁻¹⁶.

Thus factor coordinates affect sparse execution on real activations even with the dense update and routing fixed. The intervention is controlled; naturally occurring changes and their task effects remain untested. Repeated singular values retain the subspace-level qualification.

Why can poorer reconstruction coexist with better accuracy? Table 15 of the submitted manuscript measures how well a trained router reconstructs adapter responses. Downstream routers instead optimize task CE. Qwen sparse-oracle results favor SC while learned reconstruction favors Raw; Mistral Hard/S2H effects are setting-dependent and oracle intervals often loose. In a new local intervention, reconstruction improves in all 48 question–seed comparisons, yet CE worsens in 27. Reconstruction is not a sufficient task objective, nor an explanation of the accuracy gain.

RA's role. RA has setting- and training-stage-dependent effects: it uses response magnitude, not signed task utility. In our new control with θ\theta fixed, CE changes for learned α\alpha versus α=0\alpha=0 are mixed at round 18, while aggregate CE worsens in all three seeds at round 20. With the mixed accuracy results, this supports SC as the core contribution and RA as a setting-dependent refinement.

2. Baselines & Q2: Implementation fidelity and gating fairness. When implementing Raw, we found no complete runnable code for SmartFed's reported experiments in its public repository. We implemented its rank-wise experts from SmartFed's equations and pseudocode. One difference is gating: we use selected-softmax ×K\times K to give additive components unit weight under uniform selected logits; SmartFed's equations mask after full softmax.

In our paired experiment, SC outperforms Raw under SmartFed's gate by 4.43pp on average over three seeds; under our gate, the mean advantage is 11.98pp. Every seed favors SC under both rules. Routers are separately trained, not switched at inference. The gain survives both gates but differs in size. This is not a reproduction of SmartFed's benchmark scores.

Whole-adapter, static-spectral, and FedAvg-LoRA controls compare routing granularity, weighting, and adaptation choices. They are not official MoLE/LoRA-Flow/LoRAHub/SMoRA/SVD-merging reproductions; coverage remains limited.

3. Norm/gain & Q1: Does the gain survive scale controls? In our new controls, symmetric validation-based global-gain tuning leaves +2.34pp under the original parser, with two positive seeds and one negative. Raw selects the grid boundary. Per-matrix routed-response RMS matching instead yields −3.78pp, negative in all seeds and under alternative parsers. Fixed-scaling training does not reliably restore the advantage, and matching drifts afterward. Thus the mean advantage persists after finite-grid global calibration, but not per-matrix RMS matching: representation, scale, and adaptation jointly affect realized benefits. RMS matching is not strict σ\sigma-normalized expert control.

Selecting gain with a strict parser that uses the first four-hash marker gives +11.72pp under that rule. Rescoring the same outputs with the original parser gives +1.69pp. Changing the extractor alone does not reproduce the full official GSM8K protocol.

4. Evaluation reliability. We independently re-scored the base model's saved outputs and confirmed 14/256≈5.47%14/256\approx5.47\%: Qwen2.5-7B Base, no adapter/router, single-question chat-template prompt, greedy decoding, 192-new-token cap, original numeric extraction. This verifies the score, not standard few-shot comparability. Without an alternative-protocol base comparison, its low value remains unexplained; newer 512-token controls are reported separately.

Non-IID uses difficulty-based Dirichlet(0.1) partitioning with balanced client sizes. Higher Raw accuracy than IID does not measure skew severity; we emphasize within-partition comparisons. The 512-example/20-client/three-seed scope remains limited.

In Table 2 of the submitted manuscript, the strong Non-IID Qwen SC entries are not repeat runs: 33.72% uses validation-selected checkpoints in the Raw–SC study; 38.28% uses fixed round 18 on a different split in the SC–RA study. Table 1 of the same manuscript reports +16.40pp for seed 1 on 128 questions; the separate RA comparison uses three seeds and 256 questions per seed. We will clarify both captions.

Thank you for the additional controls and the candid response; the clarification on the separate Non-IID studies resolves that concern. However, the new results do not fully address my main concern. The advantage appears to shrink under SmartFed's original gate, becomes less clear under scale-matched controls, and seems sensitive to the answer parser. Given these points, together with the remaining questions on the mechanism, the Response-Aware component, and the evaluation protocol, I will keep my current score.

AuthorsOpenReview ↗

Response to Reviewer vQEZ

Thank you for clarifying your assessment. We agree that the downstream gain is smaller under SmartFed's published gate and reverses under per-matrix RMS matching. We would, however, distinguish these findings from parser sensitivity. Using the Original Parser, SC still outperforms Raw in all three seeds under the published gate (+4.43 pp on average). Meanwhile, the RMS-matched comparison favors Raw under all four parser variants tested. Thus, parser choice affects the measured gain, but does not change the direction of the RMS-matched result.

We therefore distinguish the structural contribution from downstream accuracy: equivalent factor coordinates can change Raw's sparse output even with routing fixed, whereas SC provides invariant spectral experts under the stated non-degeneracy conditions. The resulting task-accuracy advantage is conditional on the gating and scale configuration, rather than guaranteed across all settings.

We sincerely appreciate your careful assessment and constructive feedback. Your comments have helped us sharpen the distinction between FLoSER's structural contribution and its configuration-dependent downstream benefits. We hope the revised presentation makes this distinction clearer and better communicates the scope and significance of our work.

Reviewer_NzxoOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes FLOSER, which further improves upon a previous method, SmartFed, in two folds. FLoSER adopts spectral rank one experts to replace channel-wise rank one experts in SmartFed to resolve the non-unique decomposition issue. Also, FLoSER further augment SmartFed’s routing with response-magnitude-based correction.

Critical strengths

The paper reasonably connects prior work SmartFed with existing federated-LoRA finetuning challenges, LoRA non-unique decomposition, and uses empirical examples to analyze the effectiveness of the mechanism.

Critical weaknesses
  1. The paper’s mechanism is largely based on the previous work SmartFed, yet SmartFed is not comprehensively discussed in the paper. The author should introduce SmartFed somewhere in the paper to both help reader understand what SmartFed is, and how FLOSER differs from SmartFed.
  2. Lots of related work is missing in the paper. Many relevant federated-LoRA based fine-tuning works are not discussed in the paper: Towards Building the Federated GPT: Federated Instruction Tuning; FLoRA: Federated Fine-Tuning Large Language Models with Heterogeneous Low-Rank Adaptations; Federated Fine-tuning of Large Language Models under Heterogeneous Tasks and Client Resources, etc.
  3. The mechanism design seems to be incremental from SmartFed. The algorithmic design is largely based on SmartFed with incremental changes, more effort will be expected from more substantial methodological developments or maybe theoretical insights.
  4. From the results in Table 1 and 2, it seems FLOSER works better for Qwen2.5-7B and not for Mistral-7B (with at most 1% of performance improvement), could the author provide further justification on why the performance differs between two models?
Questions for authors

See weakness

1 comment
AuthorsOpenReview ↗

Response to Reviewer Nzxo: Methodological Distinctions, Related Work, and Cross-Model Evidence

Thank you for your comments.

1. What SmartFed does and what FLoSER changes (Weakness 1). SmartFed reuses pretrained LoRA adapters. For an update BABA, column bib_i of BB and row aia_i of AA form the rank-one expert biaib_i a_i, with BA=∑ibiaiBA=\sum_i b_i a_i. The backbone and expert bank are frozen. For each token, a learned router scores the experts, selects a sparse subset, and combines their weighted responses. Clients optimize router parameters on local data; the server aggregates these parameters through FedAvg. EEQA allocates activation budgets across matrices.

These experts come directly from the stored LoRA factor coordinates: expert ii is defined by the corresponding column/row pair, not by an intrinsic component of the complete update. FLoSER retains the frozen-bank, router-only federated training framework but constructs experts from the SVD of each complete LoRA update. Spectral Construction (SC), isolated in FLoSER-S, is the core structural contribution: it defines the candidate functions. Response-Aware Routing (RA) additionally incorporates token-dependent response magnitude into routing logits while retaining the same Top-KK expert budget.

The innovation addresses expert-representation dependence, not just a gate formula. We will clarify this distinction in Sections 1–3 of our paper. Raw is our independent, paper-guided baseline; Appendix A.4 of our submission specifies our gate, not SmartFed's.

2. Positioning within federated LoRA research (Weakness 2). We will discuss the three works you identified in our Related Work (Section 2). FedIT studies federated instruction tuning across heterogeneous instructions; FLoRA uses stacking-based aggregation of heterogeneous LoRA adapters; FlexLoRA uses SVD to redistribute aggregated updates at client-specific ranks.

These address adapter training or aggregation; our focus is sparse expert representation of frozen updates. FlexLoRA's SVD adapts aggregated updates to client resources, whereas ours defines independently selectable functions.

3. Why the structural contribution is substantive (Weakness 3). Equivalent factors satisfy BA=(BQ)(Q−1A)BA=(BQ)(Q^{-1}A): the dense update is unchanged, but its rank-one expert functions and their sparse combinations can change. Thus, dense equivalence does not imply sparse-routing equivalence. This ambiguity affects the functions available to the router even when adapters are frozen and never aggregated.

SC provides a canonical representation for expert identities: for distinct nonzero singular values, rank-one spectral expert matrices are invariant under equivalent factorizations. For repeated nonzero singular values, invariance holds for the associated singular subspaces and aggregate spectral contributions, rather than individual rank-one terms.

In a new control, we tested the operational consequence using real task hidden states and a trained router. Holding the dense update, hidden states, logits, supports, and gate weights fixed, we applied equivalent factor transformations. At the query projection in Qwen layer 14, over 64 inputs and three transformations, the median relative sparse-output change was 1.7024 for Raw versus 1.51×10−151.51\times10^{-15} for SC; dense-matrix discrepancies remained below 10−1510^{-15}. This isolates an execution-level effect of expert representation under unchanged routing decisions.

This changes the router's candidate-function definition and resolves a structural ambiguity within the stated spectral conditions, rather than tuning hyperparameters. RA remains a complementary refinement with setting-dependent gains.

4. Cross-model performance and additional evidence (Weakness 4). The settings in Tables 1 and 2 of our submission differ in more than the backbone: Qwen uses rank-8 adapters and K=12K=12 out of 24 experts (50%), whereas Mistral uses rank-16 adapters and K=12K=12 out of 48 (25%); adapter sources also differ. We therefore tested K=12K=12 and K=24K=24 within Mistral, pairing Raw/SC initialization, data, and client schedule, with RA/EEQA off and the expert banks fixed across budgets.

At fixed R18, on 256 questions per seed with greedy 512-token generation and the Original Parser, SC's mean accuracy advantage was +4.17pp at K=24K=24, versus +0.13pp at K=12K=12. All three K=24K=24 pairs favored SC (+5.08/+5.86/+1.56pp). Thus, Mistral's small original gain is not a fixed backbone limitation. KK changes gate mass and optimization as well as support size; SC itself did not improve from K=12K=12 to K=24K=24 in every seed.

We also tested Gemma-2-2B-IT with a different code/SQL/translation bank. At prespecified R18, SC had lower validation CE in all three seeds (0.50948 vs. 0.51481). On the same 256 GSM8K questions per seed, mean generation accuracy was 43.23% vs. 42.58%, with SC-Raw differences of +5.08/-2.73/-0.39pp. This extends task evaluation to a smaller, third model family, with a modest, seed-dependent mean generation gain.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.