- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates how to decompose an existing LoRA into experts and select a sparse subset of experts for each token in federated fine-tuning. The authors first formalize the coordinate dependence created when LoRA factor components are materialized as independently routable experts, and show that dense equivalence does not imply sparse-routing equivalence. They then propose FLoSER, combining a function-defined spectral expert bank with Response-Aware routing as a complementary, setting-dependent selection refinement that does not increase the active expert budget. Experimental results show that FLoSER-S improves matched Raw rank-wise routing by up to 16.40 percentage points, while full FLoSER yields additional setting-dependent gains.
Critical strengths
1)This paper investigates a very valuable question: how to effectively utilize pretrained LoRA knowledge for task adaptation. Furthermore, they discovered that while equivalent LoRA factorization preserve the same dense update, treating each rank-one component as an independent expert for Top-K routing can result in different sets of sparse functions.
2)A spectral expert construction method is proposed. For distinct non-zero singular values, the spectral expert is determined by a complete LoRA update, which can eliminate the ambiguity caused by arbitrary factor coordinates. The paper also accurately distinguishes between distinct singular values and repeated singular values. In addition, the compact SVD construction method can avoid constructing a large dense update matrix.
3)The authors demonstrated the effectiveness of the method through extensive experiments. In Table 2, FLoSER-S improved performance relative to Raw by 15.62 and 6.51 percentage points in IID and strong Non-IID federated settings, respectively. Meanwhile, Table 3 shows that even with EEQA disabled and expert quotas fixed for each matrix, the average improvement remained at 5.34 percentage points.
Critical weaknesses
1)The method has limited model generalization. In Table 2, the method effectively improves model performance for Qwen2.5-7B, but the performance improvement on Mistral-7B is very limited, with a maximum improvement of only 1.04. Therefore, the generality of the method needs to be verified on more model architectures and different model sizes.
2)The federation's experimental setup has certain limitations. The experiment was set up with 20 clients and two participants per round. The effectiveness of this method in large-scale federated training could not be evaluated. The performance of this method can be tested with 100 or more clients.
3)The experimental section only compared the performance of SmartFed, but did not compare other metrics such as resource overhead and convergence speed. It is recommended to add these additional overhead analyses as well.
4)The effectiveness of the response-aware routing module has not been fully analyzed. Table 2 shows that the average gain of the response module is zero under the Qwen strong Non-IID setting, but decreases by 0.78 percentage points under the Mistral IID setting. Therefore, the current experimental analysis more fully demonstrates the effectiveness of FLoSER-S than the complete FLoSER.
Questions for authors
See Weaknesses.
Response to Reviewer pzrK: Cross-Model Generalization, Client-Pool Expansion, Computational Cost, and Response-Aware Routing
Thank you for your comments. We address your four concerns below.
1. Generalization across model families and sizes (Weakness 1). We added Gemma-2-2B-IT, a smaller model from a third family, and a controlled Mistral K12/K24 study.
The original Qwen/Mistral comparison also changes adapter sources, ranks, and sparse budgets: K12 of 24 experts versus K12 of 48. We therefore compared K12/K24 within Mistral, fixing the banks and pairing Raw/SC initialization, data, and client schedules, with RA/EEQA off. At fixed R18, SC's mean generation-accuracy advantage is +0.13pp at K12 and +4.17pp at K24, with all three K24 pairs positive (+5.08/+5.86/+1.56pp). This shows that the limited original gain is not a fixed backbone limitation. K changes gate mass and optimization alongside support size, so the result concerns the routing configuration as a whole.
On Gemma-2-2B-IT with a different pretrained adapter bank, SC lowers R18 validation CE in all three seeds (mean 0.50948 vs. 0.51481). Mean generation accuracy is 43.23% vs. 42.58%, with SC-Raw differences +5.08/-2.73/-0.39pp. Thus, the third-family evidence shows consistent validation-loss improvements but mixed generation gains. Both studies use 256 shared GSM8K questions per seed, greedy 512-token decoding, and the Original Parser.
2. Extension to 100 clients (Weakness 2). Following your suggestion, we expanded to 100 clients, completing Raw/SC training and generation evaluation at 20/100 clients across three seeds.
Both settings use the same 2,560-example training pool, with two participating clients per round, 20 rounds, and 400 client optimizer updates per run. This matched design evaluates client-pool expansion under fixed participation and optimization budgets. Within each size, Raw/SC share partitions, schedules, and initialization; RA/EEQA are off.
At fixed R18 on the same 256 test questions, mean Raw/SC accuracy is 31.12%/42.58% at 20 clients and 24.22%/42.58% at 100 clients. Paired SC-Raw gains are +11.46 +/- 3.32pp and +18.36 +/- 2.03pp (three-seed sample SD); all three seeds favor SC at each size. The frozen evaluation uses greedy 512-token, dual-EOS decoding and the Original Parser.
SC's relative task benefit persists after a fivefold client-pool expansion, beyond the original 20-client configuration. Two clients still participate per round: participation falls from 10% to 2%. Over 20 rounds, the schedules covered 17/20 and 31/100 distinct clients, respectively, so realized client-data coverage also differs.
3. Communication, computation, and convergence (Weakness 3).
Communication. Qwen Raw/SC have identical FP32 router payloads: 18.38 MiB per state, or 73.52 MiB per round for two clients each downloading/uploading once. RA adds 0.0279% model payload, excluding initial distribution and network overhead.
Computation. On one RTX 4090, with matched batch-1 inputs padded to 384 tokens and three interleaved blocks, pooled median training-update times are 2.896/2.862/3.219 s for Raw/SC/RA (16 microbatches). RA's paired overhead over SC is 1.04%-14.28% across blocks. Despite its small parameter overhead, RA incurs measurable runtime cost. One-time Qwen CPU bank construction takes 0.135/0.050 s for SC/Raw. These are fixed-input, not generation-latency, measurements.
Convergence. At matched update and communication budgets, R18 Raw/SC validation CE is 0.3891/0.3344 at 20 clients and 0.4012/0.3562 at 100 clients. Full curves will accompany these fixed-budget comparisons.
Communication is reported as protocol-based payload accounting; the sequential simulation does not measure real network transfer or distributed aggregation latency.
4. RA mechanisms and conditional gains (Weakness 4). Our additional analyses show that RA changes expert supports and weights but does not consistently improve downstream accuracy. Response magnitude reflects response strength, but is insufficient to reliably indicate task-benefit direction.
On 16 questions, the three-seed mean Spearman correlation between response magnitude and selected experts' local first-order utility estimates is 0.015 for signed utility versus 0.461 for absolute utility. This suggests a stronger association with influence strength than beneficial direction.
Teacher-forced CE evaluates gold-prefix predictions; generation accuracy evaluates model-produced answers. In a separate paired Qwen seed-1 R18 diagnostic, SC+RA lowers validation CE 0.358938 to 0.356513, yet both score 105/256 under the same greedy 192-token Original-Parser protocol.
With theta and experts fixed and only alpha trained, all-token CE versus alpha0 improves in one seed and worsens in two at R18; it worsens in all three at R20. SC remains our main structural contribution; RA is an auxiliary, input-dependent correction with observable routing effects and conditional task benefits. We retain Table 2's zero and negative RA results.