acceptodds
Under review as a conference paper at ICLR 2027

SVD-IMOE:PRETRAINEDSPECTRALDIRECTIONSAS IMPLICITROUTERSFORLOW-RANKADAPTATION

Abstract

Parameter-efficient fine-tuning with low-rank adapters typically relies on dense rank usage or randomly initialized routing directions, which may ignore the spectral geometry of pretrained weights. We propose SVD-IMoE, a spectral implicit mixture-of-experts framework that uses pretrained right-singular directions as frozen routing bases for sparse low-rank adaptation. SVD-IMoE-W selects directions through calibration-free stratified Weight-SVD and applies inverse-square-root spectral scaling. SVD-IMoE-TA further uses task activations to select task-aligned directions within the same spectral bins, together with a heuristic alignment gate and damped adaptive normalization. Both variants use token-wise top- routing and train only the LoRA up-projection. With matched and , SVD-IMoE-W improves GSM8K over FlyLoRA by 4.39 points on Llama 3.1-8B and 2.27 points on Qwen 2.5-7B. SVD-IMoE-TA improves Qwen HumanEval Pass@1, Pass@5, and Pass@10 over SVD-IMoE-W by 3.60, 3.27, and 2.21 points, respectively. In four-adapter parameter averaging, SVD-IMoE-TA outperforms merged FlyLoRA on all twelve evaluated backbone-metric comparisons. These results suggest that pretrained spectral geometry provides a useful routing prior, while activation geometry enables selective task specialization and improves cross-task adapter compatibility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
PEFT, LoRA, SVD, spectral routing, implicit MoE, sparse adaptation, task adaptation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Lq1jOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SVD-IMoE, a parameter-efficient adaptation framework that replaces the frozen random routing basis in implicit rank-wise MoE-LoRA with spectral directions derived from pretrained weights. It introduces two complementary variants: SVD-IMoE-W, which performs calibration-free stratified selection of right-singular vectors, and SVD-IMoE-TA, which uses calibration activations to select task-aligned directions with quality gating and adaptive normalization. Both variants freeze the routing basis and train only the LoRA up-projection.

Critical strengths
  1. The paper identifies a key limitation of implicit rank-wise routing: frozen random projections ignore the geometry of pretrained weights. Replacing them with the right-singular directions of pretrained weights is intuitive and technically well motivated, while requiring neither a learnable router nor an explicit expert bank.
  2. The two variants provide complementary perspectives. SVD-IMoE-W serves as a robust, task-independent default, whereas SVD-IMoE-TA leverages activation geometry to explore task-specific specialization. The improvements on code-generation benchmarks provide some evidence that task-adaptive routing can be beneficial in certain settings.
Critical weaknesses
  1. The improvements in the main results reported in Table 2 are limited and inconsistent across benchmarks. Although the proposed methods achieve noticeable gains on some tasks, particularly GSM8K and selected code-generation metrics, they do not consistently outperform the baselines on MMLU, ScienceQA, and several HumanEval settings. In addition, the paper provides only a theoretical comparison of trainable parameter counts and lacks empirical analysis of training time, inference latency, memory consumption, and actual computational efficiency. This is particularly important because the current implementation uses dense kernels and therefore does not realize practical speedups from sparse rank activation.
  2. The baseline coverage is relatively limited, with the main comparisons focusing on standard LoRA and FlyLoRA. Given the rapid development of spectral LoRA, rank-wise routing, and LoRA-MoE methods, it is unclear whether the proposed approach remains competitive against a broader set of recent methods. Comparisons with stronger and more diverse baselines are needed to establish the current empirical advantage of SVD-IMoE.
  3. The paper motivates its approach partly through the potential advantages of MoE-style adaptation in multi-task settings, but the experiments do not include genuine multi-task training or evaluation. The four-adapter parameter-averaging experiment measures post-hoc adapter composition rather than joint multi-task learning, shared routing across tasks, or interference under mixed-task inputs. Therefore, the current evidence is insufficient to support claims about the advantages of the method in multi-task settings.
  4. The analysis of the proposed mechanism is insufficient. In Figure 3, the task-guided and weight-only alignment curves largely overlap, while the quality gate accepts task guidance in only 15–21% of the layers. Moreover, because task-guided selection explicitly optimizes alignment with the calibration subspace, evaluating it using the same alignment measure is partly self-confirming. The figure does not establish whether the observed differences are statistically meaningful or whether increased alignment leads to improved downstream performance. More targeted ablations and analyses are needed to isolate the effects of task-guided direction selection, spectral normalization, routing-bias initialization, and the quality gate.
Questions for authors

Please refer to the Weakness

Reviewer_y2zuOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SVD-IMoE, a low-rank adaptation method based on pre-trained weight spectral information. Unlike FlyLoRA, which uses random projections for implicit routing, this method constructs a routing space using the singular vectors of pre-trained weights. Two variants, SVD-IMoE-W and SVD-IMoE-TA, are designed to implement weight-based spectral direction selection and task-adaptive direction selection, respectively. Experiments were conducted on Llama 3.1-8B and Qwen 2.5-7B, and the results demonstrate that this method achieves good performance in partial inference, code generation, and multi-task adapter fusion tasks.

Critical strengths

S1: The motivation of this paper is clear. In this paper, the authors use spectral information from pre-trained weights to replace random routing directions. I think this is a reasonable approach. Besides, the proposed method does not require the introduction of an additional learnable routing network, thereby maintaining a low parameter cost.

S2: The paper includes sufficient experiments and compares the results with some promising prior work. In addition, detailed ablation experiments are also conducted. The experimental results demonstrate the effectiveness of this method to a certain extent.

Critical weaknesses

W1: I have some questions about the rationale of the hierarchical spectrum selection approach, i.e., since directly using Top-r singular vectors can also preserve important information in the pre-trained weights, what exactly are the advantages of the current hierarchical selection strategy?

W2: The design of Quality Gate relies on empirical parameters, e.g., why did the authors choose 1.5 as the threshold coefficient? Is this setting stable across different setting models and tasks? I think the authors should provide more explanation.

W3: In the experiment, all sparse routing methods used (r=32) and (k=8). I am particularly interested in whether SVD-IMoE can still maintain its current performance advantage when the rank or Top-k changes.

W4: The current experiments focus primarily on FlyLoRA, but I think these comparisons with spectrum decomposition-based methods, e.g., PiSSA and CorDA, are not yet enough. I suggest the authors add more experiments and comparisons, which will help to more clearly highlight the innovations and advantages.

W5: Although the proposed method reduces the number of effective rank components involved in the computation, the actual implementation still relies on dense matrix operations. Furthermore, the additional SVD and calibration processes also introduce overhead. Therefore, I think the authors should conduct some extra experiments to verify the current efficiency advantage.

W6: I think Figure 2 is too simplistic; some details are missing. This makes the paper difficult to read. I suggest that the author include more details.

Questions for authors

Please refer to W1-W6. I think some concerns should be explained and clarified. Besides, some extra experiments need to be added.

Reviewer_hnf8OpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper replaces FlyLoRA's frozen random projection with selected right-singular vectors of pretrained weights for token-wise top-k rank routing. A task-adaptive variant additionally selects directions using an activation-derived subspace, with heuristic gating and normalization. Results are reported on two LLMs across reasoning, knowledge, and code benchmarks, including adapter averaging.

Critical strengths
  • The approach retains FlyLoRA's parameter-efficient, router-free training setup and reports gains over FlyLoRA on GSM8K for both evaluated backbones.
  • The presentation is clear.
Critical weaknesses
  • The motivation for spectral routing is unconvincing. Singular directions of pretrained weights do not necessarily correspond to directions requiring downstream adaptation. Relative to FlyLoRA, the method mainly replaces a frozen projection basis and modifies routing scores, while related LoRA methods already exploit spectral information. The paper does not establish why its particular stratified selection and inverse-singular-value scaling should be preferable for adaptation.

  • The empirical evaluation is too narrow to support the claimed contributions. The main baselines are standard LoRA and FlyLoRA, with no convincing comparison to stronger contemporary spectral or parameter-efficient adaptation methods. Essential ablations, including random orthogonal directions, top-r SVD directions, scaling choices, and isolating task-guided selection from other TA changes, are missing. Consequently, the gains cannot be attributed to the proposed spectral mechanism, and the averaging experiments do not validate the stated shared-routing fusion condition.

Questions for authors
  1. Why should right-singular directions of pretrained weights better identify directions useful for downstream adaptation, and what evidence directly tests this hypothesis?
  2. Can the authors add stronger recent PEFT baselines and controlled comparisons that isolate spectral direction selection, routing-score scaling, and the task-adaptive components?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.