acceptodds
Withdrawn from ICLR 2027

Decoupling Rotation and Deformation via Frozen Reservoirs for Efficient Fine-Tuning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Parameter-efficient fine-tuning (PEFT) is widely used for low-cost adaptation of pretrained models. From a geometric perspective, full fine-tuning permits flexible, high-rank updates that alter the orientation and scale of weight mappings, whereas representative PEFT methods restrict this flexibility through low-rank or orthogonality constraints. To enable high-rank weight updates and independently control rotation and deformation in weight transformations under a limited parameter budget, we propose Geometrically Decoupled Reservoir Adaptation (GeDRA). GeDRA decomposes a frozen operator reservoir into rotation-generator and deformation-generator templates and learns one modulation vector per template, enabling separate control over local rotation and deformation updates in the tangent space. Theoretical analysis reveals an expressivity advantage of geometric decoupling in terms of expected optimal approximation error under matched trainable parameter budgets. We further observe broad directional support for both rotation and deformation in GeDRA, qualitatively similar to full fine-tuning. Experiments on commonsense reasoning, image classification, and natural language understanding demonstrate favorable performance–parameter trade-offs against strong baselines. Qualitative comparisons with LoRA further illustrate GeDRA's effectiveness in subject-driven generation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large language model, parameter-efficient fine-tuning, geometric decoupling

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_75MGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary
  • The paper proposes GeDRA (Group Decoupled Reservoir Adaptation), a PEFT method that indepentently modulates rotation and deformation in weight transformations. The motivation behind GeDRA is to perform high rank updates while lowering trainable parameters.
  • GeDRA structures the parameter updates as Wθ=(In+Aθ)W0W_\theta = (I_n + A_\theta)W_0, where Aθ∈Rn×nA_\theta \in \mathbb{R}^{n \times n} is the geometric adaptation generator.
  • AθA_\theta is constructed as $U_\ell = \operatorname{Diag}(\xi_\ell) P_\ell, \qquad Q = I_n + \frac{1}{\sqrt{2M}} \sum_{\ell=1}^{M} U_\ell$ where PlP_l is sampled uniformly from a set of n×nn\times n permutation matrices, and ξℓ∈{−1,+1}n\xi_\ell \in \{-1,+1\}^n.
  • The reservoir is decomposed into skwe-symmetric component RR which is the rotation-generator template and symmetric component DD which is the deformation-generation template.
  • AθA_\theta is obtained by adding two matrices Ar=Diag⁡(r) R+R Diag⁡(r)2A_r = \frac{\operatorname{Diag}(\mathbf{r})\, R + R \,\operatorname{Diag}(\mathbf{r})}{2} and Ad=Diag⁡(d) D+D Diag⁡(d)2A_d = \frac{\operatorname{Diag}(\mathbf{d})\, D + D \,\operatorname{Diag}(\mathbf{d})}{2}, where r\mathbf{r} and d\mathbf{d} are trainable vectors.
  • The method is evaluated on commonsense reasoning, natural language understanding, visual recognition and sibject driven generation.
Critical strengths
  • The rotation and deformation analysis of full fine-tuning, LoRA and OFT, and the result that GeDRA update shows pattern similar to full fine-tuning is interesting.
Critical weaknesses
  • The performance of LoRA in all the benchmarks looks much lower than expected. For example, on commonsense reasoning, [1] reports 86.3% accuracy for LoRA with Llama3 8B. This paper reports 80.19%. While a few point difference is expected, this difference is too large, which suggests that the hypserparameters for LoRA were not tuned well or the evaluation settings like sampling are different.
  • This makes the gains over LoRA questionable.
  • The does not cite WeGeFT [1], which is highly related and has the same structural form: WeGeFT uses an update Wθ=(I+A)W0W_\theta = (I + A)W_0 when applied to the output dimensions of the weights (or Wθ=W0(I+A)W_\theta = W_0(I + A) when applied to the input dimensions), where A=ϕψA=\phi\psi is a low rank matrix with ϕ\phi and ψ\psi learned directly instead of . The work should include a comparison with WeGeFT in shared and unshared modes.

References

[1] Savadikar et al., "WeGeFT: Weight-Generative Fine-Tuning for Multi-Faceted Efficient Adaptation of Large Models", ICML 2025

Questions for authors
  • How are the hyperparameters tuned for LoRA?
Reviewer_ikBzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces GeDRA, a parameter-efficient fine-tuning method that decomposes a frozen random operator reservoir into skew-symmetric and symmetric templates. Two trainable vectors modulate these two templates separately to control local rotation and deformation. The authors show that GeDRA can represent certain target transformations more accurately than two-sided scaling of the same reservoir. Experiments on commonsense reasoning, image classification, and natural language understanding show competitive performance-parameter trade-offs. The paper also presents qualitative comparisons on subject-driven image generation.

Critical strengths
  1. The proposed method is simple and parameter-efficient. By learning only two modulation vectors per adapted weight matrix, GeDRA supports high-rank updates while separately controlling local rotation and deformation.

  2. The evaluation covers multiple models across commonsense reasoning, image classification, and natural language understanding. The reported results show competitive performance-parameter trade-offs across different settings, suggesting that the approach has potential beyond a single application.

Critical weaknesses
  1. The theoretical analysis and ablations do not sufficiently establish the advantage of geometric decoupling.

    (a) The theory compares GeDRA with two-sided scaling on targets constructed from GeDRA's templates, T=αR+βDT = \alpha R + \beta D. GeDRA has the advantage since it can change entries that two-sided scaling must keep at zero. However, the analysis does not show whether geometric decoupling is better than other methods that can also change those entries.

    (b) The component ablation does not isolate the contribution of geometric decoupling. As shown in Figure 4 a) i), combining rotation and deformation improves VTAB accuracy from 61.56% to 62.66% over deformation alone, but doubles the number of trainable parameters. The experiment does not distinguish the advantage of combining rotation and deformation from that of increasing trainable capacity. A parameter-matched experiment would help clarify the contribution of the rotation component.

  2. The baseline settings and reported results do not provide enough evidence to confirm the reported performance improvements.

    (a) There are substantial discrepancies from published baseline results. For example, the original VeRA paper reports RoBERTa-base scores of 65.6 on CoLA and 78.7 on RTE, compared with 52.08 and 58.48 in this paper at a similar adapter parameter count. Although experimental protocols differ, these gaps raise concerns about whether the baselines are adequately tuned.

    (b) The paper reports averages over five random seeds but does not show how much the results vary across runs, making it difficult to judge whether the smaller gains are consistent.

Questions for authors

Please refer to the weaknesses discussed above.

Reviewer_9V8wOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a parameter efficient fine-tuning (PEFT) parameterization based on the (core) ideal of first creating a pool of rotation & deformation operators by constructing transformation matrices with Rademacher (-1/0/+1) entries and then expressing the PEFT module as a function of them. Since one vector for each one of these two are learned per layer, it can deliver complexities below the complexity of RANK-1 LoRA components, following the recent works on PEFT.

Critical strengths
  • The method provides clear flexibility in terms of the PEFT complexity.
  • Thanks to its construction, it is not limited by low-rank updates as in LORA.
  • The experimental analysis coverage in terms of task variety is strong, covering both NLP and image classification and generative model personalization tasks. Promising results are reported, for example GeDRA seems to bring PEFT performance better than / much closer to that of full-fine tuning compared to LORA and many PEFT variants in Table 1 and Table 2. Similarly, Table 3 GLUE benchmark results are promising.
Critical weaknesses
  • One particularly relevant baseline is missing: SHiRA, which is (similarly) randomization based and provides full-rank updates.
  • SDXL results are qualitative only, despite the existence of prior works providing quantitative metrics, both on PEFT-based personalization and one-shot subject-driven generation . Quick options include CLIP-I, CLIP-T, DINO -- as in the DreamBooth paper. More sophisticated metrics, especially for human ID personalization, do exist in more recent works.
  • Figure 4 suggests that the method performs degeneratively beyond M=2, which is confusing and suggests a fundamental limitation in the construction of the method.
Questions for authors
  • Any principled explanations of what is observed in Figure 4?
  • Should we understand that the method scales poorly beyond M=2 in general?
  • Why no SHiRA comparison?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.