acceptodds
Withdrawn from ICLR 2027

Curvature-Guided Rank Scheduling for LoRA Fine-Tuning of Vision Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Low-rank adaptation (LoRA) makes transfer learning parameter-efficient, but its rank is usually fixed before optimization and shared across layers. We introduce Curvature-Guided Rank Scheduling (CGRS), which uses a diagonal empirical-Fisher signal to change LoRA capacity during training. A global scheduler grows or shrinks one shared rank, while layer-wise schedulers allocate heterogeneous ranks across Vision Transformer (ViT) blocks; within each block, the query and value projections share a common rank. Rank changes preserve the learned update through a singular value decomposition (SVD)-based transfer and are delayed by a grace period to avoid reacting to noisy early curvature. We evaluate ImageNet-pretrained ViT-Base/16 on CIFAR-100, Street View House Numbers (SVHN), and Oxford Flowers-102 against full fine-tuning, a frozen backbone, a 14-rank LoRA sweep, and AdaLoRA. Global CGRS reaches 91.16% on CIFAR-100 and 97.45% on SVHN, improving over fixed LoRA at the same final rank by 0.70 and 0.47 percentage points, respectively. It also exceeds the best tested AdaLoRA accuracy by 1.10 and 1.54 points on these datasets. On Flowers-102, however, AdaLoRA is better (98.78% versus 98.52%), and some layer-wise schedules collapse to the maximum rank. These results support curvature as a useful capacity signal while identifying low-data calibration and one-way rank growth as important failure modes. Given single-seed evaluation, we treat the results as a pilot study establishing feasibility rather than a statistically validated comparison; multi-seed replication is the immediate next step.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
PEFT, LoRA, Adaptive Rank Allocation, Vision Transformers, Dynamic Rank Scheduling, Image Classification, Transfer Learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_nEzpOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Curvature-Guided Rank Scheduling (CGRS), an approach that dynamically adjusts LoRA rank capacity during Vision Transformer fine-tuning. While LoRA fixes adapter rank prior to training and shares it across all layers, CGRS uses an approximation of the diagonal empirical Fisher information as an inexpensive signal of how sensitive the loss is to each adapter. When curvature indicates an adapter is constrained, the framework increases capacity; when sensitivity is low, it can reduce capacity.

The proposed global scheduler adjusts a single shared rank along a discrete ladder, using singular value decomposition (SVD) to transfer learned weights without loss of function. Layer-wise variants instead assign heterogeneous ranks per transformer block under growth-only policies.

Experiments on ImageNet-pretrained ViT-Base/16 across CIFAR-100, SVHN, and Oxford Flowers-102 show modest gains on the (saturated) benchmarks, outperforming fixed LoRA by 0.70 points on CIFAR-100 and 0.47 points on SVHN. However, on Flowers-102, authors suggest that weak curvature signals cause erratic rank oscillations globally and trigger irreversible collapse to maximum rank across all blocks in layer-wise schedules.

Critical strengths

The paper is honest about the limitations and failure modes of curvature guidance on Flowers-102. This transparency provides actionable insights to researchers developing dynamic parameter-efficient fine-tuning methods.

Using singular value decomposition to project or embed adapter weights across rank changes is conceptually practical. Retaining the leading singular components ensures that functional progress is preserved during resizing, avoiding the disruption associated with cold reinitializations.

Critical weaknesses

The datasets reported are small and saturated benchmarks for large vision transformers pretrained on Imagenet. Harder tasks or NLP tasks where LoRA is typically used would be more relevant and would most likely highly more capacity failures of LoRA (finetuned tasks highly different to the pretrained tasks).

The reported improvements over fixed LoRA are modest (+0.70% on CIFAR-100 and +0.47% on SVHN) and fall well within standard run-to-run variance for Vision Transformer fine-tuning. This makes it questionable to claim statistical significance of the improvements.

Each rank change recreates the Adam optimizer state and rescales the learning rate proportionally. The paper provides no control baseline to test whether fixed-rank LoRA benefits equally from periodic optimizer restarts and rate adjustments. As a result, it is unclear whether the slight accuracy gains originate from curvature-guided capacity or secondary optimization side effects.

As a result, although the initial idea of the paper is relevant and deserves to be studied further, the paper is not ready to be accepted at ICLR. I encourage the authors to strengthen their evaluation further in view of a re-submission.

Questions for authors

What is the measured wall-clock time and memory overhead of computing diagonal empirical Fisher probes and performing SVD resizes compared to standard LoRA and AdaLoRA?

Can thresholds be normalized online during training to eliminate the need for a separate pre-training profiling run at rank 16?

Reviewer_T4wUOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies dynamic rank selection for LoRA. While LoRA enables parameter-efficient transfer learning, its rank is typically fixed throughout optimization and often shared across layers. The key intuition of this work is that if the task loss remains consistently sensitive to a particular adapter, its current low-rank subspace may be too restrictive; conversely, persistently low sensitivity provides little evidence that additional rank is needed.

Based on this intuition, the paper proposes Curvature-Guided Rank Scheduling (CGRS), which dynamically adjusts LoRA ranks during training using curvature estimates derived directly from squared task gradients. The framework includes global schedulers that support both rank growth and shrinkage, as well as several layer-wise variants that only increase rank. Rank transitions are handled through an SVD-based migration mechanism, together with several stabilization heuristics.

Experiments on CIFAR-100, SVHN, and Flowers-102 show that CGRS outperforms fixed-rank LoRA with the same final rank on CIFAR-100 and SVHN. However, performance degrades on Flowers-102, where the learned rank allocation is also unstable.

Critical strengths
  • The problem studied in this paper has practical significance: how to select and allocate rank for LoRA in the training process.
  • The experimental evaluation was relatively complete: a large-scale fixed rank scan, conventional fine-tuning baseline, and comparison with AdaLoRA were made.
  • The authors truthfully reported the negative results and distribution failure phenomenon, did not only show the full positive experimental results, and clearly pointed out several key limitations.
Critical weaknesses
  • The technical increment with existing work is not yet clear. There have been relevant studies on adjusting LoRA rank and allocating parameter budget according to importance during training: AdaLoRA[1] adopts importance driven rank budget allocation, and IncreLoRA[2] gradually increases adaptation parameters according to module importance. Furthermore, E²LoRA[3] uses gradient information to guide adaptive rank allocation, while FiLoRA[4] uses Fisher information to construct adaptation subspaces. These works coincide with the dynamic capacity allocation, curvature control signal and scheduling stabilization in this paper.
  • The core mechanism has not been independently verified. CGRS will change the adapter rank, migration parameters, reset the optimizer status, and modify the learning rate at the same time. Therefore, even if the performance is improved compared with the fixed rank LoRA, it cannot be proved that the curvature signal has found a suitable adaptive capacity. Section 6 of the paper also explicitly recognizes that "establishing which transition capacity growth, SVD transfer, or optimizer restart causes the gain requirements exemptions not available in the current experiments."
  • Insufficient statistical reliability. All experimental configurations use only a single random seed evaluation. Compared with the final rank fixed LoRA, the core improvement of the paper is only 0.70% on CIFAR‑100 and 0.47% on SVHN. Section 4 of the paper explicitly acknowledges "given single seed evaluation, we treat these results as a pilot study establishing feasibility rather than a statistically validated comparison", and section 7 explicitly acknowledges "every configuration has one seed, so 0.5 – 1.5 point differences may include run to run noise"
  • Heuristic rules and failure causes lack validation. The method relies on a predefined rank ladder, percentile-based thresholds, a shrink threshold of 0.3\tau, a grace period, a cooldown period, and a hand-specified minimum-rank safeguard for blocks 4–7. Although the paper compares three percentile thresholds and several allocation strategies, it does not verify the necessity of other rules, nor does it systematically analyze the sensitivity of the results to these settings. The empirical results also show the unresolved allocation failure: in Table 2, PL-C3-Live of Flowers-102 and Ordinal-K5 of SVHN eventually degenerated to uniform rank 64. Section 6 also explicitly acknowledges that "the negative flowers result is more consistent with a calibration failure, alternate the present experiments do not establish a causal mechanism."
  • The actual income is lack of complete evaluation. Section 7 explicitly recognizes "final adapter counts do not measure peak training memory, wall clock time, or the overhead of curvature probes and SVD transfers.". Threshold calibration also requires additional fixed rank training and curvature analysis. Therefore, the final adapter parameter can only reflect part of the storage cost, and cannot directly explain the overall training efficiency.

[1] Zhang Q, Chen M, Bukharin A, et al. Adalora: Adaptive budget allocation for parameter-efficient fine-tuning. 2023. [2] Zhang F, Li L, Chen J, et al. Increlora: Incremental parameter allocation method for parameter-efficient fine-tuning. 2023. [3] Li M, Ye P, Ye J, et al. E²lora: Efficient and effective low-rank adaptation with entropy-guided adaptive sharing. 2026. [4] Han D, Guo S. FiLoRA: Parameter-Efficient Fine-Tuning With Fisher Information-Guided Low-Rank Adaptation. 2026.

Questions for authors
  • What is the key technical distinction between CGRS and prior adaptive-rank methods such as AdaLoRA, IncreLoRA, E²LoRA, and FiLoRA? Can the authors provide direct experimental comparisons with the most closely related methods?
  • Can the authors disentangle the effect of the curvature signal from other changes introduced during rank transitions, such as SVD migration, optimizer restart, and learning-rate modification?
  • Can the authors report results over multiple random seeds? Given that the main gains over fixed-rank LoRA are below 1 percentage point on CIFAR-100 and SVHN, it is difficult to assess whether the improvements are statistically meaningful.
  • How sensitive is CGRS to the rank ladder, percentile threshold, shrink threshold, grace period, cooldown period, and minimum-rank safeguard? Which of these heuristics are essential?
  • Why do some configurations collapse to a uniform rank of 64, e.g., PL-C3-Live on Flowers-102 and Ordinal-K5 on SVHN? Is this caused by threshold calibration, curvature estimation, or the scheduling rule itself?
  • Can the authors report end-to-end training efficiency, including peak GPU memory, wall-clock time, curvature-estimation overhead, SVD migration overhead, and any additional cost required for threshold calibration?
Reviewer_VQ33OpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Curvature-Guided Rank Scheduling (CGRS), which dynamically adjusts LoRA ranks during ViT fine-tuning using an empirical-Fisher proxy. The authors study global and layer-wise scheduling on three image-classification datasets and compare against fixed-rank LoRA, AdaLoRA, full fine-tuning, and classifier-only training.

Critical strengths
  1. The paper addresses a relevant problem in parameter-efficient fine-tuning, and the proposed method is straightforward to understand.

  2. The authors provide a broad fixed-rank LoRA sweep, comparisons with AdaLoRA, and a candid discussion of limitations and failure cases.

Critical weaknesses
  1. The evaluation uses one backbone, three classification datasets, and only one random seed. The gains over final-rank-matched LoRA are modest. Moreover, the strongest CGRS thresholds and AdaLoRA budgets are selected using test accuracy, introducing selection bias.

  2. The method lacks central ablations: curvature-based decisions are confounded with rank changes, SVD transfer, optimizer resets, and learning-rate rescaling. The paper also reports final adapter counts without training time, peak memory, peak rank, or the overhead of curvature estimation and resizing.

Questions for authors
  1. Provide multi-seed results for the main comparisons using validation-based configuration selection.

  2. Explain how CGRS compares with a simple rank schedule that does not use curvature information

  3. Provide information on what additional training time and peak-memory cost arise from curvature estimation and SVD-based resizing.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.