- Rating
- 1
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Curvature-Guided Rank Scheduling (CGRS), an approach that dynamically adjusts LoRA rank capacity during Vision Transformer fine-tuning. While LoRA fixes adapter rank prior to training and shares it across all layers, CGRS uses an approximation of the diagonal empirical Fisher information as an inexpensive signal of how sensitive the loss is to each adapter. When curvature indicates an adapter is constrained, the framework increases capacity; when sensitivity is low, it can reduce capacity.
The proposed global scheduler adjusts a single shared rank along a discrete ladder, using singular value decomposition (SVD) to transfer learned weights without loss of function. Layer-wise variants instead assign heterogeneous ranks per transformer block under growth-only policies.
Experiments on ImageNet-pretrained ViT-Base/16 across CIFAR-100, SVHN, and Oxford Flowers-102 show modest gains on the (saturated) benchmarks, outperforming fixed LoRA by 0.70 points on CIFAR-100 and 0.47 points on SVHN. However, on Flowers-102, authors suggest that weak curvature signals cause erratic rank oscillations globally and trigger irreversible collapse to maximum rank across all blocks in layer-wise schedules.
Critical strengths
The paper is honest about the limitations and failure modes of curvature guidance on Flowers-102. This transparency provides actionable insights to researchers developing dynamic parameter-efficient fine-tuning methods.
Using singular value decomposition to project or embed adapter weights across rank changes is conceptually practical. Retaining the leading singular components ensures that functional progress is preserved during resizing, avoiding the disruption associated with cold reinitializations.
Critical weaknesses
The datasets reported are small and saturated benchmarks for large vision transformers pretrained on Imagenet. Harder tasks or NLP tasks where LoRA is typically used would be more relevant and would most likely highly more capacity failures of LoRA (finetuned tasks highly different to the pretrained tasks).
The reported improvements over fixed LoRA are modest (+0.70% on CIFAR-100 and +0.47% on SVHN) and fall well within standard run-to-run variance for Vision Transformer fine-tuning. This makes it questionable to claim statistical significance of the improvements.
Each rank change recreates the Adam optimizer state and rescales the learning rate proportionally. The paper provides no control baseline to test whether fixed-rank LoRA benefits equally from periodic optimizer restarts and rate adjustments. As a result, it is unclear whether the slight accuracy gains originate from curvature-guided capacity or secondary optimization side effects.
As a result, although the initial idea of the paper is relevant and deserves to be studied further, the paper is not ready to be accepted at ICLR. I encourage the authors to strengthen their evaluation further in view of a re-submission.
Questions for authors
What is the measured wall-clock time and memory overhead of computing diagonal empirical Fisher probes and performing SVD resizes compared to standard LoRA and AdaLoRA?
Can thresholds be normalized online during training to eliminate the need for a separate pre-training profiling run at rank 16?