acceptodds
Under review as a conference paper at ICLR 2027

Water-Filling LoRA: Data-Free Rank Allocation via Conditional Estimation–Approximation Bounds

Abstract

Choosing a LoRA rank for each layer is a resource-allocation problem when the total number of trainable parameters is fixed. We analyze this problem using a PAC-Bayes bound for Gibbs risk and an explicit spectral-tail approximation model. Under a common norm envelope and the stated approximation assumptions, the continuous optimum is proportional to pretrained stable rank, with clipping at the rank limits. The contribution is this conditional budgeted analysis, rather than the use of stable rank itself. We distinguish the continuous optimum, the proportionally rounded implementation, and a budget-feasible integer rule with a computable surrogate suboptimality certificate. An exact integer audit on Pythia-160M and Pythia-410M exposes a small budget overrun in the rounded 410M allocation and verifies feasibility of the budget-feasible rule. In four-seed adapter-only experiments, the proportionally rounded heuristic has lower mean validation perplexity than AdaLoRA on both models (28.353 versus 29.898 and 15.373 versus 16.078), while its advantage over uniform rank is not resolved. A separate four-seed comparison of the budget-feasible integer rule finds lower mean PPL than proportional rounding at 160M and lower mean PPL than uniform at 410M, but no consistent advantage across both comparators and models. Neither comparison verifies the assumptions linking the surrogate to downstream risk. The analysis identifies when a data-free allocation is optimal for a specified bound and where empirical validation is still necessary.

Primary area:
foundation or frontier models, including LLMs
Keywords:
LoRA, rank allocation, stable rank, PAC-Bayes, parameter-efficient fine-tuning, data-free adaptation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YCrWOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper proposes a method for allocating different LoRA ranks across layers based on the stable rank of pretrained weight matrices, without requiring any training data. The authors provide theoretical analysis to support their rank allocation method and compare it with several baselines. The experimental results show some improvements over AdaLoRA, but the improvements over uniform rank allocation are not consistent.

Critical strengths

The paper proposes a simple and interesting method for allocating LoRA ranks across different layers without requiring training data. The authors also provide interesting theoretical insights to support their approach.

Critical weaknesses
  1. I think the baseline comparisons are not enough. In particular, I would like to see a direct comparison with SR-LoRA, since it uses a similar idea of allocating ranks based on pretrained stable rank.
  2. I found the paper a bit hard to read, especially with the amount of theoretical and technical details throughout the paper.
  3. The abstract also includes too many technical details, which makes it harder to understand the main idea and contribution of the paper.
Questions for authors
  1. Could the authors clarify the empirical advantages of Water-Filling LoRA compared to SR-LoRA?
  2. Could the authors simplify the abstract and focus more on the main contributions rather than technical details?
1 comment

Thank you for suggesting a direct comparison with SR-LoRA and a simpler presentation.

  1. Comparison with SR-LoRA.

On EuroSAT, Water-Filling with standard LoRA updates achieves 94.81% ± 0.12%, versus 93.20% ± 0.61% for the official SR-LoRA implementation including stochastic partial updating (SPU). This is a 1.60 percentage-point improvement, with higher accuracy in all three seeds. Both methods use the same backbone, data split, optimizer, learning rate, batch size, 20-epoch schedule, and allocated adapter budget of 3,889,152 parameters. Results are final-epoch accuracy, reported as mean ± sample standard deviation.

  1. Clearer presentation.

We will simplify the abstract around the allocation problem, our conditional budgeted analysis, and the empirical findings. Detailed derivations and implementation audits will be moved out of the abstract, and the allocation procedure will be presented separately from its proof.

Reviewer_WJEWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Water-Filling LoRA, a data-free rank allocation method based on pretrained weight statistics. By combining PAC-Bayes bounds with spectral approximation assumptions, the authors derive a theoretically optimal rank allocation strategy under a fixed parameter budget. Experiments on Pythia models demonstrate competitive performance against existing baselines.

Critical strengths
  1. The paper presents a well-structured theoretical framework for LoRA rank allocation by integrating PAC-Bayes generalization analysis with constrained optimization.

  2. The proposed method enables layer-wise rank allocation without requiring downstream training data or additional fine-tuning for rank selection, making it practical for resource-constrained scenarios.

  3. The paper clearly distinguishes between continuous optimization, integer allocation, and empirical performance, while providing a transparent discussion of experimental limitations.

Critical weaknesses
  1. The use of pretrained stable rank for LoRA rank allocation has already been investigated in SR-LoRA. Therefore, the methodological novelty appears relatively limited, despite the additional theoretical analysis.

  2. The main controlled experiments are limited to two relatively small Pythia models on WikiText-2. Moreover, the proposed method does not demonstrate consistent or statistically significant improvements over uniform and random allocation.

  3. The theoretical optimality results depend on restrictive spectral approximation assumptions, whose validity and practical relevance are not sufficiently supported by empirical evidence.

Questions for authors
  1. Could the authors further clarify the key distinctions and practical advantages of Water-Filling LoRA over SR-LoRA, particularly given their shared reliance on pretrained stable rank?

  2. Could the authors provide additional experiments on larger models and more diverse downstream tasks to better assess the generalizability and effectiveness of the proposed method?

  3. Could the authors provide further empirical justification for the spectral approximation assumptions, especially considering the weak correlations between pretrained stable rank and task-update approximation residuals reported in Appendix E?

3 comments

Thank you for asking us to clarify the contribution and strengthen the empirical evidence.

  1. Distinction from SR-LoRA.

Our contribution is the conditional budgeted analysis, rather than pretrained stable rank itself: deriving the continuous allocation and a budget-feasible integer rule with a computable surrogate suboptimality certificate. This connects the allocation objective directly to the parameter constraint.

  1. New empirical evidence.

On EuroSAT, Water-Filling with standard LoRA updates achieves 94.81% ± 0.12%, compared with 93.20% ± 0.61% for the official SR-LoRA implementation including stochastic partial updating (SPU). All three seeds favor Water-Filling. Both methods use the same backbone, data split, optimizer, learning rate, batch size, 20-epoch schedule, and allocated adapter budget of 3,889,152 parameters. Results are final-epoch accuracy, reported as mean ± sample standard deviation. On Gemma-3-1B-it/WikiText-2, mean perplexity improves over uniform and random allocation by 2.66% and 2.93%, with improvements in all five seeds. These experiments add a larger language model than the original Pythia models and a vision task.

  1. Spectral assumptions.

The spectral-tail assumption supplies the rank-dependent approximation term used in the allocation objective. Together with the common norm envelope, it yields the stated proportional allocation. The new performance results complement, but do not verify, this assumption. We will clarify this distinction and explain how Appendix E relates to the assumed approximation model.

Thank you to the authors for their patient and thoughtful responses. Unfortunately, I have decided to maintain my original score.

Thank you for taking the time to consider our response and the additional experiments.

We will revise the paper to make the distinction from SR-LoRA clearer: the contribution is the budget-constrained analysis and the integer allocation rule with a computable certificate, rather than stable rank itself.

The Gemma and EuroSAT results will be included in the main text. Gemma improves over both uniform and random allocation in all five seeds, while the EuroSAT comparison adds the direct SR-LoRA baseline you requested. We will specify the update schemes in that comparison and explain how the spectral assumptions relate to the diagnostics in Appendix E.

Your feedback has helped us identify what needs to be clearer in the paper. Thank you again for the careful review.

Reviewer_m1bkOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors propose a new method for distributing LoRA ranks among layers. The approach is supported by theoretical formulas from PAC-Bayes approach. Experiments have been conducted on three models and one dataset with two simple competitors.

Critical strengths
  1. The authors describe the contribution of their work in a careful manner that shows respect for other studies.

  2. The theory provides a complementary addition to the proposed approach.

Critical weaknesses
  1. Current experiments show no advantage even over the simplest baselines—namely, random and uniform approaches. This is a critical shortcoming that undermines the case for the proposed method.

  2. In line 88, the authors list a large number of competing methods but do not compare their own approach against them in the experiments.

  3. The experimental setup is limited to just two rows in a table, which is clearly insufficient—both in terms of the number of models (including those from different developers) and in terms of the range of benchmarks used.

Questions for authors

See Weaknesses

3 comments

Thank you for highlighting the need for stronger baseline comparisons and broader experiments.

  1. Gains over uniform and random allocation.

In our five-seed Gemma-3-1B-it experiments on WikiText-2, Water-Filling achieves mean perplexity 14.187, compared with 14.574 for uniform allocation and 14.614 for random allocation—reductions of 2.66% and 2.93%, respectively. Water-Filling performs better in all five seeds against both baselines.

  1. Additional architecture, task, and baseline.

We also completed a three-seed ViT comparison on EuroSAT against the official SR-LoRA implementation including stochastic partial updating (SPU). Water-Filling with standard LoRA updates achieves 94.81% ± 0.12%, versus 93.20% ± 0.61% for SR-LoRA with SPU, improving accuracy in all three seeds. Both methods use the same backbone, data split, optimizer, learning rate, batch size, 20-epoch schedule, and allocated adapter budget of 3,889,152 parameters. Accuracy is measured at the final epoch; ± denotes the sample standard deviation across seeds. These additions extend the evaluation beyond Pythia to another language-model family and image classification.

Thank you to the authors for the response!

It is great that, in certain experiments, the proposed approach outperforms the simplest baselines (random and uniform). It is also a positive step to have an additional domain for evaluation. However, I still have some unresolved concerns that prevent me from raising my score.

  1. The number of baselines remains quite small. As the authors themselves note, there is a vast number of competing methods. Comparing against only one or two of them is clearly insufficient to fully grasp the contribution of this paper—especially given that, in the experiments involving Gemma, the proposed approach actually underperforms compared to other methods.

  2. I am concerned by the results for ViT + CIFAR-100. A quick look suggests that a standard ViT + CIFAR-100 combination should yield an accuracy of over 90%. While I understand that the authors are using LoRA and following a different protocol, such low performance is troubling. It seems questionable to evaluate the viability of methods in a setting that inherently produces poor results—results that are, in fact, unusable in practice—particularly when better alternatives exist.

Thank you for the follow-up. We appreciate the opportunity to provide the exact comparison set and CIFAR-100 protocol.

The revised evaluation includes five comparison methods across the language and vision experiments: uniform allocation, random allocation, AdaLoRA, a budget-adapted stable-rank allocation, and the official SR-LoRA pipeline with SPU. In the Gemma experiment, the budget-adapted stable-rank baseline is indeed slightly better than Water-Filling: 14.165 versus 14.187 validation perplexity, a difference of 0.022. The revised manuscript states this result explicitly. Water-Filling is better than uniform and random allocation in all five Gemma experiments, while the close result against the stable-rank baseline shows that the two pretrained-spectrum allocations perform similarly on this model. The distinction is that Water-Filling provides a budget-feasible integer allocation and a computable certificate for its allocation objective. In the direct official SR-LoRA comparison, Water-Filling improves final-epoch accuracy from 93.20% to 94.81% on EuroSAT and from 60.14% to 61.61% on CIFAR-100, with all six matched experiments favoring Water-Filling.

The CIFAR-100 experiment uses the VTAB-1k protocol, with 1,000 training images and 10,000 test images, rather than the full 50,000-image CIFAR-100 training set. The 90%+ figures commonly reported for ViT on CIFAR-100 use the full training set and describe a different training-data regime. Both methods here use the same 1,000-image split, pretrained ViT-B/16 checkpoint, classifier head, optimizer, learning rate, and 20-epoch schedule. The revised manuscript states the 1,000-image training-set size next to the CIFAR-100 result and distinguishes this VTAB-1k evaluation from full-data CIFAR-100 training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.