- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes a seed-decoupled evaluation protocol for LoRA rank allocation and uses it to audit whether learned/heuristic allocations genuinely outperform uniform rank. The core concern is valid: under coupled seeds, allocation and initialization are confounded. By fixing head, LoRA, and data streams per master seed and pairing each candidate against a shared equal-budget uniform r8 reference, the protocol removes 94.1% shared variance and tightens comparison error 4.7–6.6× at the cost of one extra reference run per seed. Empirically, a ~1-point advantage under coupled seeds vanishes (−0.63 pp, Holm p = 0.578), while permuting positions at fixed budget and rank multiset shifts accuracy by 4.3 pp—larger than typical claimed effects. The conclusion that fair comparison requires equal budget, fixed rank multiset, and a shared reference is well supported.
Critical strengths
Consider other factors affecting fine-tuning performance.
Critical weaknesses
Evaluating only on Llama-2-7B is not sufficiently convincing. In addition, there is a lack of extensive ablation experiments.
Questions for authors
None
2 comments
I appreciate the substantial additions, but the following concerns remain unresolved:
1 The q/v contrast is 7.11 pp on Llama-2 but only 2.40 pp on Llama-3. A threefold gap means the deployable gain cannot be predicted from the paper's characterization. → Add a limitation statement, or analyze how effect size scales with model width / GQA grouping.
2 Evaluation surface is narrow. The main conclusion rests almost entirely on MNLI-m, with QNLI, CoLA, and GSM8K serving only as consistency checks. → Add at least one long-form generation or instruction-following benchmark under the same paired protocol, to rule out that the positional effect is specific to classification heads and short sequences.
3 No comparison with scaling variants (rsLoRA, DoRA). Scaling and rank position may interact, and this omission weakens the "position rather than magnitude" conclusion. → Clarify whether scaling is orthogonal to the positional effect, and if not, test it.
Therefore, I keep my original score unchanged.
Reply to Reviewer iUJf
Based on your critique and suggestions, we have made further additions: the Llama-3-8B experiments promised in our first reply (noise decomposition and 8-seed paired panels), a 7B Qwen model, the generative reasoning task GSM8K, a
q,k,v,o_projadapter variant, and a full set of ablations that the revised paper now reports in the main text. The positional preference is consistent across three model families and three tasks, and the same permutation ordering holds in all five model–task cells (Kendall's W=1.000). This supports our central claim: rank allocation has a real positional effect, it can be measured only under paired, equal-budget evaluation, and Profile-then-Retrain (PtR) recovers it (+0.88 pp, 24/24 seed pairs).1. Beyond Llama-2-7B: delivered as promised.
From §6.2 of the revised paper:
v_heavy+0.59 / +0.52 (7/8, 8/8);q_heavy−1.81 / −1.62 (0/8, 0/8)q,k,v,o_projadaptersq,vcells; span 3.19 ppOn the pre-registered test, §6.1 states:
Llama-3 and Qwen use grouped-query attention; there the winning allocations carry fewer parameters than uniform (§3.1), so the gains do not come from capacity.
2. Ablations. Each row below is a controlled comparison at a fixed parameter budget with paired seeds.
The ablations agree. Searched optima fall below uniform; they looked positive because picking the best single reading inflates it by about as much as the gains reported in the literature. The rank map that AdaLoRA learns carries real positional information. PtR keeps that information and removes the max step: from §5.1, "§4 and this section use the same protocol, budget and seed pairing; §4 gives a negative verdict and this section a positive one (24/24)."
Thank you again for pushing us toward broader evidence. We hope the additional models, the generative task and the ablations address your concerns, and we would be grateful if you reconsidered your assessment.