- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The paper proposes SEA, a static layer-wise rank allocation scheme for low-rank fine-tuning. Ranks are assigned by greedily retaining singular modes with the highest gradient-energy density under a parameter budget, where the gradient spectrum is estimated once via SVD of the averaged gradient over high-loss calibration samples. Experiments on five models and several benchmarks show improvements over uniform and adaptive rank-allocation baselines.
Critical strengths
- The method is simple, static, and adds no per-step training overhead; it is compatible with both SVD reparameterization and LoRA.
- The weight-vs-gradient spectral analysis is nicely presented, and the empirical coverage across models and benchmarks is broad.
Critical weaknesses
-
The core objective is a heuristic rather than a justified criterion. The paper optimizes retained normalized gradient singular energy, but it never shows that this surrogate actually tracks downstream loss, either theoretically or through targeted ablations (such as comparing against a magnitude-weighted variant). In particular, normalizing the energy per layer is an unexamined design choice that discards gradient magnitudes and could easily introduce bias rather than remove it. Meanwhile, the theoretical results (convexity, threshold rule, and near-optimality) are just standard fractional-knapsack derivations and do not bridge the gap between the surrogate objective and the true loss. Overall, the central premise of the method relies on assertion rather than solid evidence.
-
The gradient calibration step is methodologically questionable. SEA relies on applying SVD to the averaged gradient over high-loss samples. However, averaging gradients across samples allows individual directions to cancel out, which can severely distort the singular spectrum being measured. In addition, the claimed norm relationship between the full-batch and calibration gradients does not hold as stated, and selecting only high-loss samples tends to over-represent outliers. A more principled alternative, such as using the uncentered second moment of the gradients (as done in GaLore), is neither discussed nor compared against. Because the entire rank allocation depends on this calibration signal, these methodological flaws are a major concern.
-
The empirical gains cannot be clearly attributed to the proposed score. Because the experiments lack a random non-uniform allocation baseline, it is impossible to tell whether the improvements come from SEA's specific ranking or simply from using a non-uniform rank distribution. The inverted-order ablation only shows that the method is better than an adversarial worst-case assignment. Given that the main contribution of the paper is the importance score itself, failing to isolate its effect from generic non-uniformity is a critical gap.
Questions for authors
- How does the spectrum of the averaged calibration gradient compare to the full-batch gradient spectrum (e.g., subspace alignment), and to the spectrum of E[GG⊤]?
- Can you add a random non-uniform allocation baseline at matched budget to isolate the value of SEA's ordering?
- Can you ablate the per-layer normalization against a magnitude-weighted objective?