- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper introduces GA-SVD, which compresses LLMs via SVD with bidirectional whitening, and it also applies evolutionary search for layer-wise rank allocation.
Critical strengths
The paper evaluates SVD-based compression beyond perplexity and multiple-choice benchmarks, covering generation tasks such as code generation, mathematical reasoning and summarization.
The empirical coverage is broad, spanning several model families and sizes up to 70B parameters.
The authors also show that the method can be combined with quantization for further compression, and they provide sensitivity analyses of the warm-up ratio, calibration set size and rollout length.
Critical weaknesses
The paper lacks novelty compared to prior art.
The bidirectional whitening has already been proposed in OBD-LLM (Li et al., 2026) and GFWSVD (Chekalina et al., 2025); in particular, OBD-LLM’s whitening, SVD and recoloring steps are identical to Eqs. 6–7 and 33 here. Neither work is cited or compared against.
The evolutionary search for rank allocation is also a standard technique, and for its small modifications (specific mutation operators, etc.) the paper does not compare against a standard GA to show a significant benefit.
The paper also does not compare its rank allocation, on the same whitened weights, against prior rank allocation methods such as ASVD’s STRS (Yuan et al., 2023), AdaSVD (Li et al., 2025), or PGSVD (Solgi et al., 2025).
There also seem to be theoretical gaps. In the derivation of generation-aware whitening, the authors drop the first-order term after Eq. 15. Prior work (e.g., OBD-LLM) justifies this by assuming the gradient of the pretrained model is small. Here, however, the gradients are computed on the student, which is already compressed by the input-based warm-start step, so they may not be small (or near zero).
Also, the expansion in Eq. 14 is taken around the teacher’s output, while the statistics are computed from the student. The authors need to justify this derivation.
LM Loss and Prefix KD are used as alternative loss functions in the ablation but are not clearly defined.
From both the contribution perspective and writing clarity, the paper requires substantial revision.
References Y. Li, et al. Optimal Brain Decomposition for Accurate LLM Low-Rank Approximation. 2026.
V. Chekalina, et al. Scalable Kronecker-Factored Fisher Approximation for Neural Network Parameter Sensitivity. 2025.
R. Solgi, et al. Activation-Informed Pareto-Guided Low-Rank Compression for Efficient LLM/VLM. 2025.
Z. Yuan, et al. ASVD: Activation-Aware Singular Value Decomposition for Compressing Large Language Models. 2023.
Z. Li, et al. AdaSVD: Adaptive Singular Value Decomposition for Large Language Models. 2025.
Questions for authors
In Table 5, do the LM Loss and Prefix KD variants use ERA? If not, please explain how a bidirectional whitening method achieves worse results than GA-SVD w/o GAWW.
What is the novelty of this work compared with prior two-sided whitening methods? If using the one-sided compressed student to compute the gradients is the only difference, then the proposed solution seems less rigorous and suboptimal, as some of its assumptions may not hold in practice. What, then, makes this method better?