- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies the limitations of LogDet compression metrics for evaluating model capabilities across families. It terms the phenomenon in which small eigenvalues inflate compression scores while accompanying stronger anisotropy “Compression Hacking,” and proposes three refined metrics: SE, Semantic CV, and PCS. Across 18 models from three families, these metrics achieve Spearman correlations of 0.917–0.965 with a comprehensive score aggregated from eight external benchmarks, compared with 0.445 for the original metric.
Critical strengths
The paper addresses a practically relevant problem: compression metrics that work within a model family may not reliably compare models across families. Table 2 shows substantial improvements in correlation with comprehensive capability scores for all three refined metrics, with improvements extending across multiple individual tasks. If reproducible, these results would be valuable for model evaluation based on hidden representations.
The experiments include relevant baselines such as Diff-eRank, Whitening, and LW shrinkage, together with parameter sensitivity, within-family comparisons, and leave-one-model-out analyses. PCS’s LOO correlations remain between 0.958 and 0.983, suggesting that the main result is not entirely driven by a single model.
Critical weaknesses
Important computational details remain unclear. Section 2 describes the centered covariance matrix as positive definite, but under the per-input computation in Section 3.2, it will be singular if the token count does not exceed the hidden dimension. The paper does not sufficiently explain how the original DE and condition number handle zero eigenvalues. Additionally, OPT-13B’s CE value of 42.09 in Figure 1 is inconsistent with the rank aggregation rule in Appendix B. Clarification of the numerical treatment and data sufficient to recompute the results are needed to establish whether these issues affect Table 2.
The mechanistic explanation is stronger than the available evidence. Section 2.3 interprets low-variance directions as noise, but Table 1 and Figure 2 mainly show spectral concentration, while Figure 3 shows a correlation between compression and anisotropy. Since both metrics depend on small eigenvalues, a high correlation alone cannot establish that these directions are noise or impair model capability. Appendix A already acknowledges the absence of a complete causal explanation. The main text should narrow its claims accordingly or provide evidence that distinguishes these explanations.
Questions for authors
Which samples are used for each covariance estimate in Section 3.2? How do the original DE, condition number, and relevant baselines handle zero or extremely small eigenvalues, and do they use consistent numerical treatment?
How is OPT-13B’s CE value of 42.09 in Figure 1 obtained from Equation (10) in Appendix B? Could you provide the raw scores and calculation steps needed to recompute CE and the correlations in Table 2?
Beyond the correlation between the two metrics, what evidence shows that tail directions are noise and cause misleading capability evaluations? If such evidence is not yet available, could this mechanism be explicitly presented as a hypothesis?
2 comments
Dear Reviewer 1cFk,
As the discussion period will end in about a day, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. If our responses have indeed resolved these issues, we kindly ask you to consider updating your rating and confidence. Should you have any remaining questions before the deadline, we would be more than happy to answer them at any time.
Response to Reviewer 1cFk
Dear reviewer 1cFk
Thank you for carefully reviewing our paper and providing valuable feedback, which has helped us improve the paper. We respond to your comments below.
Handling singular covariance matrices and zero or very small eigenvalues
Thank you for pointing out this missing explanation. The analysis in Section 2 assumes a positive-definite covariance matrix. We have added the diagonal regularization term that ensures positive definiteness, along with its default value, to the revised paper. We compute the metrics separately from the final-layer token representations of each input, then average them over the projection set. Inputs are truncated or padded to 64 positions, and padding positions are included in the calculation. In Section 2, mean-centered means subtracting the mean of the token representations within each input, yielding a zero-mean matrix. The preprocessing in Diff-eRank [1] also includes subtracting the sample mean. To prevent zero eigenvalues from making LogDet and the condition number undefined, we also add diagonal regularization:
ΣZ=∣V∣1Z⊤Z+αID,α=10−8. We will clarify this in the revised paper.
The distinction between 42.09 and CE
Two quantities need to be distinguished here: the value 42.09 for OPT-13B in Figure 1 is the average of its measured scores across the benchmarks, whereas CE aggregates the model's ranks across benchmarks using Eq. (10) in Appendix B. The former averages scores, while the latter aggregates ranks, so 42.09 does not need to be derived from Eq. (10). We will clarify this in the caption using “Mean benchmark score” to avoid confusion.
Insufficient mechanistic evidence that low-variance tail directions are noise and harm capability
Thank you for your suggestion. Compression hacking concerns the following issue: some low-variance directions contribute little to the task, but their very small eigenvalues substantially inflate the compression score after the negative-log transformation, leading to misleading capability evaluations. Existing intervention studies support this interpretation.
Removal experiments show that some low-variance directions make limited contributions to task performance. When 25% of the dimensions are removed without recovery fine-tuning, SliceGPT [2] retains approximately 99% and 96% of the original zero-shot performance of OPT-66B and Llama-2-70B, respectively. Research on sentence embeddings also finds that removing some low-variance directions with PCA can improve performance on some semantic tasks [4]. A vision-language study further finds that removing 164 low-variance directions from CLIP changes ImageNet Top-5 accuracy only from 91.2% to 91.1%. In contrast, removing the same number of random directions reduces it to 89.3% [5].
Low-variance directions can also reduce a score's ability to distinguish samples. In near-OOD detection experiments on CIFAR, Ren et al. [6] find that some low-variance directions contribute heavily to Mahalanobis scores despite providing little separation between in-distribution and out-of-distribution samples. Keeping only the top 76 principal directions improves detection AUROC from 74.98% with all directions to 79.72%. Semantic Volume [3], which also uses LogDet, likewise finds that removing some low-variance directions with PCA before computing the uncertainty metric improves detection compared with using all directions. In the revised version, we will incorporate these supporting evidence to make our claims more solid.
In Table 2, the correlations of SE and PCS with comprehensive capability CE are 0.917 and 0.965, respectively, compared with 0.445 for DE. These results show that the corrected scores better reflect measured capability. Our focus is that changes in tail eigenvalues may substantially change the compression score without corresponding changes in capability. The relevant tests are therefore whether compression scores accurately reflect model capability and whether reducing the tail's influence improves this relationship.
Thank you again for your constructive feedback. Please let us know if you have any further questions, and we will respond promptly. If our responses address your concerns, we kindly ask you to consider updating your score and confidence accordingly.
[1] Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models
[2] SliceGPT: Compress Large Language Models by Deleting Rows and Columns
[3] Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs
[4] Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings
[5] Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers
[6] A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection