acceptodds
Under review as a conference paper at ICLR 2027

Compression Hacking: Towards a Unified Informatics and Geometric Perspective on Intrinsic Evaluation of Large Language Models

Abstract

Recently, the concept of “compression as intelligence” has provided a novel informatics metric perspective for large language models (LLMs), emphasizing that highly structured representations signify the intelligence performance of LLMs. However, from a geometric standpoint, the representation space of highly compressed LLMs tends to degenerate into a highly anisotropic state, which hinders the LLM's ability to comprehend instructions and directly impacts its performance. We found this compression-anisotropy synchronicity is essentially the “Compression Hacking” in LLMs representations, where noise-dominated directions tend to create the illusion of high compression rates by sacrificing spatial uniformity. Based on this, we propose three refined compression metrics by incorporating geometric distortion analysis and integrate them into an intrinsic-evaluation framework. The refined metrics exhibit strong alignment with the LLM's comprehensive capabilities, achieving Spearman correlation coefficients above 0.9, significantly outperforming both the original compression and other internal structure-based metrics. This confirms that compression hacking substantially enhances the information-theoretic perspective of LLMs by incorporating geometric distortion of representations.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Model, Intrinsic Evaluation, Representation Space, Hidden States, Information Compression, Anisotropy, Matrix Theory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_1cFkOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies the limitations of LogDet compression metrics for evaluating model capabilities across families. It terms the phenomenon in which small eigenvalues inflate compression scores while accompanying stronger anisotropy “Compression Hacking,” and proposes three refined metrics: SE, Semantic CV, and PCS. Across 18 models from three families, these metrics achieve Spearman correlations of 0.917–0.965 with a comprehensive score aggregated from eight external benchmarks, compared with 0.445 for the original metric.

Critical strengths

The paper addresses a practically relevant problem: compression metrics that work within a model family may not reliably compare models across families. Table 2 shows substantial improvements in correlation with comprehensive capability scores for all three refined metrics, with improvements extending across multiple individual tasks. If reproducible, these results would be valuable for model evaluation based on hidden representations.

The experiments include relevant baselines such as Diff-eRank, Whitening, and LW shrinkage, together with parameter sensitivity, within-family comparisons, and leave-one-model-out analyses. PCS’s LOO correlations remain between 0.958 and 0.983, suggesting that the main result is not entirely driven by a single model.

Critical weaknesses

Important computational details remain unclear. Section 2 describes the centered covariance matrix as positive definite, but under the per-input computation in Section 3.2, it will be singular if the token count does not exceed the hidden dimension. The paper does not sufficiently explain how the original DE and condition number handle zero eigenvalues. Additionally, OPT-13B’s CE value of 42.09 in Figure 1 is inconsistent with the rank aggregation rule in Appendix B. Clarification of the numerical treatment and data sufficient to recompute the results are needed to establish whether these issues affect Table 2.

The mechanistic explanation is stronger than the available evidence. Section 2.3 interprets low-variance directions as noise, but Table 1 and Figure 2 mainly show spectral concentration, while Figure 3 shows a correlation between compression and anisotropy. Since both metrics depend on small eigenvalues, a high correlation alone cannot establish that these directions are noise or impair model capability. Appendix A already acknowledges the absence of a complete causal explanation. The main text should narrow its claims accordingly or provide evidence that distinguishes these explanations.

Questions for authors

Which samples are used for each covariance estimate in Section 3.2? How do the original DE, condition number, and relevant baselines handle zero or extremely small eigenvalues, and do they use consistent numerical treatment?

How is OPT-13B’s CE value of 42.09 in Figure 1 obtained from Equation (10) in Appendix B? Could you provide the raw scores and calculation steps needed to recompute CE and the correlations in Table 2?

Beyond the correlation between the two metrics, what evidence shows that tail directions are noise and cause misleading capability evaluations? If such evidence is not yet available, could this mechanism be explicitly presented as a hypothesis?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1cFk

Dear reviewer 1cFk

Thank you for carefully reviewing our paper and providing valuable feedback, which has helped us improve the paper. We respond to your comments below.


Handling singular covariance matrices and zero or very small eigenvalues

Thank you for pointing out this missing explanation. The analysis in Section 2 assumes a positive-definite covariance matrix. We have added the diagonal regularization term that ensures positive definiteness, along with its default value, to the revised paper. We compute the metrics separately from the final-layer token representations of each input, then average them over the projection set. Inputs are truncated or padded to 64 positions, and padding positions are included in the calculation. In Section 2, mean-centered means subtracting the mean of the token representations within each input, yielding a zero-mean matrix. The preprocessing in Diff-eRank [1] also includes subtracting the sample mean. To prevent zero eigenvalues from making LogDet and the condition number undefined, we also add diagonal regularization:

ΣZ=1∣V∣Z⊤Z+αID,α=10−8.\Sigma_{\mathbf Z}=\frac{1}{|\mathcal V|}\mathbf Z^\top\mathbf Z+\alpha I_D,\qquad \alpha=10^{-8}. We will clarify this in the revised paper.


The distinction between 42.09 and CE

Two quantities need to be distinguished here: the value 42.09 for OPT-13B in Figure 1 is the average of its measured scores across the benchmarks, whereas CE aggregates the model's ranks across benchmarks using Eq. (10) in Appendix B. The former averages scores, while the latter aggregates ranks, so 42.09 does not need to be derived from Eq. (10). We will clarify this in the caption using “Mean benchmark score” to avoid confusion.


Insufficient mechanistic evidence that low-variance tail directions are noise and harm capability

Thank you for your suggestion. Compression hacking concerns the following issue: some low-variance directions contribute little to the task, but their very small eigenvalues substantially inflate the compression score after the negative-log transformation, leading to misleading capability evaluations. Existing intervention studies support this interpretation.

Removal experiments show that some low-variance directions make limited contributions to task performance. When 25% of the dimensions are removed without recovery fine-tuning, SliceGPT [2] retains approximately 99% and 96% of the original zero-shot performance of OPT-66B and Llama-2-70B, respectively. Research on sentence embeddings also finds that removing some low-variance directions with PCA can improve performance on some semantic tasks [4]. A vision-language study further finds that removing 164 low-variance directions from CLIP changes ImageNet Top-5 accuracy only from 91.2% to 91.1%. In contrast, removing the same number of random directions reduces it to 89.3% [5].

Low-variance directions can also reduce a score's ability to distinguish samples. In near-OOD detection experiments on CIFAR, Ren et al. [6] find that some low-variance directions contribute heavily to Mahalanobis scores despite providing little separation between in-distribution and out-of-distribution samples. Keeping only the top 76 principal directions improves detection AUROC from 74.98% with all directions to 79.72%. Semantic Volume [3], which also uses LogDet, likewise finds that removing some low-variance directions with PCA before computing the uncertainty metric improves detection compared with using all directions. In the revised version, we will incorporate these supporting evidence to make our claims more solid.

In Table 2, the correlations of SE and PCS with comprehensive capability CE are 0.917 and 0.965, respectively, compared with 0.445 for DE. These results show that the corrected scores better reflect measured capability. Our focus is that changes in tail eigenvalues may substantially change the compression score without corresponding changes in capability. The relevant tests are therefore whether compression scores accurately reflect model capability and whether reducing the tail's influence improves this relationship.


Thank you again for your constructive feedback. Please let us know if you have any further questions, and we will respond promptly. If our responses address your concerns, we kindly ask you to consider updating your score and confidence accordingly.

[1] Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models

[2] SliceGPT: Compress Large Language Models by Deleting Rows and Columns

[3] Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs

[4] Evaluating Unsupervised Dimensionality Reduction Methods for Pretrained Sentence Embeddings

[5] Your CLIP has 164 dimensions of noise: Exploring the embeddings covariance eigenspectrum of contrastively pretrained vision-language transformers

[6] A Simple Fix to Mahalanobis Distance for Improving Near-OOD Detection

Dear Reviewer 1cFk,

As the discussion period will end in about a day, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. If our responses have indeed resolved these issues, we kindly ask you to consider updating your rating and confidence. Should you have any remaining questions before the deadline, we would be more than happy to answer them at any time.

Reviewer_E8qhOpenReview ↗
Rating
4(+1)
Confidence
1(−1)
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper works on intrinsic evaluation of large language models, where the authors proposes the ideaa of compression hacking, where high compression is strongly correlated to high anisotropic, which hurts LLM performance. Based on this, the ahuotrs propose three refined compression metrics, by incoporating gemometric distortion analysis. The refined metrics aligns with LLM capabilities better.

Critical strengths
  • This paper identifies a surprising gap in existing literature.
  • This paper is well motivated, and proposes solid solutions.
Critical weaknesses

Methodolodgy soundness

  • Zero-mean assumption. Line 095 states that Z is a zero-mean matrix right after stating that hidden states are normalized, but no centering step is described, and unit-norm normalization alone does not give zero mean. (Besides, Table 1’s λ_max = 1.25 seems impossible if the vectors were unit-normalized, so the paper’s “normalized” is undefined.) Real LLM representations have a large non-zero common mean, which the works cited for anisotropy (Ethayarajh, 2019; Mu & Viswanath, 2018) identify as its dominant source. Assuming zero mean removes the linear term 1ᵀZc in Eq. (3), which is exactly the term that makes Z(c) direction-dependent for uncentered data. The anisotropy measured here (eigenvalue spread after centering) is therefore not the anisotropy of the cited literature. Please (a) state the exact preprocessing order (normalize then center, or the reverse), and (b) discuss how the findings change for uncentered representations.
  • Eq (6) is derived from eq (5) by canceling |V| term in both numerator and denominator. The step could be biased (and in fact, the scale of |V| in experiments is not well explained). Please provide addtional justication on the potential bias of this step.
  • The signifance analysis in section 2 claims “the dependent variable is well and significantly explained by anisotropy”, however, correlation analysis does not indicate causality. Please justify / correct this claim.

Evaluation

  • The evaluation is limited on 2024 or before models, would the authors consider involve evaluation on newer models, like qwen3, gemma3? Please justify this in response.

Presentation

  • Section 3 introduces three refined metrics, but it is not explained how the community would use them, or how they connect to the "compression is intelligence" claim. The paper describes the metrics as "jointly considering the compression and anisotropy of representations" (line 262), which suggests two equal properties. I believe, the paper still follows "compression is intelligence", but what the paper claims is that compression metrcs should be ``calibrated'' by anisotropy information. These are different narratives; please clarify which one the paper makes. Relatedly, Compression (SE) is closely tied to effective rank (lines 272–273), which Diff-eRank already uses; please clarify the difference and why SE reaches 0.917 while Diff-eRank reaches 0.492.
  • The naming of “Semantic Coefficient of Variation” is not clear — it is neither semantic nor a coefficient of variation (std/mean).
  • The paper may benefit from a (compacted) related work in main paper body.
  • a substantial amount of words are abused / unclear. for instance, “concentrating” in line 201; “secondary components”, “homologous” in lines 267-268;
  • Font in Figure 2 is too small to be clearly understood.

Overall I enjoyed reading the paper, I will raise my ratings if my concerns are addressed.

Questions for authors
  1. What exactly does "normalized" mean in line 095, and at which step are the hidden states mean-centered (normalize then center, or the reverse)? Given that Table 1 reports λ_max = 1.25 for Qwen2.5-3B-Instruct, which would be impossible if each vector had unit L2 norm.

  2. As a follow up of 1, since the works cited for anisotropy (Ethayarajh, 2019; Mu & Viswanath, 2018) identify the common mean as its dominant source, and the zero-mean assumption removes the linear term 1ᵀZc in Eq. (3), how do the compression–anisotropy synchronicity and the refined metrics' correlations change when computed on uncentered representations?

  3. Is Eq. (6) intended as an approximation of Eq. (5), or as a separate design choice? If it is an approximation, how do the authors justify dropping the |V| terms? What is |V| in the experiments: the number of distinct words in a single sample, or across the whole projection set? How are repeated words, which have different contextual vectors, handled?

  4. Given that the analysis in Section 2.3 is correlational, and that compression and anisotropy are both functions of the same eigenvalues (so a strong correlation may be expected by construction), how do the authors justify stating that compression "is well and significantly explained by anisotropy" and that the synchronicity is "caused by compression hacking" (line 260)? Would the authors consider revising this wording?

  5. Would the authors consider evaluating newer models such as Qwen3 and Gemma 3, with β and the layer choice kept fixed, as a held-out test? If not, why should the results be expected to transfer beyond the 2024-or-earlier models evaluated?

  6. Does the paper claim that (a) compression and anisotropy are two jointly necessary properties of capable models, as "jointly considering the compression and anisotropy of representations" (line 262) suggests, or (b) that compression remains the measure of intelligence and anisotropy serves to calibrate it? How should practitioners use the three refined metrics, e.g., which one to report and for what decision?

  7. Since Compression (SE) is closely tied to effective rank (lines 272–273), which Diff-eRank also builds on, what explains the gap between them (0.917 vs. 0.492)?

  8. Why is the metric in Eq. (8) called "Semantic Coefficient of Variation", when it is the ratio of the condition number to C_DE rather than a standard deviation over a mean, and contains no semantic component?

9 comments
AuthorsOpenReview ↗

Response to Reviewer E8qh (1/3)

Dear Reviewer E8qh,

Thank you for carefully reviewing our paper and providing constructive comments on the theoretical derivations, experimental settings, and interpretation of the metrics. Your comments highlighted definitions and statements that need clarification, which we address below.

Weakness 1 & Question 1 & Question 2: Normalization, centering, and the reported eigenvalues

Thank you for pointing this out. Our original wording was ambiguous: when describing mean centering, we mistakenly used “normalized” instead of “mean-centered”. In Section 2, “represents the token hidden state, which has been normalized” should read “represents the token hidden state, which has been mean-centered”. Mean centering means subtracting the mean of the token representations within each input: zi=hi−μ,μ=1n∑j=1nhjz_i=h_i-\mu,\qquad \mu=\frac{1}{n}\sum_{j=1}^{n}h_j. This is the basis for the following statement, “Z\mathbf Z is a zero-mean matrix,” and does not require normalizing each token representation to unit norm. Mean centering does not impose an upper bound on covariance eigenvalues, so a maximum eigenvalue of 1.25 does not itself contradict the zero-mean condition.

For the common mean, we focus on how differences among covariance eigenvalues affect the compression score. Adding the same vector to all representations may change the partition function across directions, but it does not change the shape of the point cloud, its covariance, or the covariance-based LogDet compression score. Thus, changes in anisotropy caused by the common mean do not necessarily correspond to changes in compression. We use centering to remove this common offset and examine the eigenvalue distribution directly related to compression; we do not assume that raw LLM representations naturally have zero mean. Centering alone only shifts the point cloud; its shape and covariance remain unchanged. Diff-eRank [1] also includes subtracting the sample mean in its preprocessing, before computing effective rank from the normalized eigenvalues.

Weakness 2 & Question 3: The basis for moving from Eq. (5) to Eq. (6), and the definition of |V|

Thank you for noting that the derivation needs clarification. Eq. (6) is a spectral anisotropy metric motivated by second-order analysis, rather than a numerical approximation to Eq. (5) obtained by ignoring ∣V∣|\mathcal V|. In the second-order expansion after centering, ∣V∣|\mathcal V| is a direction-independent constant. After subtracting this term, the maximum and minimum of the second-order term over all unit directions correspond to the largest and smallest eigenvalues. When the covariance is positive definite, their ratio is the condition number. This metric is unchanged when all eigenvalues are scaled by the same factor. It captures the gap between the directions of maximum and minimum variance and helps identify very small eigenvalues that may inflate the compression score.

Regarding ∣V∣|\mathcal V|, Section 3.2 explains that we extract representations and compute metrics separately for each input, then average them over the projection set. Appendix E specifies the default of 800 inputs, and Appendix E.2 further examines the effect of sample size. Thus, ∣V∣|\mathcal V| is the number of token positions included in the calculation for a single input. Each input is truncated or padded to 64 positions.

Weakness 3 & Question 4: Correlational evidence and causal wording

Thank you for your suggestion. We agree that statistical association should be clearly distinguished from causation. We have replaced “explained by” in Section 2.3 with a strong, statistically significant linear association between compression and the logarithm of anisotropy, and changed “caused by compression hacking” to “characterizing compression hacking”, to match the evidence from the regression analysis.

AuthorsOpenReview ↗

Response to Reviewer E8qh (2/3)

Weakness 4 & Question 5: Evaluating new models not used in metric development with fixed settings

Thank you for your suggestion. We chose 18 models from LLaMA3, Qwen2.5, and OPT for the initial evaluation mainly because each family includes models of different sizes, allowing comparisons both within and across families. Within-family comparisons test whether the metrics reflect capability differences between models of different sizes in the same series. Cross-family comparisons test whether the relationship between the metrics and actual capability also holds across model series.

To evaluate additional model families, we add four models from Qwen3 and Gemma: Qwen3-8B and Qwen3-14B from Qwen3, and Gemma4-12B and Gemma4-26B from Gemma4.

Evaluation setNumber of modelsDESESemantic CV (direction-aligned)PCS
Original three model families180.4450.9170.9260.965
+ Qwen3 family200.5590.9350.9350.974
+ Gemma4 family200.4690.9080.9350.962
+ Qwen3 and Gemma4 families220.5720.9280.9440.972

As shown in the table, after adding the new models, the correlation between DE and CE is 0.4690.469–0.5720.572, while the three refined metrics achieve 0.9080.908–0.9740.974. This shows that the three refined metrics continue to reflect measured capability more accurately than DE on the additional models.

Weakness 5 & Question 6 & Question 7: The paper's claims, how to use the metrics, and the distinction from Diff-eRank

Thank you for asking us to clarify the purpose and use of the metrics. Our central claim is that when using compression scores to evaluate capability, we should also examine the eigenvalue distribution, so that scores inflated by near-zero eigenvalues are not mistaken for stronger capability. Thus, “jointly considering” means accounting for both compression and the eigenvalue distribution when constructing metrics. It does not claim that high compression and low anisotropy are two necessary conditions for intelligence. The three metrics in Section 3.1 examine representations from the following perspectives:

MetricPerspective
Compression (SE)Overall distribution: Spectral entropy measures how concentrated the eigenvalues are.
Semantic CVRelative dispersion: The ratio of the largest to the smallest eigenvalue is assessed relative to overall compression.
Compression (PCS)Compression after smoothing: Raises tail eigenvalues before computing the LogDet compression score.

Intuitively, the three metrics are like three ways of summarizing the same report card. They may therefore be highly correlated while producing different rankings for particular spectral shapes.

We recommend reporting SE, Semantic CV, and PCS together in practice, and using agreement and disagreement among their rankings to interpret differences between models. HELM [2] and RAGAS [3] likewise use multiple metrics to assess complementary aspects of language models and RAG systems, respectively.

Diff-eRank and Compression (SE) both use matrix entropy, but assess different quantities: Diff-eRank measures how much effective rank decreases relative to random initialization, while SE directly evaluates the representations after training. Following the Diff-eRank paper [1] and its official dataset implementation, let M0M_0 be a randomly initialized model with the same architecture as the trained model MM. With the same preprocessing and eigenvalue normalization, their relationship is:

sSE(M)=h(M),Diff-eRank(M)=eh(M0)−eh(M).s_{\mathrm{SE}}(M)=h(M),\quad\quad\quad\quad\quad \text{Diff-eRank}(M)=e^{h(M_0)}-e^{h(M)}.

Here, h(M)h(M) is the average spectral entropy of the model's representations over the projection set. We evaluate model capability after training, so we assess the final representations directly, without using entropy before training as an additional baseline. Diff-eRank instead measures the change from random initialization, and its baseline varies across models. The two metrics can therefore produce different rankings and correlations with capability.

AuthorsOpenReview ↗

Response to Reviewer E8qh (3/3)

Weakness 6 & Question 8: The name Semantic Coefficient of Variation

We use “Semantic” because the metric analyzes LLM representations that encode contextual semantic information. Semantic Volume [4] likewise describes semantic distributions through the geometry of embedding vectors. Our work focuses on the eigenvalue distribution and compression of these representations.

“Coefficient of Variation” draws on the idea of variation relative to an overall level. Specifically, the numerator uses the ratio of the largest to the smallest eigenvalue to capture relative eigenvalue dispersion, while the denominator summarizes negative log-eigenvalues through the compression score. Their ratio measures spectral imbalance relative to overall compression. The conceptual analogy is:

\qquad\longleftrightarrow\qquad \underbrace{\frac{\text{Condition number: relative eigenvalue dispersion}}{\text{Compression score: overall compression}}}_{\text{Semantic CV}}.$$ At a fixed dimension, the compression score is proportional to the mean of the negative log-eigenvalues. **This is a conceptual analogy based on relative variation, not an equivalent definition of the standard statistical CV.** ## Weakness 7 & Weakness 9: Related work in the main paper and the readability of Figure 2 Thank you for your suggestions. In the revised paper, we will **add a brief Related Work section before the Conclusion** to summarize related studies and clarify how our work differs. We will also **increase the font sizes of the axis labels, legends, and annotations in Figure 2** to improve readability. Thank you again for your constructive feedback. We will respond promptly to any further questions. If our responses address your concerns, we kindly ask you to consider updating your score and confidence accordingly. [1] Diff-eRank: A Novel Rank-Based Metric for Evaluating Large Language Models [2] Holistic Evaluation of Language Models [3] RAGAs: Automated Evaluation of Retrieval Augmented Generation [4] Semantic Volume: Quantifying and Detecting both External and Internal Uncertainty in LLMs
Reviewer_HHdJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Building on the "compression as intelligence" perspective, prior work has suggested that an LLM's capability can be estimated by analyzing how its internal representations are well-strucutred. However, when such compression metrics are applied directly, their correlation with ground-truth comprehensive capability is limited.To address this, the authors analyze the representations of real models and propose three refined compression metrics that correlate substantially better with capability than existing metrics.

Critical strengths
  • As shown in Table 2, the proposed metrics achieves higher Spearman correlations with ground-truth capability than existing metrics.
  • Table 1 and Figure 3 systematically characterize the relationship between compression and anisotropy, and the metric design is grounded in this analysis.
Critical weaknesses

The proposed metrics are evaluated only on the same three model families (Qwen, Llama, OPT) on which they were developed. Therefore, it is unclear whether the metrics generalize to other model families or are specific to these three families.

Questions for authors
  • Could you provide the experimental results on different model families which is not used for developing the proposed metrics?
  • The metrics are computed on a sampled projection set, so the ranking of the models may vary with which samples are drawn. Could you provide results on several independently sampled projections sets?
4 comments
AuthorsOpenReview ↗

Response to Reviewer HHdJ

Dear reviewer HHdJ,

Thank you for carefully reviewing our paper and providing valuable feedback. We respond to your comments below.

Weakness & Question 1: Generalization to model families not used in metric development

Thank you for your suggestion. We chose 18 models from LLaMA3, Qwen2.5, and OPT for the initial evaluation mainly because each family includes models of different sizes, allowing comparisons both within and across families. Within-family comparisons test whether the metrics reflect capability differences between models of different sizes in the same series. Cross-family comparisons test whether the relationship between the metrics and actual capability also holds across model series.

To evaluate additional model families, we add four models from Qwen3 and Gemma: Qwen3-8B and Qwen3-14B from Qwen3, and Gemma4-12B and Gemma4-26B from Gemma4.

Evaluation setNumber of modelsDESESemantic CV (direction-aligned)PCS
Original three model families180.4450.9170.9260.965
+ Qwen3 family200.5590.9350.9350.974
+ Gemma4 family200.4690.9080.9350.962
+ Qwen3 and Gemma4 families220.5720.9280.9440.972

As shown in the table, after adding the new models, the correlation between DE and CE is 0.4690.469–0.5720.572, while the three refined metrics achieve 0.9080.908–0.9740.974. This shows that the three refined metrics continue to reflect measured capability more accurately than DE on the additional models.

Question 2: Ranking stability after independently resampling the projection set

Thank you for your suggestion. Figure 11 already shows that the cumulative means of the metrics converge quickly as the number of projection samples increases. They are largely stable at the default sample size of 800, with only small changes as more samples are added. This indicates that the metric estimates are already insensitive to sample size at the default setting, supporting the use of a relatively small projection set for evaluation.

For the 18 models in the main paper, we computed the metrics using projection sets of 200, 400, 600, and 800 samples, allowing for sample overlap across draws. Representation extraction, preprocessing, metric hyperparameters, and CE scores remained unchanged.

As shown below, the correlations between the four metrics and CE, as well as the model rankings, remain stable across sampling settings. This indicates that, within the range of 200–800 samples, the evaluation conclusions are insensitive to sample size and sample selection. The three refined metrics retain their correlation advantage over DE, and the main comparative findings do not depend on the specific choice of 800 samples.

Number of projection samplesρ(DE, CE)ρ(SE, CE)ρ(Semantic CV, CE)ρ(PCS, CE)
2000.4450.9170.9260.965
4000.4450.9170.9260.965
6000.4450.9170.9260.965
8000.4450.9170.9260.965

Thank you again for your constructive feedback. Please let us know if you have any further questions, and we will respond promptly. If our responses address your concerns, we kindly ask you to consider updating your score and confidence accordingly.

Dear Reviewer HHdJ,

As the discussion period will end in about a day, we would greatly appreciate it if you could let us know whether our responses have addressed your concerns. If our responses have indeed resolved these issues, we kindly ask you to consider updating your rating and confidence. Should you have any remaining questions before the deadline, we would be more than happy to answer them at any time.

I appreciate to the authors for the additional experiments. The results on new model families and across different projection sample sizes address my main concerns and increases my confidence in the findings. On balance, I believe Weak Accept remains the most appreciate rating for the contribution.

Dear reviewer HHdJ,

We are glad to know that our responses have boosted your confidence in our paper. We would be extremely grateful if you could consider raising your confidence score in the reviewer system. Thank you once again for your recognition of our responses!

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.