acceptodds
Under review as a conference paper at ICLR 2027

Same Behavior, Different Compression: Auditing Coordinate Dependence

Abstract

Can a compression audit change its verdict when reference behavior stays fixed? On 2,669 Pets examples previously unevaluated in this project, we test a transformation that doubles the feature components in the reference nullspace while preserving the fixed reference behavior. At the fixed setting of random projection to 64 dimensions, agreement falls from 91.7% to 81.2%. The change from passing to failing occurs in all 20 fresh projection draws. We also compare maps without reducing dimension and maps that retain the same subspace on ViT/CIFAR-10, DINOv2/CIFAR-10, DINOv2/Pets and BERT/TREC. These maps retain the same available linear score functions but can yield different fitted predictions. When the regularization objectives match, separate fits give the same class predictions as the orthonormal-coordinate baseline in all tested pairs. Some logit and probability differences still exceed the declared tolerances. Classical covariance equivalence and whitening principles give explicit invariance conditions for the complete randomized audit. Using verified LayerNorm support greatly improves conditioning. None of the four settings gains a feasible compressed point at the tested under the ordinary isotropic Gaussian-RP probe. The tests of matching objectives and correcting support use benchmark examples examined earlier. They are not independent confirmation on new data. Compression audits should distinguish reference behavior, retained information and access through a fitted procedure.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, representation analysis, behavioral compression, dimensionality reduction, linear probing, coordinate dependence, random projection, regularization

Reviews

Among:
Shape:
1234this paper, Bottom 1%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.00 over 3 reviews, above 0% of the others.
Reviewer_bgjWOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
3
Read review
Paper summary

This work investigates how compression audits can be sensitive to transformations of the reference features that do not change the behavior of the original model. They design such transformations that leave the behavior of the reference model the same and show that, after compression using either PCA or Random Projections, the results of the compression audit change. Evaluating on different pretrained models, the authors showcase how the success of individual compression methods is dependent on coordinate transformations that do not affect the initial model behavior or reduce the information in the reference uncompressed features. In addition to the observations of the compression dependence on non-behavior-changing transformations, they investigate ways to achieve invariance in such transformations, like feature whitening.

Critical strengths
  • The authors make a substantial effort to study the dependency of compression audits to transformations that leave the original feature information and model behavior unchanged. Such results that study the behavior of compression audits to such non behavioral changing transformations can be of interest to the community.
Critical weaknesses
  • One of the main weaknesses of this paper is the writing quality and lack of clarity. The paper's presentation is very disorganized and does not allow the reader to understand and evaluate the author's contributions. Additionally, prior work or datasets used are referred to without proper citations, with the citations appearing much later in the paper. Such an example is the reference to Pets, which is used as one of the main results in the abstract and the introduction and is first cited in Section 5.
  • There is limited novelty, with the main contribution of the work being an extensive experimental study of techniques proposed in previous works.
Questions for authors
  • What is the main experimental observation that establishes the novelty of this work and hasn't been explored in previous works
Reviewer_jAafOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies compression audits: whether a low-dimensional representation, combined with a newly fitted probe, preserves the behavior of a fixed reference classifier. It asks three questions. First, can an invertible coordinate change that leaves the reference behavior unchanged flip the compression verdict? Second, can maps that retain the same linear information still yield different fitted behavior? Third, what can a coordinate correction such as whitening guarantee? The authors support these with experiments on frozen ViT, DINOv2 and BERT features and several propositions. The main held-out result is on Pets: scaling the reference nullspace by 2 changes RP (q=64) agreement from 91.7% to 81.2%, a pass-to-fail flip in all 20 draws.

Critical strengths

None

Critical weaknesses

1. The connection between the research questions and the evidence is hard to follow.

The research questions are understandable, and the paper reports a large number of experiments. However, it is often unclear which result answers which question, and what conclusion the reader should draw from it. The reader has to reconstruct these links from scattered statements across the main text and appendices. Two aspects of the writing make this harder:

  • Much of the text reads as a report of the project’s internal workflow rather than a self-contained exposition. Examples include previously unevaluated in this project, fresh projection draws, inherited test maps and the H1/H2 labels.
  • Limitations and caveats frequently appear before the meaning of the corresponding result has been established.

As a result, it is difficult to assess the validity and significance of the claims.


2. The contribution relative to known results is unclear.

The paper itself describes Propositions 1–3 as applications of classical results (equal-Gram equivalence, rotation invariance of Gaussian matrices, the beta law of random subspaces). The objective-matching result (Fig. 1 right, Table 2) also appears to follow directly from a change of variables. Since Θ=WGRT\Theta = W_G R^{T} with R invertible, JG,match(WG)=JU(Θ)J_G,match(W_G) = J_U(\Theta). The matched arm is moreover optimized in common Θ\Theta coordinates, so agreement with the unit fit seems to verify solver precision rather than an empirical property of the representation. It would help if the paper stated explicitly which findings are new and what role each proposition plays in supporting the experimental claims.

Questions for authors
  1. Could the authors map each of the three research questions in Section 1 to the specific experiment that answers it, and state the resulting claim in one sentence each? In particular, for the Pets result (Table 1), why is this the decisive evidence for the first question? And what should a practitioner running a compression audit do differently because of it? A clear answer here would substantially improve my assessment of the paper’s validity.

  2. Given that the matched objective equals the unit objective under reparametrization, what does Table 2 establish beyond confirming the optimizer? Relatedly, are the BERT/TREC tolerance failures (24/240 cells) purely numerical, with no effect on any behavioral conclusion? If the authors can clarify what is non-trivial in this experiment, or reframe it accordingly, I would be open to raising my score.

Reviewer_pJC2OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The paper shows that scaling directions the reference classifier ignores can change compression audit results (from pass to fail), while not changing the reference classifier's predictions. Whitening the data before random projection removes the effect of this scaling, but after whitening, Gaussian random projection fails every tested compression audit. Mixing and stretching using invertible matrices (no compression) can also degrade agreement with the reference classifier, because of weight regularization, and it is solved by a simple penalty adjustment.

Critical strengths
  1. A useful guideline for practitioners. A classifier performing worse on transformed data does not necessarily mean that the information was lost. Under an invertible linear transformation that stretches directions unevenly, the linear classifier that was trained on the transformed data agreed less with the reference classifier, and the original performance was restored by just changing the penalty.
Critical weaknesses
  1. The results are predictable. It is well known that strengthening specific directions changes the PCA compression toward these directions, and if the reference classifier ignores these directions, a classifier learning from the compressed data can agree less with the reference classifier. Additionally, scaling and mixing the data using an invertible linear transformation that stretches directions unevenly changes the scale of the features, thus a different scale of weights will be needed for classification, which requires a change to the penalty (using the transformation matrix), otherwise the classifier predictions can get worse. These are established ideas, which limits the paper's contribution.

  2. Poor presentation. The paper is very hard to follow. Setups are scattered across many sections, terms are used without definition in the main text (like "inherited" test maps, "thin" coordinates). The Appendix includes a lot of earlier experiments, and it is hard to understand which results support which claims.

Questions for authors
  1. In all main experiments, you use C=1C = 1. In practice one would tune CC using a validation set. How much of the agreement gap in Table 2 remains when doing so?

  2. Which of your results would you consider novel? In other words, which results do you think one would not expect in advance, and why? See Weakness 1 for examples.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.