acceptodds
Under review as a conference paper at ICLR 2027

Nuisance removal is a rotation problem

Abstract

Frozen encoders are reused in settings their training never anticipated, and their representations carry nuisance factors such as the layout a document was rendered in or the camera it passed through. Cleaning them up is almost always done one dimension at a time, by thresholding, masking, reweighting or selecting the largest entries of a sparse code. Practitioners assume that this per-dimension filtering separates content from nuisance. We show that this family has a measurable ceiling. Every per-dimension operation weights a dimension by that dimension’s own statistics, so it can only exploit differences between dimensions, and its headroom is governed by one quantity: the spread of the per-dimension content-to-nuisance variance ratio. The ceiling is measured rather than proved: our derivation is idealised and over-estimates what a real encoder yields, so we use it to predict ordering and check every prediction against the measurement, over eight encoders and six domains. This analysis motivates a complementary strategy: mix dimensions rather than reweight them. A fitted diagonal escapes the ceiling, at a price we quantify: up to seven times more ratio out of sample, but less retrieval in eight of the twelve settings. The ratio is therefore a diagnostic, not a training target. Our remedy mixes dimensions instead of weighting them: a closed-form rotation estimates the nuisance subspace by PCA on the differences between two renderings of the same content and projects it out, using pairs only at fit time and no nuisance labels, and improves held-out retrieval by about ten points on the document grid. Finally, an input-dependent per-dimension readout can be worse than neutral: in a released sparse autoencoder the sparse code its global top-k produces retrieves far less than the dense head it replaces. To separate content from nuisance, rotate; do not rescale.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
invariance and nuisance factors, subspace projection, sparse autoencoders

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_xb1zOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
3
Read review
Paper summary

The paper studies nuisance removal in frozen representations. A nuisance is a change that should not affect the underlying content, such as document layout, camera conditions, or background.

It first argues that simple per-dimension reweighting has little room to help when each feature dimension has about the same content-to-nuisance ratio.

The paper then proposes a simple method:

  1. take pairs of the same content under different nuisance conditions,
  2. compute the difference between their embeddings,
  3. apply PCA to these differences to estimate a nuisance subspace, and project this subspace out.

On the main rendered-document experiment, retrieval improves from 0.845 to 0.945. A trained full linear map on the same data reaches 0.950. The paper also contains a case study showing that a global top-k sparse-autoencoder readout can discard a large amount of useful information.

Critical strengths

S1. The paper contains many experiments and controls across domains and encoders.

S2. The proposed correction is simple and gives a large improvement on the main document experiment, nearly matching a learned full linear map.

Critical weaknesses

W1. The claimed limitation of per-dimension methods seems weak. The paper reports small ratio gains when each dimension's weight is computed from its own statistics. Jointly optimizing the diagonal weights produces much larger gains, as the paper itself shows. More importantly, the bound is on the paper’s content-to-nuisance ratio, not on retrieval. The paper itself shows retrieval improving from 0.845 to 0.938 while this ratio improves by only 1.9%. The paper acknowledges this gap. I therefore do not see how the ratio bound supports its broad recommendation to rotate rather than rescale.

W2. The proposed method relies on strong assumptions, and I do not see a clear advantage over simply fitting a linear map once these assumptions are granted. The method requires pairs of the same content under different nuisance conditions, assumes that the nuisance can be represented by a reusable linear subspace, and assumes that projecting out this subspace does not remove important content. The learned subspace does not transfer across domains. Given these pairs, a trained full linear map slightly outperforms the proposed PCA projection on the main experiment, 0.950 versus 0.945. The paper emphasizes that its method is closed form and requires no training objective, but it does not show that this gives a meaningful advantage over a regularized or low-rank linear fit.

Questions for authors

Q1. Why should the proposed PCA projection be preferred to fitting a regularized or low-rank linear map on the same paired data? Could the authors compare these methods across domains and as a function of the number of calibration pairs?

Q2. The paired-difference PCA estimator appears closely related to classical within-class / within-content covariance estimation. What is the methodological difference from these classical constructions?

Reviewer_XUSVOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper argues that per-dimension operations (IDF, masking, reweighting) can’t separate content from nuisance in frozen encoder embeddings, and proposes projecting out a nuisance subspace estimated by PCA on embedding differences between paired renderings of the same content. Experiments cover six custom grids and eight encoders, plus a case study on a sparse autoencoder readout.

Critical strengths

Simple, closed-form method that needs only paired renderings, with a solid in-domain gain on rendered documents.

Critical weaknesses
  1. The method isn’t new. Estimating a nuisance subspace from pairs that share content but differ in nuisance, then projecting it out, is a known recipe. Speaker recognition does it with recordings of the same speaker over different channels (Solomonoff et al., 2005); word-embedding debiasing does it with word pairs differing only in gender (Bolukbasi et al., 2016); Chuang et al. (2023) do it on frozen CLIP embeddings with prompt pairs differing only in the spurious attribute. None of these is cited or compared.

  2. Generalisation to unseen nuisances isn’t tested. Most tables fit and test on the same fixed layouts, with half the test contents seen at fit time, and the subspace transfers nothing across domains.

  3. The theory is thin and oversold. The “bound” is a weighted-average inequality plus a first-order expansion that the authors themselves call close to definitional, and the idealised ceiling misses measured headroom by a factor of about 20.

  4. x − BBᵀx is not a rotation but is a projection that zeroes r axes in a learned basis, i.e., a per-dimension mask, which undercuts “rotate, do not rescale”.

Questions for authors
  1. Does the gain hold when the subspace is fitted on some layouts, scanners or devices and tested on unseen ones?
  2. How does it compare with suppressing nuisance latents in an SAE, and with Chuang et al. (2023) and NAP on the same grids?
Reviewer_VvGzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies nuisance removal from frozen representations and argues that coordinate-wise operations such as masking, thresholding, and reweghting have an inherent limitation: they can exploit differnces in content-to-nuisance statistics across dimensions but cannot effectively remove nuisance that is distributed across directions in representation space. Motivated by this observation, the authors propose estimating nuisance directions using PCA on representation differences between paired observations of the same content and projecting out the resulting nuisance subspace. The paper evaluates the proposed analysis and method across multiple encoders and domains and reports retrieval improvements in some settings.

Critical strengths
  • The central observation is intuitive and potentially useful: nuisance information need not align with individual representation coordinates, making coordinate-wise filtering inherently restrictive.

  • The proposed PCA-based approach is simple, computationally attractive, and easy to implement. I appreciate that the authors explicitly distinguish their theoretical/idealized analysis from empirical measurements and acknowledge that improving the proposed variance ratio does not necessarily improve downstream retrieval.

  • The paper examines the phenomenon across multiple encoders and domains rather than relying on a single representation.

Critical weaknesses
  1. My main concern is the gap between the breadth of the paper's central claim and what is theoretically established. The ceiling for per-dimension methods is derived under idealized assumptions and is ultimately validated empirically rather than established as a general limitation for realistic representations. Consequently, the strong framing that nuisance removal is fundamentally a "rotation problem" appears broader than the analysis supports.

  2. The comparison between coordinate-wise methods and the proposed rotation-based method is not fully convincing. The proposed method has access at fit time to paired observations corresponding to the same content under different nuisance conditions. Such pairs provide substantial information about nuisance variation. It is therefore important to separate the benefit of this paired supervision from the benefit of using a non-diagonal transformation. A stronger evaluation would compare diagonal and full/subspace methods under exactly the same paired information.

  3. The requirement for paired observations also limits the general applicability of the method. Such pairs are natural in controlled rendering experiments, but may be expensive or unavailable for many realistic nuisance factors and naturally occurring distribution shifts. This makes it unclear how broadly the method and conclusions extend beyond the experimental settings considered here.

  4. The empirical results do not yet seem sufficiently broad to support the general conclusion. While the paper studies several encoders/domains, the most compelling downstream gains appear concentrated in particular settings (e.g., the document retrieval experiments). I would like to see stronger evidence that rotation-based removal consistently improves downstream performance across qualitatively different nuisance factors and representation families.

  5. There is also an unresolved connection between the proposed content-to-nuisance variance ratio and downstream utility. The paper itself shows that a fitted diagonal can improve this ratio substantially while making retrieval worse in many settings. This is an interesting negative result, but it also raises questions about how informative the proposed quantity is as a practical diagnostic and how strongly conclusions about nuisance removal can be drawn from it.

  6. Finally, the PCA-based solution itself is relatively straightforward once matched observations are available: compute representation differences, identify high-variance directions, and project them out. This is not necessarily a weakness by itself, but it places more burden on the theoretical characterization and empirical generality of the paper. In my view, these aspects are not yet strong enough to establish the broader contribution claimed by the paper.

Questions for authors
  1. Can you clarify whether the diagonal baselines have access to exactly the same paired same-content observations as the proposed PCA method? If not, can you disentangle the benefit of paired supervision from the benefit of allowing non-diagonal transformations?

  2. How robust is the method when the paired observations are imperfect, i.e., when the representation difference contains some content variation in addition to nuisance variation? Could the resulting projection remove content-relevant directions?

  3. The paper shows that the content-to-nuisance variance ratio can improve substantially while retrieval becomes worse. Given this result, what evidence supports using this ratio as a meaningful diagnostic of the limitations of per-dimension methods?

  4. To what extent do the retrieval improvements hold consistently across the different domains/encoders evaluated, rather than being driven primarily by the document experiments? A clearer breakdown of where rotation helps, has little effect, or hurts would materially affect my assessment.

  5. Can the authors clarify the precise scope of the main claim? Is the claim that coordinate-wise methods have a fundamental limitation under the assumptions of the analysis, or that nuisance removal in realistic frozen representations generally requires rotation/subspace removal?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.