Nuisance removal is a rotation problem

Frozen encoders are reused in settings their training never anticipated, and their representations carry nuisance factors such as the layout a document was rendered in or the camera it passed through. Cleaning them up is almost always done one dimension at a time, by thresholding, masking, reweighting or selecting the largest entries of a sparse code. Practitioners assume that this per-dimension filtering separates content from nuisance. We show that this family has a measurable ceiling. Every per-dimension operation weights a dimension by that dimension’s own statistics, so it can only exploit differences between dimensions, and its headroom is governed by one quantity: the spread of the per-dimension content-to-nuisance variance ratio. The ceiling is measured rather than proved: our derivation is idealised and over-estimates what a real encoder yields, so we use it to predict ordering and check every prediction against the measurement, over eight encoders and six domains. This analysis motivates a complementary strategy: mix dimensions rather than reweight them. A fitted diagonal escapes the ceiling, at a price we quantify: up to seven times more ratio out of sample, but less retrieval in eight of the twelve settings. The ratio is therefore a diagnostic, not a training target. Our remedy mixes dimensions instead of weighting them: a closed-form rotation estimates the nuisance subspace by PCA on the differences between two renderings of the same content and projects it out, using pairs only at fit time and no nuisance labels, and improves held-out retrieval by about ten points on the document grid. Finally, an input-dependent per-dimension readout can be worse than neutral: in a released sparse autoencoder the sparse code its global top-k produces retrieves far less than the dense head it replaces. To separate content from nuisance, rotate; do not rescale.