acceptodds
Under review as a conference paper at ICLR 2027

Where Does a Representation's Value Live? The Realizability Decomposition of Dimensionality Reduction and Depth

Abstract

Methods grouped under dimensionality reduction are used to obtain lowerdimensional descriptions of data, but “fewer coordinates” does not specify the geometric operation that produced them. Against the reference of a basis change, which keeps all D coordinates, we separate two constructions that both return k < D numbers: a projection onto a subspace of the original ambient space, and an embedding into a distinct target space. Every rank-k linear map is of the first kind, an orthogonal projection followed by an invertible coordinate change with a closed-form metric that restores the projected geometry, so the distinction has content only for nonlinear and learned maps. To measure it we introduce the realizability decomposition: any finite centered representation Z of a declared source X splits orthogonally as Z = Z∥ + Z⊥, where Z∥ is exactly the part expressible as coordinates of one fixed source-subspace projection and Z⊥ is what no such projection can produce. The residual ρ = ∥Z⊥∥ / ∥Z∥ comes with an exact finite-sample caveat and a random-representation null. The decomposition also makes precise what a linear autoencoder identifies (a subspace and a metric, never a basis) and why a decoder trained with a pointwise loss returns to the original space rather than a symmetric copy of it. Empirically, a fixed projection of pixel space carries 82–95% of the energy of t-SNE, UMAP and PaCMAP layouts of MNIST and Fashion-MNIST, and 91–95% on CIFAR-10, yet that part has only PCA-level neighborhood fidelity; the embeddings’ advantage lives in the small remainder Z⊥. For autoencoders, end-to-end training produces codes with Z⊥ ̸= 0 whenever it beats PCA, but a nonlinear decoder trained on PCA scores (Z⊥ = 0) beats PCA as well, so the remainder describes the code training chose, not a requirement of reconstruction. The decomposition is a diagnostic, not a theory of representation: it says how much of a representation is a projection of a declared source, and what follows from that.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Dimensionality reduction, Representation geometry, Realizability decomposition, Source projection, projection-realizable representations, Nonlinear embeddings

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_mx1vOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper provides a useful perspective on the differences between linear and nonlinear dimensionality reduction methods. It is well written and clear. I particularly appreciated the characterization of the decomposition between the part of a representation that can be reproduced by a linear transformation and the part that cannot. I also found the discussion of the equivalence between linear dimensionality reduction methods, and of how coordinate transformations can affect their geometric properties, particularly insightful. Finally, the numerical experiments provide an interesting illustration of how a relatively small nonlinear component can have a substantial impact on the quality of a representation.

Critical strengths

The paper presents a clear and rigorous analysis of the differences between linear and nonlinear dimensionality reduction methods. They introduce a mathematical framework that separates a low-dimensional representation into a part that can be reproduced by a linear transformation of the original data and a part that cannot. They show that, although the latter often accounts for only a small fraction of the representation, it can play a crucial role in preserving the neighbourhood structure of the data. The framework is illustrated through experiments with popular dimensionality reduction methods and autoencoders. A good paper for a lecture.

Critical weaknesses

Some of the theoretical results, particularly Theorems 1 and 2, are elementary consequences of standard linear algebra. Similarly, the central realizability decomposition is the familiar decomposition of multivariate linear regression into fitted values and residuals. The main novelty of the paper therefore appears to lie in the geometric interpretation of these constructions and in their empirical application to nonlinear representations, rather than in the mathematical results themselves. I believe the authors should make this distinction more explicit, and maybe avoid calling "theorems" textbook results. A most important limitation concerns the practical implications of the proposed framework. I found unclear how the decomposition can be used in practice. In particular, a practitioner comparing different dimensionality reduction methods is typically interested in identifying which method best preserves the information relevant to a given task. How this can be achieved is not explicitly discussed (see question below)

Questions for authors

The authors convincingly demonstrate that the realizability residual is not, by itself, a measure of representation quality. However, this raises the question of what additional actionable information the decomposition provides beyond conventional measures of representation quality. I believe the paper would benefit from a more explicit discussion, supported by a concrete example, of how the proposed analysis can guide the interpretation and selection of dimensionality reduction methods.

Reviewer_qxy6OpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3
Soundness
2(+1)
Read review
Paper summary

The paper proposes the realizability decomposition: a representation ZZ of a source XX is split by least-squares regression into Z∥=ΠXZZ_\parallel = \Pi_X Z, which the authors show is exactly the coordinates of a fixed orthogonal projection of the source, and an orthogonal remainder Z⊥Z_\perp. The residual ratio ρ=∥Z⊥∥/∥Z∥\rho = \|Z_\perp\| / \|Z\| is used to measure how far a representation is from being a projection of its source. Empirically, the authors report that for t-SNE, UMAP and PaCMAP on MNIST, Fashion-MNIST and CIFAR-10, Z∥Z_\parallel carries most of the layout's energy (82–95%) but only PCA-level neighborhood fidelity, so that the embeddings' advantage over PCA resides in Z⊥Z_\perp. The paper also studies nonlinear autoencoders, local and layer-wise variants of the decomposition, and a layer-linearization certificate.

Critical strengths
  1. The experiments are broad and credible. Every residual is reported alongside an analytically derived random-representation null whose expected value matches the observed values. Seed variability is reported, rich feature sources are evaluated out of sample, and negative results are reported rather than omitted. The numbers are consistent across the abstract, the main text and the appendices.
Critical weaknesses
  1. The theoretical contribution is minimal. Essentially all formal results are classical facts from linear algebra or differential geometry, restated in the paper's notation:

    • Theorem 2 is the thin QR factorization of W⊤W^\top; Theorems 1 and 3 follow in one or two lines.
    • ρlin\rho_{\text{lin}} is 1−R2\sqrt{1 - R^2} of a multivariate linear regression, which the authors acknowledge coincides with the linear-regression similarity index of Kornblith et al. (2019).
    • Proposition 2 is the classical result of Bourlard & Kamp (1988).
    • Proposition 6 restates the known fact that the differential of a smooth idempotent map is a projection onto the tangent space of its image.
    • Proposition 7 is a standard Taylor-expansion argument, and Proposition 8 is the standard Lipschitz-margin argument.

    Presenting these as theorems overstates the theoretical depth of the work.

  2. The experiments largely quantify a known qualitative fact. That nonlinear neighborhood embeddings outperform linear projections of pixel space in neighborhood preservation is well established. The finding that Z∥Z_\parallel carries most of the energy but little of the neighborhood fidelity is a quantitative version of this known picture. This quantification has some value, but it does not constitute a substantially new empirical insight.

  3. Pretrained network features are missing as a source. The central claim is explicitly source-relative, and the most natural next step is to use pretrained features (e.g., ResNet or CLIP embeddings of CIFAR-10) as XX. If Z⊥Z_\perp largely vanishes relative to such a source, the paper's conclusion would change from "the value of t-SNE does not lie in a projection" to "the value of t-SNE does not lie in a projection of raw pixels", which is a much weaker statement. The supervised-MLP source in Table 8 does not substitute for this experiment: its rank is only about 110, far below the 2000-dimensional kernel sources it is compared against.

Questions for authors
  1. How do ρ\rho and the utilities of Z∥Z_\parallel change when the source is a pretrained feature extractor such as ResNet or CLIP?
  2. Given that the formal results are classical, which results do the authors regard as the paper's novel theoretical contribution?
Reviewer_p3QUOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The paper distinguishes reporting coordinates of a projection in the source space from embedding samples in a different target space. It first shows that every rank-k linear map into a k-dimensional target can be factored into an orthogonal projection followed by an invertible change of coordinates. For centered source data X and a centered representation Z, it then defines the realizability decomposition Z = Z_parallel + Z_perp, where Z_parallel = Pi_X Z is the least-squares prediction of Z from X. The normalized residual satisfies rho^2 = 1 - R^2, with R^2 computed from the total residual and total sums of squares across representation coordinates.

The analysis includes a random-representation null and a finite-sample degeneracy: every centered representation has zero residual when the centered source matrix has rank n - 1, a condition that generically occurs when n <= D + 1. The paper also discusses latent reparameterization, linear autoencoders, and the distinction between pointwise reconstruction losses and objectives that preserve output symmetries. Local and layer-wise variants extend the diagnostic to neighborhoods and intermediate network representations.

Experiments on MNIST, Fashion-MNIST, and CIFAR-10 report that the fitted component captures most of the energy of t-SNE, UMAP, and PaCMAP layouts, while retaining substantially less neighborhood quality than the complete representation. Autoencoder experiments examine the association between reconstruction gains and a nonzero residual, while a nonlinear decoder trained on PCA scores demonstrates that such a residual is not necessary to improve on PCA reconstruction.

Critical strengths
  1. Careful calibration and an explicit account of the diagnostic’s limitations.

The paper supplies a random-representation baseline, explains the rank-dependent finite-sample degeneracy, and makes the dependence on the declared source explicit. These qualifications are important for interpreting the residual: they help distinguish linear predictability from informativeness and explain why a zero residual in an interpolation regime may provide little evidence about meaningful structure. The local and layer-wise variants also provide concrete ways to investigate the same question at different levels of a representation. I appreciate the effort to make the scope and limitations of the diagnostic explicit.

  1. Useful autoencoder controls that clarify the relationship between encoding and reconstruction.

The PCA-score decoder experiment is a useful control. Table 4 reports that a nonlinear decoder trained on PCA scores, whose encoding has zero realizability residual, can match or outperform the end-to-end autoencoder in the reported settings with k >= 10. This separates the properties of the learned encoder from the capacity of the decoder and shows why outperforming PCA reconstruction does not establish that a non-projective code is necessary. Including this result makes the empirical discussion more balanced and helps clarify what conclusions can be drawn from the decomposition.

Critical weaknesses
  1. The additional diagnostic insight could be demonstrated more explicitly.

Several central ingredients build on established linear algebra and regression. The factorization in Theorem 2 follows from thin QR factorization, while the least-squares decomposition in Proposition 1 yields the identity rho^2 = 1 - R^2. The paper acknowledges these connections, and evaluating the fitted and residual components separately can provide useful information beyond a single regression score. Appendix J already provides a concrete layer-linearization experiment and a certificate that combines the approximation residual with downstream sensitivity and classification margins. My remaining concern is what additional diagnostic insight the realizability interpretation provides beyond standard regression residuals or R^2, together with the same downstream sensitivity information. A clearer explanation using these existing results would help distinguish the framework's interpretive contribution from the established tools on which it builds.

  1. The relationship between representation energy, spatial scale, and utility needs further clarification.

The main empirical interpretation compares global Frobenius energy with neighborhood quality measured at a small spatial scale. These quantities emphasize different properties. For example, cluster placement could account for substantial layout energy while smaller within-cluster variations strongly affect neighborhood rankings. The current global decomposition leaves this possible scale dependence difficult to separate from the distinction between realizable and residual components.

The reported weak trustworthiness of the residual alone also suggests that the improvement depends on how the two components interact. Orthogonality provides an additive decomposition of squared energy, but it does not provide a corresponding additive decomposition of neighborhood quality. A more precise account of the residual’s contribution when combined with the fitted component would therefore help interpret where the representation’s value resides.

The local analysis in Appendix I is relevant to this question. Connecting it more directly to the headline claims, and providing scale-controlled comparisons where feasible, would strengthen the empirical interpretation. These clarifications could potentially be addressed using the existing analyses or a small number of targeted controls.

Overall, I appreciate the paper’s careful calibration, random-representation baselines, and informative PCA-decoder control. My current recommendation is Weak Reject, driven mainly by uncertainty about the added diagnostic insight and the interpretation of the energy-versus-utility results. Clearer evidence on these points, including how the existing local analyses address spatial scale, could change my assessment during the rebuttal.

Questions for authors
  1. Separating spatial scale from realizability. Could the authors evaluate the decomposition after within-cluster or local centering, or use an energy measure aligned with the neighborhood scale used by trustworthiness? How do the local results in Appendix I support the interpretation of the global energy split? Evidence that the main finding persists after controlling for spatial scale would strengthen the empirical contribution.

  2. Additional diagnostic value beyond regression. Appendix J already demonstrates replacing individual layers with affine fits. Using those results, could the authors clarify which interpretation or practical decision follows from the proposed realizability framework beyond what a standard regression-residual or R^2 analysis, combined with the same sensitivity and margin information, would provide? This would help establish the additional value of the proposed framework.

  3. Interaction between the two components. Given the weak neighborhood quality of Z_perp alone, could the authors clarify how its contribution depends on the fitted component Z_parallel? An interpolation or perturbation analysis of Z_parallel + alpha Z_perp, potentially compared with a scale-matched residual control, could help explain how their interaction produces the observed improvement.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.