- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper provides a useful perspective on the differences between linear and nonlinear dimensionality reduction methods. It is well written and clear. I particularly appreciated the characterization of the decomposition between the part of a representation that can be reproduced by a linear transformation and the part that cannot. I also found the discussion of the equivalence between linear dimensionality reduction methods, and of how coordinate transformations can affect their geometric properties, particularly insightful. Finally, the numerical experiments provide an interesting illustration of how a relatively small nonlinear component can have a substantial impact on the quality of a representation.
Critical strengths
The paper presents a clear and rigorous analysis of the differences between linear and nonlinear dimensionality reduction methods. They introduce a mathematical framework that separates a low-dimensional representation into a part that can be reproduced by a linear transformation of the original data and a part that cannot. They show that, although the latter often accounts for only a small fraction of the representation, it can play a crucial role in preserving the neighbourhood structure of the data. The framework is illustrated through experiments with popular dimensionality reduction methods and autoencoders. A good paper for a lecture.
Critical weaknesses
Some of the theoretical results, particularly Theorems 1 and 2, are elementary consequences of standard linear algebra. Similarly, the central realizability decomposition is the familiar decomposition of multivariate linear regression into fitted values and residuals. The main novelty of the paper therefore appears to lie in the geometric interpretation of these constructions and in their empirical application to nonlinear representations, rather than in the mathematical results themselves. I believe the authors should make this distinction more explicit, and maybe avoid calling "theorems" textbook results. A most important limitation concerns the practical implications of the proposed framework. I found unclear how the decomposition can be used in practice. In particular, a practitioner comparing different dimensionality reduction methods is typically interested in identifying which method best preserves the information relevant to a given task. How this can be achieved is not explicitly discussed (see question below)
Questions for authors
The authors convincingly demonstrate that the realizability residual is not, by itself, a measure of representation quality. However, this raises the question of what additional actionable information the decomposition provides beyond conventional measures of representation quality. I believe the paper would benefit from a more explicit discussion, supported by a concrete example, of how the proposed analysis can guide the interpretation and selection of dimensionality reduction methods.