acceptodds
Under review as a conference paper at ICLR 2027

What Can We Infer from t-SNE, UMAP, and Other Nonlinear 2D Maps of High-Dimensional Representations?

Abstract

A common practice in machine learning and scientific data analysis is to reduce a high-dimensional representation to two dimensions using methods such as t-SNE or UMAP, then color the resulting map by semantic variables—cell type, digit identity, topic label—and read the visible organization as evidence about what the representation contains. A variable that forms clear islands in 2D may be interpreted as strongly represented, while one that appears mixed may be taken as weak or absent. We show that this backward inference can break in ways not captured by geometric fidelity alone: a variable carrying more information in the high-dimensional representation can appear less structured than a weaker one on the same fixed map — the visual ranking can reverse the information ranking. Furthermore, the mixed appearance itself is not diagnostic: nearly identical coarse 2D mixing statistics can arise from opposite underlying information states, one in which all information carried by the original noisy Y-channel is retained and another in which it is entirely absent. These results reveal a decoupling between information present before mapping and semantic expression after it. At the same time, nonlinear 2D maps can retain useful semantic structure: class-conditioned neighborhood relations — which classes tend to occur near one another — can remain similar even when most individual neighbors change. This semantic stability can coexist with substantial reorganization of between-class geometry and class-local persistence profiles. Taken together, these results highlight that semantic patterns in 2D can be informative about selected relationships in a high-dimensional representation without faithfully encoding its information content or geometry. A 2D map can therefore provide a useful view of a semantic relation without establishing the strength of the underlying information, the geometry from which it arose, or the broader scientific interpretation attached to it.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
dimensionality reduction, semantic interpretation, representation visualization, representation learning, semantic structure, post-hoc analysis, t-SNE, UMAP

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_GGhUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents an interesting study of what can be inferred from the geometric structures generated by dimensionality-reduction (DR) methods. The authors investigate four nonlinear DR methods: t-SNE, UMAP, PaCMAP, and TriMap.

In contrast to the often subjective visual interpretation of DR results, this study introduces quantitative measures for analyzing the relationships between the geometric structures of low-dimensional representations and the semantic information they express.

The findings, if presented and contextualized clearly, could provide useful insights for practitioners in data visualization, particularly in interpreting low-dimensional representations of high-dimensional data.

Critical strengths

The paper is generally well written and has the following strengths:

  1. The motivation for investigating the relationship between low-dimensional geometric structures and their semantic information is clearly stated.

  2. The analytical framework, consisting of controlled experiments addressing Q1–Q3, is systematically developed and clearly explained.

  3. The information contained in the high-dimensional representations is quantitatively characterized using mutual information. Furthermore, the semantic expression in the low-dimensional representations is quantified using the measure defined in Eq. (2).

  4. The controlled experimental design allows the authors to investigate the relationship between semantic information and geometric structure under well-defined conditions.

Critical weaknesses

The reviewer has several concerns that should be addressed.

  1. Selection of dimensionality-reduction methods

The authors do not sufficiently explain their rationale for selecting UMAP, t-SNE, PaCMAP, and TriMap. The reviewer understands that these are widely used nonlinear DR methods. However, the extent to which the conclusions can be generalized beyond these particular methods remains unclear.

The authors should provide a clearer justification for their selection and discuss the scope and limitations of generalizing their findings to other DR methods.

  1. Motivation and interpretation of Q1 (page 4)

The authors state that "practitioners routinely read clear regions in a colored 2D map as evidence that a variable is strongly represented in X."

The reviewer questions the generality of this assumption. An informed practitioner would normally interpret the resulting geometric structure in the context of the objective function and properties of the particular DR method being used.

For example, t-SNE is designed to represent local stochastic neighborhood relationships. Therefore, one should not necessarily expect its resulting low-dimensional representation to preserve arbitrary semantic information that is not explicitly considered in its optimization objective.

In this context, it is unclear whether Q1 reflects realistic expectations of informed practitioners or primarily demonstrates a limitation already implied by the objectives of unsupervised DR methods.

Similarly, the authors state that "information present in high dimensions need not be visually expressed." This observation is reasonable, but also largely expected, since the visual expression of semantic information depends on how that information relates to the original data geometry and the objective of the DR method.

The authors should clarify what new understanding their quantitative results provide beyond these known properties of dimensionality reduction, and under what conditions their conclusions are expected to hold.

  1. Comparison with supervised dimensionality reduction

The reviewer suggests that the authors consider including supervised DR methods as a complementary analysis.

Such a comparison could help clarify how explicitly incorporating semantic information into the DR objective affects its preservation and visual expression in low-dimensional representations.

In particular, comparing supervised and unsupervised methods within the same original data structure could provide further insight into the relationship between the objectives of DR methods and the semantic information expressed in their resulting representations.

For the authors' reference, some past research has demonstrated that different semantic information given to the same dataset results in substantially different representations. For example:

P. Hartono, Mixing autoencoder with classifier: conceptual data visualization, IEEE Access, Vol. 8, pp.105301 -105310 (2020) DOI: 10.1109/ACCESS.2020.2999155

Although this comparison may be beyond the primary scope of the paper, it could strengthen the interpretation and practical relevance of the findings.

  1. Practical implications of the findings

The most significant concern is the limited discussion of how practitioners can utilize the insights obtained from this study when analyzing high-dimensional data using DR methods.

The authors demonstrate that low-dimensional geometric structures do not necessarily reflect the amount of semantic information contained in the original representations. However, the paper provides limited guidance on how practitioners should evaluate or interpret the resulting visualizations in light of these findings.

For example, can the proposed quantitative measures be incorporated into a practical evaluation procedure? Under what conditions can a practitioner reasonably trust the semantic interpretation of a low-dimensional representation?

Addressing these questions would substantially strengthen the practical contribution of the paper.

Questions for authors
  1. Label-blind partitioning (pages 3–4, Eq. 2)

The authors define the semantic expression measure using a label-blind partition of the low-dimensional representation. Appendix B.2 specifies that the partition is obtained using K-means with four clusters.

However, how sensitive are the analytical results to the choice of partitioning method and the number of partitions?

Furthermore, to what extent does this partition-based measure reflect the semantic organization perceived by a human observer? A discussion or sensitivity analysis would help establish the validity and robustness of the proposed measure.

  1. Supervised encoding of semantic information (page 8)

The authors describe an experiment in which semantic information is incorporated through supervised representation learning. The implementation details are provided in Appendix E.1.

Could the authors briefly explain the supervised encoding procedure in the main text and clarify how the choice of semantic grouping and encoding method affects the analytical results?

In particular, to what extent do the conclusions depend on the specific supervised encoding procedure adopted in this experiment?

Reviewer_2QUHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates if and how well properties of the high-dimensional can be inferred from a dimensionality reduced 2D plot with a neighbor embedding method. In particular, it compares if if the visual prominence of certain parts of the data structure aligns with their prominence in the original data. They find mixed results, with ranks of visual prominence being not always preserved, and geometry often distorted, even if the embedding does show important structure. They also find that coarse structure like classes might be preserved even if finer structure like exact neighbors is not.make clearly visible salient structure of the data.

Critical strengths
  • S1 2D visualizations of high-dimensional data are very prominent for gaining insights into the data's structure. Flagging shortcomings of these maps is therefore important. The authors list several such shortcomings.
Critical weaknesses
  • W1 The novelty of the proposed analysis is limited. There are numerous works that describe pitfalls of low-dimensional maps, several of which were cited. While the exact framing of the "backward pipeline" might be novel, several of the key findings are not: The cited work by Bergam et al. (2026) already provably describes that Silhouette scores get exaggerated by t-SNE, which is very similar to the analysis in Sec 3.4. Sec 3.3 is, as the authors state, an extension of Lause et al. (2024) who already make the point that the classes of neighbors are better retained than the actual neighbors. I find their extension rather incremental here.

  • W2 Impact of the findings. First, I am not convinced that the main premise of the paper, coloring an embedding by some semantic variables to judge how prominent they are in the data, is really the main application of low-dimensional maps. If the semantic variable is available, one can simply investigate how well it is supported in the real data. To my mind, low-dimensional visualizations are mainly used for data exploration [de Bodt et al. 2025], so that it is mainly the uncolored scatterplot that users look at. After an hypothesis about the data structure is thus formed and tested on the real data, a colored version of the plot might be used for communication of the finding.
    Second, even when buying the "backward pipeline" relevance, I find some of the analyses unconvincing. The purpose of visualizations is to extract salient structure of the data and it make clearly visible. So exaggerated structure (Se 3.4) might even be desirable.
    Moreover, the setup in 3.2 seems forced: While the mutual information of Y with X might be larger than that of Y', one could very plausibly argue that Y is indeed much less prominent, because it is so scaled down. Thus, the visualizations correctly identify Y' rather than Y. This is also not a particularly surprising finding: As a visualization needs to distort some aspects of the data, it is usually possible to find some metric of the high-dimensional data that it does not preserve well (here mutual information).
    Lastly, the paper exclusively focuses on neighbor embedding methods, which are by far the most prominent class of non-linear dimensionality reduction methods, but not the only one, while the title seems to promise a broader range.

References:
de Bodt, Cyril, et al. "Low-dimensional embeddings of high-dimensional data." arXiv preprint arXiv:2508.15929 (2025).

Questions for authors
  • Q1: The authors often write about "the practitioner" (e.g. l 160, 198). Did the authors conduct a user study or was the Mix metric the stand-in for what a hypothetical practitioner might infer?
  • Q2: In Sec 3.3 the easier problem of retaining neighbors of the correct classes was contrasted with retaining the actual neighbors. However, very different metrics were used to measure both, making the comparison difficult. Why not define an analogous metric to R_nbr, where the fraction at the end gets replaced by something that directly measures if the neighbors have the correct classes like ∑cmin(∣NNX,c(i)∣,∣NNZ,c(i)∣)/k,\sum_c min(|NN_{X,c}(i)|, |NN_{Z, c}(i)|)/k, where NN_c(i) indicates the neighbors of i of class c. Setting c to the point identity directly recovers R_nbr, setting it to an actual class is a strictly coarser (and thus by construction larger) metric.
  • Q3: Why was the retention of cavities chosen as a to-be-preserved structure, when, by construction, a 2D scatter plot cannot possibly contain cavities (other than those artificially introduced by the combinatorics of VR filtration)?
  • Q4: What do the authors mean by the questions in line 422.
Reviewer_v53cOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2(+1)
Presentation
2
Soundness
3(+1)
Read review
Paper summary

Coloured t-SNE and UMAP plots are often used to draw conclusions about what a representation encodes. The authors look at which of these conclusions hold, in three parts. First, a synthetic example shows that a variable the representation carries more information about can look more mixed on the map than a weaker one (Q1). The construction rescales a block of features, which changes distances and leaves the information as it is. Second, across a range of datasets, representations and methods, most exact neighbours change while the class make-up of neighbourhoods changes little (Q2). Third, this class relation can survive large changes in class separation and in the overall shape of the point cloud (Q3). The authors conclude that a map shows which classes sit near which, and says little about how much the representation knows about a variable or about its geometry.

Critical strengths
  1. The rescaling example (Q1) is a counterexample to the common habit of taking a mixed colouring to mean "the model does not encode this". The two variables have the same number of classes, so the reversal cannot be put down to cardinality. In the Hide/Erase pair, nearly the same picture comes with and without the information.

  2. I found the RankBand control the most interesting result. Replacing each neighbourhood by the next band of source neighbours keeps class composition much better than a random replacement does, and not far from what UMAP keeps. This suggests the map keeps locality in the source space even when it loses the original neighbours. The controls for the third part (Q3) are in the appendix, and each deals with a different objection. I think they should be in the main text.

Critical weaknesses
  1. I am unsure what the class-destination relation adds over neighbour purity. It is meant to show which other classes stay nearby, and that is the stated extension over Lause et al. (2024). The results are aggregate discrepancies, though, and I could not find a relation matrix on natural data, or an example of an off-diagonal pattern. Each row sums to one, so the total variation is at least the change on the diagonal, and equals it when the map only makes neighbourhoods purer. The rows (.99, .01, 0) and (.99, 0, .01), for instance, differ by only .01, although the cross-class neighbour has changed completely. A small discrepancy can therefore hide a change in which other classes are nearby, and that is the change the relation is supposed to capture.

The effects of more separation and of a free layout on the map are known from earlier work (e.g. Wattenberg et al. 2016, Böhm et al. 2022), and I see the third part (Q3) as supporting evidence for the main point. I agree with the overall message, but beyond the rescaling example (Q1) I find less that is new, which is why I lean towards rejection.

Questions for authors
  1. Is there anything in the rescaling example (Q1) that is specific to nonlinear 2D maps? What do kNN accuracy and Mix look like in the original space, and under PCA, for your constructions?

  2. Can you show one natural dataset where the off-diagonal part of the class relation tells something that same-class neighbour accuracy does not? An example like this, or a good answer to question 1, could move me to weak accept.

  3. What would you advise a practitioner to do differently? For example, would you recommend reporting kNN accuracy in the original space next to every coloured map?

  4. In single-cell biology these maps are often used to look at trajectories and gradients more than classes. Does the class-relation view extend to continuous structure like this?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.