- Rating
- 3(+1)
- Confidence
- 3(+1)
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes COMET, a tool for high-dimensional data visualization in 2D or 3D. It builds upon existing methods like t-SNE and UMAP by avoiding "subjectively tunable" hyperparameters, separation of continuous structures into separate clusters, and scale/layout distortions. COMET's pipeline begins by building a nearest-neighbor graph, then uses a random walk commute time to record graph distances (approximated by eigenvectors of the graph Laplacian). Nearby neighborhoods are ranked by local similarity and "long-distance" similarity (from randomly sampled points). This ranking than provides target probabilities (whereas UMAP would assign a probability of 0 throughout). COMET fits this layout with a cross-entropy loss and kernel wifths and regularization are calculated from data. Runtime scales linearly with dataset size.
Critical strengths
-
COMET addresses very practical problems with t-SNE and UMAP, which are known to be over-interpreted in certain applications. Tuning until the plot "looks right" is obviously not a sufficient scientific approach, so I appreciate the author's proposed pipeline to address this. Moreover, the central idea of the pipeline is clear and well motivated. It uses the data shape to compute the data repulsion in lower dimensions. Ablation experiments are convincing that removing this modifications leads back to the usual separations and distortions.
-
The method is scalable and thus practical. Authors show it embeds 86k cells in about 20 seconds, with no need for subsampling (unlike PHATE).
Critical weaknesses
-
Weak metrics: First, the headline comparison metrics are proposed by the work itself. Without established measures of global structure, the main claim is weakened in its obvious tie to the scoring design... Authors use an established metric for trustworthiness, and t-SNE beats COMET on every dataset there, sometimes significantly (on Newsgroups, for example). As such, the results on global structure preservation would be significantly strengthened if there was some independent metric that authors could report. In particular, the proposed geodesic rank correlation ρGeo is closely related to DEMaP (Moon et al., 2019), which correlates geodesic distances in the data with distances in the embedding. Second, the quantitative topological metrics are really limited. For one, the only topological check (at least in the main paper) is whether an embedding stays in one piece. Loops (in COIL). and holes (Swiss roll) are qualitatively checked. It would be valuable to include quantiative checks for a paper with "expected topology" in the title... for example with persistent homology, at a minimum.
-
Hyperparameters involved: The claim that COMET avoids hand-tuned hyperparameters seems overstated. COMET uses fixed parameters like neighborhood size, sampled size, spectral dimension, and so on. Sensitivity analyses show that at least one of them noticeably changes results. So it would be important to adjust claims like COMET "removes the analyst's choices". General
Questions for authors
Clarifications
Were the three proposed measures and their settings (e.g., the floor of 5 in N_comp) fixed before COMET was developed? Appendix A reports results at floors of 2 and 15, which suggests the alternatives were seen before the choice was made.
Which topological properties beyond connectivity does “expected topology” refer to, and does anything in the objective explicitly preserve them?
Were COMET’s fixed defaults (k, s, l = s, κ_t, the fallback threshold) chosen or adjusted using the evaluation datasets?
Suggested future work (satisfying these suggestions would convince me to change my score)
Report at least one established global-structure measure that the authors did not design (e.g., DEMaP, see comment above in Weaknesses).
Quantify topology preservation with some topological metric (like persistent homology), at least on the Swiss roll and COIL-20.
Report sensitivity to k, and subsequently adapt the “no hyperparameter tuning” claims accordingly.
6 comments
We thank the reviewer for their constructive feedback
Question: Which topological properties beyond connectivity does “expected topology” refer to, and does anything in the objective explicitly preserve them?
Answer: The name refers to how the objective’s targets are constructed. “Expected” refers to the expected ranks (Equation 4) that assign target probabilities to the sampled non-neighbors. “Topology” refers to the k-nearest-neighbor graph from which those targets are derived. The graph encodes topological structure such as holes indirectly: paths between points on opposite sides of a hole must go around it, which lengthens the commute times between them. We want to clarify that no term in the objective preserves such structure explicitly. It is preserved only implicitly, through the commute-time targets.
Question: Were COMET’s fixed defaults (k, s, l = s, κ_t, the fallback threshold) chosen or adjusted using the evaluation datasets?
Answer: and were both fixed at 15 before evaluation, and was chosen based on the motivation given in the final paragraph of Section 2.3. and the fallback threshold are numerical-stability safeguards, set during the development of COMET on a wide range of datasets that included some of the evaluation datasets.
Thank you for the detailed response and additional experiments. The DEMaP results, persistent-homology evaluation, and sensitivity analysis address my main requests. I also appreciate the changes to the hyperparameter claims and clarifications about topology preservation.
I still have a few unanswered questions:
- How were the evaluation measures and their settings chosen, particularly the component-count floor of 5? Were these adjusted after inspecting comparative results, and do the alternative floors change the conclusions?
- For the numerical safeguards developed using some evaluation datasets, could you clarify whether the choices were based only on numerical stability, or also on the quality of the resulting embeddings?
Also, I would appreciate if the main text could better present the new clearer picture of where COMET performs best versus other methods, particularly the DEMaP results on trees and hierarchical Gaussians, the loop-detection results on COIL-20, and the lower trustworthiness scores.
We thank the reviewer for their constructive feedback
Comment: Weak metrics: First, the headline comparison metrics are proposed by the work itself. Without established measures of global structure, the main claim is weakened in its obvious tie to the scoring design... Authors use an established metric for trustworthiness, and t-SNE beats COMET on every dataset there, sometimes significantly (on Newsgroups, for example). As such, the results on global structure preservation would be significantly strengthened if there was some independent metric that authors could report. In particular, the proposed geodesic rank correlation ρGeo is closely related to DEMaP (Moon et al., 2019), which correlates geodesic distances in the data with distances in the embedding. Second, the quantitative topological metrics are really limited. For one, the only topological check (at least in the main paper) is whether an embedding stays in one piece. Loops (in COIL). and holes (Swiss roll) are qualitatively checked. It would be valuable to include quantiative checks for a paper with "expected topology" in the title... for example with persistent homology, at a minimum.
Answer: We address both points in our answers to the first two requests below.
Comment: Hyperparameters involved: The claim that COMET avoids hand-tuned hyperparameters seems overstated. COMET uses fixed parameters like neighborhood size, sampled size, spectral dimension, and so on. Sensitivity analyses show that at least one of them noticeably changes results. So it would be important to adjust claims like COMET "removes the analyst's choices".
Answer: We agree with all three reviewers that the claim of requiring no hyperparameter tuning was overstated. Therefore, we have revised the wording throughout the manuscript (Abstract, Sections 1, 3.1 and 4, Appendix B.5) to soften the claim. In particular, it should now be clear that while COMET aims to reduce the need for per-dataset tuning it does not eliminate all tunable hyperparameters, and changing the hyperparameters can still affect the results.
Request: Report at least one established global-structure measure that the authors did not design (e.g., DEMaP, see comment above in Weaknesses).
Answer: Thanks for the suggestion. We now report DEMaP for every method and dataset in Appendix B.6 (Table 13). DEMaP and ρGeo differ only in the LD distances: DEMaP and ρGeo compare the HD shortest-path distances with the Euclidean distances and the shortest-path distances in the embedding, respectively. We keep ρGeo as the primary measure because DEMaP penalizes an embedding for reproducing a hole present in the data. For example, the ground-truth embedding of the Swiss roll with a hole scores 1.000 on ρGeo but 0.991 on DEMaP (Figure 7). For the real datasets both measures select the same best method (Tables 2 and 13). On the synthetic datasets (Tables 1 and 13). they agree on the flat disc and the Swiss roll, while on the hierarchical Gaussians and the tree COMET ranks first on ρGeo but third on DEMaP, behind SASNE and TriMAP.
Request: Quantify topology preservation with some topological metric (like persistent homology), at least on the Swiss roll and COIL-20.
Answer: Thanks for the suggestion. We added the hole detection score of Damrich et al. (2024), a persistent-homology measure of how clearly a loop stands out in the embedding, to Section 3.2. Appendix A.1 defines and reports the hole detection score for the Swiss roll and COIL-20 (Table 3). Table 3 shows that for the Swiss roll, COMET achieves the highest score by a wide margin. For COIL-20, where each object’s rotation forms a loop, PaCMAP and TriMAP separate these loops most clearly.
Request: Report sensitivity to k, and subsequently adapt the “no hyperparameter tuning” claims accordingly.
Answer: We added a sensitivity analysis of k in Appendix B.4. Table 11 varies k from 5 to 50 on the Swiss roll and the hierarchical Gaussians and reports the runtime on the C. elegans atlas dataset. From k=10 to k=30, ρGeo stays between 0.975 and 0.986 on the Swiss roll and between 0.933 and 0.980 on the Gaussians, and the default k=15 is within 0.006 of the best k on both. The largest change occurs at k=5 on the Gaussians, where the 5-NN graph is disconnected and ρGeo falls to 0.884. Repeating the analysis for UMAP (Figure 6), COMET’s results vary less across k than UMAP’s on four of the five measures. The “no hyperparameter tuning” claims have been adapted accordingly (Abstract, Sections 1, 3.1 and 4, Appendix B.5).