- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper discusses a method for optimizing the parameters of the similarity kernel, which defines data similarity in the output space (the low-dimensional space) during t-SNE, and examines the relationship between these parameters and the underlying latent dimensionality. Specifically, the authors propose minimizing the KL divergence between the data distributions before and after embedding with respect to , treating (where the t-distribution degrees of freedom is defined as ) as a tunable parameter of the Cauchy similarity kernel. The study demonstrates that this optimal is related to the dimensionality of the low-dimensional space under certain ideal circumstances, and validates, through real data experiments, that the optimized generally holds this theoretical relationship across various latent dimensions. Furthermore, the experiments demonstrate that when the latent dimensionality is low, the theoretically expected optimal parameter tends to decrease rapidly when noise is introduced into the k-NN graph. This empirically supports the intuition and practical observation that the heavy-tail nature of the t-distribution is effective when noise is present.
Critical strengths
To the best of the reviewer's knowledge, there are no prior studies that parameterize the similarity kernel used in t-SNE for embedding and discuss its optimality in relation to the dimensionality. The results obtained are reasonably coherent. The finding that the relationship found between latent dimensionality and the optimal parameter is largely consistent with experimental data is beneficial. Furthermore, the analysis of how the system behaves when noise is introduced to the k-NN graph was also convincing.
Critical weaknesses
- The most significant issue I find with this paper is the failure to properly formulate the problem, and the overall inaccurate description.
Firstly, it is unclear what is being optimized and how. The objective function is supposedly minimized with respect to what? It is likely that the optimization is a joint, gradient-based process over both the embedded coordinates and the parameter . However, this is not stated in the paper. Furthermore, the notation is inconsistent. Section 3.1 states that the input data has dimension and the output data has dimension , but from Section 5 onwards (at least), is treated as the intrinsic dimension. Did Proposition 4.3 not show that the optimal depends on the embedding dimension ? Additionally, the intrinsic dimension is used without being defined, and is immediately treated as a target for estimation using the Levina & Bickel method.
A minor point, but the use of to mean both the number of neighbors and the kernel is sloppy. Such negligence detracts from the overall readability and comprehensibility of the paper.
- Issues with the theoretical section
Firstly, Theorem 1 lacks novelty; it merely states a relatively self-evident fact. Again, the inaccuracy of the description is a major problem. In the proof of Proposition 4.2, the argument—"perplexity calibration ensures is finite; the mapping $2A/D \mapsto \operatorname{Perp}(p_m)$ is strictly monotone and injective for any , therefore converges"—is incorrect or at least lacks sufficient argument. The final conclusion ("therefore") is unfounded. Injectivity alone does not guarantee the convergence of the inverse function for each . Uniform convergence plus uniqueness is required. Furthermore, the meaning of "average of values" must be clarified: is it an average over the population or over the observed data? While it would be acceptable if it were over the population, the argument in this proposition is somewhat inaccurate if it applies to empirical data.
Questions for authors
- How was the optimization performed (i.e., what was optimized and how)?
- Is the usage of and consistent?
- What is the definition of the Intrinsic Dimension used here?
2 comments
Thank you for reply to my questions and explanation to the point I enumerated in the "weakness". The notational problems are revolved. Proposition 4.2 seems to become accurate. Overall, the readability and accuracy of the paper have been improved.
This work has certain amount of novelty, and the coherence between theory and experiments.
Considering that the improved readability and soundness, in addition to the contribution itself, and taking into account the upcoming inter-reviwer discussion/cross-check, I will consider the score again.
Rebuttal
Dear reviewer, thank you for your comments. We appreciate your critical reading; we have updated the paper and we believe we could address all your concerns.
Thank you. We added an explicit statement to Section 3.2:
Thank you. We renamed the ambient dimensionality in Section 3.1 to
D_\text{amb}, and added the following clarification to the beginning of Section 4:Yes indeed. To make it more explicit, we now write
\hat\alpha^*(D,d)after Proposition 4.3 when we talk about finding the optimal alpha numerically.Indeed. We now added the definition to the beginning of Section 4, see above.
Thank you. We changed the kernel notation to T (this only affects Section 3.2).
Fair enough, and we did not want to claim novelty here. We renamed this statement to Proposition. The subsequent content of Section 4 is the actual substance.
We thank the reviewer for spotting this gap. Injectivity alone indeed does not imply convergence. We now prove convergence in a short lemma in the Appendix (Lemma~A.1): since the perplexity is strictly decreasing in 2A/D and continuous in 2/D, the calibrated value 2A/D is squeezed into any neighborhood of β for large D.
Thank you. pm denotes the population average (expected value); we now state this explicitly, and clarify that the approximation consists of replacing the random neighbor distances by their typical values.
We hope that our edits make the manuscript more precise and address your concerns! As you appreciated the novelty and convincingness of our results, we hope that you will be willing to revisit your score.