- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes CLUSTOPT, which predicts the utility of clustering effectiveness metrics based on data features, constructs a weighted search objective, and then reorders the candidate clustering results. The experiments primarily focus on automatic clustering selection in two-dimensional data.
Critical strengths
- The motivation for the research is clear; different data structures require different evaluation metrics, and this approach has practical significance.
- The paper analyzes the effects of metric weighting and candidate reordering separately and reports that no significant improvements were achieved; the experimental results are sufficiently well-documented.
Critical weaknesses
- On external benchmarks, the single-view ARI for ARI is 0.319, and for AutoClust it is 0.321; the relative advantage is primarily concentrated in certain non-convex datasets, and its practical applicability remains limited.
- A comparison with a “reordering algorithm using non-conditional criteria combined with independent training” has not yet been conducted; therefore, the additional contribution of data conditioning in a complete system remains unclear.
Questions for authors
- If the reordering algorithm is trained separately using non-conditional criteria, does CLUSTOPT still offer an advantage?
- Could you provide information on the time required for meta-feature extraction so that the computational cost of the entire process can be evaluated?
3 comments
Resposne to Reviewer pYXv (part 2)
W1. External results and practical applicability. The reviewer's reading is correct and matches Section 7. On the deployable single view the two systems are level (0.319 against 0.321), and no overall advantage over AutoClust is established. The advantage is concentrated in non-convex, connectivity-defined data, on both benchmarks: arcs and rings (+0.030) and de-interleaving (+0.013) on the controlled benchmark, concentric circles and two moons on the external one. AutoClust leads on isotropic blobs. On the 28 real tabular datasets, 26 of them projected to 2D with PCA, every system stays below 0.16 ARI.
The non-convex regime is the one the method was built for. The work began with a concrete need, separating radio-frequency emitters such as radars in a stream of intercepted pulses. Pulses are clustered in planes of two measured parameters (for example, time of arrival against amplitude or frequency). There, clusters are arcs, parallel bands and interleaved trains, and compactness-based indices can choose the wrong partition. The same holds for other natively two-dimensional data with shape-defined clusters: spatial point data (incidents, disease cases, seismic events), single-molecule localization microscopy, wafer defect maps and 2D laser scans.
High-dimensional data can be brought into scope through 2D embeddings. t-SNE and UMAP maps are widely used to identify groups, for example in single-cell biology [1] and for learned text embeddings at the scale of millions of documents [2]. Clustering in a UMAP embedding instead of the original feature space has also been shown to improve density-based clustering [3]. Our indices and pipeline apply to such a map without change. We have not evaluated this route. We used PCA, which keeps far less cluster structure than these maps do [4], and the embedding test is a natural next experiment.
We state the boundary of the evidence clearly. Our radar-like data are synthetic, and the paper has no result on real data from these domains or on t-SNE or UMAP maps. The reviewer's assessment that practical applicability is not yet demonstrated is fair. The setting is narrow by design, and the paper should have explained why. The revised introduction and limitations now do.
W2. Missing comparison with an independently trained reranker. This is the experiment reported under Q1. It answers the question that the submission left open and narrows our claim about conditioning.
A note on Strength 2. The summary says that the paper reports no significant improvements. For the record, this holds for the overall comparison with AutoClust, but not for the components: profile weighting against uniform weighting (+0.227 ARI) and reranking (+0.058 on the controlled benchmark; +0.083 externally, unadjusted p = 0.0013). Against ML2DAC, the external benchmark shows a positive paired difference of +0.067 [+0.001, +0.157], unadjusted p = 0.039; the best ML2DAC arm was chosen as comparator after unblinding. We withdraw the submission's controlled ML2DAC comparison (+0.042): it was computed against a historical ML2DAC run whose index scorer was defective, and no corrected controlled run exists. On the controlled benchmark the system comparison is therefore with AutoClust only, where the two are at parity (0.616 against 0.614).
We thank the reviewer. The two questions led to the two most informative experiments we have run since submission, whatever the decision.
- [1] D. Kobak, P. Berens, "The art of using t-SNE for single-cell transcriptomics," Nature Communications, 2019.
- [2] R. González-Márquez, L. Schmidt, B. M. Schmidt, P. Berens, D. Kobak, "The landscape of biomedical research," Patterns, 2024.
- [3] M. Herrmann, D. Kazempour, F. Scheipl, P. Kröger, "Enhancing cluster analysis via topological manifold learning," Data Mining and Knowledge Discovery, 2024.
- [4] J. Lause, P. Berens, D. Kobak, "The art of seeing the elephant in the room: 2D embeddings of single-cell data do make sense," PLOS Computational Biology, 2024. ----- END COMMENT 2 -----
The author's response has addressed part of my concerns.
Resposne to Reviewer pYXv
We thank the reviewer for their constructive review. Both questions led to new experiments. We report them here in full and address the weaknesses in the next comment. We have prepared a revised paper that includes them (section and table numbers below refer to it).
Q1. Does CLUSTOPT keep its advantage when a reranker is trained separately for the non-conditioned criterion? We ran this comparison. We trained two rerankers with the same model, hyperparameters, target and training data (48,039 candidate banks from the development splits). One serves the conditioned criterion (the predicted utility profile). The other serves the non-conditioned criterion (the mean profile, with the same 10 indices for every dataset). Each was evaluated on the 1,055 held-out Split-1 datasets, on the candidates visited under its own criterion.
The answer is no. When both rerankers see the predicted utilities, the two criteria show no detectable difference: +0.0014, p = 0.70, with 337 wins, 390 ties and 328 losses. This is not a formal equivalence test: the interval rules out neither a small advantage nor a small disadvantage. The revised paper no longer claims that conditioning the criterion adds to the complete system (abstract, Section 5.2 with Table 3, Section 8; details in Appendix V.1).
The experiment also shows where the gain sits.
Split 1 informed our design decisions, and this comparison was not run on the external corpus. Two statements are corrected in the revision: the reranker is trained on each candidate's regret relative to the best candidate in its bank, not on ARI directly, and the submission's estimate that this comparison needed 77-107 hours of new search was wrong, because the rerankers are trained on fixed candidate banks that do not depend on the criterion (Appendix V.1).
Q2. Meta-feature extraction time and total cost. We measured the complete pipeline of the three primary systems (the deployed CLUSTOPT and the best AutoClust and ML2DAC arms) on the 50 external datasets (450 runs, one worker, one numeric thread).
These figures include meta-feature extraction, model loading, search, index evaluation and reranking. They exclude only the refits that our harness performs to log partition hashes, which never affect the selection (with them: 17.29 and 42.14 seconds for CLUSTOPT).
Meta-feature extraction has a median of 1.98 seconds per run, covering all three views. This is 11.5% of CLUSTOPT's execution time. Reranking has a median of 0.019 seconds. CLUSTOPT and AutoClust have similar typical cost (paired median ratio 0.97). ML2DAC is about 2.5 times faster than CLUSTOPT. The timings describe our implementations, including our re-implementations of the two baselines. Appendix A of the submission said that extraction was logged as zero; in fact it lies inside the timed path. The original records also ran with two to four concurrent workers, so they are not comparable with these single-worker figures. The revision corrects this and reports the measurements (Section 7, Appendix S.1); the other arms keep only their original records, labelled as such.