- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper developes a novel workflow for validating unsupervised scientific discovery, centered around the principle of representational stability. Primarily the paper addresses the notion of clustering similar physical phenomena based on stochastic embeddings and it asks whether perturbations in latent space affect the type of clusters found. The central premise is to build on an earlier framework, NEMI, which proposes to overcome any such instabilities by an ensemble-based approach where the same features are embedded multiple times, then clustered and returned to physical coordinates. These clusters are then aligned using one of two matching techniques and a consensus cluster is established using geographic entropy. The paper evaluates this approach on four datasets ranging from Ocean ecology over Atmospheric dynamics to Ocean biogeochemistry.
Critical strengths
Overall, I think the approach in the paper is sensible and contains an interesting approach to stability, that is very much used in other areas of scientific computing as well: ensemble-based approaches are widely used to enhance stability and robustness of forecasting and classification algorithms, so using them in the present context is a worthwhile endeavour. I think the authors provide sufficient mathematical details to describe their methodology (including clustering, alignment and consensus extraction) and the datasets used in the numerical experiments appear thorough. Presentation-wise I would like to highlight that the paper appears human written.
Critical weaknesses
- I found the paper to be quite difficult to read. I actually want to emphasise that this is not because it is overloaded with information (I would definitely not consider this as unedited AI output) but rather because the paper is formulated really abstractly and misses some details in crucial places. For example:
(a) In the abstract the authors refer to "the same ensemble that NEMI uses for clustering". This assumes some familiarity with the NEMI technique which until this point is only introduced as an "ensemble-based workflow".
(b) On p. 2 the authors mention "In general terms, the space where the interpretation should take place is the coordinate system in which scientific entities have natural relationships, constraints, or causal structure.". It would be great if they could be more explicit here as well, this is a very broad statement that does not really convey a practical application in my opinion.
(c) At the beginning of section 2 the authors mention that "The NEMI workflow comprises four stages: 1) stochastic embedding, 2) deterministic clustering in latent space, 3) ensemble construction, and 4) projection back to domain space" there is no mention what type of input data or indeed computation the principle is supposed to make. Are the authors intending to use this to make predictions of physical fields (e.g. weather forecasting), or is this intended to classify certain events? Or to characterise similar phenomena. In my opinion this is not clear at least by this point in the paper.
-
While the datasets are overall a good choice for validation, the paper does not place this novel approach sufficiently well in the broader literature. In particular, the introduction cites several alternative clustering stability and consensus clusterin methods (Ben-Hur; Lange; von Luxburg; Minti et al.) but does not benchmark against any of them in the numerical experiments section. It is thus very difficult to ascertain the practical benefit of the novel approach.
-
One might argue that the novelty of the contribution is somewhat limited with the main differences to prior work highlighted to be "1) we perturb the learned representation before applying deterministic clustering, and 2) we evaluate agreement for each location in domain space, rather than as a global score." This suggests the current approach is incremental and together with the lack in baselines, the algorithmic contribution might not be enough to merit publication in ICLR.
-
The labels in Figures 2-6 are unusually small making it impossible to read and understand any of the numerical results in a printed version of the manuscript, and even on a screen the results are not easy to read.
Questions for authors
-
I wonder if the authors have explored the effect of and possibility for adverserial perturbations rather than just random perturbations on their models?
-
I wonder if the authors can explain more clearly what the role of random perturbation in latent space might correspond to in real scenarios? It seems to me more likely that observational data might be perturbed through noise in the state space and that the effects of such perturbations on downstream clustering should be examined. Is the NEMI approach able to cope with this?