- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
Starting from the aggregated posterior obtained from training a VAE, we perform K-means-like reduction to K components by repeatedly merging pairs according to the moment matching rule of Eqs 1-2 in successive rounds of mutual nearest neighbours where each round constitutes a perfect matching reducing the number of components by half and is defined via scoring all pairs with diagonal Bhattacharyya distance. We thus get a deterministic hierarchical nested mixture for any choice of K in the range [1,N]. A co-training version of the model re-partitions the training data into K parts every epoch and introduces a cluster reconstruction loss together with a free bits constraint. The performance of this mixture is assessed in three separate tasks: As a latent density for OOD detection, our hypothesis gets falsified as the pre-registered. It underperforms on MNIST compared to both EM and kNN (AUROC 0.821 vs 0.922 and 0.932 respectively), it does worse than 0.5 on CIFAR-10 vs SVHN, like any latent score and on CelebA-64 (post-registration) it scores near chance with respect to SVHN. For clustering it has a smaller NMI variation across beta and seeds than EM (largest across seed std 0.04 vs 0.12) and the co-trained partition achieves 0.740 NMI on conv-MNIST (below DEC at 0.861). As a generative prior it outperforms N(0,I) in sampling and is roughly on par with EM. When substituted in as the prior within the pixel likelihood it has no effect on that score (whereas an EM mixture can sometimes cause it to collapse). As a frozen codebook in an SQ-VAE style model it outperforms a codebook learned from scratch.
Critical strengths
-
The experimental technique is thorough. In particular, the VaDE and DEC reproductions (see Appendices G and H) stand out on their own merits. For example, the discovery that DEC's claimed accuracy relies on a dropout procedure at finetune time, which the DEC paper states does not apply.
-
The negative OOD result is presented in full with explanations of part of it. Under calibrated Gaussian likelihood, the reconstruction error, reconstruction probability, and IWAE score deliver very similar AUROCs on SVHN (0.174, 0.174, and 0.172, respectively; Table 14). While the pixel score correlates with PNG file size (Spearman's ), the latent score shows no correlation (). Changing the CelebA-64 likelihood model from a Bernoulli to a calibrated Gaussian reduces the pixel AUROC on SVHN from 0.769 to 0.153 (Table 16).
-
The authors state where the method loses: EM samples as well, k-means and DEC cluster MNIST better, and GPU EM is faster.
Critical weaknesses
The merged mixture beats EM in some settings, but each time a simple alternative does as well or the control that would settle the comparison is missing.
-
Novelty. The moment-preserving merge of Eqs. 1-2 is standard Gaussian mixture reduction (Salmond, 1990; Runnalls, 2007; Goldberger & Roweis, 2004). Ward's (1963) method already gives a deterministic, nested hierarchy, and merging mutual nearest neighbours is the reciprocal-nearest-neighbour scheme (Murtagh, 1983). None of this is cited or compared against, and the new parts, the Bhattacharyya cost and the forced perfect matching, are not shown to beat Runnalls' cost or Ward linkage.
-
The perfect matching forces near-equal weights: true cluster sizes 490, ..., 6 come back as 256, ..., 124 (Section 3.2). All clustering results are on class-balanced data, so the gains over EM may come from this built-in balance. As a latent density the mixture is also far worse than EM (held-out log-density against at ; Table 28).
-
The density-safety comparison is confounded. The EM prior drops the MNIST pixel score from 0.9997 to 0.85 or 0.80 on some seeds because its components shrink to . But EM is fit to the means only, while the merge uses the posterior scales (Appendix K.3). EM on the full aggregated posterior, or with a variance floor, may well be just as safe, and until that is run the advantage is not established.
-
The baselines are weak. EM samples as well or better, and beating repeats Ghosh et al. (2020). k-means on the posterior means reaches 0.700 NMI on conv-MNIST, above the merge (0.594) and EM (0.641) (Table 35); the authors attribute the co-trained partition's 0.740 to "the co-trained encoder, not the mixture fitter". The CIFAR-10 gap in Table 11 comes from fixed models that reconstruct each image as its mean colour (Appendix D.10). The codebook study has no frozen k-means or EM codebook as a control (cf. Łańcucki et al., 2020).
-
The unifying principle is not tested. Every latent score is below 0.5 on CIFAR-10 vs. SVHN (0.27 to 0.30 in Table 1), and the abstract says density "depends only on the encoder", so the failure belongs to the representation rather than the compression. "Preserves where the data lives ... not how likely it is" summarizes the results; no experiment in the paper could have contradicted it.
-
Presentation. Most claims rest on 35 appendix tables. Table 1 uses one seed and the degenerate CIFAR-10 models, and the -path results use a different merger from Section 3.2 (Appendix D.9). There are many typos ("disentengle", "algoirthm", "wehre fittet"), a garbled contribution bullet (lines 105-106) and leftover template text in the Acknowledgments.
Questions for authors
-
How does the merge compare with Ward linkage on the posterior means and with greedy mixture reduction under Runnalls' cost? Both are deterministic and nested, so they can be compared on clustering, sampling, and use as a prior density.
-
Can you fit EM to the full aggregated posterior (using the ), or with a variance floor, and repeat the experiment in which the EM prior collapses the MNIST pixel score?
-
Since the perfect matching forces near-equal weights, do the clustering results hold on class-imbalanced data such as long-tailed MNIST or CIFAR-10? Does re-estimating the weights after merging change them?
-
In Table 27, how does a frozen k-means or EM codebook on the same latents compare with the frozen merged codebook?
-
Could Table 1 be recomputed with the calibrated CIFAR-10 models and 10 seeds, the configuration the appendix treats as the reference?
-
Where is the pre-registration recorded, and what did it fix (hypothesis, metrics, analysis plan, values of )? An anonymized link or timestamp would help.
2 comments
Thanks for your reply and for doing the comparison of Ward with the suggested merge. It proves that all my concerns were true: Ward linking of the means is not worse, but at least as good as the suggested merge; the only algorithmic difference between the two is the order of merging; the claim of safety in relation to EM is retracted; and the "pre-registration" was just an internal design document.
The comparison to Runnalls' cost has not been done, and neither has the experiment with EM with the full aggregated posterior as the prior on the pixel likelihood, and class imbalanced data and a frozen k-means or EM codebook. All of these issues have not been addressed. The only thing left standing now is a deterministic hierarchy where the sampling prior is equal to EM (FID 65.7 on CelebA-64), and an OOD negative result, which also applies to EM.
We thank the reviewer for an accurate summary and a review in which every weakness is paired with the experiment that would address it. We agree with almost all of it. The revised PDF has not been uploaded yet, as some of the results are still in the finish line (next few hours); it will be followed by a list of changed numbers when the last runs finish. Numbers below are from a re-run on the full training split (10 seeds, calibrated decoders, one evaluator). Two answers go against the paper.
Q1. Ward on the posterior means: run, and it is as good as or better than anything of ours. 230 plain VAEs, NMI at K = 10, pooled over β, paired by encoder:
Our Bhattacharyya merge is lower still (0.607 on conv-MNIST). For clustering, the posterior scales, the moment-preserving update and the matching add nothing over the 1963 algorithm. We withdraw the clustering claim for the method.
Runnalls' cost: implemented, not finished. We implemented the exact closest-pair reduction under Runnalls' bound and verified it against brute force on small inputs, but it was too slow on the real posteriors to finish in time. So the reviewer's statement stands: the Bhattacharyya cost and the perfect matching are not shown to beat Runnalls' cost. Sampling and prior-density comparisons for both reductions were not run either. We will, of course, run it, and we do plan to incorporate that in future versions.
Q2. Not run in the rebuttal period. The comparison is confounded, as the reviewer says: EM saw only the means. Elsewhere we added EM on one posterior draw per training point, but not yet as the prior in the pixel likelihood. Until then, the revision does not call the merged prior safer than EM.
Q3. Not run, for lack of time. We agree it matters most: the perfect matching forces near-equal cluster sizes and all our data are balanced. Long-tailed MNIST and CIFAR-10, with and without re-estimated weights, are planned; we make no claim about imbalanced data and state the balance as a limitation.
Q4. Not run (needs training). Without a frozen k-means or EM codebook the study only shows that a frozen data-derived codebook beats one learned from scratch, as Łańcucki et al. (2020) suggest. The revision says so.
Q5. Yes. β = 1, K = 50, 10 seeds, AUROC:
The picture is unchanged: every score inverts against SVHN, and on MNIST the merge trails EM. kNN and Mahalanobis are being recomputed.
Q6. It is not an external registration, and we should not have used a word that suggests one. It is a design document in our own repository, first committed on 3 June 2026, before the result files (the earliest is dated 8 June). It fixed the claim ("matching or beating EM-GMM latent density at equal K on OOD AUROC"), score, protocol, baselines and failure modes. Deviations: Omniglot and Textures were never run; CelebA was added later. The revision says "fixed in advance in a dated design document" and provides it.
Novelty. Agreed. The revision cites Salmond, Runnalls, Goldberger & Roweis, Nielsen et al., Ward and Murtagh, and describes the method as classical moment-preserving reduction applied to a VAE's aggregated posterior with a different merge order. For a reducible criterion like Ward's, reciprocal-nearest-neighbour merging gives the closest-pair tree; with our cost it does not, and the closest-pair order chains under the Bhattacharyya distance. That is the only algorithmic difference we claim.
Principle. Agreed: it is a summary, not a tested principle, and the OOD failure belongs to the representation (EM inverts too). The remaining claim is small: the known failure also holds for the reduced aggregated posterior.
Baselines, presentation. Beating N(0, I) confirms Ghosh et al.; EM samples as well. Tables are regenerated by one script; typos and the template text are fixed.