- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes converge-to-surprise, a self-supervised method for clustering images without knowing the number of clusters in advance. It compares cluster predictions from two independently augmented checkerboard views, rewarding agreement that is unlikely under an i.i.d. pixel-noise assumption.
An evolution strategy explores parameter variations to increase the surprise score, while periodic gradient-descent training reinforces pseudo-labels from surprising, balanced clusters. On MNIST, Fashion-MNIST, and USPS, a ResNet-9 with achieves competitive results and discovers cluster counts close to 10. The paper reports less satisfactory performance on CIFAR-10.
Critical strengths
- The core idea is distinctive and easy to follow. The paper uses complementary checkerboard masks to create two views with no shared pixels, then measures how often they receive the same cluster assignment.
- The optimization strategy is clearly presented. Evolution strategies explore parameter space, while periodic gradient-descent steps reinforce clusters identified as surprising.
- The experiments include useful component analyses. The ablations examine augmentation, the gradient-training inner loop, and USPS-specific anisotropic augmentation. The authors also report less satisfactory performance on CIFAR-10.
Critical weaknesses
- The generality of the claims is not yet established. Experiments cover MNIST, Fashion-MNIST, and USPS, with mixed results across datasets and metrics. For example, the method is close to DeepDPM on MNIST ACC but has lower NMI on Fashion-MNIST. These results are promising on the evaluated benchmarks, but provide limited evidence for broader claims about image clustering.
- The statistical assumptions behind the surprise score need further support. The null match probability is defined as which assumes that both views have the same cluster marginal. Since the masks retain different pixel positions, it would help to explain when their output distributions should be identical and whether this holds in practice. The paper also estimates from the same batch used to count matches. The authors treat each cluster's match count as binomial and state that dependence across clusters has only a weak effect; clarification of the conditions and limitations of this approximation would strengthen the score's statistical interpretation.
- The checkerboard masking strategy lacks a direct ablation. The “no augmentation” experiment retains checkerboard masking and removes only augmentation. It therefore does not show whether the checkerboard pattern itself is important or whether a different disjoint partition would work similarly.
- One reference appears to have inconsistent bibliographic details. The Batina et al. (2011) entry pairs the title “Web text corpus for natural language processing” with Journal of Cryptology, 24(2), 269–291, which appears to refer to a different article. The authors should verify the reference and confirm that the corrected source supports the independence claim.
Questions for authors
- Checkerboard ablation: The checkerboard pattern is central to the construction, but the current ablations do not isolate its contribution. Have you tried random disjoint half-pixel masks under the same training setup? Comparing those results with checkerboard masking could show whether the specific pattern matters while preserving the no-shared-pixel property. A separate comparison with no masking and independently augmented views could also help assess the effect of using disjoint views, although it would change both masking and pixel overlap.
- Equal marginals: The score uses which seems to rely on the two masked views having equal cluster marginals. Have you checked the marginals separately for each view? If they differ, it would be interesting to see whether using changes the surprise scores or clustering results.
- Sensitivity to spatial structure: Have you tried randomly permuting pixel locations within each image while preserving its pixel-value histogram? Comparing the surprise scores with those on the original data could help clarify whether the score reflects spatial image structure.
- Equation (64): At , the sigmoid limit is , whereas the strict indicator shown in the equation is zero. The KL term vanishes at equality, so this may not affect the limiting score, but could the equation be clarified? It would also help to specify how the relaxation is handled at boundary values such as .
- Binomial bound: Since is estimated from the same batch used to count matches, could you clarify the conditions under which the binomial tail bound in Equation (5) applies with this estimate? The paper also notes weak dependence across cluster counts; it would be helpful to explain how that dependence affects the interpretation of the summed score.
- Number of runs: Table 1 reports results over ten runs, and Table 2 specifies ten runs for the full model and five for each ablation. Appendix F, however, states that five independent experiments were run for each dataset. Could you clarify which run counts apply to the main results and ablations?
2 comments
Thanks for the new experiments and the clear answers. Most of my questions on the statistical side are resolved.
But my main concerns remain. The random-mask result shows that the gain depends on the chessboard pattern itself, since both masks keep the views independent under the null. This is not explained by the statistical argument of the paper, and the comparison without masking is still missing. On generality, the new experiments vary K and K* within the same three datasets, so the evidence is still limited to small grayscale benchmarks, and the revised abstract and conclusion still claim state-of-the-art results. Also, Appendix L cites Table 1 numbers (98.3 / 64.5 / 94.9%) that do not match Table 1.
So I will keep my rating at 2. I do appreciate the effort you put into the discussion.
Thank you again for your detailed and constructive review. Your suggestions were very concrete, and we have now run the checks you proposed. The revised manuscript reports them in Section 4.4, Table 2 and Appendices H and I. We summarize them below.
Q1: checkerboard ablation
Thank you for this suggestion. We trained with random disjoint half-pixel masks under the same setup (five runs per dataset): each image gets a fresh random set of exactly half of the pixel positions for one view and the complement for the other. This keeps the no-shared-pixel property, but in our runs it gave clearly lower results than the checkerboard (Table 2):
The random mask tends to merge classes. One possible reason is that a random half can leave gaps of several pixels along a stroke, whereas under the checkerboard every missing pixel has its four neighbors in the same view.
Q2: equal marginals (Appendix I)
Because the two patterns are swapped between the views by a per-image coin flip, the two views have equal marginals by construction. To check your stronger question directly, we fixed the white positions to one view and the black positions to the other on every test image:
Q3: sensitivity to spatial structure (Appendix I)
Following your suggestion, we permuted the pixel positions within each test image before masking. This removes 97–99% of the score, and NMI drops below 9%. On i.i.d. pixel noise, the score falls further, to the level of an independent pairing of the views, and no cluster is surprising. The small residual on permuted images is expected, since the two views still share the image's pixel histogram.
Q4: Equation (64)
Thank you, you are right. The limit holds only for q̂ₖ ≠ qₖ; at q̂ₖ = qₖ the sigmoid equals 1/2. Since D(q̂ₖ ‖ qₖ) = 0 at equality, the limit of the soft score still equals the hard score, and we will state this explicitly. For boundary values, the hard score uses 0 · log 0 = 0: a cluster with no match contributes nothing, and any match implies qₖ > 0, so every term is finite. In the relaxation, q̂ₖ and qₖ are clamped to [10⁻¹², 1 − 10⁻¹²] before the logarithms.
Q5: binomial bound with a plug-in pₖ
We agree that, with pₖ estimated from the same batch, the binomial tail bound is an approximation rather than an exact test. Likewise, because of the dependence across clusters, the summed score is best read as a test statistic rather than the log-probability of a joint event. To calibrate it, we used the exact null given the batch marginals: shuffling one view's predictions across images (the "independent pairing" row in Appendix I). Under this null, the score is 0.6–1.2 × 10⁻³ and no cluster passes τ, compared with about 1.4 on real images. So at our batch size (N = 3000), the plug-in estimate and the dependence across clusters appear to have little practical effect. We will add this clarification to Section 3.2.
Generality
We fully agree that our evidence is limited to three small grayscale datasets, and we will word our claims accordingly. Section 4.4 adds two robustness experiments:
Thank you again for the careful reading, including the reference and the run counts, which we corrected earlier. We would be glad to provide any further details.