acceptodds
Under review as a conference paper at ICLR 2027

Coverage Risk Minimization: A Unified Framework for Long-Tailed Deep Fine-grained Clustering

Abstract

Long-tailed distributions and fine-grained semantics frequently co-occur in real-world visual data: tail classes lack sufficient data support, while semantically similar categories share highly similar visual representations, rendering reliable intra-class aggregation and inter-class separation considerably more challenging. To this end, we revisit fine-grained long-tailed clustering from the perspective of forward representation optimization and propose **D4C**, a hierarchical coverage risk optimization framework with density-aware forward control that continuously shapes clustering-conducive geometric structures across different representation levels. Theoretically, starting from the PAC upper bound of coverage risk, we isolate an optimizable average term and a geometric bottleneck governed by the worst-case covering radius, and adopt optimal transport and geometric repulsive potentials as their differentiable surrogates to coordinate sample coverage and center separation. Concurrently, we introduce a vMF mixture model at each representation level to characterize the local density structure, guiding samples to aggregate toward local density peaks. To tackle imbalanced support among long-tailed categories, we further design long-tail-aware transport marginals, and over-segmentation with Ward merging and EMA teacher freezing keep coverage-risk minimization stable. Extensive experiments on multiple long-tailed and fine-grained clustering benchmarks demonstrate that D4C achieves state-of-the-art performance.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Longtailed Clustering, Optimal Transport, vMF Mixture Model, PAC

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_StAvOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes D4C, a framework for long-tailed clustering that integrates optimal transport, layer-wise density modeling, geometric repulsion, and prototype merging.

Critical strengths
  1. The proposed method jointly addresses class imbalance, local density variations, and prototype separation.

  2. The experiments cover multiple benchmark settings and evaluate performance using complementary metrics.

  3. The ablation studies, training analyses, and theoretical discussions provide useful insights into the proposed framework.

Critical weaknesses
  1. The methodological novelty needs to be more clearly articulated. Sinkhorn assignments, vMF density modeling, geometric repulsion, and prototype merging are widely adopted techniques. While the coverage-risk perspective offers a coherent integration of these modules, the paper has not sufficiently demonstrated what algorithmic properties or theoretical conclusions this formal framework brings that differ from existing methods. Therefore, the reviewer has concerns regarding the contributions of this work.

  2. The coverage-risk theory does not establish a guarantee for the implemented method. The bound does not account for learning the representation from the same data, and the paper does not show that its Sinkhorn-target cross-entropy reduces the bounded nearest-center cost. Center separation also does not generally guarantee a smaller worst-case covering radius.

  3. The fine-grained long-tailed claim lacks targeted validation. The paper reports overall results on CUB, Dogs, and Flowers, but its key component ablations are conducted on CIFAR-10. Without fine-grained ablations and tail-class analyses, it is unclear which components improve this specific setting.

  4. The benefit of layer-wise control is not isolated. The paper does not directly show that fine-grained information is progressively lost across layers, or compare its layer-wise design with a final-layer-only alternative of matched capacity.

  5. The baseline comparisons lack a clear common protocol. The tables do not identify which results come from published sources and which are reruns, or consistently clarify the encoders and evaluation settings used for each method.

  6. The reference list contains minor inaccuracies. For example, “Self-labelling via Simultaneous Clustering and Representation Learning” is cited as a 2019 arXiv preprint, although the work was subsequently published at ICLR 2020.

Questions for authors
  1. Could the authors clarify the sources and experimental settings of the MiniClustering results reported in Tables 3 and 4? For example, the reported ACC on CIFAR-10 at an imbalance ratio of 5 is 62.3, compared with 74.3 in the published MiniClustering paper. This discrepancy changes the relative ranking between MiniClustering and D4C, which achieves an ACC of 68.0. If the reported MiniClustering results were obtained through independent reruns, why was this baseline rerun while other baselines appear to rely on previously published results? Please clarify the pretrained encoder, data split, and evaluation protocol used, and explicitly distinguish published results from independently reproduced results.

  2. Could the authors provide controlled ablation studies of the three key components on one or two fine-grained long-tailed benchmarks to better substantiate their individual contributions in the target setting?

6 comments

W1 : The Coverage-Risk Framework Licenses Predictions the Modules Alone Do Not

The novelty is not Sinkhorn, vMF, or the determinant kernel themselves, but how constraints at different granularities jointly construct the hierarchical coverage geometry.

First, a risk decomposition assigns each module a place in the bound. For any layer the coverage risk is Rτ(C)=Pr⁡x∼P[min⁡mR_\tau(C)=\Pr_{x\sim P}[\min_m||x−cmx-c_m||>τ]> \tau]. The finite-sample bound (Eq. 10, adaptive version Eq. 17 in Appendix B) decomposes it into an optimizable average term Ψ(C)=1N∑imin⁡m\Psi(C)=\frac{1}{N}\sum_i\min_m||xi−cmx_i-c_m||2^2 and a geometric bottleneck governed by the worst-case covering radius rmax⁡(C)r_{\max}(C) (Eq. 11). Optimal transport minimizes Ψ\Psi, the geometric repulsive potential keeps the center set non-degenerate, and the vMF pull improves local density geometry, which empirically lowers Ψ\Psi and rmax⁡r_{\max}. A list of tricks description provides no such assignment and offers no prediction about which module can be removed.

Second, the theoretical statements are ones the components alone do not yield, and they differ from existing mechanisms. Any global-assignment cost is a differentiable upper bound on the average coverage risk (Eq. 2). The log-det repulsion yields a quantitative lower bound on the minimum inter-center distance dmin⁡d_{\min}, transferred to the output prototypes via Lemma E (Appendix D). The long-tail-aware marginal mk∝πkαm_k\propto\pi_k^{\alpha} is derived rather than assumed, because under long tails the uniform marginal is statistically mismatched with the class support. Existing deep clustering confines rebalancing and separation to the final feature space. D4C instead places a class-level long-tail marginal at the output layer, uses peak-level masses in the hidden layers so that class priors and local density structures are never conflated, and ties prototype separation to the coverage-radius analysis through the log-det bound. The logical status of each statement is given in our reply to W2.

Third, the framework predicts algorithmic properties that the experiments confirm. Because each objective controls a different term of the bound, the framework predicts complementarity rather than redundancy, and Table 5 confirms it. Removing Lgeo(l)L_{geo}^{(l)} or Lvmf(l)L_{vmf}^{(l)} collapses training to 32.7 and 38.2 ACC even though over-segmentation remains active, so no other module can substitute. It also predicts that coverage risk is necessary but not sufficient for clustering accuracy, a falsifiable statement verified in Appendix L and Fig. 7, where Ψ\Psi, rmax⁡r_{\max} and RτR_\tau keep improving while ACC saturates. This statement justifies reporting checkpoints from the stable window rather than the final epoch.

W2 : The Theory Covers Coverage Risk in a Fixed Representation and Nothing Stronger

The coverage-risk theory gives a per-layer bound on a fixed representation, and this reply states exactly what it proves and what it does not.

The bound characterizes coverage risk in a given representation space, not an end-to-end guarantee for joint training. Appendix B (Eq. 17) upgrades the fixed-center bound (Eq. 10) to a uniform bound over all center sets CM=(Sd−1)MC_M=(S^{d-1})^{M}. It holds for any data-dependent center set C^\hat C, adds only an O(dM/N)O(\sqrt{dM/N}) complexity term, and keeps Ψ(C^)\Psi(\hat C) as the optimizable leading term. Both bounds are per-layer statements conditional on θ\theta. They cover the features f(l)=hθ(f(l−1))f^{(l)}=h_{\theta}(f^{(l-1)}) the losses observe, not the dependence of θ^\hat\theta on the same training sample, which would require uniform control over the forward-stack parameter class that we do not claim. The revision has made this conditioning explicit.

For the Sinkhorn cross-entropy we separate three statements of distinct logical status.

  • Provable. The plan tikt_{ik} is a feasible coupling with row marginal 1/N1/N and column marginal mkm_k, so by Eq. 2 the teacher-plan cost ∑i,ktik\sum_{i,k}t_{ik}||$f_i-w_k||$$^{2}\ge\Psi(W)$ upper bounds the nearest-center cost, and on the sphere ||f−wf-w||2=2−2⟨f,w⟩^{2}=2-2\langle f,w\rangle it is affinely equivalent to the inner-product cost of LskL_{sk}.
  • Not claimed. We do not claim that minimizing LskL_{sk} monotonically decreases this bound. Its gradient involves pik−tikp_{ik}-t_{ik}, which is not the gradient of the fixed-plan transport cost, so the two descent directions are not interchangeable.
  • Experimental. Fig. 7 shows Ψ\Psi, rmax⁡r_{\max} and RτR_\tau decreasing in both layers throughout training, and the revision has been added the teacher-plan cost curve alongside LskL_{sk}.

On separation versus covering radius, the revision has stated three claims of distinct logical type. OT provides an optimizable upper bound on the average nearest-center cost (Eq. 2). The geometric repulsive potential provides a deterministic lower bound dmin⁡2≥σ2ln⁡11−det⁡(S)d_{\min}^{2}\ge\sigma^{2}\ln\frac{1}{1-\det(S)} (Appendix D), transferred to the output prototypes via Lemma E. This non-degeneracy is necessary, not sufficient, for improving the worst-case covering radius, and the equatorial-ring counterexample in Appendix E documents the lack of a monotone equivalence.

Finally, the entropic OT equivalence of Appendix C holds for the full-dataset EM posterior qEMq^{\mathrm{EM}} at convergence, where ∑iqimEM=Nωm\sum_i q^{\mathrm{EM}}_{im}=N\omega_m. The mini-batch assignment q^\hat q used in the training regularizer is a detached approximation that does not satisfy the same global marginal constraints, and the revision has stated this boundary explicitly.

W3 & Q2 : All Components Contribute on Oxford Flowers and Tail-Sensitive Metrics Suffer the Most

We ran the controlled single-variable ablation of every component on Oxford Flowers (imb_ratio = 10, frozen MoCo v3, the exact protocol of Table 1), where ConMix leads in ARI (49.3 vs. 47.2), so it is the least favorable test environment for our components.

All three key components contribute, consistent with CIFAR-10 (Table 5). Removing the density-aware forward layers costs the most at −7.1-7.1 ACC and −18.3-18.3 ARI, so fine-grained long-tailed data depends more, not less, on layer-wise geometry control, and final-layer rebalancing alone cannot recover structure lost earlier in the forward pass.

The tail evidence is directly readable from CAA, the unweighted per-class recall. Removing LvmfL_{vmf} costs 5.3 CAA, prototype anchoring 6.5, LTAM (α=0\alpha = 0) 6.0, and density-aware forward layers 8.7, while head-dominated ACC stays flat or even rises for w/o EMA (53.5) and w/o LvmfL_{vmf} (53.4). This is the head-masks-tail failure mode the long-tailed design targets, and these components contribute through class-balanced tail-sensitive performance rather than head-dominated ACC.

The components are complementary rather than redundant. Each controls a different term of the coverage-risk bound, and each removal produces a distinct measurable loss, mirroring the collapse on CIFAR-10 when both LgeoL_{geo} and LvmfL_{vmf} are removed (Table 5).

HyperparameterACCCAANMIARI
Full D4C53.051.273.247.2
w/o prototype anchoring49.844.769.640.9
w/o long-tail marginal(α=0\alpha = 0)50.445.269.940.2
w/o density-aware forward layers45.942.564.928.9
w/o EMA53.548.970.242.7
w/o LgeoL_{geo} only51.347.770.543.0
w/o LvmfL_{vmf} only53.445.970.443.2

This table has been added to the revision.

Reviewer_DFKWOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes D4C for long-tailed fine-grained clustering. It refines frozen pre-trained features through hierarchical coverage risk optimization, combining optimal transport, geometric repulsion, vMF density pulling, and prototype anchoring, with over-segmentation and Ward merging.

Critical strengths
  • Addressing long-tailed distributions and fine-grained semantics together is practically valuable.
  • The motivation to correct the mismatch between uniform transport marginals and imbalanced class distributions is reasonable.
  • The experiments cover multiple benchmarks and provide extensive comparisons with existing methods.
Critical weaknesses
  • The method refines frozen pre-trained features, and its effectiveness depends on their quality. Its performance with encoders pre-trained on class-imbalanced data remains unclear.
  • Hyperparameter analyses are limited to CIFAR-10 and do not sufficiently justify the different settings used on fine-grained datasets.
  • It is unclear whether the baselines in Table 2 use the same DINOv3 features.
  • The analysis of over-segmentation is limited to comparing the true class count with twice that count.
  • Results are reported from a single random seed, limiting evidence of stability.
  • The accuracy of the estimated class prior under different imbalance ratios is insufficiently analyzed.
Questions for authors
  • Can the authors clarify whether the baselines in Table 2 are re-run with DINOv3 features or taken from their original papers?
  • Can the authors extend the analysis of the over-segmentation factor and justify the hyperparameter settings used on fine-grained datasets?
  • Can the authors report results across multiple random seeds to verify stability?
  • Can the authors analyze how class-prior estimation accuracy changes with the imbalance ratio?
  • Can the authors evaluate the method using encoders pre-trained on class-imbalanced data?
6 comments

W1 & Q5 : D4C Ablation Value on a Long-Tail-Pre-Trained Encoder

We added this control : following SCAN's SimCLR specification, we pre-trained ResNet-18 from scratch on the long-tailed CIFAR-10 training set itself (imb_ratio=10), froze the encoder, and ran K-means and D4C on exactly the same features. The value-selection rule for D4C is the same as in the main text.

Ablations on CIFAR-10ACCCAANMIARI
K-means41.241.846.829.8
D4C, ep.40 (CAA peak)50.351.755.243.6
D4C, ep.80 (end of plateau)47.446.853.536.1
D4C (Original)70.667.565.159.9

Backbone quality determines the absolute upper bound. Long-tailed pre-training drops D4C from 70.6 to 47.4 ACC (−23.2-23.2), CAA −20.7-20.7, and ARI −23.8-23.8, but NMI by only 11.6, less than half of the ACC/ARI drops-so boundary and tail discriminability erode while the dominant mode structure largely survives.

The decisive point is that the mechanism still holds on low-quality inputs. The contribution is not ''obtaining a good backbone'' but ''correcting coverage geometry layer by layer on top of any given input geometry.'' Input quality is measured by the K-means row, and D4C adds value stably on top of it. This reading matches the oracle decomposition in Appendix H. The backbone sets the upper bound and the clustering mechanism determines how closely it is approached.

The direction of decay within the plateau is also informative. From epoch 40 to 80, CAA drops by 4.9 and ARI by 7.5 while ACC drops by only 2.9, i.e., tail classes are gradually reabsorbed by neighbouring clusters (Appendix M). It also shows that the main-text rule of reporting the end of the plateau rather than the peak is conservative, and the gains here are not artifacts of peak selection.

W2 & Q2-2 : Justifications of the Settings Changed on Fine-Grained Datasets

There are four differences in total between the fine-grained configuration and the CIFAR-10 default : hidden width 512→1024512\to1024, βhid ⁣:0.2→0.4\beta_{\mathrm{hid}}\colon 0.2\to0.4, ξ ⁣:0.6→0.5\xi\colon 0.6\to0.5, and disabling over-segmentation (ρ=1\rho=1). We sweep the width, βhid\beta_{\mathrm{hid}} and ξ\xi differences on Oxford Flowers (imb_ratio=10) (table below) and justify the remaining difference (ρ=1\rho = 1) in the text. All other hyperparameters follow the CIFAR-10 defaults. ''Oracle'' denotes the experiment in the appendix that uses ground-truth labels. Rows marked with an '' * '' are the original values from the paper.

HyperparameterValueACCCAANMIARI
hidden_dim/n_layers512/252.450.769.845.3
1024/2 *53.051.273.247.2
βhid\beta_{hid}0.249.644.369.538.1
0.4 *53.051.273.247.2
0.652.546.070.243.3
ξ\xi0.5 *53.051.273.247.2
0.650.645.569.841.4
OracleTRUE97.897.797.992.4
FALSE *53.051.273.247.2

The width 512→1024512\to1024 is capacity compensation. MoCo v3 ResNet-50 outputs 2048-dimensional features, whereas the SimCLR ResNet-18 for CIFAR-10 is 512-dimensional, so a 512 hidden layer is dimension-preserving on CIFAR-10 but a bottleneck on fine-grained data (52.4/50.7/69.8/45.352.4/50.7/69.8/45.3), and 1024 mainly restores NMI/ARI (+3.4+3.4/+1.9+1.9). The choice follows input dimensionality, not a dataset-specific search.

βhid=0.4\beta_{\mathrm{hid}}=0.4 is the peak of a concave response. On Oxford Flowers the peak budget is capped at 200 while the mass of 102 classes is dispersed over many low-mass peaks, so each peak receives a smaller π~m\tilde{\pi}_m (Eq. 9) and the same β\beta exerts weaker anchoring force than on CIFAR-10-hence a larger β\beta is needed. The sweep shows 0.2 under-anchors (49.6/44.349.6/44.3), 0.4 is the peak, and 0.6 over-anchors with a clear CAA drop (46.0, −5.2-5.2), as tail-class peaks shared by multiple classes are pulled toward different prototypes.

ξ=0.5\xi = 0.5 : fine-grained features converge faster, so the teacher should be frozen earlier. MoCo v3 fine-grained features are already close to the clustering solution at initialization, and ξ=0.6\xi = 0.6 lets the teacher follow the later-drifting Sinkhorn target (''tail classes are gradually re-merged after the plateau'', Appendix L/M), whereas ξ=0.5\xi = 0.5 locks the target within the plateau. In the sweep, 0.5 is superior on all four metrics (ACC +2.4, CAA +5.7, NMI +3.4, ARI +5.8), and Oxford Flowers shows no collapse under any configuration swept here.

Disabling over-segmentation is a principled choice for fine-grained data. Over-segmentation exists because at K=K∗K = K^{*} adjacent separable pairs are forced to share prototypes (Sec. 3.4), but fine-grained datasets have many classes and few samples per class, so forced sharing rarely occurs. These datasets have few samples per class overall (8,189 images across 102 classes in Oxford Flowers, 11,788 across 200 in CUB-200-2011, and 20,580 across 120 in Stanford Dogs), and at ratio 10 the Oxford Flowers tail classes retain only single-digit samples, which leaves extra prototypes without statistical support, so we set ρ=1\rho = 1 and do not sweep it on fine-grained data.

W3 & Q1 : Compares Native Protocols Rather Than a Shared Frozen DINOv3 Backbone

We thank the reviewer for pointing out that the encoder settings in Table 2 are insufficiently clear.

First, the baselines in Table 2 do not uniformly use the same frozen DINOv3 features. D4C extracts initial representations with DINOv3 and optimizes the clustering structure with a frozen encoder, while the other baselines follow their original training protocols. Tiny-ImageNet-LT uses values reported in the MiniClustering paper, and the ImageNet-100-LT and ImageNet-50-LT results are independent runs from public implementations.

If we uniformly changed them to frozen DINOv3 and compared again, what is compared would no longer be the strict original methods. ConMix jointly learns representations through SDCLR and subsequent ConMix stages; LFSS relies on a trainable online encoder, a momentum-updated key encoder, and representation stability across training stages; MiniClustering also explicitly includes the encoder and two clustering heads in backpropagation. Converting these methods to frozen-encoder versions would not represent their original methods.

We acknowledge that the current Table 2 is not a strictly controlled-variable experiment and cannot be used to attribute all performance differences solely to D4C's algorithm design. In the revised version, we have clarified the source of each baseline result and accurately define the table as a performance reference under the native training protocols of different methods. We wish to emphasize that D4C can obtain competitive clustering performance through subsequent coverage-risk refinement without updating a large-scale pre-trained encoder, which constitutes a different application setting and optimization setup from end-to-end representation learning methods. For the MiniClustering rows of Tables 3 and 4 see our reply to StAv W5.

To preempt a likely follow-up, we clarify the MiniClustering (SimCLR) rows of Tables 3 and 4. That row is initialized from SCAN's public SimCLR weights and keeps the original encoder fine-tuning, and our reproduction came close to the published values, so the substitution is a faithful same-backbone reference. Full sources are given in our reply to StAv W5.

Reviewer_STnNOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper studies the problem of clustering with imbalanced data. The proposed method is justified as minimizing the clustering coverage risk, ensuring that representations remain close to the respective cluster centroids. The proposed architecture uses a stack of clustering layers, each of which extends vMF mixture models by (1) adjusting the mixture weights to reduce the influence of dominant clusters, (2) a geometric repulsive potential to keep cluster centroids well-separated, (3) a regularizer to keep cluster centroids in the hidden layers close to the final prototypes of the output layer. To increase clustering stability, these prototypes are smoothed via an exponential moving average, and to favor the discovery of small cluster, training is performed with a larger number of clusters which are aggregated via hierarchical clustering using Ward linkage as a final post-training step. Several experiments show the method achieving higher performance compared to alternative baselines.

Critical strengths

None.

Critical weaknesses

The method relies on knowing the true number of classes K* as well as their true marginal distribution to instantiate over-segmentation (L209 and L300). This not only makes the method inapplicable to real-world unsupervised learning where classes and their distribution are truly unknown, but also makes the problem of clustering fine-grained, long-tailed data considerably easier. It is therefore unclear how much of the performance gains come from algorithmic improvements and how much are due to merely knowing true class priors. Moreover, the method relies on a large number of hyper-parameters while offering no guidance on tuning them when the true class priors are unknown.

Some ideas are poorly justified, and their necessity is called into question by the fact that using more than one hidden layer has considerable negative effects on performance (Fig. 2d) . For example, why are "Long-tail-aware marginals [...] applied only to the final-layer risk objective" (L202)? For very rare classes, could this unjustified architectural choice be the reason why "adjacent yet separable class pairs are often forced to share a single prototype" (L300)? The "geometric repulsive potential" and over-clustering seemingly solve this very problem; why are both of them needed? The repulsive potential prevents "several centers [to] nearly coincide" (L214), but later on over-clustering is motivated as "separable class pairs [...] often share a single prototype", suggesting that the repulsive regularizer does not work very well. In fact, the connection between coverage risk and the final clustering performance is broken by the introduction of over-clustering, since increasing the number of clusters is a trivial way to reduce coverage risk without necessarily improving the final clustering performance. L294 seems to suggest that prototype anchoring to "propagate the density structure to the classification layer" (L265) is needed as a solution, unfortunately Lemma E mentioned there is nowhere to be found in the appendix. "Square root weighting of peak masses" to "suppress the monopoly of dominant peaks over gradient directions" (L252) seems to be another trick made necessary by inappropriate cluster weighting in the hidden layers. Moreover, changing vMF weights in such an arbitrary way calls into question the entire theoretical grounding of the paper, since vMF itself is justified in light of optimal transport and thus coverage risk minimization ("In hidden layers this proxy is the per-batch posterior of Sec 3.2", L195, and "It therefore instantiates the same cost as the optimal transport objective of Sec. 3.1", L250).

Perhaps the answer to some of these questions is in the paper, but the exposition is extremely unclear, not following any apparent logical order, frequently name-dropping new concepts and minor details without proper explanation or connection to previous ideas, as is typical of unpolised LLM writing. For example, the very first sentence of the method section "recovered at evaluation as K∗ groups by Ward merging (Sec 3.4)" already introduces new concepts without any justification or explanation, and similarly many terms in the next paragraph, starting with "D4C minimizes the coverage risk" (L140) can only be understood after reading the rest of section 3. Another key aspect of the method, namely constraining representations on a sphere, is introduced there without any explicit motivation. Most importantly, the relation between coverage risk and optimal transport is introduced first in section 3.1, but the coupling weights \Pi in equation 2 are not used anywhere else, leaving the reader confused as for their need. The following explanation starting at L190 is highly opaque without first understanding the remainder of section 3 and appendix F, and the key motivation for optimal transport and its connection to coverage risk is only justified in the middle of the method section. The Sinkhorn targets, another key architectural decision, are only mentioned in passing while explaining optimal transport (L195) without justification or even definition. These are only a few of the offending examples, but almost every part of the paper, from introduction to appendix, is very difficult to understand without substantial effort and re-reading.

Questions for authors

How can the method be tuned and evaluated without knowing the true class priors? How does the method perform on the benchmarks in the paper?

Could you clarify and justify key aspects of the method in light of the questions raised in the weaknesses box?

6 comments

W1 & Q1: Clarification on the Use of K* and the Estimated Marginals

All baselines are given the true number of classes KK. The MiniClustering paper states that ''The number of target-clusters KK, as in most deep clustering methods (...citations...), is a known prior, corresponding to the true number of classes'' (Page 8, Implementation Details, L3) and cites ConMix/LFSS/SCAN as precedents; our setting matches theirs.

No ground-truth class frequency enters any loss. The predefined distribution at L209 is not any ground-truth prior. It is built from the data itself. The final-layer marginal uses the FCM pseudo-class mass on the KK prototypes estimated on frozen features during warm-up, and the hidden-layer marginals use the vMF posterior peak masses. Specifically, π^k\hat\pi_k is the FCM pseudo-class mass on the K=ρK∗K=\rho K^* prototypes from warm-up. The blend πk=(1−η)π^k+η/K\pi_k = (1-\eta)\hat\pi_k + \eta/K with η=0.5\eta = 0.5, raised to the power α\alpha, gives the Sinkhorn column marginal bk∝πkαb_k \propto \pi_k^{\alpha}. The geometric prior πk∝r−k/(K−1)\pi_k \propto r^{-k/(K-1)} at L209 belongs to an unused code branch and never entered any reported run. The revision has state that L209 refers to the FCM-constructed final-layer marginal, with vMF masses in the hidden layers.


W2: Hyperparameter Settings are Independent of the True Class Prior

K∗K^* is a protocol prior shared by all methods, not information specific to ours, and we document this in our reply to W1. The paper already contains the control that answers this. With the estimated class distribution prior removed (α=0\alpha = 0), D4C still reaches 66.4 ACC / 61.6 CAA on CIFAR-10 (ratio 10), above the strongest baseline at 59.9 / 58.2.

K∗K^* has exactly three uses in D4C, none of which injects a class frequency into a loss, as shown in our reply to W1. The first is a capacity hyperparameter whose sensitivity is covered by ρ\rho, and the latter two are evaluation steps.

As an auxiliary diagnostic we replace Ward merging with DBSCAN at evaluation, to test whether D4C features carry local density structure without hierarchical aggregation. This row is a qualitative check, not a magnitude comparison, since DBSCAN keeps 17 sub-clusters instead of 10 classes and ACC/CAA reflect a different partition granularity. Both runs use the same unsupervised kk-NN elbow rule for eps. The decisive contrast is the first row of the table, where DBSCAN on frozen SimCLR features is near random while D4C features form separable sub-clusters, so D4C learns more density-separable representations even before Ward aggregation.

CIFAR-10ACCCAANMIARI
SimCLR + DBSCAN25.911.25.9-1.2
Replace Ward with DBSCAN73.066.158.835.0
D4C (Original)70.667.565.159.9

Tuning does not rely on the true prior, and the estimated prior enters through a single fixed knob. Only α\alpha is connected to the estimated prior, with the fixed default 0.4 shared by all datasets and never tuned per dataset. Across the full Fig. 2c sweep our ACC stays above the strongest baseline, so the default lies on a plateau and selecting it needs no true prior. The ρ\rho knob is covered in our reply to DFKW W4, and the remaining knobs (βhid\beta_{hid}, λhid\lambda_{hid}, width) are swept in Fig. 2. All defaults are fixed before training and transferred across datasets, and the fine-grained changes follow input dimensionality rather than prior information, see DFKW W2. The α=0\alpha = 0 endpoint is the uniform marginal, which depends on no prior estimate at all and serves as the prior-free fallback.


W3: Sensitivity of the Multi-Level Design to Depth

''Hierarchical'' means geometric control applied at several representation levels, not ''deeper is better''. The three levels are the encoder output, one hidden layer, and the output/prototype layer. The row of Fig. 2d that tests this claim is 0/1 vs. 512/2: disabling the hidden layer drops ACC from 70.6 to 62.7 (−7.9-7.9). A degradation at greater depth therefore does not imply that multi-level control is ineffective; what the comparison tests is the presence of hidden-layer geometric control.

The 512/3 row was obtained without re-tuning, and its total geometric weight grows linearly with depth. The hidden-layer terms of Eq. 13 accumulate as λhid∑l=1L−1\lambda_{hid}\sum_{l=1}^{L-1}: when LL goes from 2 to 3 with λhid\lambda_{hid} unchanged, the combined weight of LgeoL_{geo} and LvmfL_{vmf} doubles. The 512/3 row therefore changes depth and total geometric constraint strength at once and cannot serve as evidence that depth itself is harmful.

Layers L=2L=2 is fixed on all datasets and was never tuned per dataset. We do not claim that deeper stacks are unusable, only that the default weights are set for L=2L=2.


W4: Scope of the LTAM(Long-Tail-Aware Marginals)

LTAM has nothing to operate on in the hidden layers. Hidden-layer OT runs between samples and MM vMF density peaks with M≠KM \ne K, and peaks relate to classes many-to-many head classes typically form several peaks while tail classes share peaks-so there is no ''hidden-layer prototype of class kk'' for a KK dimensional class marginal to assign mass to. The term amk(l)a^{(l)}_{mk} in Eq. 9 encodes precisely this many-to-many soft correspondence; imposing a class-level marginal over peaks would overwrite the data-estimated peak masses ωm\omega_m with a guess and conflate semantic balancing with density modeling.

Therefore, the placement of LTAM follows from where a fixed-marginal mismatch actually exists, which is not an arbitrary choice.


W5: Prototype Sharing and the Placement of LTAM

Prototype sharing arises when the number of prototypes equals the number of classes, independently of the form of the marginal. With K=K∗K=K^* the prototype budget is fully consumed, so at a fixed KK changing α\alpha changes no prototype count and cannot remove sharing, as measured by the w/o over-segmentation row of Table 5 (52.3 ACC). Sharing is determined by the prototype count, not by where LTAM is imposed.

What LTAM changes is an aggregate quantity, not the per-sample cost. It only replaces the final-layer column marginal with mk∝πkαm_k\propto\pi_k^{\alpha}, i.e.\ the total mass each prototype receives; the sample--prototype cost ⟨fi,wk⟩\langle f_i,w_k\rangle is unchanged, and individual assignments are affected only indirectly, as the result of solving that OT problem. The prototype count is set by ρ\rho and does not pass through LTAM, so moving LTAM from the final layer to a hidden layer would reduce no sharing.

Sharing has two sources: a rare class with too few samples does not support an independent prototype, and neighbouring class prototypes may lie too close in feature space. The first cannot be resolved at the prototype level; the second is handled by over-segmentation, which provides extra prototypes before Ward merging back to KK. LTAM governs mass allocation and is orthogonal to both.

LTAM's necessity is given by the α=0\alpha=0 row of Table 5: removing it lowers CAA from 67.5 to 61.6.

W6: Complementary Roles of Geometric Repulsion and Over-Segmentation

Over-segmentation is not a remedy for a failing repulsion. If repulsion really did not work, over-segmentation should be able to substitute for it; the opposite holds. In Table 5 the w/o LgeoL_{geo} row still retains over-segmentation (K=20K = 20) and still collapses to 32.7 ACC, while keeping repulsion but removing over-segmentation gives 52.3. Neither mechanism substitutes for the other. The prototype sharing the reviewer mentions has a fixed cause, equality of prototype and class counts at K=K∗K = K^*, as shown in our reply to W5, so it is not evidence that repulsion fails.

Over-segmentation is motivated by insufficient representational capacity at K=K∗K=K^*, not by reducing coverage risk. ρ\rho is fixed before training and takes no part in optimization; all evaluation is performed after Ward merging back to K∗K^*. Were the gain due to reporting more clusters, the two variants would not differ by more than 18 points after the cluster count is equalized.

We never claim that coverage risk is equivalent to clustering quality. Appendix L and Fig. 7 state that it is necessary but not sufficient, and the falsifiable status of this claim is discussed in our reply to StAv W1. The theory clarifies what each term controls, formalizing the design principle rather than guaranteeing performance.


W7: Notational Correction for Lemma E

We thank the reviewer for pointing this out. The cited content does exist in Appendix E, in the ''Separation transfer'' paragraph; the label was not rendered, so the main-text reference is dangling. In the revision we have fixed it.


W8: Motivation of the Square-Root Peak-Mass Weighting

The hidden-layer peak masses need no correction, so the ''patch'' framing does not apply. ωm\omega_m is the maximum-likelihood vMF estimate at the EM fixed point (Appendix C), and it is neither modified by the square root nor involved in computing π~\tilde\pi, which is a separate coefficient applied only to the downstream LvmfL_{vmf} and LprotoL_{proto}. The square root is a design choice, not a remedy for a defective weighting.

What it governs is how the auxiliary losses budget gradient across peaks. LvmfL_{vmf} and LprotoL_{proto} are shaping terms, not likelihood terms: their total magnitude is set by λhid\lambda_{\mathrm{hid}} and βhid\beta_{\mathrm{hid}}, but how that budget is split among peaks is a separate design decision. Under linear mass weighting, a head peak's mass share becomes its share of shaping strength (4: 1), leaving tail peaks almost unshaped; the square root compresses this to 2: 1, preserving the ordering while retaining substantial shaping strength for tail peaks.

It belongs to the same power family as the final-layer α\alpha, and the fixed midpoint is deliberate. The family πβ\pi^\beta reduces to uniform at β=0\beta = 0 and to mass-proportional at β=1\beta = 1. The final-layer α\alpha is adjustable because the warm-up provides a data-estimated class marginal for interpolation. The peak layer has no such estimate, so the fixed midpoint 1/21/2 is used, never tuned on any dataset and not exposed as a hyperparameter.

The revision has stated explicitly that the square root compresses the spread of per-peak shaping strength relative to linear mass weighting.


W9: Relation between the Peak Reweighting and the vMF-OT Grounding

The theoretical concern rests on a notation collision, which our code audit has located. The paper uses a single qimq_{im} for two different quantities, and the vMF weights are untouched in the implementation. The revision has been added the missing sentence distinguishing the two.

The two quantities are the global EM posterior and a per-batch assignment. qEMq^{EM} is the full-dataset EM posterior, satisfies ∑iqimEM=Nωm\sum_i q^{EM}_{im} = N \omega_m, and is the quantity characterized by the Appendix C proposition, used to estimate the peak directions μm\mu_m and masses ωm\omega_m. q^\hat q is a detached temperature-scaled softmax computed per batch (T=0.15T = 0.15) that only decides batch-internal membership and is not an OT plan under the same marginal constraints.

The square-root weight only rescales the downstream regularizers once both posteriors are available. π~\tilde\pi is computed from the column sums of qEMq^{EM} without gradient and held constant during training, so it modifies neither ωm\omega_m nor qEMq^{EM} and does not enter q^\hat q. The vMF weights are untouched, and the Appendix C proposition and the coverage-risk chain remain valid.

We have clarified the notation and the related descriptions to avoid ambiguity, and add a brief note in the appendix where necessary.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.