- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The work analyzes how uniform optimal-transport capacities can distort relational supervision when learned representations have uneven cluster sizes. It proposes PACI, which estimates cluster capacities from the evolving representation and uses them to construct transport assignments and a relational teacher. The paper combines theoretical analysis, long-tailed clustering benchmarks, and controlled interventions to show that matching capacities to representation geometry can improve clustering performance.
Critical strengths
- The experiments and their broadness are meaningful and interesting.
- The ablations and analyzes connects a concrete design choice to both supervision and downstream performance.
Critical weaknesses
- The paper does not quantify the additional training time and memory required by periodic feature extraction, clustering, and transport computation.
- The size of the experiments ends with CIFAR-100 and should be extended to imagenet size.
- Unclear usefulness of the theoretical bound.
- Evaluation focuses on artificially constructed long-tailed benchmarks, leaving the effectiveness of geometry-matched capacities under naturally occurring imbalance insufficiently established.
Questions for authors
- How should the finding that reducing contamination did not improve accuracy affect the proposed explanation of PACI’s gains?
- What evidence distinguishes contamination reduction from other changes induced by the assignment intervention?
- Do the assignment-versus-normalization effects on final accuracy also hold on CIFAR-100 or Tiny-ImageNet?
4 comments
Response to Reviewer PuWS (2/3)
Reply to W1. Training time and GPU memory
PACI's measured step times, including teacher refresh, are close to BYOL. We measured the methods in separate GPU-exclusive processes on one RTX 4090. Both used ResNet-18, batch size 256, AMP, and the same data pipeline and starting optimizer state. Times are mean ± standard deviation over three timing repetitions. Memory is the steady-window peak GPU allocated memory.
| Dataset | Method | Step (ms) | Including refresh (ms/step) | Peak allocated (MiB) | GPU bank (MiB) |
|---|---|---|---|---|---|
| CIFAR-100 | BYOL | 78.60±2.12 | 78.60±2.12 | 1571.58 | — |
| CIFAR-100 | PACI | 75.82±0.44 | 76.20±0.45 | 1654.97 | 82.59 |
| Tiny-ImageNet | BYOL | 67.90±0.12 | 67.90±0.12 | 820.12 | — |
| Tiny-ImageNet | PACI | 67.86±0.48 | 68.84±0.49 | 897.73 | 75.91 |
Complete teacher refresh includes feature extraction, preprocessing, clustering, OT, and bank construction. A subsequent refresh took 1.436±0.045 s on CIFAR-100. At its 50-epoch interval, this adds 0.378 ms/step, about 0.50% of PACI's normal step time. On Tiny-ImageNet, a complete refresh took 2.060±0.026 s and adds 0.981 ms/step under its refresh schedule.
The memory increase is 83.39 MiB (5.31%) on CIFAR-100 and 77.61 MiB (9.46%) on Tiny-ImageNet, largely accounted for by the retained bank. Training warm-up peaks are 3642.02/3719.52 MiB for BYOL/PACI on CIFAR-100 and 2931.37/3006.58 MiB on Tiny-ImageNet. These measurements include the additional relation storage and complete refresh cost requested in the review.
Reply to W2 & W4. Larger datasets and natural imbalance
ResNet-50 on ImageNet-LT and iNature500. We extended evaluation to ImageNet-LT with 1,000 classes and iNature500 with 500 naturally imbalanced classes. All methods use ResNet-50 without external pretraining, with their own warm-up or self-supervised learning stages on the target data. Results are CAA (%).
| Method | ImageNet-LT | iNature500 |
|---|---|---|
| BYOL | 10.6 | 28.1 |
| ConMix | 12.3 | 29.4 |
| MiniClustering (MaxiClustering variant) | 12.7 | 30.7 |
| PACI | 13.4 | 31.2 |
PACI exceeds the strongest comparison by 0.7/0.5 CAA points on ImageNet-LT/iNature500. These results extend the evaluation to a larger backbone, 1,000-class clustering, and natural class imbalance.
Shared-DINO ViT-B/16 on iNature500. We also evaluated the naturally imbalanced dataset under a common ImageNet-pretrained DINO initialization. Initial DINO is the feature-clustering reference before additional adaptation. All metrics are percentages.
| Method | CAA | ACC | NMI | ARI |
|---|---|---|---|---|
| Initial DINO | 29.3 | 36.9 | 65.8 | 13.0 |
| P²OT | 32.8 | 40.2 | 67.1 | 13.4 |
| SP²OT | 34.5 | 38.4 | 67.8 | 13.9 |
| PACI | 36.7 | 39.1 | 67.3 | 14.2 |
PACI improves CAA over the strongest OT baseline by 2.2 points and over the common starting point by 7.4 points. The iNature500 results provide natural-imbalance evidence under both target-data learning with ResNet-50 and shared pretrained initialization with ViT-B/16.
Response to Reviewer PuWS (3/3)
Reply to W3. Interpretation of the finite-batch bound
Theorem 1 connects the full-bank semantic composition analysis to the normalized minibatch teacher used during optimization. It quantifies the effects of relation preprocessing and batch sampling through measurable properties of the relation rows.
We evaluated its terms at the actual training batch size B=256. Here κ measures row concentration and δ measures the change introduced by preprocessing. The overall gap is the absolute difference between full-bank contamination and the Monte Carlo estimate of mean minibatch contamination. The per-anchor gap measures the absolute discrepancy for individual anchors before averaging. Bounds are computed per anchor and then averaged. δ, bounds, and gaps below are in percentage points.
| Dataset | Mean κ | Mean δ | Upper bound | Overall gap | Per-anchor gap |
|---|---|---|---|---|---|
| CIFAR-10 | 7.814 | 0.0784 | 14.13 | 0.026 | 0.991 |
| CIFAR-20 | 10.257 | 0.1135 | 16.60 | 0.519 | 1.246 |
| CIFAR-100 | 8.346 | 0.1052 | 19.10 | 0.587 | 1.524 |
| Tiny-ImageNet | 14.187 | 0.2761 | 20.61 | 0.381 | 1.957 |
The analytical bounds are conservative. Observed overall discrepancies remain below 0.6 percentage points, and mean per-anchor discrepancies remain below 2.0 points across all four datasets. These results show close empirical agreement between full-bank relations and the minibatch teacher used during training.
Does the capacity effect reach the training teacher? We held the final features, prototypes, and transport cost fixed and compared uniform with matched capacities through raw relation H, normalized full-bank relation R, and minibatch teacher T. The uniform-minus-matched contamination gaps were 8.145→8.221→7.992 points on CIFAR-100 and 3.478→3.284→3.688 on Tiny-ImageNet. CIFAR-10/20 also favored matched capacity at every stage. Thus, the capacity-matching advantage persists through preprocessing and minibatch row normalization.
The bound makes explicit how batch size, row concentration, and preprocessing jointly control an upper bound on the full-bank–minibatch discrepancy. Larger batches reduce the sampling term, while more concentrated rows increase it. We will include these measured quantities and their interpretation in the revision so that readers can relate the full-bank analysis to the actual training configuration.
Thank you again for your time and helpful comments. We hope these additions address your concerns and would be happy to clarify any remaining questions.
Thank you for the detailed response and substantial additional experiments. My concerns have been sufficiently addressed, and I maintain my weak acceptance recommendation.
Response to Reviewer PuWS (1/3)
Thank you for your thoughtful review. We appreciate your recognition of the breadth of our experiments and the analyses connecting capacity choices to supervision and downstream performance. We added semantic interventions, four-arm training on larger datasets, cost measurements, and evaluations at larger scale and under natural imbalance. We address your comments and questions below.
Reply to Q1 & Q2. Semantic supervision and the filtering result
Why filtering gives small accuracy gains. Filtering removes wrong-semantic relations and also discards useful correct-semantic supervision. We compared stability filtering with label-assisted semantic repair. Repair changes which samples receive the relation weights while preserving each row's weight multiset, entropy, and L2 norm.
Preserving these row statistics provides a targeted test of semantic connections. Similarity and graph structure also change under rewiring. We interpret the results as evidence that semantic connections contribute to learning, alongside the overall effect of capacity choices measured by the four-arm experiment.
The table reports contamination (ΔC) and CAA changes in percentage points. Retention is the percentage of original correct-semantic mass kept by filtering.
Repair produces larger accuracy gains with smaller contamination reductions. Random filtering lowers CAA by 0.82/0.48 points on CIFAR-10/20. These comparisons explain the stability-filter result through the loss of useful supervision and support preserving correct connections when removing incorrect relations.
Direct damage and repair on larger datasets. We tested two levels of semantic damage and a repair condition on CIFAR-100 and Tiny-ImageNet, preserving the same per-row weight statistics. Changes below are relative to the corresponding control, in percentage points.
Greater semantic damage produces larger losses in all three clustering metrics on both datasets, and repair improves all three. These bidirectional effects connect the teacher's semantic relations to final performance while retaining its row-weight statistics. Together with the filtering comparison, they support a contribution of semantic supervision to PACI's gains.
Reply to Q3. Assignment and normalization on larger datasets
Yes. Assignment capacity also has the larger effect on final clustering accuracy on CIFAR-100 and Tiny-ImageNet. We extended the paper's four-arm comparison by independently using estimated or uniform capacities for transport assignment and relation normalization.
The table reports final CAA loss relative to EE, in percentage points. Positive values measure the accuracy decrease from EE.
Replacing estimated assignment capacity with uniform capacity lowers CAA by 3.981/3.667 points on CIFAR-100 under estimated/uniform normalization. The corresponding losses on Tiny-ImageNet are 2.756/2.433 points. Changing normalization produces absolute CAA differences of at most 0.170/0.429 points on the two datasets.
The assignment effect therefore dominates under both normalization choices on both datasets, extending the original final-accuracy finding to larger class counts.