acceptodds
Under review as a conference paper at ICLR 2027

Preserving Fine-Grained Structure under Coarse Supervision via EMA-Guided Similarity-Weighted Contrastive Learning

Abstract

Fine-tuning a pretrained model with coarse labels can destroy existing fine-grained structure in latent space. Preserving this latent structure is important when detailed annotations are unavailable. Standard supervised contrastive learning (SupCon) amplifies this problem by attracting all samples sharing a coarse label uniformly, despite potentially substantial within-class heterogeneity. We introduce , a similarity-weighted extension of SupCon that uses a stop-gradient exponential moving average (EMA) teacher to estimate within-class pairwise similarity and selectively redistribute positive-pair attraction. Similar positives therefore receive greater learning pressure, while dissimilar positives are attracted less strongly, without requiring fine-grained labels, an auxiliary self-supervised objective, or an external memory bank. We evaluate TaxoCon on iNaturalist 2021 and FGVC-Aircraft using only coarse labels for representation fine-tuning and model selection, with finer-grained labels withheld for post-hoc evaluation. Across ResNet-50, ViT-S/16, and Swin-T pretrained backbones, TaxoCon consistently improves preservation of fine-grained structure across neighborhood retrieval, linear-probing, and hierarchy-sensitive metrics on evaluated baselines. These results show that TaxoCon can mitigate the loss of fine-grained structure during coarse-supervised fine-tuning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
contrastive learning, supervised contrastive learning, representation learning, fine-grained representation learning, coarse supervision, hierarchical representation learning

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_fhFGOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work introduces an extension of standard supervised contrastive learning framework called TaxoCon.

The proposed TaxoCon uses a stop-gradient EMA to estimate within-class pairwise similarity and redistribute positive-pair attraction.

Without requiring the finegrained labels, the proposed model is able to assign higher learning pressure to similar positives.

The evaluations are performed on two datasets, covering 3 pretrained backbones. They show improved performance.

Critical strengths
  • The evaluations show clear improvements compared to existing works on two benchmarks.

  • The organisation of the paper is clear and clean.

Critical weaknesses

Major:

  • The novelty of this work appears to be limited. It sounds more like an extension of existing supervised contrastive learning framework. Fig 1 and fig 2 are quite poorly presented. It does not show what are the key innovations and how this work differs from existing works.

  • The evaluations appear to be very limited. (i) The authors should explore more pretrained models in the literature, only 3 pretrained models presented in the paper showing improvements appear to be cherry picking and biased. (ii) The evaluations should be performed on more challenging datasets. In addition, large scale evaluations and comparisons are need to show the wide applicable of the proposed framework.

  • Most experimental results are presented in the form of tables, and there are no enough figures/plots to show why and how the proposed framework works compared to existing methods/frameworks in the current literature.

  • The method section does not provide proper justifications of why choosing these blocks/modules, what are the design principles, etc, and how each component contributes to what tasks/purposes etc. After reading the method section several times, I could not figure out what are the key innovations of this work. In addition, it is unclear to me what are the losses used for training and for downstreaming tasks etc.

  • Sec 4.4 could be much clearer. In additon, the variants of the proposed framework could be clearer in the method section, and why those variants, how each variant contributes to the improvements.

  • I feel the current evaluations and comparisons are not thorough enough.

Minor:

  • Figure font sizes are too small to read, eg, fig 1.

  • Some redundant parts presented in the paper due to the use of eg "overall", etc.

Questions for authors

Refer to my raised concerns.

Reviewer_rATJOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies fine-tuning a pretrained vision backbone with coarse labels only, aiming to preserve the fine-grained structure the backbone already encodes. TaxoCon modifies SupCon by replacing the uniform weight on same-coarse-label positives with a softmax over pairwise similarities from a stop-gradient EMA teacher; a multi-view variant averages teacher similarities over augmented views. With ResNet-50, ViT-S/16 and Swin-T, the authors train on order labels of iNaturalist 2021-mini (one model per iconic group) and on manufacturer labels of FGVC-Aircraft, and evaluate the finer ranks withheld during training with Recall@1/5 and linear probing.

Critical strengths
  • The problem is practically relevant: fine-grained labels are costly in domains such as biodiversity monitoring, and the paper shows that common coarse-supervised objectives such as SupCon and cross-entropy can degrade fine-grained structure present in the pretrained model (e.g., ViT-S/16 species Recall@1: frozen 18.09 vs. SupCon 12.98, Table 1).
  • The method is simple and easy to adopt: a drop-in reweighting of SupCon positives with no extra trainable parameters, auxiliary loss or memory bank.
  • The evaluation design, in which finer ranks are withheld from training and assessed with both local (Recall@K, linear probing) and hierarchy-level metrics, is a sensible way to study this problem.
Critical weaknesses
  1. Limited novelty; the contribution over MaskCon is incremental. Weighting or softening positives with similarities from a teacher or from neighboring samples is an established idea in contrastive learning (e.g., SCE, X-CLR, ReSSL, MSF, NNCLR). Relative to MaskCon, the difference is a design choice: reweighting same-label positives inside SupCon instead of building coarse-masked soft targets, and using an EMA teacher instead of the current encoder. Sec. 3.3 gives only motivation and no analysis of why this reweighting should preserve fine-grained structure. Moreover, the main empirical message, that coarse supervision can improve fine-grained structure beyond both SupCon and the frozen backbone, is already achieved by MaskCon in Table 1 (ResNet-50 species Recall@1: 13.35 vs. 7.23 and 8.01), and TaxoCon(MV) adds only 0.83 points on top of it (14.18). Even this margin is established only for ResNet-50: on iNaturalist, MaskCon is omitted for ViT-S/16 and Swin-T "due to computational constraints" (Sec. 4.4), although TaxoCon(MV), which uses the same batch size and an additional teacher forward pass, is run for all three backbones, and MaskCon is run for all three on FGVC-Aircraft (Table 3). Consequently, the largest gains in Table 1 (e.g., species Recall@1 +7.60+7.60 for ViT-S/16 and +5.38+5.38 for Swin-T) are computed without the closest baseline, while on FGVC-Aircraft, where MaskCon is included, the margins over it are small and inconsistent (e.g., Swin-T variant Recall@1: TaxoCon(MV) 51.97 vs. MaskCon 52.06; ViT-S/16 manufacturer Recall@5: 93.67 vs. 93.79).

  2. The comparison with the closest baseline is neither fair nor verifiable. MaskCon is run "without modification" with its original settings (App. D), which were tuned for other datasets, whereas TaxoCon's τp\tau_p and mm were searched on iNaturalist; the hyperparameters of SupCon and MS-loss are not reported. On iNaturalist, MaskCon is evaluated only with ResNet-50 (Sec. 4.4), so the large gains reported for ViT-S/16 and Swin-T in Table 1 do not include it. Where MaskCon is included, the margins are small or mixed (e.g., +0.03+0.03 family Recall@5 for ViT-S/16 on FGVC-Aircraft), Table 3's Δ\Delta takes the best of three TaxoCon variants per metric, and no seeds or variance are reported. None of MaskCon's published benchmarks (CIFAR-100, ImageNet32, Stanford Online Products, Cars196) are used, so the results cannot be checked against published numbers.

  3. Model selection appears to rely on the held-out ranks, and the method is sensitive to mm at exactly those ranks. Sec. 4.3 states that τp\tau_p and mm were selected by order-level Recall@1 on a validation split, which is essential to the claim that gains at finer ranks owe nothing to fine-grained supervision. However, at the order rank the three tested values of mm are nearly indistinguishable, and m=0.95m = 0.95 is not the best for any backbone (m=0.90m = 0.90 is best for ResNet-50 and m=0.999m = 0.999 for ViT-S/16 and Swin-T; Fig. 7–8). m=0.95m = 0.95 is clearly best only at the genus and species ranks for the transformer backbones, and App. E.2 itself justifies the choice by its performance "across backbones, taxonomic ranks, and evaluation protocols". This matters because fine-rank performance varies substantially with mm (e.g., Swin-T species linear-probe accuracy ranges from about 21 to 26 across m∈{0.90,0.95,0.999}m \in \{0.90, 0.95, 0.999\}, comparable to the reported Δ=+6.78\Delta = +6.78 in Table 1), contrary to the claim that TaxoCon is "not highly sensitive" to mm. With only three values tested and a selection criterion that barely separates them, the choice of mm appears to be driven by the held-out ranks, which undermines the held-out-rank protocol.

  4. Parts of the evaluation are unreliable. The frozen ViT-S/16 species linear probe reaches 0.06% on Insects and 0.07% on Plants, while 1-NN retrieval on the same features gives 15.05% and 10.83% (Tables 6 and 9). This indicates a failed probe, and it inflates the reported Δ=+6.73\Delta = +6.73 in Table 1 (about +3.9+3.9 if the 1-NN values are used as a proxy). The full-hierarchy experiment (Table 2) feeds 32×3232 \times 32 inputs to a backbone pretrained at 224×224224 \times 224, reaches only about 1% species Recall@1, and compares against MaskCon alone, yet it is presented as evidence that TaxoCon's advantage persists.

  5. The practical scenario is unclear, and the experiments lack the reference points needed to define it. The paper restricts itself to the setting where the pretrained model already encodes fine-grained structure and the goal is to preserve rather than discover fine-grained classes. In that setting, simpler alternatives seem natural: using a stronger frozen backbone directly (e.g., DINOv2 or CLIP with kk-NN), or pseudo-labeling the data at the fine-grained level with such a model and training with standard supervision. None of these is evaluated, and the paper reports no fine-label-supervised upper bound, so it is impossible to tell where TaxoCon stands between the frozen backbone and what fine labels would achieve. All backbones are ImageNet-pretrained ResNet-50, ViT-S/16 and Swin-T, and the main results rely on per-group training, in which some groups have only 2–4 coarse classes. The authors should clarify in which practical scenario coarse-supervised fine-tuning with TaxoCon is preferable to these alternatives, and support it with the corresponding baselines.

Questions for authors
  1. MaskCon baseline. Which of MaskCon's dataset-specific configurations, optimizer and schedule were used, and was any hyperparameter of MaskCon, SupCon or MS-loss tuned on iNaturalist? Please also report MaskCon for ViT-S/16 and Swin-T on iNaturalist, as was done on FGVC-Aircraft, and results on at least one of MaskCon's published benchmarks (e.g., CIFAR-100 or Cars196).

  2. Model selection. Please report the validation-split, order-level numbers used to select τp\tau_p and mm, and state whether Fig. 7–8 are computed on the validation or the test set.

  3. Linear probing. Please verify the convergence of the linear probe (e.g., the frozen ViT-S/16 species probe at 0.06% on Insects and 0.07% on Plants in Table 9) and re-report the affected entries and the Δ\Delta values in Table 1.

  4. Reference points and alternatives. Please report a fine-label-supervised upper bound, a strong frozen backbone (e.g., DINOv2 with kk-NN), and a baseline trained on fine-grained pseudo-labels from such a model, and explain in which practical scenario TaxoCon is preferable to these.

  5. Evidence of preservation. Fig. 2 illustrates the intended effect only schematically. Could you visualize the actual embeddings (e.g., t-SNE or UMAP of one coarse class, colored by finer-rank labels) for the frozen backbone, SupCon, MaskCon and TaxoCon(MV), to show whether TaxoCon preserves the within-class structure of the pretrained model or creates a different one? A quantitative counterpart, such as kk-NN neighborhood overlap or CKA between each fine-tuned model and the frozen backbone, together with a comparison to a teacher fixed at the pretrained weights (m=1m = 1), would make the preservation claim directly testable.

Reviewer_egcvOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies how to retain fine-grained information when pretrained visual models are fine-tuned using only coarse labels. It proposes TaxoCon, a supervised contrastive objective that replaces uniform positive weights with within-class similarity weights estimated by a stop-gradient EMA teacher. A multi-view variant averages teacher similarities across augmented views to obtain more stable weights. Experiments use order-level supervision on iNaturalist 2021 and manufacturer-level supervision on FGVC-Aircraft. Finer labels are excluded from representation learning and hyperparameter selection, but are used for post-hoc evaluation and fitting linear probes on frozen representations. Across three pretrained backbones, the paper reports improvements in fine-grained retrieval, linear separability, and hierarchy alignment. Hyperparameters are selected using coarse-rank validation on iNaturalist and transferred to Aircraft.

Critical strengths
  1. Meaningful problem and motivation. Preserving fine-grained information during coarse-supervised adaptation is a relevant problem. The frozen-backbone comparison demonstrates that improved coarse-task performance can coincide with degraded fine-grained retrieval (Table 1).
  2. Simple and coherent objective. TaxoCon provides a straightforward extension of SupCon and recovers it under uniform positive weighting. The method requires no additional fine-grained annotations, auxiliary self-supervised loss, or external memory bank (§3).
  3. Appropriate separation of supervision and evaluation. Hyperparameters are selected using coarse-rank validation and transferred to Aircraft. Fine labels are excluded from representation learning and model selection, while remaining available for post-hoc evaluation and linear-probe fitting (§4).
  4. Complementary evaluation and encouraging gains. Retrieval, linear probing, and hierarchy metrics provide complementary evidence across two datasets and three backbones. On Aircraft/ResNet-50, variant Recall@1 improves from MaskCon’s 34.38 to 38.34 with TaxoCon(MV) (Table 3).
Critical weaknesses
  1. The distinction from MaskCon needs greater precision. MaskCon already uses coarse-masked similarity targets, and its official implementation includes a momentum encoder. Since TaxoCon’s Eq. (2) also admits a soft-target cross-entropy interpretation, §3.4 should identify the substantive differences more precisely.
  2. The component ablations contain confounding factors. EMA and no-EMA use different weighting temperatures (0.1 versus 0.3), while SV and MV use different batch sizes (800 versus 300), also changing update counts under fixed epochs. These comparisons do not cleanly isolate the benefits of EMA and multi-view estimation (§4.3; Appendix D).
  3. Reported aggregates are internally inconsistent. Table 13’s Swin-T no-EMA results average to 21.31 for Spearman and 20.30 for Dendrogram–Taxonomy Agreement, versus 17.51 and 16.09 in Table 1. Under the stated macro-average protocol, these discrepancies exceed rounding and materially affect the apparent EMA benefit.
  4. Uncertainty and reproduction details are insufficient. Independent-run counts, variability, the numerical contrastive temperature (\tau), and linear-probe optimization settings are missing. This limits reproducibility and confidence in small gains, including the reported 0.21 percentage-point species probing improvement over MaskCon on iNaturalist/ResNet-50 (Table 1).
Questions for authors
  1. Relationship to the closest prior work. Can you present TaxoCon and MaskCon in a common formulation and identify the substantive differences in target construction, encoder usage, and contrastive sample sets? Please clarify the discussion in §3.4 in light of the momentum encoder used by the official MaskCon implementation.
  2. Controlled component attribution. In one representative setting, can you compare no-EMA, EMA single-view, and EMA multi-view variants while fixing the positive-weighting temperature, batch size, student augmentation, and student update budget? Please also specify whether the weights are detached in the no-EMA variant. A limited comparison would help separate component effects from the benefits of independently selected configurations.
  3. Reconciliation of existing results. Can you reconcile the no-EMA aggregates in Table 1 with the group-level results in Tables 12 and 13, confirm whether they originate from the same runs, and provide corrected values? How do these corrections affect the conclusions about the benefit of EMA?
  4. Reproducibility and uncertainty. How many independent runs underlie the main results? Please provide the numerical contrastive temperature (\tau), the linear-probe training configuration, and available per-run results or variability for one representative TaxoCon(MV)–MaskCon comparison and the matched ablation. I am not requesting a rerun of the entire experimental suite.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.