acceptodds
Under review as a conference paper at ICLR 2027

Coverage-aware Semantic Representation Learning for Underrepresented Visual Concepts

Abstract

Modern vision models increasingly rely on rich semantic representations that extend beyond class labels to include descriptive concepts, attributes, and contextual cues. However, semantic concepts are not uniformly represented across classes: a concept may be frequent globally yet remain underrepresented within a class, resulting in low class-concept coverage. We formalize this class-concept phenomenon as Semantic Coverage Imbalance (SCI) and study its relationship with learned semantic representations. We find that lower class-concept coverage is consistently associated with weaker semantic representations. To address this, we introduce Coverage-aware Semantic Representation Learning, which shares more semantic structure for low-coverage or uncertain relations while preserving more pair-specific structure for high-coverage relations. We instantiate this principle in SemCovNet with Coverage-Calibrated Semantic Sharing (CCSS), a coverage- and uncertainty-aware partial-pooling mechanism. Using a range of vision datasets spanning facial attributes and fine-grained recognition, as well as real-world medical datasets, we show that SCI is widespread, degrades representation quality, and can be mitigated through coverage-aware semantic representation learning.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semantic Coverage Imbalance, semantic representation learning, coverage-aware learning, concept-based learning, visual representation learning, semantic representations

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WnkwOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper defines Semantic Coverage Imbalance (SCI), the uneven class-conditional concept coverage C_{y,k}. It documents SCI on CelebA, CUB, Waterbirds, Derm7pt and MILK10k, and reports that low-coverage class–concept pairs have lower frozen-probe concept Macro-F1. It proposes SEMCOVNET (DINOv3, per-concept cross-attention queries, residual path) with Coverage-Calibrated Semantic Sharing (CCSS). CCSS shrinks each pair prototype toward an additive prior μ0+μ̃y+μ̃k with weight α=nτ²/(nτ²+σ²), used as a cosine-alignment target. Validation consists of an observational head/tail analysis, a CCSS ladder, an evidence-removal intervention on 24 CUB pairs, clinical datasets, and appendix comparisons and ablations.

Critical strengths
  1. A clean question with good hygiene: the pair is the unit of analysis, CIs are bootstrapped over pairs, comparisons are paired, and the intervention removes a pair's evidence while keeping class/concept marginals. Unfavourable results (Derm7pt, WaterBirds vs DRO methods, non-significant Tables 6/15) are reported openly.
  2. CCSS is cheap (training-only) and derived from an explicit hierarchical model. The pair-level coverage lens is a reusable diagnostic for concept-based and medical models.
Critical weaknesses
  1. The central claim is not supported by the paper's own controlled evidence, and the mechanism is not exercised.
    • Table 5 is the only causal test of "lower coverage → weaker representation". Quality is flat as evidence is removed. For 4/5 methods the 0%-evidence value is ≥ the 100% value (Δ100→0 = −0.012, −0.005, −0.006, −0.003), including the matched LoRA Concept control (0.664→0.670). Only frozen Concept falls (0.675→0.665; paired CI not reported). The Introduction and Sec. 4.4 nevertheless state that the interventions show degradation.
    • SEMCOVNET's edge over Coverage Reweight/GroupDRO is already +0.029/+0.024 at 100% evidence and +0.021/+0.022 at 0%, so there is no coverage-specific growth. Over concept supervision at 0% it is +0.017 with CI [−0.001, +0.036]. In Table 6, fixed specialization (0.684), no class sharing (0.678) and shared-prior-only (0.667) are indistinguishable from CCSS (0.682).
    • L_CCSS (Eq. 11) is weighted by a_{i,k}, so a pair with n_eff=0 contributes no term. The α=0 branch invoked in Sec. 4.4 is never exercised.
    • The observational evidence (Fig. 5) is confounded. Class-conditional Macro-F1 depends mechanically on within-class prevalence and the number of probe positives, and many zero-coverage CUB pairs are semantically impossible (App. E.1). Prevalence-invariant metrics (AUROC/AUPRC at matched prevalence, or fixed positives) are needed.
  2. Effects are tiny and headline quantities are inconsistent across tables.
    • CUB Tail Macro-F1 is 0.003→0.005 in Table 1 but 0.53–0.63 in Tables 12–14.
    • The concept-supervised CUB Head–Tail gap is 0.031 (Sec. 4.2), 0.171 (Table 1) and −0.005 (Table 12, where Tail ≥ Head contradicts SCI). On CelebA it is 0.094 vs 0.426.
    • The full model's CUB Tail F1 is 0.599/0.608/0.619/0.634 in Tables 15/14/12/13, with seed SDs of 0.05–0.07. Most comparisons in Tables 12–15 are therefore within noise.
    • CUB's quantiles are degenerate (Q25 = median = 0), and the "predefined support criteria" are never defined.
    • Task gains over the paper's own controls are 0.002–0.005 (CelebA 0.955 vs 0.953/0.954; CUB 0.891 vs 0.886). Table 9 omits the DINOv3-ERM control and does not say whether baselines share backbone and tuning. SEMCOVNET is not best on Derm7pt (0.851 vs 0.870) and is far behind on WaterBirds WGA (0.830 vs 0.923).
    • Additional: novelty is modest. SCI is class-conditional prevalence, and CCSS is James–Stein-style shrinkage with an additive class+concept prior. Compositional zero-shot, prototype-calibration and center-loss-style baselines are neither discussed nor compared. "Bayes-optimal" (App. D) holds only for known variances, whereas τ² and σ² are plug-in estimates. Fig. 1(b) is a schematic captioned "we show". Table 13's caption duplicates Table 12's.
Questions for authors
  1. Define Tail/Head pairs in Table 1, Fig. 5, Tables 12–15 and Fig. 6: criteria, counts, pairs with ≥1 positive test example, and F1 handling. Why do the gaps differ by up to 5×?

  2. How is evidence removed: dropping images, or setting a_{i,k}=0, which makes them negatives in L_concept? Does the probe still see the pair's positives?

  3. Through what pathway does CCSS affect pairs with n_eff=0? Are σ² and τ² computed once after warm-up or refreshed?

  4. (suggestion) Report AUROC/AUPRC at matched prevalence, and a null simulation of probe F1 vs coverage for a fixed representation.

  5. (suggestion) Add a shrinkage-free prototype-consistency control and a compositional baseline, and state the backbone and tuning budget of every Table 9 baseline.

Reviewer_5eZfOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the issue of "imbalanced semantic coverage" in concept-supervised visual representation learning: even when individual classes and concepts are well-represented by sufficient samples, certain class-concept combinations may lack adequate training evidence. The authors propose SemCovNet, which employs concept queries to extract concept representations from DINOv3 features. Through a mechanism called Coverage-Calibrated Semantic Sharing (CCSS), the model dynamically weights the contribution between combination-specific prototypes and shared semantic structures, based on the effective support (sample size) and estimated uncertainty for each combination. This objective constrains the concept representations, while a residual visual branch is maintained for downstream task prediction. Experiments conducted on both natural and medical image datasets demonstrate that combinations with low coverage yield weaker representations, and that adaptive semantic sharing can effectively mitigate this problem.

Critical strengths
  1. The research problem holds practical significance. Focusing on category-concept combinations—rather than merely on category frequencies—helps reveal fine-grained representation deficiencies that might otherwise be obscured by aggregate task metrics; this is particularly relevant for scenarios involving rare attributes or clinical concepts.
  2. The sharing mechanism is grounded in a clear statistical rationale. Appendix D derives the shared weights using a hierarchical Gaussian model, providing a sound justification for a design that relies more heavily on shared structures when evidence is scarce, while preserving compositional characteristics when evidence is abundant.
  3. The experiments incorporate valuable controls and mechanistic analyses. Table 1 compares performance with and without CCSS, while keeping the conceptual supervision and backbone adaptation settings consistent. The appendix provides further analysis of the shared structures, calibration rules, LoRA adaptation, and loss weighting. Additionally, "coverage" intervention experiments help distinguish between cases where compositional evidence is insufficient versus those where a category or concept is entirely absent; the experimental scope spans multiple datasets.
Critical weaknesses
  1. The innovative boundaries of the proposed problem and methodology remain insufficiently clear. Intra-class attribute imbalance has already been addressed in works such as GLT (Tang et al., ECCV 2022) and CALA (Zhou et al., AAAI 2025). While the manuscript acknowledges these studies, it needs to more explicitly articulate the new insights provided by SCI that cannot be captured by existing frameworks. Since the CCSS formulation falls under the category of partial sharing and shrinkage estimation within a hierarchical model, its contribution should be grounded in its unique role in conceptual representation learning, rather than relying merely on a reformulation of established statistical mechanisms.
  2. The correspondence between metrics and experimental configurations across key results is unclear, which hinders the verifiability of the conclusions. In the main Table 1, the "Tail" concept Macro-F1 for the CUB dataset is approximately 0.003–0.005, whereas in Appendix Tables 12–15, the "Tail" F1 values ​​are around 0.55–0.63. Furthermore, the performance of the "full" or "default" method varies across different tables, appearing as 0.619, 0.634, and 0.599, respectively. These discrepancies—potentially stemming from differences in evaluation sets, detectors, or training configurations—lack explicit clarification.
  3. The current ablation studies provide stronger support for the sharing mechanism than for the added value of full calibration. In Table 12, both the "Shared" variant and the "CCSS frozen" variant yield a Tail F1 of 0.577. In Table 13, the full CCSS model shows a marginal improvement of 0.009 over a baseline using only the "effective support size," yet this lacks an uncertainty analysis of the paired differences. In the "zero-evidence" scenario (Table 6), the version with fixed specialization performs comparably to the full model; moreover, the confidence interval for the improvement of the full model over standard conceptual supervision includes zero. Consequently, the necessity of each component—calibration, uncertainty estimation, and class-level sharing—remains inadequately substantiated.
  4. There remains a gap between the statistical derivations and the actual data/optimization process. The proposed method calculates weights primarily based on the "effective support size," while the experiments categorize concepts into "Tail" or "Head" based on intra-class coverage ratios; the correspondence between these two approaches requires further clarification. For soft-concept scores, using the sum of weights directly as the effective sample size also necessitates additional assumptions regarding observation noise.
  5. The presentation of the methodology and the comparative results are insufficient to support broad conclusions. Figure 3 fails to clearly distinguish between training-specific statistics, learnable parameters, and inference paths; furthermore, it remains ambiguous whether prototypes and variances are re-estimated following updates to the encoder. While the baselines cover a wide range of approaches—including recent ones—the proposed method does not demonstrate a consistent advantage across the relevant task metrics for the CUB, Derm7pt, and Waterbirds datasets.
Questions for authors
  1. Please explain the order-of-magnitude difference in the "CUB Tail F1" metric between Main Table 1 and Appendix Tables 12–15. For each table, specify the evaluation set, detector, aggregation method, and training configuration. Which of these results are directly comparable?
  2. Based on the five existing sub-results, what are the pairwise performance differences (deltas) between the full CCSS mechanism and the variants using only "effective support," "simple sharing," or "fixed specialization"? Is the current evidence sufficient to substantiate the independent contribution of the full calibration mechanism?
  3. When using soft concept scores as weights, the prototype estimation variance is expressed as (σk2/∑iai,k)(\sigma_k^2/\sum_i a_{i,k}). What observation model or independence assumptions are required for this formulation? How does this derivation relate to the cosine alignment loss actually employed in the implementation?
  4. Following the warm-up phase, are the combined prototypes, variances, and shared weights held constant, or are they updated periodically? Please describe the update sequence, and specify which quantities have their gradients stopped (detached) and which parameters receive gradients.
  5. In the context of existing research on intra-class attribute imbalance, what is the most critical new insight offered by SCI? Among the current experimental results, which one most directly substantiates this insight—rather than merely demonstrating that shared regularization is beneficial?
Reviewer_azp9OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses Semantic Coverage Imbalance in concept representation learning, where individual categories and attributes are common but certain combinations are underrepresented. To address this issue, the authors propose SemCovNet, which adaptively adjusts information sharing based on available evidence and uncertainty. Specifically, underrepresented combinations rely more on shared knowledge, while well-represented combinations preserve their distinctive features, thereby improving the representation of low-coverage combinations. The method is evaluated in several settings, including medical imaging.

Critical strengths
  1. The method has a clear design rationale, and its treatment of zero-coverage pairs is internally consistent. For example, Equation (8) clearly specifies the sampling-noise correction used to estimate τ².

  2. The experiments directly assess concept representation quality, and the controlled comparisons are reasonably fair. For example, Figure 5 and Table 1 evaluate whether the learned representations encode concept information and compare matched variants with and without CCSS. This helps identify the source of the reported gains.

  3. The authors acknowledge the method’s limitations. They explicitly report comparisons in which the gains are not statistically significant, as well as settings in which other methods perform better, without claiming uniformly superior performance.

Critical weaknesses
  1. The evidence that lower coverage directly causes weaker semantic representations remains insufficient. In Figure 5, the lower scores of low-coverage pairs could also reflect differences in concept difficulty, test-set positive prevalence, or annotation noise. These factors have not been sufficiently ruled out, and the controlled coverage interventions do not show a clear degradation trend.

  2. The zero-coverage experiments do not yet establish that the method handles missing pair-specific evidence more effectively. In Table 5, reducing training evidence from full coverage to zero changes the score of standard concept supervision only from 0.675 to 0.665, indicating limited degradation under the current protocol. Although SEMCOVNET significantly outperforms Coverage Reweighting and Semantic GroupDRO at zero coverage, its advantage over standard concept supervision is not statistically significant. A broader evaluation and a direct comparison of the performance degradation across methods would strengthen this claim.

  3. The core evaluation metric is insufficiently specified, and some reported results show unexplained discrepancies. For example, CUB Tail F1 is approximately 0.003–0.005 in Table 1 but approximately 0.55–0.63 in Tables 12–15. The paper does not sufficiently explain whether these results use different evaluation pairs, metrics, or aggregation procedures.

Questions for authors
  1. CUB evaluation protocol. Table 1 reports CUB Tail F1 of about 0.003–0.005, whereas Tables 12–15 report about 0.55–0.63. Please explain how these differ in pair sets, probe training, positive/negative label handling, and support-based eligibility criteria, and report all CUB comparisons under one unified protocol. Also, several paired Δ values in Table 15 do not match the differences between the listed means. Is a different aggregation used, or do these entries need correction?

  2. Coverage intervention. In Table 5, most baselines show little degradation from 100% to 0% evidence, and some improve (e.g., Coverage reweighting, −0.012). Does the intervention remove whole training images of the target pair, or only mask their annotations for concept k? Please also report each method’s own 100%→0% change with a paired 95% CI, so readers can judge whether the “limited degradation” is statistically supported.

  3. Medical splits and scope of claims. Are the Derm7pt and MILK10k splits made at the patient or lesion level, so that related images do not cross train/validation/test? On MILK10k, Tail F1 is below concept supervision in some settings and Soft Spearman is lower throughout (Table 8). Should this experiment be read as showing stability under soft supervision, or improved representation of low-coverage pairs over the baseline?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.