- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper presents a post-hoc diagnostic of how Semantic Coverage Imbalance (SCI), introduced in SemCovNet (https://arxiv.org/abs/2602.16917), is reflected in trained image classifiers. It aims to answer whether per-class concept distribution can explain per-class failure beyond class imbalance A concept function maps images to concept scores z(x), using a linear probe on annotated attributes, CLIP image–text similarity (CIFAR-100), or MONET descriptors (medical data). From class-conditional means of z, the paper defines Semantic Imbalance (variance of class means), Semantic Reliance (concept-score gap between correct and incorrect predictions) and a Unified Reliance Score (per-concept reliance weighted by the source–target change in mean concept score). These are correlated with worst-class performance and shift degradation across 4 backbones on spurious-correlation and context-shift datasets (WaterBirds, CelebA, NICO++), CIFAR-100 and two dermoscopy datasets, and used to guide a feature-attenuation intervention compared with Deep Feature Reweighting (DFR; last-layer retraining on group-balanced data).
Critical strengths
- The problem is relevant.
- The definitions of main metrics are easy to follow.
- experiments are conducted on various domains
Critical weaknesses
Novelty:
- The paper seems rather incremental over SemCovNet, which is under-credited throughout. The intro frames SemCovNet as merely "training-time" - the novelty compared to it needs to be stated more precisely.
- The concept evidence metric is the same and some of the post-hoc analysis overlaps. The newly introduced measures are simple first-order moments of the concept scores, given new names.
- SR (concept/attribute evidence between correctly and incorrectly classified samples) is a widely used concept from related literature on bias, given a new name. The works are not cited. For example: B2T (Kim et al., "Discovering and Mitigating Visual Biases through Keyword Explanation", CVPR 2024), Spuriosity Rankings (Moayeri et al., NeurIPS 2023), CB-SLICE (Konforti et al., ICML 2026). Overall these should be cited and discussed
- the related work discussion overall omits a lot of important literature, as evidenced above. Also, I believe it is an essential part of the paper and should not be placed in the supplementary.
- the propositions add little beyond their assumptions. No experiment depends on ie 2.1 and 2.3
Clarity:
- . L51–52 adopts SCI from SemCovNet as uneven class–concept coverage in the training data. Fig. 4 then labels SI, SR and SI×SR as "SCI variables" / "SCI (SI+SR+SI×SR)". The paper never measures training-data coverage, nor relates it to SI/SR, so the claim to diagnose SCI rests on the name. The Reproducibility Statement promises "coverage computation" and "controlled coverage interventions", neither of which appears in the paper.
Methodology and correctness:
I have many methodological concerns with the work. Some examples are listed here, I could go on.
- SR is correlational only, there is no evidence the concepts are actually used by the classifier. See for example "Representation Is Not Reliance: Concept-Based Causal Diagnostics of Shortcut Learning in Vision" (ICML 2026 Mech Interp Workshop).
- App. F.1 claims that prior concept methods do not characterize "how concept usage varies across classes within a model"; Spuriosity Rankings and B2T do that.
- the framing rests on “modern visual models” and proceeds to cite mostly 2019-2020 work. Moderen encoders (DINo family, SigLIP2, …) are missing.
- SI is not model-specific in half the settings: ISI is identical across the 4 architectures. This means that z(x) there is the dataset-provided MONET score, not g(f_θ(x)). This contradicts the "representation-level" claim. CIFAR-100 computes z from the CLIP embeddings, which is also independent of the evaluated models.
- some classifiers predict worst-class F1 = 0; which is essentially a degenerate classifier predicting the majority class. For such a model, it is unclear how SR is even computed as µ+/µ- are undefined. SImilarly frozen CLIP + linear head is broken on WaterBirds. Accuracy 78.4, worst-class F1 0.062, worst-acc 3.2%. This signals an udnertrained or degenerate baseline again.
- SI is pooled across datasets, but it scales with the number of concepts, so it is nto comparable.
- Wtaerbirds experiments ios circular: K=2 concepts = land/water, which is the spurious attribute. SR then measures whether background evidence differs between correct and incorrect samples, which is the definition of the shortcut failure. More generally, concept selection determines SI, SR and URS, yet the concept lists per dataset are given only in App. G (Tab. 7), and the main text never discusses how results depend on the vocabulary.
Presentation
- There are broken references in intro, a “Definition ??” in App. D, undefined or overloaded notation.
- Fig. 1 draws the prediction arrow from the semantic evidence z into h_θl. In the method, h_θ takes f_θ(x) and z is a side read-out. Draw z as a branch off f_θ(x).
- The input requirements are only clear from App. G
- App. B is called "Extended related work" but there is no non-extended version.
- Intro, para 2 repeats the dataset-bias and spurious-correlation citations already given in para 1
- Calling CIFAR-100 (32×32) "general object recognition" is an overstatement to me
Questions for authors
The main concerns go beyond what is resolvable with the rebuttal scope and would need a substantial paper revision. The following nevertheless have the highest chance of changing my assessment: Clarify the “diagnostic is a model property” claim Clarify the difference over prior work mentioned above.