- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper defines Semantic Coverage Imbalance (SCI), the uneven class-conditional concept coverage C_{y,k}. It documents SCI on CelebA, CUB, Waterbirds, Derm7pt and MILK10k, and reports that low-coverage class–concept pairs have lower frozen-probe concept Macro-F1. It proposes SEMCOVNET (DINOv3, per-concept cross-attention queries, residual path) with Coverage-Calibrated Semantic Sharing (CCSS). CCSS shrinks each pair prototype toward an additive prior μ0+μ̃y+μ̃k with weight α=nτ²/(nτ²+σ²), used as a cosine-alignment target. Validation consists of an observational head/tail analysis, a CCSS ladder, an evidence-removal intervention on 24 CUB pairs, clinical datasets, and appendix comparisons and ablations.
Critical strengths
- A clean question with good hygiene: the pair is the unit of analysis, CIs are bootstrapped over pairs, comparisons are paired, and the intervention removes a pair's evidence while keeping class/concept marginals. Unfavourable results (Derm7pt, WaterBirds vs DRO methods, non-significant Tables 6/15) are reported openly.
- CCSS is cheap (training-only) and derived from an explicit hierarchical model. The pair-level coverage lens is a reusable diagnostic for concept-based and medical models.
Critical weaknesses
- The central claim is not supported by the paper's own controlled evidence, and the mechanism is not exercised.
- Table 5 is the only causal test of "lower coverage → weaker representation". Quality is flat as evidence is removed. For 4/5 methods the 0%-evidence value is ≥ the 100% value (Δ100→0 = −0.012, −0.005, −0.006, −0.003), including the matched LoRA Concept control (0.664→0.670). Only frozen Concept falls (0.675→0.665; paired CI not reported). The Introduction and Sec. 4.4 nevertheless state that the interventions show degradation.
- SEMCOVNET's edge over Coverage Reweight/GroupDRO is already +0.029/+0.024 at 100% evidence and +0.021/+0.022 at 0%, so there is no coverage-specific growth. Over concept supervision at 0% it is +0.017 with CI [−0.001, +0.036]. In Table 6, fixed specialization (0.684), no class sharing (0.678) and shared-prior-only (0.667) are indistinguishable from CCSS (0.682).
- L_CCSS (Eq. 11) is weighted by a_{i,k}, so a pair with n_eff=0 contributes no term. The α=0 branch invoked in Sec. 4.4 is never exercised.
- The observational evidence (Fig. 5) is confounded. Class-conditional Macro-F1 depends mechanically on within-class prevalence and the number of probe positives, and many zero-coverage CUB pairs are semantically impossible (App. E.1). Prevalence-invariant metrics (AUROC/AUPRC at matched prevalence, or fixed positives) are needed.
- Effects are tiny and headline quantities are inconsistent across tables.
- CUB Tail Macro-F1 is 0.003→0.005 in Table 1 but 0.53–0.63 in Tables 12–14.
- The concept-supervised CUB Head–Tail gap is 0.031 (Sec. 4.2), 0.171 (Table 1) and −0.005 (Table 12, where Tail ≥ Head contradicts SCI). On CelebA it is 0.094 vs 0.426.
- The full model's CUB Tail F1 is 0.599/0.608/0.619/0.634 in Tables 15/14/12/13, with seed SDs of 0.05–0.07. Most comparisons in Tables 12–15 are therefore within noise.
- CUB's quantiles are degenerate (Q25 = median = 0), and the "predefined support criteria" are never defined.
- Task gains over the paper's own controls are 0.002–0.005 (CelebA 0.955 vs 0.953/0.954; CUB 0.891 vs 0.886). Table 9 omits the DINOv3-ERM control and does not say whether baselines share backbone and tuning. SEMCOVNET is not best on Derm7pt (0.851 vs 0.870) and is far behind on WaterBirds WGA (0.830 vs 0.923).
- Additional: novelty is modest. SCI is class-conditional prevalence, and CCSS is James–Stein-style shrinkage with an additive class+concept prior. Compositional zero-shot, prototype-calibration and center-loss-style baselines are neither discussed nor compared. "Bayes-optimal" (App. D) holds only for known variances, whereas τ² and σ² are plug-in estimates. Fig. 1(b) is a schematic captioned "we show". Table 13's caption duplicates Table 12's.
Questions for authors
-
Define Tail/Head pairs in Table 1, Fig. 5, Tables 12–15 and Fig. 6: criteria, counts, pairs with ≥1 positive test example, and F1 handling. Why do the gaps differ by up to 5×?
-
How is evidence removed: dropping images, or setting a_{i,k}=0, which makes them negatives in L_concept? Does the probe still see the pair's positives?
-
Through what pathway does CCSS affect pairs with n_eff=0? Are σ² and τ² computed once after warm-up or refreshed?
-
(suggestion) Report AUROC/AUPRC at matched prevalence, and a null simulation of probe F1 vs coverage for a fixed representation.
-
(suggestion) Add a shrinkage-free prototype-consistency control and a compositional baseline, and state the backbone and tuning budget of every Table 9 baseline.