acceptodds
Under review as a conference paper at ICLR 2027

Conditionally Minimal Sufficient Representations for Debiased Visual Learning

Abstract

Deep visual models often exploit spurious correlations in training data, degrading performance on bias-conflicting groups. Many representation-level debiasing methods reduce global dependence between representations and bias variables, which can be overly restrictive when the target and bias variables are statistically dependent. To address this limitation, we propose Conditionally Minimal Sufficient Debiasing Representation (CMDR), a conditional minimal-sufficiency framework that replaces global bias invariance with target-conditioned debiasing while limiting target-conditioned input redundancy. We theoretically characterize this formulation by establishing a feasibility separation between global invariance and conditional minimal sufficiency under a realizable setting, showing that bounds target-conditioned group-risk gaps, and that bounds dependence on input-mediated nuisances under the corresponding Markov condition. CMDR instantiates these principles with a target prediction objective, a target-conditioned adversarial discriminator, and a lightweight variational regularizer, while supporting both explicit bias supervision and confidence-weighted pseudo-bias signals when training bias annotations are unavailable. Experiments on Waterbirds, CelebA, BAR, MetaShift, and BFFHQ demonstrate improved worst-group robustness across diverse visual biases and bias-conflicting ratios. Ablations show that target-conditioned debiasing outperforms global adversarial debiasing, while conditional minimality provides additional worst-group gains and confidence weighting improves robustness under weak supervision.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
representation learning, visual debiasing, spurious correlations, conditional minimal sufficiency, group robustness, pseudo-bias discovery

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m887OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

The paper is motivated by the limitations of existing bias-mitigation methods of enforcing global bias invariance. The authors instead propose CMDR that enforces conditional minimal sufficiency. At the objective level, the authors use two additional regularizers: target-conditioned debiasing using a discriminator and variational conditional minimality regularization.

Critical strengths
  1. Principled theoretical motivation. The paper provides a clear information-theoretic motivation for target-conditioned debiasing, highlighting the incompatibility between global bias invariance and target sufficiency when target and bias variables are statistically dependent.
  2. Flexibility in bias supervision. The framework supports both explicit bias annotations and confidence-weighted pseudo-bias supervision, making it applicable to settings where training-time bias labels are unavailable.
Critical weaknesses
  1. Missing comparisons with recent baselines. The evaluation omits competitive methods such as DPE (ICML 2025) and SCER (ICLR 2026), which report higher worst-group accuracy on some overlapping benchmarks. Although evaluation protocols differ, their omission weakens the evidence for CMDR's competitiveness against recent methods.
  2. While the experiments employ shared CLIP backbones where applicable, CMDR-Pseudo additionally leverages pretrained CLIP ViT-L/14 representations to generate pseudo-bias supervision. This may provide an advantage over baselines that do not use equivalent auxiliary pretrained features, making the comparison unfair.
  3. Limited evaluation of multi-bias robustness and large-scale generalization. The experiments focus primarily on datasets involving a single explicitly modeled bias attribute and relatively few target classes. Although the appendix examines robustness to several additional attributes on CelebA, this does not constitute a systematic evaluation of simultaneous multi-bias mitigation. Furthermore, the paper does not evaluate large-scale datasets such as ImageNet-1K or associated distribution-shift benchmarks. Recent methods such as SEBRA (ICLR 2025) investigate these more challenging settings. Consequently, it remains unclear how well CMDR scales to large numbers of classes and multiple interacting spurious correlations.
  4. Although the related-work section categorizes existing debiasing approaches and briefly describes CMDR's intended differences, it provides limited critical comparison with closely related conditional adversarial representation-learning methods. In particular, the manuscript does not sufficiently explain which aspects of the proposed formulation and implementation are genuinely new relative to earlier conditional debiasing approaches.
  5. Limited novelty of the conditional adversarial debiasing component. The central idea of minimizing representation–bias dependence conditioned on the target, (I(H;Z_b\mid Y)), has substantial prior art. In particular, [1] and [2] already investigate conditional adversarial approaches to suppressing bias information while preserving target-relevant features. The current manuscript does not sufficiently distinguish its conditional debiasing component from these existing approaches.

[1] Conditional Learning of Fair Representations, ICLR 2020 [2] Conditional Adversarial Debiasing: Towards Learning Unbiased Classifiers from Biased Data

Questions for authors
  1. How do the proposed methods scale to a dataset with multiple biases and a large number of classes?
5 comments
AuthorsOpenReview ↗

W1: Comparisons with Recent State-of-the-Art Methods

We thank the reviewer for this valuable suggestion. We agree that comprehensive comparisons with recent state-of-the-art methods are important. However, we would like to clarify that our manuscript already includes comparisons with several recent methods published in 2025 and 2026. Specifically, Tables 1–2 compare CMDR with NSF (CVPR 2025), TTR (ICCV 2025), AGS (ICLR 2026), and Hierarchical DRO (ICLR 2026). Appendix J further provides comparisons with DeNetDM (NeurIPS 2024), GERNE (ICCV 2025), and SEBRA (ICLR 2025). Thus, our existing evaluation already covers multiple representative recent approaches, rather than lacking comparisons with contemporary methods.

To directly address the reviewer's suggestion regarding DPE (ICML 2025) and SCER (ICLR 2026), we additionally reproduced these methods on Waterbirds and CelebA using CLIP ResNet-50 and ViT-L/14 backbones. We organize the comparisons according to the availability of training-time subgroup supervision, comparing DPE with CMDR-Pseudo and SCER with CMDR-Bias, respectively.

Table R1. Comparison with DPE and SCER on Waterbirds (%).

MethodRN50 WGA↑RN50 Avg↑ViT-L/14 WGA↑ViT-L/14 Avg↑
Without subgroup supervision
DPE (ICML 2025)64.01995.99382.61690.132
CMDR-Pseudo (Ours)70.40593.48383.17897.777
With subgroup supervision
SCER (ICLR 2026)71.02886.93389.09796.288
CMDR-Bias (Ours)74.45594.13490.81095.560

Table R2. Comparison with DPE and SCER on CelebA (%).

MethodRN50 WGA↑RN50 Avg↑ViT-L/14 WGA↑ViT-L/14 Avg↑
Without subgroup supervision
DPE (ICML 2025)70.87996.95075.55695.618
CMDR-Pseudo (Ours)77.77896.03578.66793.878
With subgroup supervision
SCER (ICLR 2026)86.11191.11188.33392.516
CMDR-Bias (Ours)88.04593.15288.81691.070

Note: Results are reported in percentages. Bold values highlight the higher WGA within each supervision setting. DPE and SCER results are from our additional reproductions, whereas CMDR results are from Table 1 of the manuscript.

These additional experiments further support the competitiveness of CMDR: across both datasets and backbone architectures, CMDR-Pseudo consistently achieves higher WGA than DPE, while CMDR-Bias consistently outperforms SCER in WGA. Although average accuracy varies across methods and some WGA improvements are relatively small, these findings provide additional empirical evidence of CMDR's competitiveness against recent approaches. We will incorporate these comparisons into the revised manuscript and further clarify the supervision requirements and experimental protocols of the respective methods.

AuthorsOpenReview ↗

W2: Fairness of the Pretrained Backbone Comparison

We thank the reviewer for raising this important concern. We agree that ensuring fair experimental comparisons is essential. However, we would like to clarify that under the CLIP ViT-L/14 experimental setting, all reproduced baselines use the same pretrained CLIP ViT-L/14 backbone. Therefore, CMDR-Pseudo is not the only method that benefits from this pretrained representation.

The distinction is that CMDR-Pseudo additionally leverages frozen pretrained features to construct pseudo-bias supervision, rather than relying on an exclusively available source of pretrained knowledge.

More importantly, Appendix I already investigates the potential advantage introduced by using ViT-L/14 as the pseudo-bias extractor through a backbone replacement experiment. Specifically, we keep the debiased model fixed as CLIP ResNet-50 and replace only the pseudo-bias extractor, while maintaining the remaining experimental configuration.

Table R3. Effect of pseudo-bias extractor choice (Appendix I, %).

Pseudo-Bias ExtractorWaterbirds WGA↑Waterbirds Avg↑CelebA WGA↑CelebA Avg↑
CLIP ViT-L/1470.40593.48377.77896.035
CLIP ResNet-5071.02891.94975.78695.437

Note: The debiased model uses CLIP ResNet-50 in both configurations.

The results show that replacing the ViT-L/14 pseudo-bias extractor with ResNet-50 improves Waterbirds WGA by 0.623 percentage points, while CelebA WGA decreases by only 1.992 percentage points. Moreover, both configurations maintain higher WGA than the other baselines without training-time subgroup supervision under the same debiased backbone setting.

These findings suggest that CMDR-Pseudo's performance gains cannot be simply attributed to the additional use of a stronger ViT-L/14 pretrained model, but are closely associated with the proposed pseudo-bias supervision and conditional debiasing mechanisms.

We also acknowledge that pseudo-bias discovery involves additional feature processing. We will therefore clarify the associated computational overhead in the revised manuscript to improve the transparency of our experimental comparisons.

AuthorsOpenReview ↗

W3: Multi-Bias Robustness and Large-Scale Generalization

We thank the reviewer for this valuable suggestion. We agree that robustness to multiple biases and generalization to large-scale datasets are important directions for evaluating debiasing methods.

Our experiments follow widely adopted benchmarks and evaluation settings in the debiasing literature, including those used by LfF (NeurIPS 2020), JTT (ICML 2021), BE (AAAI 2023), BM (CVPR 2023), GroupDRO (ICLR 2020), DFR (ICLR 2023), CFR (CVPR 2024), TAB (ECCV 2024), NSF (CVPR 2025), TTR (ICCV 2025), AGS (ICLR 2026), Hierarchical DRO (ICLR 2026), and Zero-Shot CLIP (ICML 2021). Appendix J further includes experimental comparisons with DeNetDM (NeurIPS 2024), GERNE (ICCV 2025), and SEBRA (ICLR 2025).

We acknowledge that we have not yet evaluated CMDR on ImageNet-1K, which contains 1,000 categories and 1,281,167 training images. Nevertheless, we have conducted extensive evaluations on five datasets with diverse types of spurious correlations: Waterbirds, CelebA, BAR, MetaShift, and BFFHQ, including CelebA with 202,599 images.

It is worth noting that SEBRA (ICLR 2025) primarily evaluates on UrbanCars, CelebA, BAR, MultiNLI, and ImageNet-1K. Among these datasets, CelebA and BAR overlap with our evaluation benchmarks, and we have already included additional comparisons with SEBRA under a unified experimental setting in Appendix J. We appreciate the reviewer's suggestion regarding ImageNet-1K and plan to extend our evaluations to larger-scale datasets in future work.

Furthermore, CMDR's theoretical framework is not inherently restricted to a single bias attribute or binary classification. For multiple bias attributes, they can be jointly represented as a multidimensional variable,

Zb=(Z1,…,Zk),Z_b=(Z_1,\ldots,Z_k),

allowing target-conditioned adversarial learning to suppress conditional dependence between the learned representation and the joint bias variable. Meanwhile, conditional minimality provides an additional regularization objective for reducing residual nuisance information that is not explicitly modeled. When bias semantics or training-time bias annotations are unavailable, CMDR-Pseudo can further provide weak supervisory signals through pseudo-bias discovery.

Our existing experiments also provide preliminary evidence of robustness to multiple potential biases. Specifically, Appendix L evaluates four additional attributes on CelebA—Black Hair, Heavy Makeup, Smiling, and Straight Hair—that were not explicitly used as bias attributes during training. CMDR achieves higher worst-group accuracy than ERM for all four attributes, without showing clear evidence of bias transfer in these evaluations.

Although these results do not constitute a systematic evaluation of simultaneous multi-bias removal, they suggest that CMDR has the potential to extend to settings involving multiple latent biases.

Reviewer_B2nWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes Conditionally Minimal Sufficient Debiasing Representation (CMDR) to mitigate spurious correlations in visual learning. The authors argue that global bias invariance is overly restrictive and incompatible with target sufficiency when target and bias variables are statistically dependent. To resolve this, CMDR replaces global invariance with target-conditioned debiasing (I(H; Z_b ∣ Y)) and limits input-mediated nuisances via variational conditional minimality (I(H; X ∣ Y)). The method is evaluated on Waterbirds, CelebA, BAR, MetaShift, and BFFHQ, demonstrating improved worst-group robustness (WGA) under both explicit bias supervision and weakly supervised settings.

Critical strengths

Incompatibility feasibility separation: The information-theoretic proof that global bias independence is impossible under statistical dependence is highly elegant and cleanly motivates target-conditioned debiasing.

Comprehensive evaluations: The benchmark evaluations on five visual datasets and the detailed ablation on pseudo-bias discovery are well-designed.

Critical weaknesses

Sensitivity to Spurious Cue Detectability: The weakly supervised variant (CMDR-Pseudo) relies heavily on a frozen pretrained feature space (CLIP ViT-L/14) to discover latent bias directions. If the spurious cue is complex or not well-represented in the pretrained embeddings, the pseudo-bias quality deteriorates, limiting generalizability.

Robustness-Accuracy Trade-off: On highly biased regimes or under weak supervision (e.g., Waterbirds 0% conflict), the worst-group accuracy improvements are sometimes accompanied by a noticeable drop in overall average accuracy, which could limit practical deployments.

Questions for authors

How does CMDR-Pseudo perform if the spurious correlation is highly non-linear or relies on high-order image statistics that a frozen CLIP encoder cannot easily isolate?

What are the concrete design guidelines for tuning the conditional minimality weight β to avoid excessive feature compression?

2 comments
AuthorsOpenReview ↗

Response to Reviewer B2nW (W1, W2)

W1: Effectiveness and Limitations of Pseudo-Bias Discovery

We acknowledge that the effectiveness of pseudo-bias discovery depends on the representational capacity of pretrained features. However, we would like to clarify that CMDR-Pseudo does not rely solely on CLIP's high-level semantic representations to discover biases. As detailed in Appendix C, we incorporate complementary visual cues, including background statistics, VGG Gram-based texture features, and augmentation-response differences. We then use partial least squares (PLS) to identify latent nuisance directions in the pretrained feature space and apply confidence weighting to reduce the influence of unreliable pseudo-bias signals.

Importantly, our method does not require accurate recovery of ground-truth bias labels; instead, it leverages continuous latent nuisance signals as soft supervision.

Furthermore, the backbone replacement experiment in Appendix I demonstrates that CMDR-Pseudo is not heavily dependent on CLIP ViT-L/14. When replacing the pseudo-bias extractor with CLIP ResNet-50, WGA changes from 70.405% to 71.028% on Waterbirds and from 77.778% to 75.786% on CelebA, indicating relatively stable performance.

Table 4 further shows that confidence weighting improves WGA by 2.9 and 1.1 percentage points on BFFHQ under the 1% and 5% bias-conflict settings, respectively, supporting its effectiveness in mitigating unreliable pseudo-bias supervision.

We acknowledge that pseudo-bias discovery may be limited when certain spurious cues are insufficiently represented by the pretrained features. Nevertheless, CMDR's conditional minimality regularizer does not require explicit identification of individual bias factors and provides a complementary mechanism for discouraging residual nuisance information, although it cannot guarantee the complete removal of undetectable biases.

W2: Trade-off Between Worst-Group Robustness and Average Accuracy

We agree that the trade-off between worst-group robustness and overall accuracy is an important consideration in practical applications. However, this trade-off is not universal and should be evaluated in relation to the corresponding robustness gains.

As shown in Appendix F, under the extreme 0% bias-conflict setting on Waterbirds, CMDR-Pseudo improves WGA from 12.8% to 37.6% (+24.8 percentage points), while Avg decreases only from 94.4% to 92.1% (−2.3 percentage points). This demonstrates a substantial improvement in worst-group robustness at a relatively modest cost in overall accuracy.

More importantly, under the same setting on MetaShift, WGA increases from 42.3% to 68.7% (+26.4 percentage points), while Avg simultaneously improves from 98.6% to 99.1%. This indicates that improved worst-group robustness does not necessarily come at the expense of average accuracy.

In addition, CMDR allows this balance to be adjusted through the regularization strength. For example, Appendix K shows that, on BFFHQ with 5% bias-conflicting samples, halving the conditional minimality weight β\beta improves WGA from 90.0% to 90.8% and Avg from 95.1% to 95.3%.

Therefore, CMDR does not simply trade overall accuracy for robustness. Rather, it provides a tunable optimization framework that enables practitioners to balance worst-group performance and overall predictive accuracy according to application requirements.

AuthorsOpenReview ↗

Response to Reviewer B2nW (Q1, Q2)

Q1: Robustness to Complex and Nonlinear Biases

We agree that highly nonlinear or complex higher-order visual biases may pose challenges for pseudo-bias discovery. However, CMDR-Pseudo does not rely exclusively on CLIP's high-level semantic features to identify biases.

As described in Appendix C, we incorporate complementary visual descriptors, including VGG Gram-based texture statistics, background statistics, and augmentation-response differences, to identify latent nuisance directions in pretrained representations. Consequently, even when certain biases are difficult to capture through explicit semantic information, these complementary cues may still provide useful weak supervision.

Furthermore, Appendix F evaluates an extreme scenario in which no bias-conflicting samples are available during training (0% conflict). Even under this setting, CMDR-Pseudo achieves substantial improvements in worst-group robustness, increasing WGA by 24.8 percentage points on Waterbirds and 26.4 percentage points on MetaShift.

These results demonstrate that, even without explicit evidence from bias-conflicting training examples, CMDR-Pseudo can exploit latent visual nuisance information to improve robustness. We nevertheless acknowledge that this experiment does not establish guaranteed effectiveness against nonlinear biases that are entirely absent from the pretrained feature representation.

Q2: Selection and Sensitivity of the Conditional Minimality Weight

We thank the reviewer for this important question. Selecting the conditional minimality weight β\beta requires balancing the suppression of redundant information against the preservation of target-relevant information. Accordingly, we use a relatively small regularization weight and select it based on validation worst-group accuracy (WGA), while monitoring average accuracy (Avg) to avoid excessive compression.

Specifically, we recommend using β=10−4\beta=10^{-4} as the default value and conducting a small-range search around it. When Avg decreases substantially, a smaller β\beta can be considered.

The sensitivity analysis in Appendix D shows that, on Waterbirds, setting β=10−4\beta=10^{-4} instead of β=0\beta=0 improves WGA from 88.864% to 90.810%, while simultaneously increasing Avg from 94.991% to 95.560%. In contrast, increasing β\beta to 5×10−45\times10^{-4} results in reduced performance, suggesting that overly strong regularization may be detrimental.

Furthermore, Appendix K shows that, on BFFHQ with 5% bias-conflicting samples, halving the default β\beta improves WGA from 90.0% to 90.8% and Avg from 95.1% to 95.3%.

Together, these findings demonstrate that moderate conditional minimality regularization can improve robustness, while a relatively small β\beta, combined with validation-based selection, helps mitigate potential performance degradation caused by excessive compression.

Reviewer_st6oOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper argues that global bias invariance is too strict when the target and the bias are correlated, and proposes CMDR, which removes bias only conditionally on the target. It combines a target-conditioned adversarial discriminator with a variational penalty on I(H; X | Y), and works with either real bias labels or confidence-weighted pseudo-bias from frozen CLIP features. The theory (Appendix A) shows global invariance and sufficiency can be incompatible, and bounds group-risk gaps by I(H; Z_b | Y). Experiments cover Waterbirds, CelebA, BAR, MetaShift and BFFHQ with two CLIP backbones.

Critical strengths
  1. The motivation is clean and the ablation backs it up. Table 3 separates global vs conditional adversary and minimality alone vs full CMDR, and the conditional version is better in each case. Appendix E adds a controlled nuisance test, which I found more convincing than benchmark numbers alone.

  2. The evaluation is broad and honest about model selection. Many baselines are included (GroupDRO, JTT, LfF, DFR, plus recent ones in Appendix J), and Section 4.1 and Appendix G clearly say validation group labels are used for checkpoint selection.

Critical weaknesses
  1. The minimality theory doesn't clearly apply to the feature actually used. H is a fixed-variance Gaussian around h, but prediction and the adversary use h itself (Section 3.1, p. 4). If h is scaled down and the classifier scaled up, the KL in Eq. 10 can go to zero while h keeps all its nuisance information. I don't see any normalization or head constraint that rules this out, and the fixed variance value isn't given. So a small KL doesn't by itself show nuisances were removed.

  2. The main comparisons are hard to judge. Appendix B says three seeds, but Tables 1 and 2 have no std, and some gaps are tiny (ViT Waterbirds WGA 90.810 vs 90.745 for Hierarchical DRO). CMDR also doesn't win everywhere: on ViT CelebA, CMDR-Pseudo gets 78.667 vs 87.253 for LfF, and on ViT BAR, CMDR-Bias gets 84.507 vs 87.324. Some baselines look broken (LfF on ResNet-50 CelebA: 17.478 WGA, 38.747 Avg), and there are no per-method tuning details.

Questions for authors
  1. Is h normalized, or is the classifier norm constrained? What is the fixed variance? With β = 1e-4 (Table 5), how much does the minimality term actually change h? (clarification request)

  2. Which baseline numbers were re-run and which were copied? What tuning range and selection rule did each baseline get? The LfF CelebA result in particular. (clarification request)

  3. Could you add mean ± std to Tables 1 and 2? Also, the pseudo-bias pipeline uses frozen CLIP ViT-L/14 features that LfF and JTT don't use. Does this extra information explain part of the gain? (suggestions for future work)

6 comments
AuthorsOpenReview ↗

W1 and Q1: Conditional Minimality, Theoretical Limitations, and Feature-Scale Analysis

We thank the reviewer for the careful analysis. We would like to clarify that CMDR does not aim to completely eliminate all bias-related or irrelevant information, as target-relevant and bias-related features may exhibit complex dependencies, and excessive compression could compromise target discrimination. We acknowledge the distinction between the stochastic representation HH and the deterministic feature hh: the variational KL upper bound in Eq. (10) applies strictly to HH and does not directly guarantee the complete removal of irrelevant information from hh. Accordingly, we position Lcmin\mathcal L_{\mathrm{cmin}} as an auxiliary regularizer that complements target-conditioned adversarial debiasing, rather than treating a smaller KL value as sufficient evidence of successful debiasing.

Importantly, Appendix E provides direct empirical evidence of the effectiveness of conditional minimality against unmodeled perturbations through independently sampled low-frequency Gaussian noise:

Noise LevelMethodWGA Drop (↓)Feature Drift (↓)
0.1Cond. Adv.1.4190.208
0.1Full CMDR0.6210.182
0.2Cond. Adv.4.0490.245
0.2Full CMDR2.4570.213

At both perturbation levels, full CMDR exhibits smaller WGA degradation and feature drift while maintaining comparable performance on clean data. These results suggest that the practical role of conditional minimality is to reduce sensitivity to residual within-class nuisance factors while preserving target-discriminative information, rather than enforcing complete information removal.

Regarding the implementation, our feature projection layer uses BatchNorm and ReLU, without explicit L2 normalization of the final features. The classification head is optimized using AdamW with a weight decay of 10−410^{-4}, but no explicit upper bound is imposed on its weight norm. For the fixed variance τ2\tau^2 of the Gaussian representation, we calibrate it using feature statistics from an ERM reference model on the training set. Specifically, we first compute the empirical variance of each feature dimension and then average across all dimensions:

τ2=1d∑j=1dVar⁡x∼Dtrain[hjERM(x)].\tau^2=\frac{1}{d}\sum_{j=1}^{d} \operatorname{Var}_{x\sim\mathcal D_{\mathrm{train}}} \left[h_j^{\mathrm{ERM}}(x)\right].

For example, on Waterbirds, using 1,024-dimensional ERM features extracted from 4,795 training samples, we obtain τ2=0.121827\tau^2=0.121827, corresponding to a standard deviation of τ=0.349038\tau=0.349038. This data-driven calibration aligns the Gaussian noise scale with the empirical scale of the reference features.

We also acknowledge that a fixed variance and the optimizer's implicit regularization do not theoretically rule out the scale-degeneracy issue raised by the reviewer. To further investigate this concern, we conducted a targeted feature-scale analysis.

To examine the practical effect of the minimality regularizer on learned features, we measured feature norms, classifier weight norms, and representation changes under different values of β\beta, alongside the performance results reported in Table 5.

Table R1. Feature-scale diagnostics and minimality sensitivity.

β\betaFeature NormClassifier NormCosine Sim.Relative ChangeWGA (%) ↑Avg (%) ↑
011.30020.92290.98430.181788.86494.991
10−410^{-4}11.30030.92280.95200.309190.81095.560
5×10−45\times10^{-4}11.29870.92280.92570.391590.52594.219

Note: Cosine similarity and relative change are computed with respect to the corresponding reference model.

The results show that feature norms (approximately 11.30) and classifier weight norms (approximately 0.923) remain nearly unchanged across different regularization strengths. We do not observe substantial feature contraction or compensatory amplification of the classifier weights. Meanwhile, cosine similarity decreases from 0.9843 to 0.9257, while relative feature change increases from 0.1817 to 0.3915. These observations indicate that changes in regularization strength are accompanied by measurable differences in feature directions and learned representations, rather than merely changes in feature scale.

Notably, at β=10−4\beta=10^{-4}, WGA improves from 88.864% to 90.810%, while Avg increases from 94.991% to 95.560%. Together with the ablation results in Table 3 and the controlled perturbation experiments in Appendix E, these findings provide additional empirical evidence of the robustness benefits of the minimality regularizer.

Appendix K further demonstrates that moderate regularization helps balance worst-group robustness and average classification performance. On BFFHQ with 5% bias-conflicting samples, halving the default regularization weight β\beta improves WGA from 90.0% to 90.8% and Avg from 95.1% to 95.3%.

AuthorsOpenReview ↗

W2: Performance Trade-offs and the Anomalous LfF Result

We acknowledge that CMDR does not outperform every baseline under all experimental settings, although it remains highly competitive across most evaluations. For the ViT-L/14 results on CelebA and BAR highlighted by the reviewer, CMDR achieves lower WGA than the respective best-performing baselines. However, its average accuracy reaches 93.878% and 97.293%, respectively, exceeding the corresponding baselines' 89.092% and 94.818%. These results illustrate the trade-off between worst-group robustness and overall classification accuracy across different methods.

Regarding the anomalous performance of LfF on CelebA with ResNet-50, we further examined intermediate results from consecutive training epochs around the selected checkpoint:

EpochBiased WGABiased AvgDebiased WGADebiased Avg
432.22297.27415.92626.152
515.55695.29113.07928.425
642.77897.32317.47838.747
736.11197.03511.08232.146
842.77897.63314.21434.883

The biased model consistently maintains high average accuracy (95%–98%), whereas the debiased model exhibits substantial performance degradation across multiple consecutive epochs.

By design, LfF dynamically assigns sample weights based on the relative losses of its biased and debiased models, encouraging the latter to focus on bias-conflicting samples. We hypothesize that, under CLIP-pretrained representations and the highly imbalanced distribution of CelebA, rapid fitting of majority samples by the biased model may concentrate training weights on a small subset of difficult examples. This could prevent the debiased model from receiving sufficiently diverse and stable training signals, potentially contributing to the persistent performance degradation observed in our reproduction.

AuthorsOpenReview ↗

Q2: Baseline Reproduction and Hyperparameter Selection

We reproduced all baseline methods except Zero-Shot CLIP and TTR, as their original implementations employ heterogeneous backbone architectures. Specifically, LfF, BE, BM, and TAB originally use ResNet-18; JTT, GroupDRO, DFR, NSF, AGS, and Hierarchical DRO primarily use ResNet-50; while CFR and TTR employ CLIP-based architectures. In addition, DeNetDM, GERNE, and SEBRA adopt different ResNet architectures across datasets. ERM, OS, and UW are inherently backbone-agnostic.

To minimize architectural differences and improve comparability, we adapted these methods to CLIP ResNet-50 and CLIP ViT-L/14 using their publicly available original implementations, while maintaining consistent dataset splits and evaluation protocols.

Regarding hyperparameter selection, for Waterbirds, CelebA, and MetaShift, we prioritized the recommended configurations provided by the original implementations. For BAR and BFFHQ, where dataset-specific configurations were unavailable, we selected reference configurations based on the similarity of bias types. Specifically, we used Waterbirds as the reference for BAR because both involve background or scene-related biases, and CelebA as the reference for BFFHQ because both involve facial-attribute biases.

Comments on the paper

5 comments
AuthorsOpenReview ↗

Clarification on Baseline Reproduction and Experimental Settings

We thank the reviewer for their continued attention and valuable feedback. Regarding the discrepancy between SCER's originally reported results and our reproduced results, we believe that differences in backbone architectures, pretraining strategies, and training configurations are important potential contributing factors. The original SCER paper employs a pretrained ResNet-50 with its specific training protocol, whereas our experiments adapt SCER to the unified CLIP ResNet-50 and CLIP ViT-L/14 backbones used throughout our manuscript, with consistent data preprocessing, dataset splits, and evaluation procedures. Importantly, this adaptation is not specific to SCER but follows a consistent experimental strategy applied to all baselines in our study. Existing visual debiasing methods employ substantially different backbone architectures: LfF, BE, BM, and TAB primarily use ResNet-18; JTT, GroupDRO, DFR, NSF, and AGS mainly adopt ResNet-50; while CFR and TTR employ CLIP-based backbones. Directly comparing results obtained using different architectures and pretraining strategies may introduce confounding effects arising from differences in model capacity and pretrained representation quality, making it difficult to isolate the effectiveness of the debiasing methods themselves. Therefore, we adapt these methods to CLIP ResNet-50 and ViT-L/14 to minimize architectural differences and improve the consistency and comparability of our evaluations.

Regarding bias identification and hyperparameter settings for LfF, JTT, and SEBRA, we primarily follow their original papers and publicly available implementations, preserving their core algorithmic mechanisms while integrating them into our unified training and evaluation framework. Specifically, LfF jointly trains a biased model and a debiased model to perform dynamic sample reweighting, with both models initialized from the same pretrained weights. For JTT, the first-stage ERM model used to identify difficult examples employs the same CLIP backbone as in our main experiments. For SEBRA, the spuriosity-ranking model is trained following the self-guided learning procedure described by the original authors. For hyperparameter selection, we prioritize the recommended configurations provided in the authors' publicly available implementations. When dataset-specific configurations are unavailable, we transfer hyperparameters from datasets with similar visual characteristics and bias types. For example, when a method provides configurations for CelebA but not BFFHQ, we use the CelebA configurations as a reference, since both datasets involve facial images and attribute-related biases. We also acknowledge that, given the large number of baselines, datasets, and backbone combinations, we did not conduct an extensive, independent hyperparameter search for every configuration. Therefore, our reported results reflect the reproduced performance under a unified CLIP-based experimental protocol using the original authors' recommended configurations, rather than necessarily representing the optimal performance achievable by each baseline.

AuthorsOpenReview ↗

Clarification on Attribute Correlations and Bias Transfer in Appendix L

We thank the reviewer for this further suggestion. We agree that examining the statistical correlations between the target attribute and additional attributes can provide a more accurate interpretation of the results in Appendix L. Following the reviewer's suggestion, we compute the Pearson correlation coefficients (equivalent to the Phi coefficient for binary attributes) between Blond Hair and four additional attributes using all 162,770 samples in the CelebA training set. We further examine their correlations with Gender. The results are summarized below.

Table R. Attribute correlations on the CelebA training set.

AttributeCorr. with Blond HairCorr. with Male
Black Hair−0.2350.111
Heavy Makeup0.251−0.666
Smiling0.090−0.137
Straight Hair0.0050.063

To address the reviewer's specific concern regarding Heavy Makeup, we further examine its correlation with Blond Hair separately within each gender group:

GenderCorr. (Blond Hair, Heavy Makeup)Makeup Rate (Non-Blond / Blond)
Female0.06864.2% / 71.7%
Male0.0040.3% / 0.4%

The results show that although Blond Hair and Heavy Makeup exhibit a positive correlation in the overall training set (0.251), Heavy Makeup is much more strongly correlated with Gender (−0.666). After stratifying the samples by gender, the correlations between Blond Hair and Heavy Makeup decrease substantially to 0.068 for Female and 0.004 for Male. This suggests that their overall association is largely influenced by gender composition. We therefore acknowledge the reviewer's concern that the WGA improvement on Heavy Makeup-defined groups in Appendix L may partly arise from CMDR's mitigation of Gender bias, rather than demonstrating the independent removal of Heavy Makeup bias.

Importantly, the primary purpose of Appendix L is not to claim that CMDR independently removes all potential biases simultaneously, but rather to examine whether supervised debiasing with respect to Gender leads to noticeable robustness degradation on groups defined by other observable attributes. The existing results show that CMDR improves WGA on Heavy Makeup-defined groups from 80.30% to 87.27%. Moreover, WGA does not decrease for Black Hair, Smiling, or Straight Hair, all of which exhibit substantially weaker correlations with Gender. We therefore interpret Appendix L more cautiously: these findings provide evidence that CMDR does not exhibit an obvious shift toward the evaluated alternative biases, rather than establishing independent removal of those biases.

AuthorsOpenReview ↗

Clarification on Theoretical Novelty and Distinctions from Prior Work

We thank the reviewer for highlighting these two important related works. We acknowledge that Ghassami et al. (ISIT 2018) and Li et al. (2023) are closely related to the information-theoretic analysis underlying CMDR. However, we would like to clarify that these works differ from CMDR in their research objectives, the focus of their theoretical analyses, and their algorithmic implementations.

First, regarding Eq. (4), Ghassami et al. primarily investigate the construction of fair predictors when protected attributes are known. Their approach optimizes the probability distribution of an auxiliary representation and constructs a Bayesian decision rule based on that representation to satisfy specific group fairness criteria. In contrast, CMDR focuses on spurious correlations and worst-group robustness in visual learning, particularly on suppressing explicitly modeled biases and unmodeled nuisance factors while preserving target-discriminative information. Motivated by the tension between global bias invariance and target sufficiency, we formulate Eq. (4) around three complementary information-theoretic objectives: target sufficiency preserves discriminative information, target-conditioned debiasing suppresses within-class bias dependence, and conditional minimality further limits residual input redundancy conditioned on the target. In particular, we use the conditional data processing inequality I(H;U∣Y)≤I(H;X∣Y)I(H;U\mid Y)\leq I(H;X\mid Y) to explain how conditional minimality constrains unmodeled nuisance factors, and further establish a theoretical connection between conditional bias dependence and group-risk disparities. At the algorithmic level, rather than directly optimizing the distribution of an auxiliary representation as in Ghassami et al., CMDR instantiates these objectives through classification loss, target-conditioned adversarial learning, and class-conditional variational KL regularization for pretrained visual backbones. Furthermore, we introduce continuous pseudo-bias signals and confidence weighting, enabling the same framework to operate both with known bias labels and without ground-truth bias annotations. Thus, CMDR's primary contributions lie in its theoretical interpretation of visual group robustness, trainable deep representation learning mechanisms, and extension to weakly supervised debiasing, rather than merely proposing a particular combination of mutual information terms.

Second, regarding Proposition 1, Li et al. (2023) primarily investigate the theoretical limitations on predictive performance arising from the removal of attribute-specific information under strong attribute bias. We acknowledge that their information-theoretic analysis is closely related to Proposition 1. However, CMDR further investigates why global bias invariance may conflict with target sufficiency when the target and bias variables are statistically dependent, and how this observation can motivate a more appropriate target-conditioned debiasing objective. Specifically, we use Proposition 1 to characterize the incompatibility between target sufficiency and complete global bias independence, thereby motivating target-conditioned debiasing. Building on this observation, we further introduce conditional minimality to constrain residual nuisance information and derive a group-risk bound connecting conditional bias dependence to worst-group robustness. Therefore, Proposition 1 serves not only to characterize the predictive limitations of removing attribute-specific information, but also as a theoretical motivation for our subsequent conditionally minimal sufficient representation framework.

More importantly, we translate the above theoretical analysis into a unified, empirically validated algorithmic framework and demonstrate its applicability under different levels of bias supervision. When ground-truth bias labels are available, CMDR-Bias suppresses known biases through target-conditioned adversarial learning. When such labels are unavailable, CMDR-Pseudo provides weak supervision through continuous pseudo-bias signals and confidence weighting. We therefore emphasize that CMDR's main contribution is to develop the related information-theoretic ideas into a conditionally minimal sufficient representation learning framework for mitigating spurious correlations in visual learning, with a unified optimization mechanism that supports both bias-annotated and bias-unannotated settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.