- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes a counterexample-guided method that does not require shortcut annotations in the training objective. The authors apply MixUp or CutMix to combine biased samples with clean counterexamples from different classes, and use the labels of the clean counterexamples to supervise the mixed samples. This creates a training signal in which the shortcut evidence conflicts with the assigned label. Experiments on Hidden Watermarks, Waterbirds, and CelebA report reduced reliance on the original shortcuts and improved worst-group accuracy.
Critical strengths
The paper addresses a practically important problem: improving model robustness when per-sample shortcut annotations are unavailable. The proposed method is simple and can be readily integrated into existing training pipelines.
The Hidden Watermarks experiment provides a relatively clear demonstration that training with label conflicts can suppress reliance on a specified bias-related signal.
The paper also discusses the method’s dependence on shortcut separability and the quality of the counterexamples.
Critical weaknesses
“No shortcut annotations” does not mean “no shortcut knowledge.” The method still requires researchers to identify the factor to be suppressed in advance and to collect or construct clean counterexamples accordingly. It therefore appears better suited to suppressing prespecified attributes than to discovering previously unknown shortcuts.
The background in Waterbirds is not necessarily a shortcut. Habitat may contain genuine and useful ecological information. Whether a feature should be considered a shortcut depends on the task definition and the deployment distribution, rather than on correlation alone. Label-conflicting mixtures may introduce new shortcuts. The model may exploit mixing boundaries, region sizes, interpolation ratios, or differences between data sources instead of learning stable task-relevant features. The current experiments mainly demonstrate that reliance on the original shortcut is reduced, but they do not rule out the possibility of shortcut substitution.
Consequently, the paper’s claims regarding the learning of causal or invariant features appear somewhat overstated. The current evidence more directly supports the narrower claim that the method reduces measurable reliance on a specified attribute.
Questions for authors
Without prior knowledge of the type of potential shortcut, how can the proposed “clean counterexamples” be obtained or validated? What criteria do the authors use to determine that the background is a shortcut that should be removed, rather than legitimate contextual information?
Have the authors evaluated the model using previously unseen mixing strategies, naturally occurring unedited counterexamples, or data generated through different pipelines, to rule out the possibility that the model exploits mixing artifacts?
Can the authors provide further evidence that the model genuinely shifts toward task-relevant features, rather than merely replacing the original shortcut with a newly constructed one?