- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The author argue that selective mixup methods (e.g., LISA) confound the effect of interpolation and sampling of training samples, and that both can have an effect in improving worst-group accuracy (WGA, per the author's acronym). This is because pairing by both label and spurious attribute oversamples rare groups, which is known to already help by rebalancing the samples.
To separate the two effects, the authors propose the so-called "clean-endpoint baselines", which run the same pair sampler on unmixed examples. They classify pairs in a confusion-matrix style (SS/SD/DS/DD) based on whether they have the same label and attribute, and introduce DPE as the measure of the samples that differ in exactly one of the two. They then empirically show that much of WGA changes are due to endpoint training alone.
Critical strengths
- Clean experiments: the paper compares each mixup method against a baseline with the same exact settings but trains on two original images separately. This means that any difference must be due to the interpolation. The same goes for the synthetic experiment.
- All experiments are ran multiple times (although seeds are confusing, see weaknesses). The code is provided in the supplementary material.
Critical weaknesses
-
The paper does not frame itself in the literature. There is no dedicated Related Work section, and section 2 is a lengthy discourse on generic preliminaries and related work with not a single citation. In the journey of trying to understand what the author's contributions were, I found this paper [1], which seems to be extremely correlated, both in narrative and conclusion. The fact that not only this paper, but also the other literature, is missing, makes it extremely hard to understand what the author's contributions are. They state them briefly in the introduction, but that segment is not informative, and if all of section 2 is established literature, the contribution of the paper is narrow.
-
The experimental settings are very confusing. The authors claim to be using 5 different datasets, but only 3 are reported in the main text, and 2 in the appendix (Colored MNIST and biased CIFAR). To the best of my understanding, they are also used for different experiments (endpoint decomposition and input vs feature comparison). This needs to be mentioned explicitly and explained in the paper, because it is not clear. Also, each dataset use a different number of seeds, which is confusing. Why can't they all be standardized to the same task and seeds?
[1]. @InProceedings{pmlr-v235-teney24a, title = {Selective Mixup Helps with Distribution Shifts, But Not ({O}nly) because of Mixup}, author = {Teney, Damien and Wang, Jindong and Abbasnejad, Ehsan}, booktitle = {Proceedings of the 41st International Conference on Machine Learning}, pages = {47948--47964}, year = {2024}, editor = {Salakhutdinov, Ruslan and Kolter, Zico and Heller, Katherine and Weller, Adrian and Oliver, Nuria and Scarlett, Jonathan and Berkenkamp, Felix}, volume = {235}, series = {Proceedings of Machine Learning Research}, month = {21--27 Jul}, publisher = {PMLR}, pdf = {https://raw.githubusercontent.com/mlresearch/v235/main/assets/teney24a/teney24a.pdf}, url = {https://proceedings.mlr.press/v235/teney24a.html}, }
Questions for authors
See the two weaknesses. The framing in the literature and understanding exactly what the contribution is and what, instead, is established literature, is particularly critical.