- Rating
- 1
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
The paper focuses on the question that whether examples whose predictions change little under a label-preserving nuisance intervention form useful training subsets. It shows that a small probability change can reflect confidence rather than weak sensitivity to the intervention. For binary predictions, an exact decomposition separates the paired-logit midpoint from the intervention contrast. Experiments on synthetic data and colored MNIST examine rare-example retention and subgroup accuracy under matched class and coverage budgets. A further comparison applies the intervention during retraining while holding the selected identities fixed. Results show that augmentation improves mean worst-group accuracy in all matched configurations, while low-response selection has no consistent advantage over coverage-random selection.
Critical strengths
The analysis identifies a specific failure of probability-based scoring. It shows exactly how increasing the paired-logit midpoint reduces probability response at a fixed intervention contrast. Besides, it also includes temperature experiment which changes retained composition without changing predicted classes. In Appendix, it also distinguishes the binary result from the conditions required for multiclass saturation. Together, these results explain why response magnitude alone need not measure an example's learning value.
Critical weaknesses
-
The overall framework is obscure and hard to follow. A dozen of notations, such as , , the transformation , are not clearly defined. Besides, the practical motivation for the rule under study also needs stronger support. The manuscript defines retention by low paired-probability response but does not identify an established method whose rule closely matches it.
-
The experiments do not fully connect saturation to the loss in subgroup accuracy. The temperature sweep measures retention, and the low-nuisance-logit control appears in the linear factorial rather than matched nonlinear retraining. These results show that saturation changes selection, but they do not determine how much of the subsequent accuracy loss comes from saturation itself, as opposed to other difficulty or subtype information captured by the early scorer. This limits the proposed explanation of the learning failure. Moreover, The ordering of selection methods is sensitive to the coverage and scoring method. All five original intervals in Table 2 include zero, and with fresh seeds, the synthetic contrast drastically varies among clusters, while the digit comparisons also remain uncertain. While the evidence establishes useful failure cases, it gives limited guidance about when low-response selection will help or hurt.
Questions for authors
-
Which existing algorithm or curation practice retains factual identities with the lowest paired-probability response? Please specify how its score, ranking direction, and selected examples correspond to the rule studied here.
-
With a fixed scorer, does changing only temperature affect retrained four-group and eight-subgroup accuracy? Do you have a matched nonlinear comparison using logit contrast? These results would help establish whether saturation explains the observed learning loss.
-
How sensitive are the main selection results to scorer training duration and validation-based temperature calibration?
-
Do the main findings persist when coverage uses learned features or when the intervention introduces a controlled amount of target distortion?