acceptodds
Under review as a conference paper at ICLR 2027

Response Is Not Dispensability:A Controlled Study of Counterfactual Data Selection

Abstract

Selecting training evidence from a model's intervention response requires separating response magnitude from learning utility. We examine a low-response retention rule and show that probability saturation favors large-margin identities even at a fixed intervention-induced logit contrast. An exact binary decomposition and a multiclass saturation construction characterize this effect. We test its consequences in synthetic and rendered-digit tasks with known label-preserving interventions, matched class budgets, and shared coverage rules. In a linear factorial, increasing core strength changes rare-evidence retention from to under low-response selection, while a nuisance-logit ranking retains approximately half. Nonlinear experiments on synthetic and two colored-digit tasks separate retained composition from retrained subgroup accuracy. Across all 18 common training settings, applying the intervention during training on the same retained identities improves mean four-group worst accuracy over training without intervention. With validation-selected configurations, the gains are , , and percentage points on synthetic data, colored digits, and colored MNIST. The results establish that response, retention, and intervention use are distinct decisions that require separate validation.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Data pruning, counterfactual sensitivity, spurious correlations, rare subpopulations, group robustness, data augmentation.

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_LuB2OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper focuses on the question that whether examples whose predictions change little under a label-preserving nuisance intervention form useful training subsets. It shows that a small probability change can reflect confidence rather than weak sensitivity to the intervention. For binary predictions, an exact decomposition separates the paired-logit midpoint from the intervention contrast. Experiments on synthetic data and colored MNIST examine rare-example retention and subgroup accuracy under matched class and coverage budgets. A further comparison applies the intervention during retraining while holding the selected identities fixed. Results show that augmentation improves mean worst-group accuracy in all matched configurations, while low-response selection has no consistent advantage over coverage-random selection.

Critical strengths

The analysis identifies a specific failure of probability-based scoring. It shows exactly how increasing the paired-logit midpoint reduces probability response at a fixed intervention contrast. Besides, it also includes temperature experiment which changes retained composition without changing predicted classes. In Appendix, it also distinguishes the binary result from the conditions required for multiclass saturation. Together, these results explain why response magnitude alone need not measure an example's learning value.

Critical weaknesses
  1. The overall framework is obscure and hard to follow. A dozen of notations, such as z1(x)z_1(x), z0(x)z_0(x), the transformation TT, are not clearly defined. Besides, the practical motivation for the rule under study also needs stronger support. The manuscript defines retention by low paired-probability response but does not identify an established method whose rule closely matches it.

  2. The experiments do not fully connect saturation to the loss in subgroup accuracy. The temperature sweep measures retention, and the low-nuisance-logit control appears in the linear factorial rather than matched nonlinear retraining. These results show that saturation changes selection, but they do not determine how much of the subsequent accuracy loss comes from saturation itself, as opposed to other difficulty or subtype information captured by the early scorer. This limits the proposed explanation of the learning failure. Moreover, The ordering of selection methods is sensitive to the coverage and scoring method. All five original intervals in Table 2 include zero, and with fresh seeds, the synthetic contrast drastically varies among clusters, while the digit comparisons also remain uncertain. While the evidence establishes useful failure cases, it gives limited guidance about when low-response selection will help or hurt.

Questions for authors
  1. Which existing algorithm or curation practice retains factual identities with the lowest paired-probability response? Please specify how its score, ranking direction, and selected examples correspond to the rule studied here.

  2. With a fixed scorer, does changing only temperature affect retrained four-group and eight-subgroup accuracy? Do you have a matched nonlinear comparison using logit contrast? These results would help establish whether saturation explains the observed learning loss.

  3. How sensitive are the main selection results to scorer training duration and validation-based temperature calibration?

  4. Do the main findings persist when coverage uses learned features or when the intervention introduces a controlled amount of target distortion?

Reviewer_f9ypOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper looks at training data by an example's probability response to a known label-preserving intervention T, where a response is s = |p(x)-|p(Tx)|. The authors argue that keeping the lowest response examples conflates reliance on nuisances with model confidence. An exact binary identity s=sinh|v|/(cosh t + cosh|v|) splits a factual vs conterfactual logit pair into a midpoint margin t and an intervention half-difference v, showing that s strictly decreases in |t| when |v| is fixed. As a result, low-response selection favors confident example, and three-class softmax construction extends this. In an 8-dimension synthetic factorial, colored sklearn digits and colored MNIST, with class-matched budget, low-response ranking discards rare subtypes as core signal strengthens, while ranking by |v|keeps only about half. The authors conclude that response retention and intervention use are separate decisions that need independent validation.

Critical strengths

Experimental controls appear carefully structured, with class budgets, coverage quotas, seeds and optimization settings matching. Configurations are only selected on val loss with no test data included. Authors separated retention and learning cleanly, by reporting retained composition and retained subgroup accuracy, which appears to show a clear dissociation between the two. Result reporting also appears to be honest, the authors call overlaps with zero as non-equivalent instead of no effect, and broken results like the LfF result is properly reported.

Critical weaknesses

The theoretical novelty proposed by the authors in this paper is thin. The main result is sigmoid saturation. The identity is just a few lines of algebra and the multiclass part is one manually selected construction. The authors define the low-response retention rule, but none of the works they cite uses it. The closest CF selection method (https://arxiv.org/pdf/2211.11109) does it completely differently, they choose factual examples at random and keep the CF samples with the highest GE score. WIth no evidence that practitioners keep low-response samples, the central negative result has no clear target. On the same vein, one of the cited papers (https://arxiv.org/abs/2606.02830) already studies how sample scores can conflate spurious and core features, although using learning dynamics rather than intervention response, without assuming a known intervention. Their selector does not need group labels and is evaluated on other datasets, so it would be a natural baseline, as it uses less information than this paper's oracle->intervention setting. On a separate note, the collapse of low EL2N selection on colored digits replicates a result already reported there, where EL2N gets 0.00 on CMNIST, and should not count as new evidence. The "main" result is expected by construction, since flipping an exact oracle nuisance with p=0.5 decorrelates the nuisance from the label in training, resulting in an expected four-group worst accuracy improvement. Full data CF augmentation dominates already in every table. The 18/18 result is also measured against coverage random with no augmentation, not against the best selection method, so it doesn't really compare selection with intervention use. The phenomenon presented by the authors is not uniform across settings. On digits, low sensitivity retains more rare samples than random AND gets higher worst-group accuracy (53.3% vs 47.6%, 27.0 vs 24.0). In both synthetic conditions, low sensitivity beats random. The manuscript's narrative skips this. Also, ranking by |v| is shown to fix retention, but is never evaluated in retraining even though it seems like the natural constructive conclusion? Coverage + signed benefit is often the strongest selector while retaining few rare examples. It most likely works by selecting nuisance-minority examples, much like JTT. That seems like a more interesting finding, yet it goes undiscussed in the manuscript. While this study positions itself as a controlled study, and the exact interventions do justify synthetic settings for the mechanisms, the conclusions are framed as guidance for data selection in practice, and the evidence does not go that far. The experiments are performed on toy datasets (8 dimensional Gaussian, color MNIST, etc.). Other dataset like Waterbirds or CelebA are discussed but never run. The authors cite the lack of validated paired transformations as the obstacle, but waterbirds is built by composting CUB foregrounds onto places backgrounds, so background-swapped pairs follow directly from the segmentation masks. Some related work already evaluates on these benchmarks. At least one natural-image experiment would greatly improve this work, to show that the saturation effect and the gap between selection use, carry over beyond the known generators. Five seeds give very wide intervals (for example, [-17.56, 7.24]), and many comparisons are made without multiplicity control, while some variances are huge (for example, rate retention 69.3 ± 42.6), which suggests unstable or bimodal clustering behavior that's not investigated (or at least not reported). The related work omits CF data augmentation and invariance methods that use paired inputs. The matched subset augmentation, which flips the nuisance with probability 0.5, is CF data substitution (https://aclanthology.org/D19-1530/), and why CF pairs reduce spurious reliance has been analyzed by (https://arxiv.org/abs/1909.12434). CF logit pairing (https://arxiv.org/abs/1809.10610) and conditional variance penalties (https://arxiv.org/abs/1710.11469) also train directly on faculty-CF pairs, the same object the paper's response score is built from. They would be natural comparisons for the intervention use half of the paper.

Questions for authors

1/ The phenomenon presented by the authors is not uniform across settings. On digits, low sensitivity retains more rare samples than random AND gets higher worst-group accuracy (53.3% vs 47.6%, 27.0 vs 24.0). In both synthetic conditions, low sensitivity beats random. The manuscript's narrative skips this. Also, ranking by |v| is shown to fix retention, but is never evaluated in retraining even though it seems like the natural constructive conclusion? 2/ Coverage + signed benefit is often the strongest selector while retaining few rare examples. It most likely works by selecting nuisance-minority examples, much like JTT. That seems like a more interesting finding, yet it goes undiscussed in the manuscript. While this study positions itself as a controlled study, and the exact interventions do justify synthetic settings for the mechanisms, the conclusions are framed as guidance for data selection in practice, and the evidence does not go that far. The experiments are performed on toy datasets (8 dimensional Gaussian, color MNIST, etc.). Other dataset like Waterbirds or CelebA are discussed but never run. The authors cite the lack of validated paired transformations as the obstacle, but waterbirds is built by composting CUB foregrounds onto places backgrounds, so background-swapped pairs follow directly from the segmentation masks. Some related work already evaluates on these benchmarks. At least one natural-image experiment would greatly improve this work, to show that the saturation effect and the gap between selection use, carry over beyond the known generators.

3/ The related work omits CF data augmentation and invariance methods that use paired inputs. The matched subset augmentation, which flips the nuisance with probability 0.5, is CF data substitution (https://aclanthology.org/D19-1530/), and why CF pairs reduce spurious reliance has been analyzed by (https://arxiv.org/abs/1909.12434). CF logit pairing (https://arxiv.org/abs/1809.10610) and conditional variance penalties (https://arxiv.org/abs/1710.11469) also train directly on faculty-CF pairs, the same object the paper's response score is built from. They would be natural comparisons for the intervention use half of the paper.

Reviewer_pMYQOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The authors present a method, probably, for perturbing data while preserving the label. For what purpose it is unclear. The lack of clarity in the writing makes it hard to know.

Critical strengths

None

Critical weaknesses

Clarity Purpose

Questions for authors

Can you please share the prompt you used to write this paper?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.