- Rating
- 2
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper investigates what performance gains from data selection actually measure, distinguishing among candidate data sources, the class–group composition of selected subsets, the identities of selected examples, and the representations used by the scorer and learner. The authors introduce a same-count random control: examples are redrawn while preserving the class–group counts produced by the selector, allowing the selected examples to be compared against random subsets with the same composition. The experiments cover constructed tasks from eleven benchmark sources, music composer classification, and scorer–learner crossing experiments on two data sources.
Critical strengths
Redrawing examples while holding the learner, candidate pool, and actual class–group counts fixed provides a more informative comparison than evaluating the selector against ordinary random sampling alone. Changing the learner’s representation while keeping the selected examples and random controls fixed also helps avoid attributing learner changes to a new ranking strategy.
Critical weaknesses
The five implementations in Section 3 and Appendix A consist of mean-gap projection, two reference-versus-pool discriminators, a gradient-alignment proxy, and kernel herding. The authors explicitly distinguish these implementations from full methods such as DSIR, TracIn, and TRAK. This is an appropriate qualification, but it also means that the results cannot directly represent the practical performance of those established methods. Some benchmark sources are additionally converted into binary classification problems that differ substantially from their original tasks. For example, OfficeHome labels are defined through membership in a fixed set of class IDs, while CivilComments uses the fixed labeling rule . These constructions enable controlled experiments, but using eleven sources is not equivalent to validating the conclusions on eleven standard benchmark tasks. The rationale for the task constructions and selection rules needs further justification, as does whether the negative residuals persist in settings closer to the original tasks. Prior work has already examined how random baselines, training conditions, and transfer across models affect selection gains, including Munjal et al., CVPR 2022 and Lowell et al., EMNLP 2019. The paper cites these studies, but the authors should more clearly identify which new empirical conclusions the same-count control enables beyond reaffirming that gains depend on experimental conditions.
Questions for authors
-
Is there existing evidence that the proxy implementations behave as intended in the corresponding settings, rather than producing systematically negative residuals because of small reference sets, fixed hyperparameters, or deterministic ranking procedures? Please provide implementation-validation results or records of hyperparameter selection using an independent validation set, and clarify which conclusions apply only to the current proxy rules.
-
How were the binary label set for OfficeHome and the label threshold for CivilComments determined? Are results available for the original tasks or alternative label definitions?
-
For the fixed panel of eleven sources, what population or model-based estimand are the source-level intervals intended to estimate? What assumptions are required by the sign-flip tests? Please explain why the registered conclusions of the reference-size study primarily rely on source-level inference, and how this should be interpreted alongside repeated-randomization uncertainty on the fixed panel.
-
Using the existing paired records, could the authors report the following quantity and its conditional uncertainty?
.
This would not constitute a causal decomposition, but it would more directly show whether the ranking advantage over random sampling differs between the two candidate sources.