acceptodds
Withdrawn from ICLR 2027

What Does a Data-Selection Gain Measure?

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

A gain from selecting training data depends on the representation used to score examples and train the learner. It also depends on the candidate supply, the selected group mix and the labels used for guidance. We study what these comparisons measure using frozen feature maps and linear heads. Our central control redraws random examples at the same class–group counts as each selection. Across tasks from eleven benchmark sources, five registered comparisons show no clear worst-group benefit; secondary overall-accuracy residuals are negative in all 55 correlated source–implementation means. Two music comparisons are positive. Selecting real examples from unseen work families rather than generated variants of training works raises test accuracy by 2.88 percentage points. Random acquisition from the two supplies already gives a descriptive 2.60-point difference. On identical selected rows and random controls, retaining original descriptors alongside constructed features increases the selection advantage by 0.55 points relative to constructed features alone. This changes the learner, not the ranking rule. A larger reference gives a positive source-level estimate but an interval over repeated draws that spans zero; guidance shows no clear advantage over direct label use. Feature refresh fails its positive primary, and scorer–learner matching outside music shows no clear benefit. These findings do not provide a winning selector. They show why a selection gain should be reported with its candidate supply, scorer and learner, group-count control, and final accuracy. Code and numerical evidence are open-sourced at: https://anonymous.4open.science/r/data-selection-gains-ready-B863/README.md.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
data selection, representation learning, frozen representations, worst-group evaluation, label supervision

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_svmHOpenReview ↗
Rating
2
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates what performance gains from data selection actually measure, distinguishing among candidate data sources, the class–group composition of selected subsets, the identities of selected examples, and the representations used by the scorer and learner. The authors introduce a same-count random control: examples are redrawn while preserving the class–group counts produced by the selector, allowing the selected examples to be compared against random subsets with the same composition. The experiments cover constructed tasks from eleven benchmark sources, music composer classification, and scorer–learner crossing experiments on two data sources.

Critical strengths

Redrawing examples while holding the learner, candidate pool, and actual class–group counts fixed provides a more informative comparison than evaluating the selector against ordinary random sampling alone. Changing the learner’s representation while keeping the selected examples and random controls fixed also helps avoid attributing learner changes to a new ranking strategy.

Critical weaknesses

The five implementations in Section 3 and Appendix A consist of mean-gap projection, two reference-versus-pool discriminators, a gradient-alignment proxy, and kernel herding. The authors explicitly distinguish these implementations from full methods such as DSIR, TracIn, and TRAK. This is an appropriate qualification, but it also means that the results cannot directly represent the practical performance of those established methods. Some benchmark sources are additionally converted into binary classification problems that differ substantially from their original tasks. For example, OfficeHome labels are defined through membership in a fixed set of class IDs, while CivilComments uses the fixed labeling rule toxicity>0\mathrm{toxicity}>0. These constructions enable controlled experiments, but using eleven sources is not equivalent to validating the conclusions on eleven standard benchmark tasks. The rationale for the task constructions and selection rules needs further justification, as does whether the negative residuals persist in settings closer to the original tasks. Prior work has already examined how random baselines, training conditions, and transfer across models affect selection gains, including Munjal et al., CVPR 2022 and Lowell et al., EMNLP 2019. The paper cites these studies, but the authors should more clearly identify which new empirical conclusions the same-count control enables beyond reaffirming that gains depend on experimental conditions.

Questions for authors
  1. Is there existing evidence that the proxy implementations behave as intended in the corresponding settings, rather than producing systematically negative residuals because of small reference sets, fixed hyperparameters, or deterministic ranking procedures? Please provide implementation-validation results or records of hyperparameter selection using an independent validation set, and clarify which conclusions apply only to the current proxy rules.

  2. How were the binary label set for OfficeHome and the label threshold for CivilComments determined? Are results available for the original tasks or alternative label definitions?

  3. For the fixed panel of eleven sources, what population or model-based estimand are the source-level intervals intended to estimate? What assumptions are required by the sign-flip tests? Please explain why the registered conclusions of the reference-size study primarily rely on source-level inference, and how this should be interpreted alongside repeated-randomization uncertainty on the fixed panel.

  4. Using the existing paired records, could the authors report the following quantity and its conditional uncertainty?

Δranking=[V(Sreal)−E[V(Ureal)]]−[V(Sgenerated)−E[V(Ugenerated)]]\Delta_{\mathrm{ranking}} = [V(S_{\mathrm{real}}) - \mathbb{E}[V(U_{\mathrm{real}})]] - [V(S_{\mathrm{generated}}) - \mathbb{E}[V(U_{\mathrm{generated}})]].

This would not constitute a causal decomposition, but it would more directly show whether the ranking advantage over random sampling differs between the two candidate sources.

Reviewer_gqtbOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper is looking at how to measure whether a data selection method truly improves over the baseline: does it really pick better samples, or can random sampling also do as well if it matches the distribution of the selected samples? The example they give is a good one: suppose you are tasked to select bird images. You select images with rare backgrounds and your model's performance increases. Now, if you randomly select samples that also have rare backgrounds, they are asking whether random selection will do worse (because the rare background samples that you picked are really good) or the same. Granted, they don

Critical strengths

Their study is very interesting, of trying to diagnose why random selection is automatically the low-performing baseline in all data-selection works. I like the way they break down the factors that contribute to data selection (the candidate set, the feature space, the group mix). Table 1 shows interesting results. This makes me think that we should design random selection baselines more carefully.

Critical weaknesses

This paper is very clearly AI written and poorly written: usually humans write sequentially, explaining things from high level to low level as they keep writing. This paper is not sequentially written. For example, in the abstract, when they are presenting some findings, one of their findings is "two music comparisons are positive" (line 016). There is no explanation what a "music comparison" is, how it was done, or what they are comparing. The idea is quite interesting, so I really hope the authors can rewrite this without AI to improve the presentation of this paper.

Questions for authors
  1. What does "construction" mean in "fourteen constructions from eleven sources" (line 143/144)? I looked everywhere in the paper, and I still don't understand what a construction exactly is. I even asked Claude, and it said, "no single sentence defines it".
  2. More of a comment than a question: consider moving some of the experimental details from Appendices A and B. The "fourteen constructions from eleven sources" were not outlined in the main paper, nor was it clear that these details were in the Appendix.

One note: it's very possible that I misunderstood something and would be happy to improve my assessment during the rebuttal period.

Reviewer_dZF9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

As far as I could tell through the hazy veil of me trying to read indecipherable writing, authors feel that a big question they are asking-and-answering is whether data selection rules for classifiers just depend on the ratio of counts of each class, and they propose that a random control for active-sampling /data-selection methods should control for the count of each class in the group. I don’t think their evidence shows that in any convincing or general or statistically significant wayl. Plus, if a data selection rule is good at figuring out that you should be selecting more of a particular group, that is a legit value of that method, and so even if their contention is true, I don’t see that conclusion as interesting enough to publish.

Critical strengths

It is a fair and reasonable question if a multi-class active sampler /training data-selector is largely useful due to the ratio of class samples it selects.

Critical weaknesses

Title and text seem to have been overly written by an AI, and I don’t mean that in a good way. Besides convoluted English, important paper-specific terms are used without pre-definition or in some cases, ever being defined. Honestly, I felt disrespected by this paper’s poor writing: if the authors couldn’t be bothered to write a clear paper, why should I have to spend my expert time reviewing it?

Just one minor example of the problematic writing is this key sentence listed as one of the paper’s core contributions: “We distinguish the lack of a clear worst-group benefit from negative secondary overall residuals across eleven sources, and show why a positive music selection contrast must be read beside random acquisition and absolute accuracy.” Or “Before outcomes, 112 test masks were sealed against frozen inputs.” I don’t know what these sentences mean, and there are many like that. And titles like “What to report, and what remains open” and “A small accounting relation” just stink of AI writing and are completely useless as section titles. And AI says “rows” whereas machine learning experts say “training samples.” And so on.

For a paper that relies heavily on experimental conclusions, the experiments seemed particularly weak, and I had trouble figuring out exactly what they are doing and on what datasets and problems. A key dataset is a 10-class classification problem on 3,868 music segments, as far as I can tell, their data selection policy gets to add a mere 6 samples to the preliminary train set. This is all too small and random to tell us much of anything. Table 4 seemed to use particularly strange non-standard constructions, Table 5 had smaller datasets and budgets than usual for this type of work.

The selection uses a greedy k-center algorithm, but evidence I have seen on k-centers beyond the original paper is that it’s usually a poor selection algorithm, see for example Bootstrapping for Batch Active Sampling, KDD 2021, which also found that vanilla random sampling beat k-centers on 6 of 7 data centers (but lots of other active sampling methods compared stat. sig. beat random).

Overall, I found the contributions and conclusions logically suspect at worst, and at best not interesting or significant enough for ICLR.

Questions for authors

None

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.