acceptodds
Under review as a conference paper at ICLR 2027

Benchmaxxing: Selecting Data that Targets Many Benchmarks with Scalable Gradient-Based Methods

Abstract

Data quality is multi-dimensional: whether a sample is “good” depends on the model's downstream use, the checkpoint's weaknesses, and the composition of the surrounding dataset. Mainstay methods rely on human-like heuristics that, while useful, cannot target specialized use cases in a principled way. We advocate for benchmark-targeted data filtering, which scores samples based on their predicted effect on downstream performance. Prior approaches have been computationally expensive and limited to older models and small training scales. Our method, Benchmax, utilizes coverage selection criteria that improve the diversity and effectiveness of selected data, and combines gradient sketching with fast KNN search for efficient selection at scale. We distill the selection criteria into a multi-head classifier that simultaneously predicts a sample's impact across many benchmarks. This low-cost classifier scales to modern training regimes, scoring 1 trillion tokens for approximately $500 in cloud costs, making principled data selection practical.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
data filtering, language model

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_yfKSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Benchmax, a scalable benchmark-targeted data filtering method. It uses CountSketch to compress per-example gradients, retrieves high-alignment training examples for individual benchmark examples to preserve target coverage, and distills these gradient-based labels into a lightweight classifier for large-scale corpus filtering. The method is evaluated on OLMo-2-1B and 7B with 25B-token mid-training.

Critical strengths
  1. Benchmax provides a practical and scalable implementation of benchmark-aware gradient-based data selection, including detailed compute and storage analysis.

  2. The coverage-based construction is intuitive and empirically improves over the collapsed-gradient alternative.

Critical weaknesses
  1. The main experiments compare primarily against HQ DCLM and fastText, but do not include strong target-aware data selection baselines such as LESS, MATES, or related methods, even at a smaller tractable scale.

  2. The main mid-training results do not appear to include repeated independent training runs. The reported standard errors therefore do not establish the run-to-run robustness of the observed gains.

Questions for authors
  1. Can the authors compare Benchmax with existing target-aware gradient/data-selection methods under a smaller but compute-feasible setting?

  2. Can the authors report results across multiple independent mid-training seeds for the main 7B comparison?

  3. How sensitive is the final filtered dataset to the choice of reference checkpoint?

2 comments
AuthorsOpenReview ↗

Response to review

Target-aware baselines (W1, Q1). We added DSIR and DsDm at both scales. Please refer to the general response for a result summary, and Table 1 and Table 2 in the revised manuscript for full breakdown. DsDm is model-aware and gradient-based, and it is the strongest baseline at 7B. Benchmax is slightly ahead of it in performance and needs about 30× less selection compute. We did not run LESS or MATES. LESS computes a gradient feature for every candidate document, which at our corpus size has the same cost profile as DsDm. MATES selects data online during training, while our setting selects one fixed dataset offline.

Repeated runs (W2, Q2). We agree that seeds would strengthen the comparison. Each 7B mid-training run costs about 6 × 7.3B × 25B ≈ 1.1×10²¹ FLOPs, so two more seeds for just Benchmax and HQ DCLM would cost about 4.4×10²¹ FLOPs. That is more than DsDm's entire selection budget, and we unfortunately cannot afford the compute. In place of seeds, the gains are consistent across two training scales (1B and 7B) and two gradient models (1B and 7B gradients both improve the 7B model). Per-row standard errors are reported.

Sensitivity to the reference checkpoint (Q3). Appendix G shows label agreement across 7B checkpoints during pretraining and mid-training (Figures 10–11). The final outcome is also robust to the gradient model. Data selected with 1B gradients and with 7B gradients both improve OLMo-2-7B (+1.01% and +1.67%).

Thanks for your reply, you solved some of my problems, but I want to keep my score.

Reviewer_YK8HOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper suggests Benchmax, an approach for task-aware data selection.

Data selection is a widely studied problem in various fields. In this work, the authors focus on data selection given multiple target tasks. The main setting used in this work is mid-training, and the size of the dataset is generally large. Hence, the motivation for this work is clear.

First, the authors discuss three main challenges in finding a practical subset selection technique.

  1. Ideally, one would compute the importance of each sample by training the model on it and check how it affects the loss. However, since this is not practically feasible, they focus on the gradients. Computing the gradients for every document in the corpus is also computationally prohibitive. Thus, they address this struggle by approximating the inner product score via distillation into a transformer-based classifier.
  2. Storing these gradient is also infeasible. They address this by using the Johnson-Linderstrauss Lemma and Countsketch.
  3. The methodological challenge is reference gradients collapse. In order to avoid this, they leverage diversity in their approach.

Afterwards, the authors propose their method, which consist of three stages; gradient scoring using CountSketch, label construction using KNN, and classifier distillation.

The suggested approach is intuitive and combine multiple known techniques. The empirical evaluation validates the robustness of this approach.

Critical strengths

It was easy to follow and understand the studied problem, and the motivation behind it. I specifically liked the way the authors started by explaining the challenges, then explaining how to overcome these challenges, followed by introducing the approach. In my opinion this makes understanding the paper much easier and the transitions is very smooth. The challenges are real, and the way the authors overcome these challenges is interesting.

Critical weaknesses

I have two major weaknesses.

Empirical evaluation - not very convincing. I appreciate the extensive experiments and metrics, however, as far as I understand, the headline result (+3.0% average in Table 1), is a bit overstated. This mainly comes from the CopyColors syntething metric. On the actual accuracy average, a simple MMLU-only filter beats the multi-benchmark Benchmax. I am also not quite sure if the reference questions used for selection are kept separate from those evaluation questions, it would be great if the authors can clarify this. Furthermore, there are two lines in Table 1 that have essentially the same numbers and std: MMLU Stem zero-shot and MMLU Steam QA. Lastly, I am not sure which part of the method causes the major improvements. I am not sure if really using the gradients is the most important component. One way to show it, is to replace the gradients with text embedding for instance and keep the other ingredients the same. This is basically similar to BETR (Mizrahi et al., 2025, which by the way uses per-question coverage).

Novelty - limited: The suggest approach combines multiple known techniques:

(A) gradient-base selection (LESS, TracIN, etc...).

(B) BETR-style coverage and distillation

(C) CountSketch

I do like the combination, but a comparison with BETR and additional ablation studies are needed to understand if the contribution is legitimate as I mentioned in my previous point.

Questions for authors
  1. How well does the classifier learns? is there a real evidence that it captures more than the surface feature such as "looks like important/quiz"?
  2. I understand that this is computationally heavy, but do you think that Benchmax also helps when used for retraining?
  3. How do you explain multi-task vs mmli-only in table 1 results? what we gain from targeting many benchmarks rather than one?
2 comments
AuthorsOpenReview ↗

Response to review - Part 1

We thank the reviewers for the insightful reviews. Please refer to the following for our response.

Headline overstated, CopyColors (W1). We agree that CopyColors inflated the headline. We removed it, together with MRPC, CommonSenseQA 5-shot BPB and the duplicated MMLU STEM zero-shot row, for the reasons in the general response (item 2). Benchmax now improves by +1.67% on 17 of 23 benchmarks.

MMLU-only vs multi-task (W1, Q3). In the revised table the multi-task filter is ahead on both summaries: average accuracy 64.0 vs 63.6 (HQ DCLM 62.9), relative difference +1.67% vs +1.30%. Each filter gains most on what it targets. The MMLU-only filter gains more on the 8 MMLU rows (+1.78% vs +1.00%). The multi-task filter gains more on the other 15 benchmarks, which only it targets (+2.02% vs +1.05%). We added this to Section 6.2.

Reference vs evaluation questions (W1). Thank you for asking. We now state this in Section 5. In Tables 1 and 2, the reference examples come from the same split as the evaluated questions, so these tables measure how well a filter improves its targets. We test generalization to unseen questions in three ways:

  • Held-out benchmarks: the MMLU-only filter improves the 15 benchmarks it never targeted by +1.05% (9 of 15).
  • Held-out formats: a filter targeting only zero-shot MC on MMLU-STEM improves 5-shot MC (50.1 → 52.7) and open-ended QA (37.7 → 41.7) (Appendix B.1).
  • Disjoint splits: Appendix B reports a train/test-split experiment where the filter only sees the train split. Those runs use our earlier setup, which the appendix now notes.

On the main 7B runs, the MMLU 5-shot test split is also available. Benchmax (7B gradients) vs HQ DCLM: Humanities 55.5 → 57.3, Other 67.1 → 68.2, Social Sciences 72.6 → 72.5, STEM 51.7 → 52.3.

Duplicate MMLU STEM rows (W1). Thank you for catching this. The zero-shot row held the QA metric. We removed it and recomputed all summaries.

Comparison with other methods (W1, W2). We ran experiments on other filtering algorithms, and provide the results summary in the general response. Please refer to Table 1 and Table 2 in the revised manuscript for full breakdown of the results. Here we show results at the 1B scale, since BETR was only run at 1B due to compute limits.

OLMo-2-1B, 23 benchmarksSeedsAvg. Rel. Diff. vs HQ DCLMAverage ↑
HQ DCLM1—49.4
fastText1+9.12%52.4
DSIR1+10.26%52.8
DsDm1+5.52%51.5
BETR2+8.91%52.8
Benchmax1+10.55%53.5

The results suggest that Benchmax also outperforms DSIR and DsDm, two common baselines used in other studies. Benchmax also outperforms BETR, where the main difference is that Benchmax scores candidate data with gradients and BETR with embeddings.

Novelty (W2). We see two contributions. The first is coverage labeling in gradient space. BETR also scores candidates against individual reference examples, but by embedding similarity, and it uses that similarity as the score. Benchmax instead asks that training on the selected data reduce the loss on every reference example: each reference gradient retrieves its own most and least aligned candidates, and these per-example neighbors become the labels the classifier learns. Gradient alignment and embedding similarity measure different things, so the labeled data, and what the classifier learns from it, differ as well. The collapsed-gradient ablation (Appendix C.2) shows that this per-example requirement is critical: averaging the reference gradients into one direction loses the gain. The second contribution is making this work at corpus scale. CountSketch, distillation and linear heads make per-example coverage tractable, at about 30× less selection compute than DsDm. The embedding ablation above tests the gradient-versus-embedding difference directly.

Pretraining from scratch (Q2). We have not tested this and do not claim it. Nothing in the pipeline is specific to mid-training, but pretraining experiments from scratch are truly outside our compute budget.

AuthorsOpenReview ↗

Response to review - Part 2

This post continues the response from Part 1.

Does the classifier learn more than surface features? (Q1) Two pieces of evidence are in the paper: the held-out formats experiment (a filter trained only on zero-shot MC also improves open-ended QA, which is not quiz-formatted), and the top-scored examples in Appendix H. We also checked how well the distilled classifier reproduces the gradient labels on held-out documents for all the classifiers used in filtering. Each metric is shown next to a baseline that does not look at the document:

Mean over all classifiers1B classifier1B baseline7B classifier7B baselineBaseline
Accuracy, positive vs rest ↑0.860.840.870.85always predict "not positive"
AUC, positive vs rest ↑0.750.500.780.50any score that ignores the document
Cross-entropy, 3-way labels ↓0.640.730.590.66predict the label frequencies

Accuracy is a weak signal here, although we still report it for completeness. Only about 15% of documents are labeled positive, so a baseline that always predicts "not positive" already performs only slightly worse than the classifier in accuracy, due to the inherent class imbalance. AUC and cross-entropy are more informative. The AUC of our classifiers is around 0.75–0.78 against 0.50 for a trivial data-independent baseline, and this margin indicates that our classifiers learn non-trivial information about the utility of a training document. The cross-entropy of our classifier is also below the label-frequency baseline for every benchmark. The classifier therefore learns the gradient labels, not just how often each label occurs. We also note that a head's AUC is not strongly correlated with the downstream gain on its benchmark (Spearman ρ of 0.15 at 1B and 0.01 at 7B), and in our early experiments a classifier with higher AUC did not translate into better downstream performance.

Reviewer_Wre9OpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for data selection in language model training. The proposed method uses gradients to perform distillation tailored to a specific task. Experiments using SLM demonstrate that the proposed data selection improves performance compared to several other training methods.

Critical strengths

The concept of data distillation that takes into account the performance of specific tasks is intriguing. The fact that performance has improved in the experiments also demonstrates its usefulness.

Critical weaknesses

The main points of concern regarding this paper are the validity of the research question it seeks to address and the appropriateness of its presentation.

With regard to data selection, this paper appears to discuss the scope within which gradients are used. I believe that using gradients is a means to an end, and the ultimate goal should be to improve learning efficiency. Therefore, I question whether a discussion that assumes the use of gradients is valid.

Furthermore, if the use of gradients is not taken into consideration, it seems that [1]–[4], for example, should also be examined for comparison. The authors should explicitly state why a comparison with these works is unnecessary. Furthermore, although these papers are listed in the references of this paper, they are all marked as “not accepted”; in reality, however, they have all been accepted. While this may simply be due to the authors’ lack of research or their unfamiliarity with standard practices in this field, it raises suspicions of improper conduct—namely, intentionally downplaying the comparators to exclude them from the comparison.

Another question concerns the relationship with the size of the language model. While this paper evaluates two models—7B and 1B—the 7B model shows significant improvement, whereas the 1B model performs worse than conventional methods. Based on the proposed method, it appears that as the model size increases, the effectiveness of data selection decreases. On the other hand, for models of a size like 1B, if too much emphasis is placed on data selection, the model itself may be unable to handle the volume of information, leading to a decline in performance. While the results for the 1B model may reflect this characteristic, it remains unclear for which models this method is intended. It is unclear whether it offers value regardless of model size, or whether it targets models ranging from relatively large to medium-sized—even among those generally considered small, such as the 7B model. Furthermore, it appears that no corresponding validation has been conducted.

[1] J. Li et al., DataComp-LM: In search of the next generation of training sets for language models, NeuRIPS2024

[2] S. M. Xie et al., Data Selection for Language Models via Importance Resampling, NeuRIPS2023

[3] L. Engstrom et al., DsDm: Model-Aware Dataset Selection with Datamodels, ICML2024

[4] Z. Yu et al., MATES: Model-Aware Data Selection for Efficient Pretraining with Data Influence Models, NeuRIPS2024

Questions for authors

Please clarify the research questions addressed in this paper and explain why the comparison group is appropriate for addressing them.

From the perspective of model size, please clarify what the proposed method is intended to target.

2 comments
AuthorsOpenReview ↗

Response to review

We thank the reviewer for the effort put into the review.

Research question, and why gradients (W1, Q1). Gradients are a means, not the goal. The question is how to find the subset of a web-scale corpus that most improves a chosen set of benchmarks, at a selection cost far below that of training. A per-sample gradient inner product is a principled first-order estimate of how training on a document changes the benchmark loss. Prior methods built on it do not scale to corpus-level selection, and that is what Benchmax addresses. To test whether gradients are worth it, we compare against selectors that do not use gradients as well as one that does.

Comparison set (W2–W6, Q1).

  • DataComp-LM [1]: our fastText baseline is the DCLM classifier-filtering approach, with the target benchmarks as the positive class. Section 5 now says so explicitly.
  • DSIR [2] and DsDm [3]: now included at both scales. Please refer to Table A and Table B in the general response for summary, and Tables 1 and 2 in the revised manuscript show the full breakdown of results. DSIR is a non-gradient, n-gram selector. DsDm is a model-aware selector that also uses gradients, and it is the strongest baseline at 7B. Benchmax is slightly ahead of it and needs about 30× less selection compute (Selection FLOPs row, Figure 1, Appendix E).
  • MATES [4]: MATES selects data online during pretraining and periodically refits its data influence model to the model being trained. Our setting selects one fixed dataset offline for mid-training, so we use DsDm as the representative model-aware baseline.

Citations. We have updated the entries to their published venues. However, we respectfully disagree that our citations reflect an attempt to downplay these works. We cited their arXiv versions, which is common practice, and the bibliography style printed them without a venue. The paper does not describe any of them as unpublished. All four are cited in the paper: DSIR, DsDm and MATES in Related Work, and DataComp-LM in the introduction and Section 5. DSIR [2] and DsDm [3] are now baselines in our main tables.

Model size (W8, W9, Q2). In the revised tables Benchmax is the best method at both scales: +10.55% at 1B (DSIR +10.26%, fastText +9.12%) and +1.67% at 7B. In the submitted version the 1B result was on par with fastText (+11.42% vs +11.65%), not worse than conventional methods. Relative gains are larger at 1B, which has more headroom. No component of our algorithm ties it to a specific model size. The gradient model can also be smaller than the trained model: 1B gradients select data that improves the 7B model by +1.01%. We validate two sizes. Larger models were outside our compute budget.

Thank you for providing such a comprehensive response. The additional information has significantly improved the persuasiveness of your paper. While I still feel that certain aspects of the presentation require refinement, I look forward to your revisions and intend to adjust my final score upward.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response to all reviewers

We thank the reviewers for their careful reading. We have uploaded a revised manuscript. The main changes are:

  1. Two new baselines at both scales. DSIR (Xie et al., 2023) and DsDm (Engstrom et al., 2024) are now in the main tables for OLMo-2-7B and OLMo-2-1B.
  2. Cleaner rows. We removed four rows whose scores do not track the targeted skills:
    • CopyColors BPB, a synthetic copying task. Its score is unstable even between late checkpoints of a single run, and its relative changes are an order of magnitude larger than any other row's (up to +82% at 7B), so it dominates the average.
    • MRPC, where scores collapse to the F1 of always answering "paraphrase" (about 81). Every 7B model scores 80.9–81.3, and at 1B the highest scores (80.8–80.9) sit at the same value.
    • CommonSenseQA 5-shot BPB, which scores the same questions as the CommonSenseQA 5-shot accuracy row.
    • MMLU STEM zero-shot, which had duplicated the QA metric (thanks to Reviewer YK8H).
  3. Selection cost. Both main tables (Tables 1 and 2) now report the FLOPs needed to produce one filtered dataset from the 250B-token corpus, and Figure 1 shows them next to the results. Benchmax needs about 30× less selection compute than DsDm.
  4. Clarifications. The relation of our fastText baseline to DCLM and the reference/evaluation splits are now stated explicitly in Section 5.

The two tables below summarize the comparison with all baselines, as average relative difference over HQ DCLM. Table A uses the revised benchmark set (23 benchmarks). Table B uses the benchmark set of the submitted version (27 benchmarks), including the rows removed in item 2. We also note that all methods were rerun for correctness after our compute moved to different clusters, so the numbers in Table B may differ slightly from those in the submitted version.

Table A. Revised benchmark set (23 benchmarks).

MethodSeedsOLMo-2-7BOLMo-2-1BSelection FLOPs (7B / 1B)
HQ DCLM1——0
fastText1−1.46%+9.12%CPU
DSIR1−1.55%+10.26%CPU
DsDm1+1.48%+5.52%3.7×10²¹ / 3.7×10²¹
BETR (1B only)2—+8.91%—
Benchmax1+1.67%+10.55%1.2×10²⁰ / 8.3×10¹⁹

Table B. Benchmark set of the submitted version (27 benchmarks).

MethodSeedsOLMo-2-7BOLMo-2-1B
HQ DCLM1——
fastText1+1.39%+11.75%
DSIR1+0.73%+11.74%
DsDm1+3.34%+5.07%
BETR (1B only)2—+10.21%
Benchmax1+3.64%+11.46%

With these changes, Benchmax improves over HQ DCLM at 7B by +1.67% on 17 of 23 benchmarks. It is slightly ahead of the strongest baseline, DsDm (+1.48%), while needing about 30× less selection compute, and fastText (−1.46%) and DSIR (−1.55%) are harmful. At 1B, Benchmax improves by +10.55%, ahead of DSIR (+10.26%), fastText (+9.12%) and DsDm (+5.52%).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.