- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Benchmax, a scalable benchmark-targeted data filtering method. It uses CountSketch to compress per-example gradients, retrieves high-alignment training examples for individual benchmark examples to preserve target coverage, and distills these gradient-based labels into a lightweight classifier for large-scale corpus filtering. The method is evaluated on OLMo-2-1B and 7B with 25B-token mid-training.
Critical strengths
-
Benchmax provides a practical and scalable implementation of benchmark-aware gradient-based data selection, including detailed compute and storage analysis.
-
The coverage-based construction is intuitive and empirically improves over the collapsed-gradient alternative.
Critical weaknesses
-
The main experiments compare primarily against HQ DCLM and fastText, but do not include strong target-aware data selection baselines such as LESS, MATES, or related methods, even at a smaller tractable scale.
-
The main mid-training results do not appear to include repeated independent training runs. The reported standard errors therefore do not establish the run-to-run robustness of the observed gains.
Questions for authors
-
Can the authors compare Benchmax with existing target-aware gradient/data-selection methods under a smaller but compute-feasible setting?
-
Can the authors report results across multiple independent mid-training seeds for the main 7B comparison?
-
How sensitive is the final filtered dataset to the choice of reference checkpoint?
2 comments
Thanks for your reply, you solved some of my problems, but I want to keep my score.
Response to review
Target-aware baselines (W1, Q1). We added DSIR and DsDm at both scales. Please refer to the general response for a result summary, and Table 1 and Table 2 in the revised manuscript for full breakdown. DsDm is model-aware and gradient-based, and it is the strongest baseline at 7B. Benchmax is slightly ahead of it in performance and needs about 30× less selection compute. We did not run LESS or MATES. LESS computes a gradient feature for every candidate document, which at our corpus size has the same cost profile as DsDm. MATES selects data online during training, while our setting selects one fixed dataset offline.
Repeated runs (W2, Q2). We agree that seeds would strengthen the comparison. Each 7B mid-training run costs about 6 × 7.3B × 25B ≈ 1.1×10²¹ FLOPs, so two more seeds for just Benchmax and HQ DCLM would cost about 4.4×10²¹ FLOPs. That is more than DsDm's entire selection budget, and we unfortunately cannot afford the compute. In place of seeds, the gains are consistent across two training scales (1B and 7B) and two gradient models (1B and 7B gradients both improve the 7B model). Per-row standard errors are reported.
Sensitivity to the reference checkpoint (Q3). Appendix G shows label agreement across 7B checkpoints during pretraining and mid-training (Figures 10–11). The final outcome is also robust to the gradient model. Data selected with 1B gradients and with 7B gradients both improve OLMo-2-7B (+1.01% and +1.67%).