- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AttentionInfluence is a label-free data selection method that identifies retrieval heads in a smaller model to score documents based on performance loss when those heads are masked. By leveraging a compact model to curate training data, this approach improves larger models across various knowledge, math, and code benchmarks compared to multiple baseline filters by a small margin.
Critical strengths
- The method is, to my knowledge, novel and interesting.
- The empirical study appears properly controlled and reasonable for these expensive pre-training runs.
Critical weaknesses
-
The authors should consider incorporating the metrics presented by https://openreview.net/forum?id=JSE40ljyKm and adding them to related works to determine whether these model-size gains extrapolate to larger-scale pre-training. Doing so may also help establish statistical significance against baselines, as pre-trained-only base models initialized from scratch often exhibit high variance on standard accuracy metrics.
-
"Weak-to-Strong transfer" appears to be an over-claim. Pre-training corpora rewriting and training data filtering using previous-generation models is already standard practice and some-what known in the public domain via technical reports and conference papers.
Questions for authors
- I recommend reducing the amount of bolding in the abstract to align with standard academic conventions.
- Are there more recent potential baselines other than the FineWeb-Edu Classifier?
4 comments
Resuming from the discussion on the other global post: the authors have made their claim more convincing post-rebuttal.
However, authors should update the revised manuscript as at its current form it is not publishable quality integrating all of the reviewers' feedback. The pdf can still be uploaded during discussion period.
Also, the lack of other baselines like Dolma 3 (seems to be released between 2025-2026) is a weak point.
If these two points above are effectively addressed my score will be raised to 3. The AC can take this into consideration.
Thank you.
Thank you for your constructive reassessment and clear guidance. We are revising the manuscript to integrate the additional experiments and the reviewers’ feedback, and will upload the updated PDF before the discussion deadline. The Dolma 3 baseline is already training, and saved checkpoints are being evaluated under the same protocols as our existing comparisons. We will follow up with the revised PDF, a concise summary of the changes, and Dolma 3 comparisons at matched training-token budgets.
Updated manuscript and a strong Dolma 3 baseline
Updated manuscript and a strong Dolma 3 baseline
Thank you again for your constructive guidance. We have updated the PDF to integrate the additional experiments and the reviewers' feedback.
The revision now includes the public-checkpoint experiments, proxy and head-selection analyses, length-stratified and random-head controls, independent initialization repetitions, and explicit task-level and aggregate rBridge comparisons (Section 5.3 and Appendix X.1–X.5). We have also clarified the smaller-selector-to-larger-target-model claim, corrected Figure 2's denominator to match Equation (5) and the implementation, and reduced excessive abstract bolding. The final reproduction code is included in the Supplementary Material.
We have added Dolma 3 as a strong recent baseline, using its official released quality-classifier weights. The 1.7B target models use the same candidate corpus, domain-token quotas, initialization, 8K context, global batch and 100B training schedule. We directly apply the released classifier weights without additional fitting.
The Olmo 3 report, Section 3.4.1 describes supervised fastText training with source-derived quality labels: OpenHermes-2.5 and ELI5 provide positive examples, supplemented by UltraChat-200k and WildChat-1M; negatives are 30GB sampled from DCLM-RefinedWeb. This source-based weak supervision gives Dolma 3 the benefit of dedicated quality-supervised classifier training. AttentionInfluence is training-free at the data-selection stage: it uses a frozen selector without fitting an additional quality classifier or requiring target-corpus quality labels. This training-free construction of the selection signal is a central contribution of our work.
The results below are included in Section 5.4 and Appendix X.7 of the revised manuscript (Tables 5 and 32–34). The Dolma 3 target-model run is still training toward 100B tokens.
Edu denotes the FineWeb-Edu classifier. Our method (CoT-proxy) is AttentionInfluence with the CoT-crucial proxy.
| Task | Random | Edu | Our method (CoT-proxy) | Dolma 3 |
|---|---|---|---|---|
| Step | 60,000 | 60,000 | 60,000 | 60,000 |
| Training tokens (B) | 62.914553856 | 62.914553856 | 62.914553856 | 62.914553856 |
| CommonSenseQA | 19.49 | 19.90 | 19.74 | 19.74 |
| PIQA | 74.43 | 72.20 | 73.29 | 73.34 |
| TriviaQA | 25.86 | 27.45 | 27.48 | 25.51 |
| WinoGrande | 57.22 | 57.22 | 59.59 | 56.59 |
| HellaSwag | 57.65 | 54.54 | 56.74 | 55.91 |
| ARC-Easy | 72.60 | 74.16 | 71.42 | 72.98 |
| ARC-Challenge | 42.41 | 45.82 | 40.44 | 42.58 |
| GSM8K | 3.41 | 2.05 | 3.34 | 3.34 |
| MATH | 0.60 | 1.54 | 1.86 | 0.86 |
| HumanEval | 1.86 | 4.30 | 1.49 | 2.16 |
| SVAMP | 7.20 | 9.90 | 18.30 | 12.60 |
| Mean | 32.98 | 33.55 | 33.97 | 33.24 |
| Mean Δ vs 60k Random (pp) | — | +0.58 | +1.00 | +0.26 |
The rBridge comparison is summarized below; lower is better. Each aggregate weights its tasks equally: six tasks for 0-shot and two for 5-shot. The reported deltas use the 60k Random result.
| Data selection | 0-shot mean ↓ | Δ vs Random | 5-shot mean ↓ | Δ vs Random |
|---|---|---|---|---|
| Random | 0.66889 | — | 0.55737 | — |
| FineWeb-Edu classifier | 0.64928 | -0.01961 | 0.53669 | -0.02068 |
| Our method (CoT-proxy) | 0.64651 | -0.02238 | 0.53685 | -0.02052 |
| Dolma 3 classifier | 0.64112 | -0.02778 | 0.54077 | -0.01660 |
Dolma 3 is a strong baseline. On rBridge, it is slightly better than our method in the 0-shot mean (0.64112 vs. 0.64651), while our method is slightly better in the 5-shot mean (0.53685 vs. 0.54077; lower is better). Our 60k checkpoint also has a slightly higher 11-task mean than the reported Dolma 3 checkpoint (33.97% vs. 33.24%). We have incorporated these results into the revised manuscript. Training and checkpoint evaluations are ongoing, and we will report the completed 100B comparison when available. We will make the final results available in our anonymous repository.
Response to Reviewer 2GLr
Thank you for the feedback and rBridge recommendation. We added measurements and training repetitions to address your concerns.
1. rBridge, larger target models, and training variability
We implemented rBridge using the authors' public reasoning references and token probabilities, and will cite it in the revised manuscript. It provides a finer-grained, more sensitive signal of reference-reasoning prediction than answer accuracy alone in this training regime.
Our frozen Qwen3-1.7B-Base CoT-proxy selector identifies heads using 200 publisher-provided OpenStax examples, then selects one 100B-token corpus for 1.7B and 8B target models trained from scratch. Within each size, selected-data and random-baseline runs share initialization, token budget, per-domain quotas and training hyperparameters. This directly tests a larger target model scale; the original 1T-token study supplies evidence under longer pretraining.
At 100B, CoT-proxy selection improves the following task scores over random sampling (percentage points):
The corresponding rBridge differences below are CoT minus random; negative is better. We use fixed references and our frozen 0-shot/5-shot prompt adaptations. Intervals use 10,000 paired question-bootstrap resamples at initialization seed 20261110.
Five of six 0-shot tasks at each size have intervals entirely below zero; ARC-Challenge is inconclusive. These are nominal per-cell intervals conditional on fixed checkpoints, separate from training-run variability. rBridge evaluates reference prediction; task scores evaluate final answers or code execution. The MATH reference subset has 494 examples, versus 5,000 in the generation evaluation.
We separately added two initialization seeds at 1.7B, holding the data pools and data-order seed fixed and sharing initialization within each baseline/CoT pair. All three runs reach 100B. On the full 1,000-question SVAMP evaluation:
The paired gain averages +8.0 pp, with sample standard deviation 0.85 pp across seeds. All three pairs also show lower 0-shot rBridge on GSM8K, MATH and HumanEval. These repetitions address initialization variability separately from question-sampling uncertainty. See Part 4 of the common response for selection and task results, Part 5 for training/evaluation settings and seeds, and Part 6 for rBridge and mixing.
2. Scope of "weak-to-strong"
We agree that using pretrained models to filter or rewrite training corpora is established practice. We will describe our setting as using a smaller selector to choose data for target models at a larger parameter scale. Our 1.7B-to-8B result concerns parameter count, without assuming the selector is less capable than the target model. The contribution is the head-intervention loss signal within a frozen model and its use for data selection.
3. Abstract presentation
Thank you for the reminder about academic writing conventions. We will remove excessive bolding from the abstract.
4. More recent baselines
Yes. The Dolma 3 quality classifier used in Olmo 3 pretraining has released fastText weights under Apache-2.0. It can score our corpus without new annotation or selector training. PreSelect (2025) also provides a public fastText scorer.
We will add both to related work. Since both use fastText and compute for additional pretraining runs is limited, we choose the more recent Dolma 3 as our additional baseline. The matched Dolma 3 baseline experiment is ongoing, using the same candidate corpus, per-domain token quotas and 1.7B training/evaluation settings. We will report the results once the experiment is complete.