acceptodds
Under review as a conference paper at ICLR 2027

AttentionInfluence: Training-Free Weak-to-Strong Pretraining Data Selection via Attention Head Masking

Abstract

Recently, there has been growing interest in collecting reasoning-intensive pretraining data to improve the reasoning ability of LLMs. Prior approaches rely on supervised classifiers, requiring human or LLM labeling and often introducing domain-specific biases. Since attention heads are crucial to in-context reasoning, we propose AttentionInfluence, a training-free method without supervision signal. Our approach turns a small pretrained language model into a strong data selector: we identify its retrieval heads and score each document by the loss difference incurred by masking them. We apply AttentionInfluence to a 1.3B-parameter dense model to select from the 241B-token SmolLM corpus, and mix the corpus with the selected 73B-token subset to pretrain a 7B-parameter dense model on 1T tokens under a Warmup-Stable-Decay (WSD) schedule; the baseline is identical except that its 73B subset is drawn uniformly at random, so the only variable is which tokens are selected. Improvements range from 0.8pp to 3.5pp across knowledge-intensive and reasoning-heavy benchmarks (MMLU, MMLU-Pro, SuperGPQA, GSM8K, and HumanEval), and the overall average at 1T tokens exceeds that of the FineWeb-Edu classifier, a supervised selector distilled from a 70B annotator—with no labels and no classifier training. This is an effective Weak-to-Strong scaling property: a small model improves a larger one. Code is available in an anonymous repository at https://github.com/gofornlpsota/AttentionInfluence.

Primary area:
foundation or frontier models, including LLMs
Keywords:
pretraining data selection, data curation, large language models, attention heads, retrieval heads, weak-to-strong generalization, reasoning-intensive data

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_2GLrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AttentionInfluence is a label-free data selection method that identifies retrieval heads in a smaller model to score documents based on performance loss when those heads are masked. By leveraging a compact model to curate training data, this approach improves larger models across various knowledge, math, and code benchmarks compared to multiple baseline filters by a small margin.

Critical strengths
  1. The method is, to my knowledge, novel and interesting.
  2. The empirical study appears properly controlled and reasonable for these expensive pre-training runs.
Critical weaknesses
  1. The authors should consider incorporating the metrics presented by https://openreview.net/forum?id=JSE40ljyKm and adding them to related works to determine whether these model-size gains extrapolate to larger-scale pre-training. Doing so may also help establish statistical significance against baselines, as pre-trained-only base models initialized from scratch often exhibit high variance on standard accuracy metrics.

  2. "Weak-to-Strong transfer" appears to be an over-claim. Pre-training corpora rewriting and training data filtering using previous-generation models is already standard practice and some-what known in the public domain via technical reports and conference papers.

Questions for authors
  1. I recommend reducing the amount of bolding in the abstract to align with standard academic conventions.
  2. Are there more recent potential baselines other than the FineWeb-Edu Classifier?
4 comments
AuthorsOpenReview ↗

Response to Reviewer 2GLr

Thank you for the feedback and rBridge recommendation. We added measurements and training repetitions to address your concerns.

1. rBridge, larger target models, and training variability

We implemented rBridge using the authors' public reasoning references and token probabilities, and will cite it in the revised manuscript. It provides a finer-grained, more sensitive signal of reference-reasoning prediction than answer accuracy alone in this training regime.

Our frozen Qwen3-1.7B-Base CoT-proxy selector identifies heads using 200 publisher-provided OpenStax examples, then selects one 100B-token corpus for 1.7B and 8B target models trained from scratch. Within each size, selected-data and random-baseline runs share initialization, token budget, per-domain quotas and training hyperparameters. This directly tests a larger target model scale; the original 1T-token study supplies evidence under longer pretraining.

At 100B, CoT-proxy selection improves the following task scores over random sampling (percentage points):

Task1.7B Δ (pp)8B Δ (pp)
GSM8K+0.45+1.90
MATH+1.20+1.06
HumanEval pass@1+0.30+3.29
SVAMP+8.80+11.10

The corresponding rBridge differences below are CoT minus random; negative is better. We use fixed references and our frozen 0-shot/5-shot prompt adaptations. Intervals use 10,000 paired question-bootstrap resamples at initialization seed 20261110.

Reference task1.7B ΔrBridge [95% CI]8B ΔrBridge [95% CI]
0-shot CommonSenseQA-0.01654 [-0.01924, -0.01370]-0.00626 [-0.00895, -0.00350]
0-shot ARC-Challenge+0.00153 [-0.00100, +0.00416]-0.00092 [-0.00331, +0.00152]
0-shot MMLU-Pro STEM-0.03578 [-0.03678, -0.03477]-0.02962 [-0.03053, -0.02870]
0-shot GSM8K-0.04424 [-0.04640, -0.04208]-0.02497 [-0.02716, -0.02286]
0-shot MATH-0.04647 [-0.04947, -0.04355]-0.04599 [-0.04880, -0.04323]
0-shot HumanEval-0.02400 [-0.02941, -0.01865]-0.03003 [-0.03466, -0.02553]
5-shot ARC-Challenge+0.00006 [-0.00227, +0.00238]-0.02632 [-0.02845, -0.02419]
5-shot GSM8K-0.03641 [-0.03892, -0.03388]-0.04022 [-0.04226, -0.03822]

Five of six 0-shot tasks at each size have intervals entirely below zero; ARC-Challenge is inconclusive. These are nominal per-cell intervals conditional on fixed checkpoints, separate from training-run variability. rBridge evaluates reference prediction; task scores evaluate final answers or code execution. The MATH reference subset has 494 examples, versus 5,000 in the generation evaluation.

We separately added two initialization seeds at 1.7B, holding the data pools and data-order seed fixed and sharing initialization within each baseline/CoT pair. All three runs reach 100B. On the full 1,000-question SVAMP evaluation:

Initialization seedRandomCoT proxyGain (pp)
2026111010.6%19.4%+8.8
202611170112.2%20.3%+8.1
20261117029.9%17.0%+7.1

The paired gain averages +8.0 pp, with sample standard deviation 0.85 pp across seeds. All three pairs also show lower 0-shot rBridge on GSM8K, MATH and HumanEval. These repetitions address initialization variability separately from question-sampling uncertainty. See Part 4 of the common response for selection and task results, Part 5 for training/evaluation settings and seeds, and Part 6 for rBridge and mixing.

2. Scope of "weak-to-strong"

We agree that using pretrained models to filter or rewrite training corpora is established practice. We will describe our setting as using a smaller selector to choose data for target models at a larger parameter scale. Our 1.7B-to-8B result concerns parameter count, without assuming the selector is less capable than the target model. The contribution is the head-intervention loss signal within a frozen model and its use for data selection.

3. Abstract presentation

Thank you for the reminder about academic writing conventions. We will remove excessive bolding from the abstract.

4. More recent baselines

Yes. The Dolma 3 quality classifier used in Olmo 3 pretraining has released fastText weights under Apache-2.0. It can score our corpus without new annotation or selector training. PreSelect (2025) also provides a public fastText scorer.

We will add both to related work. Since both use fastText and compute for additional pretraining runs is limited, we choose the more recent Dolma 3 as our additional baseline. The matched Dolma 3 baseline experiment is ongoing, using the same candidate corpus, per-domain token quotas and 1.7B training/evaluation settings. We will report the results once the experiment is complete.

Resuming from the discussion on the other global post: the authors have made their claim more convincing post-rebuttal.

However, authors should update the revised manuscript as at its current form it is not publishable quality integrating all of the reviewers' feedback. The pdf can still be uploaded during discussion period.

Also, the lack of other baselines like Dolma 3 (seems to be released between 2025-2026) is a weak point.

If these two points above are effectively addressed my score will be raised to 3. The AC can take this into consideration.

Thank you.

Thank you for your constructive reassessment and clear guidance. We are revising the manuscript to integrate the additional experiments and the reviewers’ feedback, and will upload the updated PDF before the discussion deadline. The Dolma 3 baseline is already training, and saved checkpoints are being evaluated under the same protocols as our existing comparisons. We will follow up with the revised PDF, a concise summary of the changes, and Dolma 3 comparisons at matched training-token budgets.

AuthorsOpenReview ↗

Updated manuscript and a strong Dolma 3 baseline

Updated manuscript and a strong Dolma 3 baseline

Thank you again for your constructive guidance. We have updated the PDF to integrate the additional experiments and the reviewers' feedback.

The revision now includes the public-checkpoint experiments, proxy and head-selection analyses, length-stratified and random-head controls, independent initialization repetitions, and explicit task-level and aggregate rBridge comparisons (Section 5.3 and Appendix X.1–X.5). We have also clarified the smaller-selector-to-larger-target-model claim, corrected Figure 2's denominator to match Equation (5) and the implementation, and reduced excessive abstract bolding. The final reproduction code is included in the Supplementary Material.

We have added Dolma 3 as a strong recent baseline, using its official released quality-classifier weights. The 1.7B target models use the same candidate corpus, domain-token quotas, initialization, 8K context, global batch and 100B training schedule. We directly apply the released classifier weights without additional fitting.

The Olmo 3 report, Section 3.4.1 describes supervised fastText training with source-derived quality labels: OpenHermes-2.5 and ELI5 provide positive examples, supplemented by UltraChat-200k and WildChat-1M; negatives are 30GB sampled from DCLM-RefinedWeb. This source-based weak supervision gives Dolma 3 the benefit of dedicated quality-supervised classifier training. AttentionInfluence is training-free at the data-selection stage: it uses a frozen selector without fitting an additional quality classifier or requiring target-corpus quality labels. This training-free construction of the selection signal is a central contribution of our work.

The results below are included in Section 5.4 and Appendix X.7 of the revised manuscript (Tables 5 and 32–34). The Dolma 3 target-model run is still training toward 100B tokens.

Edu denotes the FineWeb-Edu classifier. Our method (CoT-proxy) is AttentionInfluence with the CoT-crucial proxy.

TaskRandomEduOur method (CoT-proxy)Dolma 3
Step60,00060,00060,00060,000
Training tokens (B)62.91455385662.91455385662.91455385662.914553856
CommonSenseQA19.4919.9019.7419.74
PIQA74.4372.2073.2973.34
TriviaQA25.8627.4527.4825.51
WinoGrande57.2257.2259.5956.59
HellaSwag57.6554.5456.7455.91
ARC-Easy72.6074.1671.4272.98
ARC-Challenge42.4145.8240.4442.58
GSM8K3.412.053.343.34
MATH0.601.541.860.86
HumanEval1.864.301.492.16
SVAMP7.209.9018.3012.60
Mean32.9833.5533.9733.24
Mean Δ vs 60k Random (pp)—+0.58+1.00+0.26

The rBridge comparison is summarized below; lower is better. Each aggregate weights its tasks equally: six tasks for 0-shot and two for 5-shot. The reported deltas use the 60k Random result.

Data selection0-shot mean ↓Δ vs Random5-shot mean ↓Δ vs Random
Random0.66889—0.55737—
FineWeb-Edu classifier0.64928-0.019610.53669-0.02068
Our method (CoT-proxy)0.64651-0.022380.53685-0.02052
Dolma 3 classifier0.64112-0.027780.54077-0.01660

Dolma 3 is a strong baseline. On rBridge, it is slightly better than our method in the 0-shot mean (0.64112 vs. 0.64651), while our method is slightly better in the 5-shot mean (0.53685 vs. 0.54077; lower is better). Our 60k checkpoint also has a slightly higher 11-task mean than the reported Dolma 3 checkpoint (33.97% vs. 33.24%). We have incorporated these results into the revised manuscript. Training and checkpoint evaluations are ongoing, and we will report the completed 100B comparison when available. We will make the final results available in our anonymous repository.

Reviewer_9kRsOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes AttentionInfluence, a training-free pretraining data selection method that scores documents by the loss increase caused by masking retrieval-related attention heads in a frozen selector model. A 1.3B selector is used to select a 73B-token subset from SmolLM, which is then used to pretrain a 7B model for 1T tokens. The selected data consistently improves downstream performance over a matched random-subset baseline and several alternative data-selection methods.

Critical strengths
  1. The large-scale evaluation is a major strength. All configurations use the same 1T-token budget and matched corpus-plus-subset mixing protocol, so the comparison directly tests which data are upsampled.
  2. AttentionInfluence provides a novel training-free selection signal and shows meaningful weak-to-strong transfer: the 1.3B and 7B selectors achieve averages of 43.16 and 43.59, respectively, compared with 42.46 for the matched random baseline.
Critical weaknesses
  1. The paper does not fully establish that AttentionInfluence specifically selects reasoning-intensive data. The authors themselves note that selected documents are systematically longer and that the current ablations do not isolate reasoning from document length or other correlated properties.
  2. The method is sensitive to the proxy task used for retrieval-head identification. Different proxy tasks produce substantially different head sets, raising concerns about the robustness of the proposed selection signal.
Questions for authors
  1. Can the authors provide a length-matched control to determine whether the gains come from reasoning-related structure rather than the preference for longer documents?
  2. How sensitive is downstream performance to the proxy task used to identify retrieval heads?
  3. Can the authors provide additional evidence that the AttentionInfluence score correlates with reasoning intensity after controlling for document length, domain, and perplexity?
1 comment
AuthorsOpenReview ↗

Response to Reviewer 9kRs — Length controls, proxy sensitivity and reasoning evidence

Thank you for recognizing the controlled pretraining study and for identifying these three closely related questions. We added public-checkpoint experiments with dense Qwen3 target models trained from scratch for 100B tokens. They supplement the original 7B / 1T-token study. Common Response Parts 4–6 give the selector configurations, training/evaluation settings and complementary rBridge results.

1. Does the benefit remain after matching document length?

We completed a length-matched random control for the CoT-proxy-selected corpus. Within each domain and exact full-document Qwen-token length, we sample uniformly without replacement from the same eligible corpus. We match document counts and boundary-document retained lengths, giving identical full and retained length histograms and exactly the same per-domain token budgets. Sampling uses no AI, raw-loss or repetition scores; overlap with the selected corpus is allowed to preserve conditional random sampling.

All three 1.7B target models share the initialization, 8,192-token context, 1,048,576-token global batch, cosine schedule and 100B budget. Completed endpoint results are:

Metric (%)RandomLength-matched randomCoT proxyCoT − length (pp)
GSM8K3.642.814.09+1.29
MATH1.121.082.32+1.24
HumanEval pass@17.933.638.23+4.60
SVAMP10.6012.4019.40+7.00
11-task mean35.0234.4035.86+1.46

The mean uses the 11 tasks and fixed protocols listed in Common Response Parts 3–5; differences use unrounded scores. CoT-proxy selection improves all four math/code tasks over the length-matched control. This supports training utility beyond the matched length and domain distributions, while other content properties may still contribute.

rBridge provides complementary evidence from conditional prediction of fixed reference reasoning. Relative to the length-matched control, CoT lowers 0-shot rBridge by 0.04482 on GSM8K, 0.04955 on MATH and 0.03390 on HumanEval. Their nominal 95% intervals exclude zero using 10,000 paired question bootstraps.

2. How sensitive is the method to the proxy?

We now explicitly compare complete public-selector configurations. JSON23 uses Qwen3-1.7B-Base with 23 heads ranked by a JSON key-value retrieval probe. The CoT proxy identifies eight heads by their intervention effect on 200 publisher-provided OpenStax worked solutions, discounting sensitivity to question text. GSM8K, MATH and AIME questions are excluded from head selection.

On the same 100-document FineWeb-Edu development panel, retaining 20 documents gives 6 high-density examples for plain needle, 6 for JSON23 and 12 for the CoT proxy, versus 5.6 expected at random. Labels were frozen assessments. This panel guided development. The development head-count sweep gives 12/10/9 hits with 8/12/16 CoT heads, demonstrating sensitivity to head count.

At 100B, JSON23 and CoT yield 34.79% and 35.86% on the 1.7B 11-task summary. These are complete-selector comparisons, not single-variable proxy ablations. Due to limited compute resources, plain needle was evaluated at the proxy/scoring stage, but we did not run its downstream pretraining experiment. We have not completed a head-fraction training ablation. For our method (CoT-proxy), the Qwen3-1.7B-Base selector masks layer 0: heads 2, 3, 7; layer 7: head 4; layer 8: head 12; layer 12: heads 7, 9; layer 13: head 13. Layer and query-head indices are zero-based (eight heads total).

3. Evidence for reasoning-related training value

The length-matched comparison in Q1 addresses this concern: our method (CoT-proxy) improves mathematical problem solving, code generation and reference-reasoning prediction while domain-token budgets and document-length distributions are matched. Together with the development-panel analysis in Q2, these results support AI as a useful ranking signal for reasoning-intensive training data, without requiring a one-to-one correspondence between a document's score and its reasoning density.

Reviewer_LyfeOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper introduces AttentionInfluence, a pretraining data selection method that requires no additional classifier training or labels for the target corpus. It first identifies retrieval attention heads using a synthetic retrieval task, then scores documents by the relative loss change induced by intervening on those heads. A 1.3B selector selects approximately 73.1B tokens from the 241B-token SmolLM corpus. The selected subset is combined with the full corpus to train a 7B model for 1T tokens. The resulting model achieves an average score of 43.16 across 21 benchmarks, compared with 42.46 for the random-subset baseline, 42.66 for the FineWeb-Edu Classifier, and 42.98 for the rule-based method. The paper provides substantial pretraining experiments, although the mechanistic interpretation, advantages over strong baselines, and reproducibility require further support.

Critical strengths
  1. Simple method with practical relevance. The method constructs a loss contrast within an existing pretrained model, avoiding target-corpus quality annotations and additional classifier fitting. This offers a useful alternative for large-scale data selection.

  2. Substantial experiments with reasonably controlled training budgets. The authors complete multiple 7B-model training runs with 1T tokens and compare against random selection, perplexity filtering, Scaling Filter, a rule-based scorer, and a supervised classifier. Total training tokens, additional subset size, and the mixing procedure are matched, supporting comparisons of selection quality within this protocol.

  3. Useful supplementary evidence. The retrieval-head versus random-head intervention supports the motivation behind the method. The 32K retrieval evaluation and subsequent SFT experiments suggest that the benefits can extend beyond the main benchmark suite or persist through further training. The authors also explicitly acknowledge limitations concerning single-run experiments, statistical testing, and auxiliary quality metrics.

Critical weaknesses
  1. The connection between the score and reasoning-related training value remains insufficiently established. The intervention measures a document’s dependence on particular attention heads, which does not necessarily identify its reasoning content or downstream training utility. Table 12 shows that selected code and mathematics documents are approximately twice as long as those selected by the FineWeb-Edu Classifier. Document length, repetition, baseline loss, and domain composition could contribute to both the scores and downstream improvements. Without controls matching these properties, it is difficult to attribute the gains specifically to reasoning-related mechanisms.

  2. The head-intervention experiments do not directly validate the necessity of retrieval heads for data selection. Tables 1 and 3 show that intervening on retrieval heads damages the selector’s capabilities more than intervening on random non-retrieval heads. However, this does not establish that data selected using retrieval-head loss contrasts is more useful than data selected using contrasts from random or other important heads. Appendix G also reports substantial variation across probing tasks: the proposed probe and Reasoning Needle share only 29.4% of their selected heads, while the corresponding downstream comparisons are not publicly reported. Robustness to the probing task, head set, and intervention proportion therefore remains unclear.

  3. The improvements over strong baselines are modest, and robustness across training runs is unmeasured. The 1.3B selector improves the average over the rule-based method and FineWeb-Edu Classifier by approximately 0.18 and 0.50 percentage points, respectively; the reported confidence intervals include zero. Each configuration has only one training run. Correlated benchmarks and multiple checkpoints from the same run cannot substitute for independent replications. The results demonstrate competitive observed performance, but do not yet establish a consistent advantage over strong selectors.

  4. The practical benefit is established only within a particular upsampling protocol. All main experiments combine the full corpus with an additional subset, effectively increasing the sampling weight of selected documents. The matched random-subset baseline appropriately compares different subset choices, but does not establish whether the proposed mixture improves over training on the full corpus alone under the same token budget. Appendix U acknowledges that this comparison has not been performed, leaving the benefit relative to a conventional corpus-only training setup unresolved.

  5. Important barriers to independent reproduction remain. The manuscript states that the 1.3B and 7B selectors are internal checkpoints, some runtime dependencies and analysis code are unavailable, and historical selection artifacts are incomplete. The supplementary SFT experiment also uses internal data. Since head identification and document rankings depend on the specific selector checkpoint, releasing core algorithm code alone is insufficient to reproduce the main selection results. Experiments with a publicly available selector and verifiable document-selection records would strengthen the paper substantially.

Questions for authors
  1. How much of the improvement remains after matching document length, baseline perplexity, and per-domain token proportions? Could the authors add a length-only selection baseline and report correlations between AttentionInfluence scores and these properties?

  2. Could the authors directly compare downstream training using data selected through retrieval-head interventions, layer-matched random-head interventions, and interventions on other important heads? How sensitive are the results to the probing task and the fraction of heads intervened on? These ablations could initially use a lower-cost training setting.

  3. Could the authors provide independent repetitions for the key comparisons, or estimate training variability in a more affordable setting? What is the measured scoring-cost versus performance trade-off relative to the rule-based method and the deployed FineWeb-Edu Classifier?

  4. How does the proposed mixture compare with training exclusively on the full 241B-token corpus for the same 1T-token budget? Does selectively upsampling high-scoring documents still provide a clear benefit against this control?

  5. Can the authors release the selector checkpoints or reproduce the method using a publicly available model? Could they also provide the selected head indices, document IDs and scores, actual per-domain token counts, and essential evaluation scripts?

  6. Two implementation details need clarification. In Appendix D, is uniform attention normalized only over positions permitted by the causal mask? Also, Figure 2 uses LrefL_{\mathrm{ref}} as the score denominator, whereas Equation (5) uses LbaseL_{\mathrm{base}}. Which definition is implemented?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Lyfe — Part 1 — Data controls and head selection

Thank you for the detailed review. During this rebuttal, we added lower-cost experiments using public Qwen3 selectors and dense Qwen3 target models trained from scratch, supplementing the original 7B / 1T study. Common Response Parts 4–6 report configurations, task results and rBridge.

Q1. Length, baseline loss, repetition and domain composition

Our method (CoT-proxy) denotes AttentionInfluence with the textbook reasoning proxy; see Common Response Part 4 for its implementation. Its length-matched random baseline uses stratified sampling by domain × exact full-document Qwen-token length, with one stratum per length. Within each stratum, uniform sampling without replacement matches document counts and boundary retained lengths, giving identical full/retained length histograms and domain-token budgets. Sampling uses no AI, raw loss or repetition; overlap with the selected corpus is allowed.

The 1.7B targets share initialization, an 8,192-token context, 1,048,576-token global batch, cosine schedule and 100B budget:

Metric (%)RandomLength-matched randomOur method (CoT-proxy)Δ vs length (pp)
GSM8K3.642.814.09+1.29
MATH1.121.082.32+1.24
HumanEval pass@17.933.638.23+4.60
SVAMP10.6012.4019.40+7.00
11-task mean35.0234.4035.86+1.46

The mean uses the tasks/protocols in Common Response Parts 3–5; differences use unrounded scores. These results support utility beyond matched length/domain distributions.

Our method (CoT-proxy) also improves rBridge over the length control on six 0-shot reference sets and two 5-shot conditions. All eight nominal 95% paired question-bootstrap intervals exclude zero. For 0-shot GSM8K/MATH/HumanEval, the differences are −0.04482/−0.04955/−0.03390. These measure prediction of fixed reference reasoning, not generated-answer accuracy or seed variability.

Development-panel reasoning-density diagnostics. The 100-document FineWeb-Edu development panel contains 28 annotated high-density documents. Our method (CoT-proxy) retains 12 high-density examples in the top 20 (60%), versus 5.6 expected under uniform random sampling. Eleven of the 12 have no flagged major errors, and two of the 20 show no reasoning. This panel was used to compare candidate proxy and head configurations before pretraining.

The within-document repetition audit also remains relevant: token-weighted repeated-eight-token coverage over common first-4K windows is 2.0183% for random data, 3.9159% for JSON23 and 3.0224% for our method (CoT-proxy) (80 FineWeb-Edu documents/group; Common Response Part 3). Repetition has not been eliminated.

At 1.7B and 100B training tokens, our method (CoT-proxy) outperforms this added length-matched random baseline on all four math/code tasks and on the 11-task mean (+1.46 percentage points).

Q2. Direct head-selection controls and proxy sensitivity

We agree that damaging the selector does not establish data-selection utility. We completed a direct training control using layer-matched random heads: the count in each layer matches its corresponding JSON set, and sampling excludes the union of the top 20% of JSON and corrected plain-needle rankings. The matched pair shares the selector, scoring window, token quotas and 100B budget. Common Response Part 4 reports the random-head arm's completed 1.7B / 100B result: 34.53% on the 11-task mean; Part 5 gives the head-matching rule and Part 6 its rBridge scores. For this matched comparison, the corresponding JSON58 arm (58 JSON-ranked heads) averages 34.25%, versus 34.53% for random heads: −0.28 percentage points (JSON58 minus random heads). The matched JSON58 result is reported here. This pair does not show a downstream advantage for retrieval heads.

We also test solution-sensitive heads. JSON23 uses Qwen3-1.7B-Base with 23 heads ranked by JSON key-value retrieval; our method (CoT-proxy) uses eight heads identified from 200 publisher-provided OpenStax worked examples, excluding GSM8K/MATH/AIME questions. Head ranking uses solution-loss sensitivity with a question-sensitivity discount (Common Response Part 4).

On the same 100-document FineWeb-Edu development panel, retaining 20 gives 6/6/12 high-density examples for plain needle/JSON23/our method (CoT-proxy), versus 5.6 expected at random. Our method (CoT-proxy) selects 12, 10 and 9 high-density documents among the top 20 when intervening on 8, 12 and 16 heads, respectively. JSON23 and our method (CoT-proxy) achieve 34.79%/35.86% on the 1.7B 100B 11-task summary, as complete-selector comparisons. Limited compute prevented us from pretraining on data selected with plain needle. In this setting, our method (CoT-proxy) outperforms JSON23 retrieval-head selection on all four math/code tasks.

AuthorsOpenReview ↗

Response to Reviewer Lyfe — Part 2 — Repetitions, cost and reproducibility

Q3. Independent repetitions and scoring cost

We completed three paired 1.7B/100B runs comparing random data with our method (CoT-proxy), using initialization seeds 20261110, 2026111701 and 2026111702. Each pair shares initialization; data and data-order seed stay fixed across pairs. SVAMP gains are +8.8, +8.1 and +7.1 pp (mean +8.0; sample SD 0.85). All three pairs improve 0-shot rBridge on GSM8K, MATH and HumanEval (Common Response Part 5). Question-bootstrap uncertainty is separate. Limited pretraining compute led us to prioritize these pairs over further repetitions of other baselines or the original 7B/1T study.

“Training-free” means no additional scorer fitting. For the same 243,295,685-document pool, successful production scoring on NVIDIA H100 GPUs cost:

ScorerInput limitGPU-hours
1.7B Our method (CoT-proxy)4,096 Qwen tokens; raw + intervened forwards2,016.07
FineWeb-Edu classifier512 native BERT tokens; FP32/TF32108.51

Costs include scoring-job setup but exclude queues, pilots, head discovery, packing and training/evaluation. Our method (CoT-proxy)/Edu 11-task means are 35.86%/34.92% (+0.94 pp); rule-based cost was not measured.

Scoring is a one-time cost for a fixed selector and corpus. We reuse the selected data across target model sizes and training seeds, amortizing approximately 2,016 GPU-hours across multiple runs. This is manageable at our study scale.

The FineWeb-Edu model card reports 450k Llama-3-70B-Instruct annotations plus classifier-head training. This upfront construction cost is excluded from the 108.51 GPU-hour application figure.

Q4. Corpus-only training and the original random baseline

We clarify the original random loader's design: after a full-corpus traversal, it uniformly redraws the additional 73.1B-token subset from the full corpus. Membership is refreshed, rather than only reshuffling a fixed subset. Under identical sampling units and domain weights, both components have marginal distribution p, so αp + (1−α)p = p. This is marginal equivalence; finite-run exposure patterns, data order and optimization trajectories need not be identical. We will document this refresh mechanism explicitly.

During this rebuttal, we added a direct corpus-only baseline: 100B tokens randomly sampled from the full four-domain pool, with no additional subset component. Our method (CoT-proxy) and length-control runs also use 100B tokens and identical domain budgets. Our method (CoT-proxy) improves the 11-task mean from 35.02% to 35.86% at 1.7B (+0.84 pp) and from 36.62% to 38.12% at 8B (+1.51 pp; Common Response Part 4). Differences use unrounded scores. This demonstrates gains outside the original upsampling protocol at this rebuttal's 100B-token budget. An 80% random-base + 20% selected-data study is ongoing, excluding shared document IDs and identical texts between branches (Common Response Part 6).

Q5. Public checkpoints and reproduction artifacts

Internal restrictions prevent public release of the original selector checkpoints. We therefore implemented this rebuttal study with public Qwen3 Base checkpoints, pinned to fixed revisions; target models train from random initialization.

We will release the proxy/head-selection scripts and a single-command selector-construction entry point, exact zero-based layer/query-head indices, actual per-domain token counts, tokenizer and scoring configurations, and essential training/evaluation scripts. This commitment covers our rebuttal experiments with public checkpoints. Common Response Parts 1, 4 and 5 describe this scope and the fixed settings.

Q6. Causal normalization and the denominator typo

Thank you for the careful reading and for catching the inconsistency. Uniform attention is normalized only over positions permitted by the causal and padding masks. For query t, each valid key j in V_t has weight 1/|V_t|; all other positions have zero weight. Zeroing selected query-projection rows makes allowed logits equal while preserving these masks; it does not zero the head output.

The original implementation uses the unmodified model's document loss as the denominator:

AI(D)=Lref(D)−Lbase(D)Lbase(D).\mathrm{AI}(D)=\frac{L_{\mathrm{ref}}(D)-L_{\mathrm{base}}(D)}{L_{\mathrm{base}}(D)}.

Here L_ref is the intervened loss. Our code matches Equation (5), using L_base as the denominator. Figure 2's L_ref denominator is a typo that we will correct. As a side note, the two formulas are equivalent for ranking the same documents with positive losses. With identical tie-breaking, selecting the same document count or token budget by rank gives the same data.

Thank you for the response, it addressed my questions, I maintain my positive score.

Comments on the paper

11 comments
AuthorsOpenReview ↗

Attention Influence with Public Checkpoints: Proxy Adaptation and Pretraining Validation — Part 1 — Public checkpoints and experimental setting

Part 1 — Public checkpoints and experimental setting

We thank all reviewers for their constructive feedback. During the rebuttal period, we conducted a supplementary study to address shared concerns about selector checkpoint accessibility, sensitivity to proxy tasks and head selection, variation across random seeds, and evidence for reasoning-related improvements.

The original selector checkpoints were trained internally and cannot be publicly released due to internal restrictions. Following the reviewers' suggestion, we independently reimplemented the selection pipeline using public Qwen3 Base checkpoints and public corpora, making the same pretrained selector weights accessible to other researchers.

We will release the proxy-based head-selection and selector-construction scripts, fixed configurations, and a single-command entry point for constructing a selector from a public Qwen3 Base checkpoint. For each released selector, we will publish the exact masked-head indices in every layer, with both layer and head indices numbered from zero.

This study evaluates the Attention Influence framework with public checkpoints and examines how proxy choice can be adapted to different selectors. The core workflow is to identify attention heads with a proxy task, then select documents using the loss change caused by head intervention. We compare proxy choices explicitly and retain the original JSON-proxy results. We agree with Reviewer 2GLr that model-assisted data curation is established practice; our contribution concerns the head-intervention selection signal.

To support more controls and repetitions within the rebuttal period, we use a 100B-token pretraining budget for 1.7B and 8B target models with the dense Qwen3 architecture. The pretrained checkpoints serve as frozen selectors; target models are trained from random initialization. In the main comparisons, selected and random corpora have exactly matched total and per-domain token budgets. Within each target size, each pair shares the initialization, tokenizer, 8,192-token training context, optimizer, cosine learning-rate schedule, and global batch size. Most control runs and seed repetitions use the 1.7B target model.

With the CoT proxy, a single frozen Qwen3-1.7B-Base selector produces one 100B-token corpus used to train both target model sizes. Both the 1.7B and 8B target models improve on mathematical and coding tasks relative to their respective same-size, same-budget random-data baselines. This provides end-to-end empirical evidence that a smaller selector can benefit a larger target model in the evaluated 1.7B-to-8B setting. In response to the reviewer's caution about “weak-to-strong,” we describe the result as using a smaller selector to select data for pretraining target models at a larger parameter scale. This refers to parameter count without assuming that the pretrained selector is less capable than the target model.

AuthorsOpenReview ↗

Part 2 — Proxy sensitivity, controls, and selector training history

Training-free selection enables inexpensive exploration. Starting from a frozen Qwen3 checkpoint, constructing and applying a selector requires forward-pass measurements without training or fine-tuning the selector. We use a small development panel sampled from the FineWeb-Edu dataset to inspect selected samples for within-document repetition and quality, and to compare candidate proxies and head sets. These diagnostics help identify obvious failure modes and form an early, qualitative expectation of downstream utility before committing to costly pretraining. Controlled training comparisons then test those expectations.

The selector's training history differs substantially from the original study. Our original selector was an intermediate 1.3B dense checkpoint trained for 1.5T tokens, without synthetic CoT training data. The public study uses the final released Qwen3-1.7B-Base checkpoint. The Qwen3 Technical Report (Qwen Team, 2025, §§3.1–3.2) describes approximately 36T pretraining tokens, including trillions of synthetic tokens and a reasoning-focused stage of about 5T tokens with an increased proportion of STEM, coding, reasoning, and synthetic data. We use “overtrained” here to describe this large token budget relative to model size. The report confirms exposure to synthetic data; CoT-style solutions may be present, but their form and proportion in the Base checkpoint are not disclosed.

Our working hypothesis is that extensive pretraining may distribute retrieval-related functions across more heads, with greater redundancy. Intervening on a limited head set identified by JSON retrieval may leave other pathways available, producing a smaller or less discriminative loss increase on reasoning-rich documents. To examine a solution-focused proxy, we also identify heads through their intervention effect on the loss of publisher-provided mathematical solutions. The CoT-proxy selector chooses eight heads using 200 OpenStax worked examples; GSM8K, MATH, and AIME problems are excluded from head selection.

We examine these choices through three complementary sets of experiments:

Proxy and head-set sensitivity. At the proxy stage, we compare plain needle, JSON key-value retrieval, and CoT proxies through head rankings, intervention effects, and document selection. Downstream pretraining comparisons use corpora selected with the JSON and CoT proxies, including the eight-head CoT-proxy configuration, together with an additional random-head control. Plain needle is evaluated only at the proxy stage. JSON23 denotes the Qwen3-1.7B-Base selector with 23 heads identified by the JSON key-value retrieval proxy for intervention during document scoring. Results are reported for each complete selector configuration.

Independent repetitions and data controls. We added two initialization seeds at 1.7B, giving three paired random-baseline/selected-data comparisons. Each pair shares its initialization, while the data pools and data-order seed remain fixed across repetitions. We compare matched training-token budgets and distinguish completed results from ongoing runs. A length-matched random control addresses the document-length concerns raised by Reviewers Lyfe and 9kRs. The comparisons with matched per-domain token budgets also include a FineWeb-Edu classifier baseline.

Additional evidence at evaluation cost. We evaluate saved checkpoints using task performance, validation NLL, and the reviewer-recommended rBridge metric. Mathematical problem solving and code generation measure task outcomes; rBridge provides a complementary measure of conditional prediction of fixed reference reasoning. Paired question-level analyses add evidence without additional pretraining runs. We report question-sampling uncertainty separately from variation across independent training seeds.

AuthorsOpenReview ↗

Part 3 — Repetition diagnostics and evaluation

Within-document repetition provides one diagnostic of the selected corpora. In the original study, we observed only a modest increase over random sampling; this historical observation is qualitative. For the public-checkpoint study, we restricted the audit to training documents originating in the FineWeb-Edu dataset and uniformly sampled 80 documents per group from each group's actual training corpus.

We report token-weighted R8 over a common first-4,096-token window, using the full text when shorter. This is the total number of tokens covered by later occurrences of identical eight-token spans, divided by the total number of tokens across these windows. The first occurrence of each span contributes no repeat coverage; overlaps are counted once, and appended document EOS tokens are excluded. Both selectors use Qwen3-1.7B-Base: JSON23 intervenes on 23 heads identified by the JSON key-value retrieval proxy, while the CoT-proxy selector uses eight heads identified from OpenStax solutions.

Repetition metricRandom baselineJSON proxy (23 heads)CoT proxy (8 heads)
Common first-4K, token-weighted R82.0183%3.9159%3.0224%

JSON23's R8 exceeds the random baseline by 1.8976 pp (approximately 1.94×), with a 95% interval of [+0.2409,+3.5660] pp. The eight-head CoT-proxy selector has a lower point estimate than JSON23, but the interval for that difference includes zero. All intervals use 20,000 document-level bootstrap resamples, drawn independently within each group. R8 captures short-span reuse, including legitimate terminology; it does not directly establish head redundancy or its causal contribution to the downstream performance gap.

The 100B-token budget also affects which evaluations are informative. Our 1.7B random baseline scores 24.95% on MMLU, close to the 25% accuracy of uniform guessing. Scores near this level can be less informative about modest improvements in the capabilities targeted by data selection. We therefore report a focused summary for this training regime, omitting near-chance broad-knowledge tasks such as MMLU from that summary and adding SVAMP as a supplementary mathematical benchmark. The 11 tasks are GSM8K, MATH, HumanEval, SVAMP, ARC-Challenge, ARC-Easy, HellaSwag, WinoGrande, CommonSenseQA, PIQA, and TriviaQA.

These public-model experiments supplement the original 7B / 1T-token study and are reported separately. We will present detailed experimental settings and results in public follow-up responses, and address reviewer-specific questions in the corresponding point-by-point replies.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.