acceptodds
Under review as a conference paper at ICLR 2027

Calibrating Long-Range Attention with Semantic Needles for Long-Context Pretraining Data Selection

Abstract

Effective long-context training requires documents with meaningful long-range dependencies, not merely long documents. Existing model-based selectors broadly follow two approaches: prediction-based contrast and attention-based measurement. Prediction-based selectors can measure the contribution of distant context but are costly to apply across a corpus, while attention-based selectors are more efficient yet often aggregate over heads without identifying those responsive to useful distant evidence. We propose SNAS (Semantic-Needle-guided Attention-based data Selection), which uses prediction contrast on a small calibration corpus to identify naturally occurring context-dependent target tokens and localize their distant evidence intervals. SNAS then calibrates model-specific attention heads by their enrichment over these intervals, without synthetic probes or downstream-task labels. For corpus-wide selection, it scores each document in a single forward pass using both distant-attention allocation and retrieved-content signals from the calibrated heads. Across two scoring models and three data domains, SNAS matches or numerically exceeds the strongest baseline in overall score across all six settings. With the Llama-3.1-8B scorer, SNAS achieves LongFilter's corpus-scoring throughput and uses 44% fewer GPU-hours; these measurements cover corpus scoring and exclude the one-time calibration cost.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Large Language Models, Long-context training data, Data selection, Attention heads

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wMgyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SNAS for selecting long-context training documents. A calibration stage identifies attention heads that respond to useful distant information. These heads then score documents in one forward pass. Experiments report quality comparable to LongFilter with higher scoring throughput.

Critical strengths
  1. The method offers a useful quality–cost trade-off, reporting about 1.8× scoring throughput while maintaining comparable downstream performance.
  2. Matched training budgets, head-selection ablations, and calibration checks support the proposed scoring signal. Results cover multiple scoring models and data domains.
Critical weaknesses
  1. The method offers a useful quality–cost trade-off, reporting about 1.8× scoring throughput while maintaining comparable downstream performance.
  2. Matched training budgets, head-selection ablations, and calibration checks support the proposed scoring signal. Results cover multiple scoring models and data domains.
Questions for authors
  1. What is the total calibration-plus-scoring cost, and how large must a corpus be for SNAS to save compute?
  2. Can you report variation across runs for the main comparison and key ablations?
Reviewer_5kDtOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SNAS, a method for selecting long-context pretraining data. It first identifies tokens whose predictions improve with distant context and uses them to select relevant attention heads. These heads then score documents in a single forward pass. Experiments show performance comparable to LongFilter with lower corpus-scoring cost.

Critical strengths
  1. The paper addresses an interesting problem: selecting documents with useful long-range dependencies rather than simply selecting long documents. Using dependencies found in natural text to identify attention heads is a reasonable approach.

  2. The experiments cover different scoring models and data domains, with results averaged over three training runs. The head-selection and scoring ablations support the main design choices. The method also achieves comparable performance to LongFilter with faster corpus scoring.

Critical weaknesses
  • The total selection cost is not fully evaluated. The reported speedup excludes head calibration, which requires repeated predictions with different context lengths. Since lower cost is a main contribution, the paper should report calibration time and the corpus size needed to recover this additional cost.

  • The evidence for the proposed scoring signals remains limited. The norm of attention-weighted value vectors does not necessarily indicate useful semantic information, and the scoring ablation is limited to one domain and scorer. The overall improvements over LongFilter are also small, and the paper reports means without variability. This makes it difficult to assess how reliably the method preserves selection quality.

Questions for authors

What is the total calibration cost, and how large must the candidate corpus be for SNAS to become cheaper than LongFilter?

Can you report standard deviations or confidence intervals for the main results and ablations?

How well does scoring only the first 64K tokens represent the quality of the remaining text in longer documents?

Does the retrieved-content signal still help after controlling for value-vector norms, and does its benefit hold across other domains and scoring models?

6 comments
AuthorsOpenReview ↗

Response to Reviewer 5kDt (1/4)

Thank you for your positive assessment and the specific questions. We have uploaded a revised version of the paper, with all changes marked in blue. We group the comments into four parts and answer them in turn. Each part lists the corresponding weaknesses (W) and questions (Q) and states where the revised paper addresses them.


A. Calibration cost and the corpus size needed to recover it (W1, Q1)

Response. Thank you for raising this point. Reducing cost is the main contribution of the paper, so the calibration cost should be counted. We report the timing of the full calibration below.

Setup. Calibration and the scoring times in Table 2 use the same setup:

  • Hardware and model: H800. All methods use the same Llama-3.1-8B scoring model in bf16. Timings are similar for another scoring model of the same size (e.g., Qwen3-8B-Base).
  • Execution: one process per GPU, batch size 1, scoring window of the first 64K tokens.
  • Timing: average per-document time on a single GPU, excluding model loading and warm-up on the first document. The wall-clock time of multi-GPU runs is converted to single-GPU time, so all GPU-h in this section are single-GPU hours and are directly comparable to the scoring cost in Table 2.

Calibration cost. The full calibration takes 49.0 GPU-h. It scans 1,002 documents and retains 987 (about 65M tokens). The repeated predictions at different context lengths run only on these ~1,000 documents and do not grow with the candidate pool:

Length tierScanned docsRetained docsGPU-h
32K4023937.4
64K40239817.4
128K19819624.2
Total1,00298749.0

The calibration data amount to less than 0.3% of the 30B-token candidate data in the paper.

Break-even scale. Calibration is done once per scoring model, whereas scoring is paid for every document. At the Table 2 throughput (SNAS 0.225 doc/s, LongFilter 0.125 doc/s), SNAS saves 3.56 GPU-s per document and recovers the calibration cost after about 50K documents (about 3.2B tokens at the average CC document length). Beyond that point it saves about 15 GPU-h per 1B tokens. The savings grow linearly with corpus size and reach about 15K GPU-h on a 1T-token corpus, which is a substantial cost advantage for data selection in large-scale long-context pretraining:

Candidate corpusSNAS scoring + calibration (GPU-h)Calibration share of SNAS totalLongFilter (GPU-h)SNAS saving incl. calibration
10B194.2 + 49.0 = 243.220.1%348.130%
30B582.6 + 49.0 = 631.67.8%1,044.340%
100B1,942 + 49.0 = 1,9912.5%3,48143%
1T19,420 + 49.0 = 19,4690.25%34,81044%

Reuse. The same head set HM\mathcal{H}_M is shared across CC, arXiv, and Books, and the calibration set itself mixes the three domains, so the three pools in the paper pay the calibration cost only once. Calibration can also be made cheaper: with 1/4 of the calibration documents (about 12 GPU-h), the resulting head set has a Jaccard similarity of 0.87 ± 0.06 with the full-calibration head set.

The revised paper reports the calibration cost in Section 4.4 (Scoring Efficiency), with the per-tier cost in Table 2(b). The timing setup, cost breakdown, and break-even analysis are in Appendix B.1.

AuthorsOpenReview ↗

Response to Reviewer 5kDt (2/4)

Response to Reviewer 5kDt (2/4)

B. Stability of the results and reliability of the quality gains (W2, last two sentences; Q2)

Response. Thank you for the suggestion. We report here the uncertainty of both the main experiments (Table 1) and the ablations (Tables 3–4 and the new controls). Every setting is trained with three random seeds (42 / 1 / 2). For each seed we average the four benchmark scores to obtain its Ovr., and we then report the mean and the sample standard deviation (n−1) of the three Ovr. values.

Table R1: Ovr. of the four methods (mean ± std over 3 seeds)

Scoring modelDomainSNASLongFilterLongAttnLADM
Llama-3.1-8BCC46.57 ± 0.1046.29 ± 0.2842.65 ± 0.1145.88 ± 0.17
Llama-3.1-8BarXiv43.77 ± 0.0943.67 ± 0.2339.50 ± 0.1342.71 ± 0.06
Llama-3.1-8BBooks47.08 ± 0.2746.40 ± 0.3545.12 ± 0.0945.43 ± 0.02
Qwen3-8B-BaseCC45.96 ± 0.1245.88 ± 0.3744.28 ± 0.1944.56 ± 0.15
Qwen3-8B-BasearXiv43.68 ± 0.1943.07 ± 0.3042.53 ± 0.3240.54 ± 0.25
Qwen3-8B-BaseBooks46.81 ± 0.1246.65 ± 0.1445.92 ± 0.2644.79 ± 0.19

Stability of the ablations. Tables 3–4 and the new cross-setting ablations also use three seeds. To avoid repeating the uncertainty in each mechanism analysis, we collect the Ovr. mean ± std of all ablations here:

SettingVariantOvr. (mean ± std)
Llama/CCSame-layer random heads (30)44.61 ± 0.29
Retrieval heads (44)45.35 ± 0.36
First-layer heads (32)44.83 ± 0.25
a only45.99 ± 0.32
a + attention-weighted value norm46.14 ± 0.27
a + v (SNAS)46.57 ± 0.10
Qwen/CCa only45.73 ± 0.28
a + v (SNAS)45.96 ± 0.12
Llama/Booksa only46.14 ± 0.36
a + v (SNAS)47.08 ± 0.27
  • Against LongFilter: in all 6 settings SNAS has the higher mean and the smaller std. Compared seed by seed, SNAS is higher in 15 of the 18 runs.
  • Against LongAttn and LADM: in all 6 settings the margin of SNAS exceeds the sum of the two methods' stds.
  • Stability: the std of SNAS is 0.09–0.27 in the main experiments, and the stds of all ablation variants are 0.10–0.36.
  • Ablation gaps: on Llama/CC, SNAS is 0.43–1.96 above every ablation variant, and each gap exceeds the sum of the stds of the two methods compared. The cross-setting results are in Part D.

Comparison with a larger trained model. You noted that the overall gain of SNAS over LongFilter is small (0.08–0.68 with the 8B trained model). To see how this gap changes with a larger trained model, we keep the scoring model (Llama-3.1-8B) and the CC candidate pool fixed and replace the trained model with Qwen3-14B-Base. All other steps follow Table 1:

Qwen3-14B-Base, CC (Llama scoring)HELMET*LB v2RULERMRCR-v2Ovr.
LongFilter66.4733.3085.7616.1550.42
SNAS67.0334.4085.5617.0551.01

SNAS is 0.59 above LongFilter in Ovr., about twice the gap in the same setting with the 8B trained model (0.28). This suggests that the margin may grow with the size of the trained model. SNAS is also higher on three of the four benchmarks, the exception being RULER. One possible explanation is that a larger model makes better use of the long-range dependencies in the selected data, so differences in data quality show up more clearly downstream. The cost of data selection depends only on the scoring model, so the cost advantage of SNAS is unchanged. These results are in Appendix A.6 of the revised paper and are briefly noted in Section 4.3 (Main Results).

The tables in the main text still report means only. The stds and the per-seed, per-task results of all methods and ablations are in Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer 5kDt (3/4)

C. Whether scoring the first 64K tokens represents longer documents (Q3)

Response. Thank you for this question. We agree that the first 64K tokens only approximate the full document, and we ran experiments to measure the effect.

Why 64K. The official implementation of LongFilter (Deng et al., 2026) scores only the first 65,536 tokens of each document. We use the same window for all methods, so the comparison does not favor SNAS. The score only decides which documents are selected; each selected document is used in full for training. Scoring only the beginning of a document is also common in data selection: FineWeb-Edu (Penedo et al., 2024) and Messmer et al. (2026) score documents on their first 512 tokens. Weighing fair comparison, prior practice, and compute, we chose the first 64K tokens as a trade-off between efficiency and quality.

Why Books. All Books documents exceed 64K tokens, and 57.3% of Books tokens lie beyond the first 64K, compared with 20.6% for CC and 8.1% for arXiv. Books is therefore the hardest test of whether the first 64K tokens represent the full document.

Scoring cost. On the subset of Books documents with at least 128K tokens, the per-document scoring time on a single GPU is as follows (same setup as Table 2):

MethodFirst 64K (s)First 128K (s)Ratio
SNAS6.0317.462.90×
LongFilter10.0526.392.63×

Extending the window to 128K raises the scoring time of both methods by 2.6–2.9×.

Window correlation and downstream results on Books. On the Books pool, the Spearman ρ between scores from the first 64K and the first 128K tokens is 0.773. We then re-scored with the first 128K tokens for both SNAS and LongFilter and retrained, with all other settings as in Table 1:

Books (Llama scoring)HELMET*LB v2RULERMRCR-v2Ovr.
LongFilter, first 64K (Table 1)64.9227.1576.0917.4546.40 ± 0.35
LongFilter, first 128K65.8327.2375.1916.6946.24 ± 0.05
SNAS, first 64K (Table 1)65.5527.7076.0519.0347.08 ± 0.27
SNAS, first 128K65.3927.3775.4519.6546.97 ± 0.05

Each entry is the mean over three seeds (42 / 1 / 2), and the Ovr. column also gives the sample std.

  • The 128K window brings no gain: Ovr. changes by −0.11 for SNAS and −0.16 for LongFilter, both smaller than the seed std of each method at 64K (0.27 and 0.35). Every benchmark changes by less than 1 point.
  • The comparison between methods is unchanged: with the first 128K tokens, SNAS is still 0.73 above LongFilter (0.68 at 64K) and is higher on all three seeds.

Since a larger window brings little benefit but costs 2.6–2.9× more in scoring, we keep the first 64K tokens as the default.

Inference about full-document scoring. SNAS outperforms LongFilter under both the 64K and the 128K window, and the margin does not shrink when the window doubles (0.68 → 0.73). From this we infer that SNAS would remain competitive under full-document scoring and that its advantage is not specific to the 64K window.

We acknowledge that this is an inference. Scoring the first 64K tokens is still an approximation of the full document, and the experiments above only reach 128K. How much of a document should be scored to balance quality and cost (for example, by sampling segments across the whole document) is a direction for future work. The revised paper states in the Limitations paragraph of Section 5 (Conclusion) that the first 64K tokens approximate the full document and lists this direction. The scoring cost, window correlation, and the downstream results above are in Appendix B.4 (Scoring Window).

References

  • Deng H, Lin Y, Lin Z, et al. Beyond length: Quantifying long-range information for long-context LLM pretraining data. In International Conference on Learning Representations, 2026: 12983–12995.
  • Messmer B, Sabolčec V, Jaggi M. Enhancing multilingual LLM pretraining with model-based data selection. Advances in Neural Information Processing Systems, 2026, 38.
  • Penedo G, Kydlíček H, Lozhkov A, et al. The FineWeb datasets: Decanting the web for the finest text data at scale. Advances in Neural Information Processing Systems, 2024, 37: 30811–30849.
Reviewer_hzZsOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary
  1. Problem and method. The paper proposes SNAS for selecting long-context continual-pretraining data. On a small calibration corpus, it compares long- and short-context token predictions, identifies targets that benefit from distant context, and assigns a coarse evidence interval to each target. It then selects attention heads whose attention to these intervals is enriched relative to their own background distance profiles.

  2. Corpus scoring. With the calibrated heads fixed, SNAS scores each candidate document in a single forward pass, using at most its first 64K tokens. It combines distant-attention allocation with a statistic based on attention-weighted value vectors, fuses their rankings, and selects whole documents under a token budget.

  3. Evaluation. Documents selected from Common Crawl, arXiv, and Books are used to continue pretraining the same Llama-3-8B-Base, followed by instruction tuning and four long-context evaluation suites. Across the six main domain-scorer settings, SNAS has slightly higher overall means than LongFilter. With the Llama scorer on Common Crawl, it reports 1.8 times the corpus-scoring throughput and about 44% fewer GPU-hours, excluding calibration.

Critical strengths
  1. A useful approach to a practical data-selection problem. Using expensive prediction comparisons for a small calibration set, then reusing the selected heads for single-pass corpus scoring, is a sensible way to combine predictive signals with a cheaper selection procedure. The enrichment score is also well motivated: it accounts for each head's background distance preference rather than selecting heads only by their raw attention mass on distant positions.

  2. Positive results with meaningful component comparisons. SNAS has slightly higher overall means than LongFilter in all six main domain-scorer settings. The reported three-domain averages are 45.81 versus 45.45 with the Llama scorer and 45.48 versus 45.20 with the Qwen scorer. The authors appropriately describe these as comparable quality rather than a large accuracy improvement. Table 3 supports the choice of calibrated heads under a fixed scoring rule, and Table 4 shows a further gain from the full score with the head set fixed. The fixed-length experiment also shows that a positive result remains in an equal-length document pool. These results give the method a useful empirical basis.

Critical weaknesses
  1. The evidence for the main quality-cost trade-off is incomplete. Table 2 measures corpus scoring with the Llama scorer on Common Crawl and explicitly excludes calibration. I appreciate this disclosure, but the total calibration cost is not reported. Calibration requires prediction comparisons across context lengths and evidence-interval localization; the number of retained documents alone does not show that this cost is small. For the workload in Table 2, the scoring-cost difference is 153.90 GPU-hours. Ignoring other setup costs, calibration would need to cost less than this for SNAS to retain a total-cost advantage on a single use of that pool. Reuse could improve the trade-off, but the actual cost and reuse conditions are needed to assess this.

    On quality, the paper reports means over three runs without per-run results or uncertainty estimates. The small margins over LongFilter are encouraging, but their stability is unclear. I do not require a significant accuracy improvement: reliably maintaining quality at lower cost would be valuable. At present, however, the complete cost and the uncertainty in quality are not sufficiently documented to establish that practical benefit.

  2. Baseline adaptation and experimental descriptions leave important questions about the comparisons. Section 3.3 says the attention-allocation component is kept identical to LongAttn, but uses min-max normalization. The published LongAttn method uses Z-score standardization (Wu et al., 2025, Section 3.2.3, Eq. 8). Separately normalizing two metrics before combining them can change their relative weights and the resulting document ranking. The paper should state which definition is used for the LongAttn baseline and for SNAS, along with the variance calculation and aggregation order. This is an unresolved adaptation issue, not a confirmed implementation error; the local controls in Tables 3 and 4 still have value.

    Several experimental descriptions also need reconciliation. Section 4.1 gives approximately 10B candidate tokens per domain, whereas Appendix A.2 gives approximately 10B across all three domains. Table 8 reports 47.77 at the 2B budget without clearly identifying the selector, scorer, domain setting, or aggregation that connects it to the main results. Appendix A.2 and Table 9 also report different packing lengths. These may be reporting omissions rather than changes in the actual comparisons, but they currently make the experiments difficult to verify.

    A secondary concern is that the empirical usefulness of the proxy scores is better supported than their semantic interpretation. The added value-based signal combines value norms, distance bands, a sink discount, and top-query aggregation, so Table 4 does not isolate which part explains the gain. Likewise, 27 of the 30 calibrated heads are in layers L9-L16, but the random-head control is not reported to match this layer distribution. These limitations do not negate the observed gains, but they narrow what the ablations establish about the method's explanation.

Questions for authors
  1. Quality and complete cost. Could the authors provide the existing per-run scores and uncertainty for the main comparisons and key ablations? Please also report the full calibration GPU-hours, the number of documents scanned and retained, and the hardware and timing setup used for Table 2. At what corpus size or number of reuses does calibration plus scoring become cheaper than LongFilter?

  2. Experimental settings. Is the candidate pool approximately 10B tokens per domain or across all three domains? Please provide the per-domain statistics. Which selector, scorer, domain setting, and aggregation are used in Table 8, and how does its 2B result relate to Table 1? Please clarify whether all non-budget settings are held fixed within the sweep and reconcile the packing lengths in Appendix A.2 and Table 9.

  3. Baseline definition and indexing. Which normalization, variance definition, and aggregation order are used for the LongAttn baseline and for SNAS's attention-allocation score? Please explain any adaptation from the published LongAttn definition. Separately, could a concrete example or code-level test show how the target token, prediction logit, attention query row, and evidence-key interval are aligned during calibration? I am asking for clarification of the causal prediction indexing, not asserting that the implementation is incorrect.

  4. Support for the interpretation. Is there evidence from a random-head control matched in both count and layer distribution, or from a focused comparison of the main parts of the value-based score? A few semantic-needle examples with checks on the selected evidence intervals would also help assess what the calibration captures. These would clarify the explanation beyond the observed downstream gains.

The first three questions mainly ask for records and implementation details underlying the reported experiments. The fourth asks for focused support for the interpretation, rather than a new large-scale training study. A narrower explanation may be appropriate where the current evidence does not separate the proposed mechanisms.

12 comments
AuthorsOpenReview ↗

Response to Reviewer hzZs (1/8)

Thank you for the detailed and constructive comments. We have uploaded a revised version of the paper, with all changes marked in blue. We group the comments into five parts and answer them in turn. Each part lists the corresponding weaknesses (W) and questions (Q) and states where the revised paper addresses them.


A. Calibration cost (W1, first paragraph; Q1, cost part)

Response. Thank you for raising this point. We agree that the cost advantage of SNAS can only be judged once calibration is included in the total cost. We report the timing of the full calibration below.

Setup. Calibration and the scoring times in Table 2 use the same setup:

  • Hardware and model: H800. All methods use the same Llama-3.1-8B scoring model in bf16. Timings are similar for another scoring model of the same size (e.g., Qwen3-8B-Base).
  • Execution: one process per GPU, batch size 1, scoring window of the first 64K tokens.
  • Timing: average per-document time on a single GPU, excluding model loading and warm-up on the first document. The wall-clock time of multi-GPU runs is converted to single-GPU time, so all GPU-h in this section are single-GPU hours and the calibration cost is directly comparable to the scoring gap in Table 2.

Calibration cost. The full calibration takes 49.0 GPU-h. This is below the 153.90 GPU-h scoring gap you estimated, so SNAS remains cheaper after calibration even when it is used only for the single pool in Table 2.

Length tierScanned docsRetained docsGPU-h
32K4023937.4
64K40239817.4
128K19819624.2
Total1,00298749.0

Calibration uses only about 1,000 documents (about 65M tokens), less than 0.3% of the 30B-token candidate data in the paper.

Break-even scale. Calibration is done once per scoring model, whereas scoring is paid for every document. At the Table 2 throughput (SNAS 0.225 doc/s, LongFilter 0.125 doc/s), SNAS saves 3.56 GPU-s per document and recovers the calibration cost after about 50K documents (about 3.2B tokens at the average CC document length). Beyond that point it saves about 15 GPU-h per 1B tokens. The savings grow linearly with corpus size and reach about 15K GPU-h on a 1T-token corpus, which is a substantial cost advantage for data selection in large-scale long-context pretraining:

Candidate corpusSNAS scoring + calibration (GPU-h)Calibration share of SNAS totalLongFilter (GPU-h)SNAS saving incl. calibration
10B194.2 + 49.0 = 243.220.1%348.130%
30B582.6 + 49.0 = 631.67.8%1,044.340%
100B1,942 + 49.0 = 1,9912.5%3,48143%
1T19,420 + 49.0 = 19,4690.25%34,81044%

Reuse. The same head set HM\mathcal{H}_M is shared across CC, arXiv, and Books, and the calibration set itself mixes the three domains, so the three pools in the paper pay the calibration cost only once. Calibration can also be made cheaper: with 1/4 of the calibration documents (about 12 GPU-h), the resulting head set has a Jaccard similarity of 0.87 ± 0.06 with the full-calibration head set.

The revised paper reports the calibration cost and timing setup in the efficiency analysis of Section 4.4. The cost breakdown and break-even analysis are in Appendix B.1.

AuthorsOpenReview ↗

Response to Reviewer hzZs (2/8)

Response to Reviewer hzZs (2/8)

B. Stability of the results (W1, second paragraph; Q1, stability part)

Response. Thank you for the suggestion. We agree that maintaining quality consistently at a lower cost is itself of value. We report here the uncertainty of both the main experiments (Table 1) and the ablations (Tables 3–4 and the new controls). Every setting is trained with three random seeds (42 / 1 / 2). For each seed we average the four benchmark scores to obtain its Ovr., and we then report the mean and the sample standard deviation (n−1) of the three Ovr. values.

Table R1: Ovr. of the four methods (mean ± std over 3 seeds)

Scoring modelDomainSNASLongFilterLongAttnLADM
Llama-3.1-8BCC46.57 ± 0.1046.29 ± 0.2842.65 ± 0.1145.88 ± 0.17
Llama-3.1-8BarXiv43.77 ± 0.0943.67 ± 0.2339.50 ± 0.1342.71 ± 0.06
Llama-3.1-8BBooks47.08 ± 0.2746.40 ± 0.3545.12 ± 0.0945.43 ± 0.02
Qwen3-8B-BaseCC45.96 ± 0.1245.88 ± 0.3744.28 ± 0.1944.56 ± 0.15
Qwen3-8B-BasearXiv43.68 ± 0.1943.07 ± 0.3042.53 ± 0.3240.54 ± 0.25
Qwen3-8B-BaseBooks46.81 ± 0.1246.65 ± 0.1445.92 ± 0.2644.79 ± 0.19

Table R2: Per-seed Ovr. of SNAS and LongFilter (the number in each column name is the seed)

Scoring modelDomainSNAS 42SNAS 1SNAS 2LongFilter 42LongFilter 1LongFilter 2Mean diff.
Llama-3.1-8BCC46.6546.6046.4646.3645.9846.54+0.28
Llama-3.1-8BarXiv43.7843.8743.6843.4443.8943.67+0.10
Llama-3.1-8BBooks47.2046.7847.2746.7546.0546.40+0.68
Qwen3-8B-BaseCC46.0945.9645.8545.5946.3045.76+0.08
Qwen3-8B-BasearXiv43.7643.4643.8143.2142.7243.27+0.61
Qwen3-8B-BaseBooks46.8846.8846.6746.8146.5846.56+0.16

Stability of the ablations. Tables 3–4 and the new cross-setting ablations also use three seeds. To avoid repeating the uncertainty in each mechanism analysis, we collect the Ovr. mean ± std of all ablations here:

SettingVariantOvr. (mean ± std)
Llama/CCSame-layer random heads (30)44.61 ± 0.29
Retrieval heads (44)45.35 ± 0.36
First-layer heads (32)44.83 ± 0.25
a only45.99 ± 0.32
a + attention-weighted value norm46.14 ± 0.27
a + v (SNAS)46.57 ± 0.10
Qwen/CCa only45.73 ± 0.28
a + v (SNAS)45.96 ± 0.12
Llama/Booksa only46.14 ± 0.36
a + v (SNAS)47.08 ± 0.27

Summary.

  • Against LongFilter: in all 6 settings SNAS has the higher mean and the smaller std. Compared seed by seed, SNAS is higher in 15 of the 18 runs.
  • Against LongAttn and LADM: in all 6 settings the margin of SNAS exceeds the sum of the two methods' stds.
  • Stability: the std of SNAS is 0.09–0.27 in the main experiments, and the stds of all ablation variants are 0.10–0.36.
  • Ablation gaps: on Llama/CC, SNAS is 0.43–1.96 above every ablation variant, and each gap exceeds the sum of the stds of the two methods compared (the individual gaps are given in E1 and E2).

Comparison with a larger trained model. You noted that the margin of SNAS over LongFilter is small (0.08–0.68 with the 8B trained model). To see how this gap changes with a larger trained model, we keep the scoring model (Llama-3.1-8B) and the CC candidate pool fixed and replace the trained model with Qwen3-14B-Base. All other steps follow Table 1:

Qwen3-14B-Base, CC (Llama scoring)HELMET*LB v2RULERMRCR-v2Ovr.
LongFilter66.4733.3085.7616.1550.42
SNAS67.0334.4085.5617.0551.01

SNAS is 0.59 above LongFilter in Ovr., about twice the gap in the same setting with the 8B trained model (0.28). This suggests that the margin may grow with the size of the trained model. SNAS is also higher on three of the four benchmarks, the exception being RULER. One possible explanation is that a larger model makes better use of the long-range dependencies in the selected data, so differences in data quality show up more clearly downstream. The cost of data selection depends only on the scoring model, so the cost advantage of SNAS is unchanged. These results are in Appendix A.6 of the revised paper and are briefly noted in Section 4.3 (Main Results).

The tables in the main text still report means only. The stds and the per-seed, per-task results of all methods and ablations are in Appendix A.5.

AuthorsOpenReview ↗

Response to Reviewer hzZs (3/8)

C. Inconsistencies in the candidate pool size, the token-budget sweep, and the packing length (W2, second paragraph; Q2)

Response. Thank you for checking these details carefully. They are errors or omissions in the writing; the experiments themselves are unchanged.

(1) Candidate corpus size. Each domain has about 10B tokens, about 30B in total across the three domains. The statement "about 10B in total across the three domains" in Appendix A.2 was a typo. The revised paper corrects it and adds per-domain statistics to Appendix A.2:

DomainDocumentsTokensShare of documents over 64K
CC156,93810.0B30.1%
arXiv290,29710.0B7.1%
Books65,22310.0B100%

(2) Token-budget sweep. Yes. Throughout the sweep, every setting other than the budget is held fixed, and only the number of CPT tokens (1B–5B) changes:

  • Model and data: Llama-3-8B-Base throughout, with no selector or scoring model, and a fixed long-to-short ratio of 8:2.
  • Training pipeline: the same 128K CPT+SFT pipeline throughout, with identical SFT settings for every checkpoint.
  • Evaluation: the same benchmarks and the same Ovr. aggregation as in the main experiments.

The sweep is a preliminary experiment used to fix the CPT token budget before the main experiments. Going from 1B to 2B improves Ovr. by 1.87, while going from 2B to 5B adds only 0.35 in total, so the main experiments use 2B. The sweep is not part of any comparison between methods in Table 1. The revised paper states these settings in Appendix A.3 (Token Budget, Table 9).

(3) Packing length. All training data are packed into 131,072-token windows with BFD (Appendix A.2). Each document is padded with EOS to a multiple of 8, so some sequences are slightly longer than 131,072. The values 135,168 (CPT) and 132,096 (SFT) in the hyper-parameter table are the maximum lengths set to accommodate these padding tokens. The revised paper states this consistently in Appendix A.2 and in the hyper-parameter table of Appendix A.4 (Table 10).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.