- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
AttentionSpan ranks training documents by how far back a model attends after correcting for positional attention sinks. It uses a filler prefix to absorb sink attention, excludes that prefix from scoring, and averages a weighted median of the top-k attention distances for each query. The authors test the resulting selections through LoRA and full-parameter training, mainly at 16K and 32K, with additional 64K evaluation and adaptation of a native-128K model. They also study domain balancing, layer selection, multiple scoring models, and self-selection. The clearest gains are on retrieval tasks; RAG and long-document QA show much less improvement.
Critical strengths
- The paper identifies a real problem with attention-distance scores: attention to early sink tokens can look like a long-range dependency. Appendix C supports the proposed correction with checks of residual sink mass, ranking stability, and attention over document tokens.
- The retrieval improvements deserve credit. In the 30K-document OLMo experiment, HELMET Recall rises from 62.0 with random selection to 75.7 with high-AttentionSpan data. Appendix E also reports a Llama-2 replication with fixed domain proportions and three training seeds, where the average Recall improvement remains 5.45 points.
Critical weaknesses
- I am not convinced that the comparisons separate the metric's benefit from the other selection choices. Table 4 gives the same Recall/RULER scores as Borda in Table 3, without clearly identifying whether AttentionSpan uses multiple scorers while the baselines use one. In Table 41, high-AttentionSpan selection scores 41.29 overall, compared with 42.05 for LongAttn; the improvement to 44.69 comes after domain balancing. That comparison needs the same balancing rule for both methods. There is also a mismatch in the OLMo domain proportions: Table 31 lists 481 government reports for the 30K high-AttentionSpan set but 1,921 for random selection. The fixed-domain Llama experiment is useful, but does not remove this confound from the OLMo result. LADM and LongAttn are relevant baselines, yet the paper also claims competitiveness with perplexity-based selection without a matched LongFilter or ProLong (Chen et al., 2024a) experiment that I could find. The evidence therefore does not yet establish how much the proposed metric adds over existing selectors.
- I see a modest methodological contribution whose practical benefit is still narrowly demonstrated. Attention-based selection and prefix-based handling of sinks have precedents; the new contribution is mainly their combination with the distance statistic. Appendix D motivates the weighted median, but I would like to see whether it improves downstream results over simpler summaries with the other settings fixed. Most substantial training experiments use 7B-8B targets at 16K-32K. The 64K evaluation and 128K adaptation are useful additions, although the latter tests only four Recall subtasks with 25 examples each. In Table 25, high-AttentionSpan reaches 97.8 against an unadapted baseline of 94.5, while random adaptation falls to 79.5. Much of the separation thus comes from avoiding degradation, and the small gain over the original model needs uncertainty estimates. The 31B model is a scorer, so this experiment does not test a larger target model. RAG and long-document QA gains remain limited. Scoring also retains quadratic time and memory costs; the layer-subsampling result at 8K does not establish its cost-benefit tradeoff at 128K. These results are promising for retrieval training, but I do not find the demonstrated advance sufficient for acceptance.
Questions for authors
- Does Table 4 use Borda for AttentionSpan? Please give the scorer and domain-balancing setup for each method, including whether LongAttn receives the same balancing in Table 41. Is there a matched experiment supporting the claim about perplexity-based selection?
- Can you explain the differing domain counts in Table 31 and provide a random control with the same quotas as high-AttentionSpan? Which OLMo comparisons have independent training repetitions, rather than only the evaluation seeds reported in Table 4?
- Do downstream ablations isolate the effects of the prefix, weighted median, and layer choice? For the 128K result, how reliable is the gain over the unadapted model, why does random adaptation degrade so much, and how expensive is scoring relative to training?
- Table 4 reports 23.5 for AttentionSpan on long-document QA, whereas Tables 38-40 give 24.1, 25.9, and 27.1, averaging to 25.7. Do these tables use different evaluation sets or aggregation rules?
12 comments
Borda and Domain Balance
Table 4 gives the same Recall/RULER scores as Borda in Table 3, without clearly identifying whether AttentionSpan uses multiple scorers while the baselines use one.
The identical Recall/RULER scores in Tables 3 and 4 are intentional, as Borda consensus is part of our default AttentionSpan method. Table 3 ablates grader choice, while Table 4 compares data-selection methods.
Crucially, AttentionSpan outperforms all competing baselines even without Borda consensus, across five individual grader configurations, including self-grading.
| Selection / Grader | Recall | RULER |
|---|---|---|
| AttentionSpan (Borda) | 78.4 [76.5, 80.3] | 66.7 [65.8, 67.7] |
| AttentionSpan (InternLM-2.5-1.8B) | 72.3 [70.3, 74.2] | 63.2 [62.2, 64.3] |
| AttentionSpan (Llama-3.2-3B) | 71.1 [68.9, 73.2] | 61.8 [60.8, 62.8] |
| AttentionSpan (Qwen-2.5-1.5B) | 68.0 [65.8, 70.1] | 63.4 [62.4, 64.4] |
| AttentionSpan (Qwen-2.5-Coder-7B-Instruct) | 65.1 [62.8, 67.5] | 63.8 [62.8, 64.8] |
| AttentionSpan (OLMo-3 self-grading) | 72.8 ± 0.5 | 62.9 |
| LADM | 56.1 [53.9, 58.4] | 61.0 [60.0, 62.0] |
| Random | 55.0 [52.8, 57.2] | 56.8 [55.7, 57.8] |
| LongAttn | 39.8 [37.9, 41.7] | 51.2 [50.2, 52.3] |
Even the weakest individual grader outperforms every competing baseline. Self-grading further demonstrates that these gains do not require an external grader. Borda improves performance further, showing that the advantage of AttentionSpan is consistent across grader choices and does not depend on consensus.
In Table 41, high-AttentionSpan selection scores 41.29 overall, compared with 42.05 for LongAttn; the improvement to 44.69 comes after domain balancing.
For a given domain/pool, the threshold for long and short is top vs bottom 25% of the pool. When we perform domain balancing, the high-AttentionSpan pool selects form the top only and likewise the low-AttentionSpan selects form the bottom 25%. As such domains with initial small selection pools will have a relatively smaller set for long and short. Random on the other hand will select from the entire pool and thus does not have the same constraints as long and short. In random’s case, domain balancing will try to get as much data from a domain as possible before backfilling using other domains. So, the domain balancing here makes sense.
Table 41 in appendix P.3 had an in the number of optimizer steps per for AttentionSpan-High and document balancing. We rerun for every pool, rule, recipe and evaluation for every selector. We have replaced table 41 with the below in appendix P.3.
| Selection | HELMET overall (mean of 5 draws, SD) | HELMET-Recall | AttentionSpan - this [95% CI] | p | Draws AttentionSpan ahead |
|---|---|---|---|---|---|
| AttentionSpan | 43.66 (0.48) | 51.30 | - | - | - |
| LongAttn | 42.60 (0.17) | 48.49 | +1.06 [+0.60, +1.53] | 0.003 | 5/5 |
| LADM | 42.68 (0.51) | 48.55 | +0.97 [-0.17, +2.12] | 0.077 | 4/5 |
| Random | 42.59 (0.55) | 48.87 | +1.06 [+0.20, +1.92] | 0.026 | 5/5 |
Table 31 lists 481 government reports for the 30K high-AttentionSpan set but 1,921 for random selection.
We rerun our evaluations for for both OLMo-3 and Llama-2-7B Long vs random is not domain-controlled, and we are specifying that in the paper now. It is important to point out that ~15pt swing is very large and that we have already shown that a near domain match with Llama-2 training runs shows that AttentionSpan win in Appendix E.for Llama-2-7B at 32k.
OLMo-3
| docs | Comparison | Domain similarity | HELMET-Recall at 32K | RULER (all 13 tasks) | MRCR |
|---|---|---|---|---|---|
| 30K | Long - Short | identical | +27.3 [+22.4, +32.2] | +9.3 [+7.1, +11.5] | +3.7 [+1.8, +5.5] |
| 30K | Long - Random | differ (10.5%) | +15.1 [+10.4, +19.8] | +3.4 [+1.3, +5.5] | +2.1 [+0.1, +4.1] |
| Setting | Comparison | Domain mix | Result [95% CI] |
|---|---|---|---|
| Llama-2-7B 16K, 5 draws | AttentionSpan - LongAttn | identical counts | +1.06 [+0.60, +1.53] HELMET overall, 5/5 draws |
| AttentionSpan - LADM | identical counts | +0.97 [-0.17, +2.12], 4/5 | |
| AttentionSpan - Random | within 2.0-3.6% | +1.06 [+0.20, +1.92], 5/5 | |
| Llama-2-7B 16K, 1 run | High - Low AttentionSpan | identical counts | +3.12 [+2.21, +4.02] overall (Recall +7.8) |
| Llama-2-7B 32K, 3 training seeds | High - Random | identical counts | +5.45 [+2.78, +8.12] Recall |
| OLMo-3-7B, 30K tier | Long - Short | identical counts | +27.3 [+22.4, +32.2] Recall |
| OLMo-3-7B, 10K tier | Long - Short | within 1.4% | +27.1 [+22.5, +31.8] Recall |
Additional Baselines
LADM and LongAttn are relevant baselines, yet the paper also claims competitiveness with perplexity-based selection without a matched LongFilter
Using training data selected from a diverse 14,300-document pool at 16k context length found llama2 trained with perplexity based selection methods like long filter to be 4.75pts below our the llama2 model trained with AttentionSpan selected data on an equal token budget selected from the same document pool.
| Method | Recall 16K | RAG 16K | LongQA 16K | All 16K |
|---|---|---|---|---|
| No sink correction | +3.06 [+0.25, +5.88]* | -0.46 [-1.67, +0.71] | +1.16 [-0.85, +3.23] | +1.26 [-0.01, +2.56] |
| LongFilter | +4.75 [+1.75, +7.75]* | -0.46 [-1.58, +0.62] | +0.34 [-1.52, +2.09] | +1.65 [+0.41, +2.90]* |
| LongAttn | +7.88 [+4.56, +11.19]* | +0.33 [-0.88, +1.58] | +1.05 [-0.70, +2.80] | +3.27 [+1.86, +4.66]* |
| Random | +8.38 [+5.19, +11.81]* | +0.29 [-0.83, +1.38] | +1.17 [-0.50, +2.88] | +3.47 [+2.12, +4.88]* |
| Bidirectional encoder | +9.81 [+6.62, +13.06]* | -0.17 [-1.29, +1.00] | +2.22 [+0.41, +4.17]* | +4.11 [+2.78, +5.51]* |
establish how much the proposed metric adds over existing selectors.
We have revised or added tables with matched pools, selection rules, recipes, ablations and evaluations to the paper that now shows how much our metric adds over other selectors. We believe this concern is addressed.
Exigence of Long Context Extension
Yes we agree; the long context extension is an important part of the modern LLM training stack. The work shows significant improvements in performance over existing baselines.