acceptodds
Under review as a conference paper at ICLR 2027

Measuring the Attention Span of Long Context Training Data

Abstract

LLMs are typically pretrained with short context lengths which are later extended through additional long-context training. As long context training is expensive, this stage requires careful data selection to maximize effectiveness. Data selection methods should distinguish a cohesive document with long dependencies across distant sections from a simple concatenation of parts with no real long distance structure. We propose to score documents based on their , which measures the look-back distance covered by the attention mechanism during a forward pass. Naive measurements of attention span are heavily influenced by attention sinks, making them a useless proxy for document structure. We present simple tricks for ignoring attention sinks, resulting in meaningful metrics of long-context structure that are useful for data filtering. When extending OLMo-3-7B from 8K to 32K using 5B tokens sampled from its own long-text mixture, additional AttentionSpan curation improves HELMET-Recall from to . The advantage remains even in a domain-matched experiment, in which context is extended using the same code, arXiv and book proportions for each checkpoint, and AttentionSpan is only used for filtering within domains. These results support sink-corrected attention distance as an interpretable curation signal for long-range retrieval training.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long Context, Data Filtering, Recursive Improving, Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_5UudOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

AttentionSpan ranks training documents by how far back a model attends after correcting for positional attention sinks. It uses a filler prefix to absorb sink attention, excludes that prefix from scoring, and averages a weighted median of the top-k attention distances for each query. The authors test the resulting selections through LoRA and full-parameter training, mainly at 16K and 32K, with additional 64K evaluation and adaptation of a native-128K model. They also study domain balancing, layer selection, multiple scoring models, and self-selection. The clearest gains are on retrieval tasks; RAG and long-document QA show much less improvement.

Critical strengths
  1. The paper identifies a real problem with attention-distance scores: attention to early sink tokens can look like a long-range dependency. Appendix C supports the proposed correction with checks of residual sink mass, ranking stability, and attention over document tokens.
  2. The retrieval improvements deserve credit. In the 30K-document OLMo experiment, HELMET Recall rises from 62.0 with random selection to 75.7 with high-AttentionSpan data. Appendix E also reports a Llama-2 replication with fixed domain proportions and three training seeds, where the average Recall improvement remains 5.45 points.
Critical weaknesses
  1. I am not convinced that the comparisons separate the metric's benefit from the other selection choices. Table 4 gives the same Recall/RULER scores as Borda in Table 3, without clearly identifying whether AttentionSpan uses multiple scorers while the baselines use one. In Table 41, high-AttentionSpan selection scores 41.29 overall, compared with 42.05 for LongAttn; the improvement to 44.69 comes after domain balancing. That comparison needs the same balancing rule for both methods. There is also a mismatch in the OLMo domain proportions: Table 31 lists 481 government reports for the 30K high-AttentionSpan set but 1,921 for random selection. The fixed-domain Llama experiment is useful, but does not remove this confound from the OLMo result. LADM and LongAttn are relevant baselines, yet the paper also claims competitiveness with perplexity-based selection without a matched LongFilter or ProLong (Chen et al., 2024a) experiment that I could find. The evidence therefore does not yet establish how much the proposed metric adds over existing selectors.
  2. I see a modest methodological contribution whose practical benefit is still narrowly demonstrated. Attention-based selection and prefix-based handling of sinks have precedents; the new contribution is mainly their combination with the distance statistic. Appendix D motivates the weighted median, but I would like to see whether it improves downstream results over simpler summaries with the other settings fixed. Most substantial training experiments use 7B-8B targets at 16K-32K. The 64K evaluation and 128K adaptation are useful additions, although the latter tests only four Recall subtasks with 25 examples each. In Table 25, high-AttentionSpan reaches 97.8 against an unadapted baseline of 94.5, while random adaptation falls to 79.5. Much of the separation thus comes from avoiding degradation, and the small gain over the original model needs uncertainty estimates. The 31B model is a scorer, so this experiment does not test a larger target model. RAG and long-document QA gains remain limited. Scoring also retains quadratic time and memory costs; the layer-subsampling result at 8K does not establish its cost-benefit tradeoff at 128K. These results are promising for retrieval training, but I do not find the demonstrated advance sufficient for acceptance.
Questions for authors
  1. Does Table 4 use Borda for AttentionSpan? Please give the scorer and domain-balancing setup for each method, including whether LongAttn receives the same balancing in Table 41. Is there a matched experiment supporting the claim about perplexity-based selection?
  2. Can you explain the differing domain counts in Table 31 and provide a random control with the same quotas as high-AttentionSpan? Which OLMo comparisons have independent training repetitions, rather than only the evaluation seeds reported in Table 4?
  3. Do downstream ablations isolate the effects of the prefix, weighted median, and layer choice? For the 128K result, how reliable is the gain over the unadapted model, why does random adaptation degrade so much, and how expensive is scoring relative to training?
  4. Table 4 reports 23.5 for AttentionSpan on long-document QA, whereas Tables 38-40 give 24.1, 25.9, and 27.1, averaging to 25.7. Do these tables use different evaluation sets or aggregation rules?
12 comments
AuthorsOpenReview ↗

Exigence of Long Context Extension

The paper identifies a real problem with attention-distance scores: attention to early sink tokens can look like a long-range dependency. Appendix C supports the proposed correction with checks of residual sink mass, ranking stability, and attention over document tokens.

The retrieval improvements deserve credit. In the 30K-document OLMo experiment, HELMET Recall rises from 62.0 with random selection to 75.7 with high-AttentionSpan data. Appendix E also reports a Llama-2 replication with fixed domain proportions and three training seeds, where the average Recall improvement remains 5.45 points.

Yes we agree; the long context extension is an important part of the modern LLM training stack. The work shows significant improvements in performance over existing baselines.

AuthorsOpenReview ↗

Borda and Domain Balance

Table 4 gives the same Recall/RULER scores as Borda in Table 3, without clearly identifying whether AttentionSpan uses multiple scorers while the baselines use one.

The identical Recall/RULER scores in Tables 3 and 4 are intentional, as Borda consensus is part of our default AttentionSpan method. Table 3 ablates grader choice, while Table 4 compares data-selection methods.

Crucially, AttentionSpan outperforms all competing baselines even without Borda consensus, across five individual grader configurations, including self-grading.

Selection / GraderRecallRULER
AttentionSpan (Borda)78.4 [76.5, 80.3]66.7 [65.8, 67.7]
AttentionSpan (InternLM-2.5-1.8B)72.3 [70.3, 74.2]63.2 [62.2, 64.3]
AttentionSpan (Llama-3.2-3B)71.1 [68.9, 73.2]61.8 [60.8, 62.8]
AttentionSpan (Qwen-2.5-1.5B)68.0 [65.8, 70.1]63.4 [62.4, 64.4]
AttentionSpan (Qwen-2.5-Coder-7B-Instruct)65.1 [62.8, 67.5]63.8 [62.8, 64.8]
AttentionSpan (OLMo-3 self-grading)72.8 ± 0.562.9
LADM56.1 [53.9, 58.4]61.0 [60.0, 62.0]
Random55.0 [52.8, 57.2]56.8 [55.7, 57.8]
LongAttn39.8 [37.9, 41.7]51.2 [50.2, 52.3]

Even the weakest individual grader outperforms every competing baseline. Self-grading further demonstrates that these gains do not require an external grader. Borda improves performance further, showing that the advantage of AttentionSpan is consistent across grader choices and does not depend on consensus.

In Table 41, high-AttentionSpan selection scores 41.29 overall, compared with 42.05 for LongAttn; the improvement to 44.69 comes after domain balancing.

For a given domain/pool, the threshold for long and short is top vs bottom 25% of the pool. When we perform domain balancing, the high-AttentionSpan pool selects form the top only and likewise the low-AttentionSpan selects form the bottom 25%. As such domains with initial small selection pools will have a relatively smaller set for long and short. Random on the other hand will select from the entire pool and thus does not have the same constraints as long and short. In random’s case, domain balancing will try to get as much data from a domain as possible before backfilling using other domains. So, the domain balancing here makes sense.

Table 41 in appendix P.3 had an in the number of optimizer steps per for AttentionSpan-High and document balancing. We rerun for every pool, rule, recipe and evaluation for every selector. We have replaced table 41 with the below in appendix P.3.

SelectionHELMET overall (mean of 5 draws, SD)HELMET-RecallAttentionSpan - this [95% CI]pDraws AttentionSpan ahead
AttentionSpan43.66 (0.48)51.30---
LongAttn42.60 (0.17)48.49+1.06 [+0.60, +1.53]0.0035/5
LADM42.68 (0.51)48.55+0.97 [-0.17, +2.12]0.0774/5
Random42.59 (0.55)48.87+1.06 [+0.20, +1.92]0.0265/5

Table 31 lists 481 government reports for the 30K high-AttentionSpan set but 1,921 for random selection.

We rerun our evaluations for for both OLMo-3 and Llama-2-7B Long vs random is not domain-controlled, and we are specifying that in the paper now. It is important to point out that ~15pt swing is very large and that we have already shown that a near domain match with Llama-2 training runs shows that AttentionSpan win in Appendix E.for Llama-2-7B at 32k.

OLMo-3

docsComparisonDomain similarityHELMET-Recall at 32KRULER (all 13 tasks)MRCR
30KLong - Shortidentical+27.3 [+22.4, +32.2]+9.3 [+7.1, +11.5]+3.7 [+1.8, +5.5]
30KLong - Randomdiffer (10.5%)+15.1 [+10.4, +19.8]+3.4 [+1.3, +5.5]+2.1 [+0.1, +4.1]
SettingComparisonDomain mixResult [95% CI]
Llama-2-7B 16K, 5 drawsAttentionSpan - LongAttnidentical counts+1.06 [+0.60, +1.53] HELMET overall, 5/5 draws
AttentionSpan - LADMidentical counts+0.97 [-0.17, +2.12], 4/5
AttentionSpan - Randomwithin 2.0-3.6%+1.06 [+0.20, +1.92], 5/5
Llama-2-7B 16K, 1 runHigh - Low AttentionSpanidentical counts+3.12 [+2.21, +4.02] overall (Recall +7.8)
Llama-2-7B 32K, 3 training seedsHigh - Randomidentical counts+5.45 [+2.78, +8.12] Recall
OLMo-3-7B, 30K tierLong - Shortidentical counts+27.3 [+22.4, +32.2] Recall
OLMo-3-7B, 10K tierLong - Shortwithin 1.4%+27.1 [+22.5, +31.8] Recall
AuthorsOpenReview ↗

Additional Baselines

LADM and LongAttn are relevant baselines, yet the paper also claims competitiveness with perplexity-based selection without a matched LongFilter

Using training data selected from a diverse 14,300-document pool at 16k context length found llama2 trained with perplexity based selection methods like long filter to be 4.75pts below our the llama2 model trained with AttentionSpan selected data on an equal token budget selected from the same document pool.

MethodRecall 16KRAG 16KLongQA 16KAll 16K
No sink correction+3.06 [+0.25, +5.88]*-0.46 [-1.67, +0.71]+1.16 [-0.85, +3.23]+1.26 [-0.01, +2.56]
LongFilter+4.75 [+1.75, +7.75]*-0.46 [-1.58, +0.62]+0.34 [-1.52, +2.09]+1.65 [+0.41, +2.90]*
LongAttn+7.88 [+4.56, +11.19]*+0.33 [-0.88, +1.58]+1.05 [-0.70, +2.80]+3.27 [+1.86, +4.66]*
Random+8.38 [+5.19, +11.81]*+0.29 [-0.83, +1.38]+1.17 [-0.50, +2.88]+3.47 [+2.12, +4.88]*
Bidirectional encoder+9.81 [+6.62, +13.06]*-0.17 [-1.29, +1.00]+2.22 [+0.41, +4.17]*+4.11 [+2.78, +5.51]*

establish how much the proposed metric adds over existing selectors.

We have revised or added tables with matched pools, selection rules, recipes, ablations and evaluations to the paper that now shows how much our metric adds over other selectors. We believe this concern is addressed.

Reviewer_Wa2LOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper focuses on improving the quality of long-context extension data. In current CPT/mid-training pipelines, long-context data tends to be many short documents concatenated together. However, this doesn't always lead to good long-context performance, as models have no incentive to look at information beyond specific documents (e.g. information at long-ranges). This paper proposes a metric, AttentionSpan, which measures the distance between a query and relevant keys. This information is then used to filter documents for long-context extension training. Because it's possible to game this metric with attention sinks, this work also proposes a number of mitigation schemes, such as using a weighted median. The metric when used for sample selection leads to mostly better performance over prior work.

Critical strengths
  1. I think the problem addressed is important and the offered solution is clearly better than alternatives.
  2. The solution is essentially long-context distillation: it needs models with strong long-context capabilities to give it good attentionspan scores, for better or for worse
  3. Very robust experimentation and theoretical analysis for the metric (mostly in the appendix)
Critical weaknesses
  1. I think the fact that it is fundamentally a distillation-based metric is also a weakness, as it limits selection of data to what the larger model thinks is useful for long context capabilities. However, I don't think there's really a better automated solution to this, as heuristics or algorithms are probably a weaker filter than attention.
  2. The main issue I have with the eval/ablations is the limited eval benchmarks. What about other standard long-context evals like MRCR and Graphwalks? i.e. evals that directly test the quality of the long-context data
Questions for authors
  1. What about using encoder models or bidirectional attention to look at relationships throughout the document? Or is only looking back important, as the data is used primarily for training causal models?
6 comments
AuthorsOpenReview ↗

Long Context Extension Exigence

I think the problem addressed is important and the offered solution is clearly better than alternatives.

Very robust experimentation and theoretical analysis for the metric (mostly in the appendix)

Thank you; we agree with these strengths of our approach to long-context extensions, which are an integral part of LLM training.

AuthorsOpenReview ↗

Improvements over Distillation

I think the fact that it is fundamentally a distillation-based metric is also a weakness, as it limits selection of data to what the larger model thinks is useful for long context capabilities. However, I don't think there's really a better automated solution to this, as heuristics or algorithms are probably a weaker filter than attention.

We agree with the reviewer that a fundamental limitation of distillation-based methods is their dependence on the choice of teacher model and its inductive biases. We explicitly recognize this limitation and investigate three complementary approaches to mitigate it, primarily in Section 4.3.

First, we show how domain balancing can help mitigate such biases. In this setup, we ensure that the grader selects documents from different domains of the pretraining mixture, rather than allowing it to favor one domain. This helps address biases such as code-specialized models preferring code documents over web or Wikipedia data. Our experiments show that domain-balanced selection can improve downstream performance by reducing the influence of such preferences.

Second, we propose consensus-based selection using Borda aggregation across three different graders (Llama-3.2-3B, Qwen-2.5-1.5B, and Qwen-2.5-Coder-7B-Instruct). This substantially improves downstream performance compared to using any individual grader. Additionally, in Appendix Q.1, we analyze agreement between graders and show that despite their different inductive biases, they often agree on document rankings. By combining their rankings, we can reduce the influence of individual grader preferences. Notably, two of these graders are smaller than the target OLMo-3-7B model.

Third, we investigate self-distillation, where the target model itself serves as the grader. Using the OLMo-3-7B base model extended to 32K, we demonstrate substantial improvements over random selection. This shows that AttentionSpan does not require a larger teacher model or a teacher with a longer context window than the target model.

Together, these experiments show that while the choice of teacher model introduces inherent limitations, we can mitigate some of these limitations through domain balancing and consensus-based selection. Furthermore, self-distillation demonstrates that a separate, larger teacher is not necessary. We discuss these findings in Section 4.3 and Appendices Q and R.

AuthorsOpenReview ↗

Additional Benchmarks

The main issue I have with the eval/ablations is the limited eval benchmarks. What about other standard long-context evals like MRCR and Graphwalks? i.e. evals that directly test the quality of the long-context data

We thank the reviewer for suggesting additional long-context benchmarks. We have added a discussion of MRCR and Graphwalks to the paper Both benchmarks have primarily been evaluated on instruction-tuned or post-trained models, whereas our experiments focus on models in the pretraining/midtraining stage. This distinction is important because these benchmarks evaluate instruction-following and reasoning capabilities in addition to long-context understanding. To quote from OpenAI release blog of the datasets: “But the task remains hard—even for advanced reasoning models…..” (https://openai.com/index/gpt-4-1/).

MRCR evaluation. MRCR requires precise instruction following, including prepending a specified alphanumeric hash to the beginning of the response. Not including this at the beginning scores the entire output as 0. Consequently, the standard evaluation can penalize models for failing to follow the required output format even when they retrieve the correct answer. We therefore also relax the hash requirement and evaluate the retrieved answer directly. We evaluate 893 prompts across three context-length bins (4K–8K, 8K–16K, and 16K–32K) with 2, 4, and 8 needles.

OLMo-3-7B, 30K-documentsMRCR (hash required)MRCR Relaxed (hash not required)
Long - Short+3.9 [+2.1, +5.7]+5.0 [+3.2, +6.9]
Long - Random+2.1 [+0.1, +4.2]+3.4 [+1.3, +5.2]

AttentionSpan-selected data outperforms both random and short-span selection, with positive confidence intervals for both comparisons. These results provide additional evidence of improved long-context capabilities without instruction-following fine-tuning under these relaxed and fairer conditions.

Graphwalks evaluation. Graphwalks is even harder than MRCR. In particular, its BFS tasks require following structured instructions, maintaining intermediate graph states, and performing multiple sequential reasoning steps; an ability only gained after specific reasoning training. These stages themselves are 10-100B token training runs in themselves that we will save for post training focused application of AttentionSpan as part of future work. As such performance on Graphwalk would not be a proper reflection of the gains achieved via AttentionSpan.

Regardless, we have included the modified MRCR protocol and results, along with the limitations of evaluating Graphwalks on Olmo-3, in the our final paper draft.

Reviewer_t4mdOpenReview ↗
Rating
4
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes a simple idea for filtering for long-context extension data: median lookback distance (dubbed 'AttentionSpan'). This is essential because some documents are long but don't actual require modelling of long-range dependencies. The paper extends context on some models (e.g. Llama 2) and shows that their method improves results. Also certain domains are identified (e.g. code) as more likely to have long-range dependencies, and thus to avoid over-training for certain domains, it is necessary to use AttentionSpan alongside other decision metrics.

Critical strengths

The paper shows very clear benefits of using AttentionSpan - it outperforms other methods for long-context extension convincingly across several base models. Data mixtures are also investigated thoroughly, and this highlights the caveat that AttentionSpan should not be used exclusively to select data for long-context extension.

Another strength is that this solves a real practical problem - training on long documents is more expensive than shorter documents, so improving the data pipeline is a valid research direction.

Critical weaknesses

There could be more motivation for removing the attention sinks (for example in the main text, we are missing a figure to show that including attention sinks when calculating AttentionSpan is bad). In general, the AttentionSpan metric is quite an obvious thing to do, so most of the strength of the paper lies in the experiments.

Questions for authors
  • Can you explain Borda in the main text?
  • Do you know why Base+YaRN works so well in Table 4?
  • Did you think about data augmentation to improve documents? One basic example is constructing documents like '[A][B][C]', where C depends on A, but B is a block of unrelated text
  • Not entirely related, but it would be interesting to consider AttentionSpan metrics on chain of thought / reasoning data. Could AttentionSpan be used to improve post-training?
7 comments
AuthorsOpenReview ↗

Exigence of Long Context Extensions

The paper shows very clear benefits of using AttentionSpan - it outperforms other methods for long-context extension convincingly across several base models. Data mixtures are also investigated thoroughly, and this highlights the caveat that AttentionSpan should not be used exclusively to select data for long-context extension.

Another strength is that this solves a real practical problem - training on long documents is more expensive than shorter documents, so improving the data pipeline is a valid research direction.

We agree, long context extensions are an important step in modern model training and we test many methods for the best recipe.

AuthorsOpenReview ↗

Motivation for Attention Sinks

There could be more motivation for removing the attention sinks (for example in the main text, we are missing a figure to show that including attention sinks when calculating AttentionSpan is bad).

Thank you for this suggestion and we agree with this point. We originally didn’t include the ablation because we found early on that the attention sinks were throwing AttentionSpan off by so much. For this review we investigated the effect of not correcting for attention sinks while calculating AttentionSpan and we will include this evidence in the main paper in order to better motivate the choice in the camera-ready version.

Attention sinks can substantially inflate the measured AttentionSpan. For example, tokens near the beginning of a document often attract disproportionate attention regardless of their relevance to later tokens. This behavior has been documented in prior work. Xiao et al. [1] demonstrate that attention sinks do not always carry meaningful semantic information. We also evaluated this qualitatively in our dataset with attention sinks landing on space or newlines or other seemingly unimportant tokens. In our dataset, a typical sink token is ~1,400 times larger than the median token in terms of attention mass.

Since these sink tokens can receive a large fraction of a query's attention mass while being far from later queries, their inclusion can artificially inflate AttentionSpan, making it difficult to distinguish documents with genuine long-range attention patterns from those whose scores are inflated by attention to early sink tokens.

Below is an example with two context lengths and on fineweb(web) and The Stack (code) comparing AttentionSpan with and without sink correction. We use Qwen-2.5-Coder-7B-Instruct as our scoring model.

Context lengthDomainDocumentsWithout correctionWith correctionInflation
8KCode4413,1249923.1×
8KWeb5001,3483094.4×
4KCode4411,5444973.1×
4KWeb5006381653.9×

Including attention sinks inflates the measured attention distance by more than 3 times across both domains and context lengths. That is a significant shift of our score and we see it in the chosen documents.

The sink correction improves downstream performance. The below evaluation is with trained llama-2 models.

Selection methodRecall 16KRAG 16KLongQA 16KOverall 16K
AttentionSpan43.5648.2927.8541.00
AttentionSpan without sink correction40.5048.7526.7039.74

This provides downstream evidence that correcting attention sinks improves the effectiveness of data selection, beyond simply changing the measured attention distances.

We note that the initial inflation analysis was conducted using 256 prefill tokens, as part of our early experiments. We subsequently reduced the prefill length to 64 tokens for computational efficiency. Our prefill-length stability analysis in the appendix shows that 64 tokens are sufficient for stable AttentionSpan rankings in the evaluated settings, supporting the use of the shorter prefill configuration.

We will include both tables and the corresponding discussion in the final manuscript to provide clearer empirical motivation for attention-sink correction.

References

[1] Guangxuan Xiao, Yuandong Tian, Beidi Chen, Song Han, and Mike Lewis. "Efficient Streaming Language Models with Attention Sinks." International Conference on Learning Representations (2024). https://arxiv.org/abs/2309.17453

AuthorsOpenReview ↗

Borda and Base+YaRN explanations

Can you explain Borda in the main text?

Yes, we thank the reviewer for this suggestion. We agree that the explanation of relevant methodological details pertaining to Borda consensus should be moved from the supplementary discussion to the main text for the camera-ready.

Do you know why Base+YaRN works so well in Table 4?

The strong Long-doc QA performance of Base + YaRN can be partly attributed to capabilities acquired during OLMo-3's midtraining stage. Approximately 5% of its midtraining data consists of instruction-style examples including QA, reasoning traces etc. [1]. Since Base + YaRN extends the context window without additional training, it retains these previously learned capabilities.

The lower Long-doc QA performance after long-context training may partly result from the degradation of previously learned capabilities during continued pretraining.. Our long-context extension from the midtraining checkpoint uses data more similar to conventional pretraining like Code, Web, Wikipedia etc, which contain little to no instruction-following or reasoning-oriented post-training. Jindal et al. [2] demonstrate that continued pretraining can degrade previously acquired instruction-following capabilities. In their experiments, Llama-3-Instruct loses nearly 10 points on the average of instruction-following tasks after continued pretraining on 100M–1B tokens. They further show that these capabilities can be recovered through instruction alignment after continued pretraining. This suggests that the Long-doc QA gap relative to Base + YaRN reflects the training stage rather than a limitation of AttentionSpan itself.

Our experiments intentionally focus on long-context continued pretraining to isolate the effect of data selection. In a complete model development pipeline, this stage would be followed by instruction-following SFT and reasoning-oriented post-training, which would help recover previously learned capabilities while retaining improvements in long-context utilization. Evaluating AttentionSpan after these stages is an important direction for future work.

References

[1] Olmo Team et al. "Olmo 3." arXiv preprint arXiv:2512.13961 (2025).

[2] Ishan Jindal, Badrinath Chandana, Pranjal Bharti, Lakkidi Vinay, and Sachin Dev Sharma. "Keep the Alignment, Skip the Overhead: Lightweight Instruction Alignment for Continually Trained LLMs." ICML 2025 Workshop on Test-Time Adaptation: Putting Updates to the Test! (2025). https://openreview.net/forum?id=qE0lPCBkD4

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.