acceptodds
Under review as a conference paper at ICLR 2027

Understanding and Accelerating Long-Context Extrapolation in Hybrid Models via State-Equilibration Curriculum

Abstract

Long-context adaptation of hybrid state-space model (SSM)–Transformer language models remains poorly understood: while multi-stage context-length training is effective in practice, it is unclear how long a model should remain at each intermediate context length before advancing to a longer sequence length. We find that long-context adaptation exhibits a two-phase behavior. When first exposed to a longer context, recurrent state dynamics rapidly stabilize: distance-dependent state drift sharply decreases and then plateaus, which we call state equilibration. Continued training beyond equilibration can still improve performance at the current context length, but these gains increasingly specialize to that length and transfer poorly to longer sequences. Based on this observation, we introduce State-Equilibration Curriculum (SEC), which advances to the next context length once recurrent-state dynamics stabilize. To support SEC efficiently, we develop a context-parallel implementation for hybrid SSM–Transformer models that removes serial cross-device state propagation while exposing chunk-boundary states as a low-overhead training signal. On Nemotron-H-8B and Bamba-9B-v2, SEC matches or improves fully trained multi-stage schedules while using about 30% fewer training tokens and reducing wall-clock training time by up to 42%. Our results suggest that intermediate context stages need not be trained to task convergence: advancing once state dynamics equilibrate provides a more efficient mechanism for multi-stage long-context adaptation.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Long-context adaptation, hybrid SSM-Transformer models, curriculum learning

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_JE73OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle a very practical unknown - we do not know how long to remain in a long context extension phase for each given extension. The authors observe that "recurrent state similarity" stabilizes but performance improves as they train a long-context phase for longer. The authors propose SEC to decide when to switch to the next long-context phase (longer sequence length). SEC simply compares a state similarity metric against a threshold. The authors also have a context parallel implementation for Mamba2.

Overall, I think the claims made are interesting and have some practical consequences. However, I don't think the results are game changing or trivially apply in all scenarios - authors do claim that this works in all recurrent language models, but I believe results are limited to Mamba based hybrid transformer architectures only. The authors provide state similary metric for GDN and pure Mamba2, but I do not see curriculum results for either of those.

Thus, the paper is a weak accept for me - it would be stronger if the authors: (1) showed that the curriculum applies for other architectures too and (2) helped explain why their metric is a good metric intuitively/mathematically.

Critical strengths

Interesting observation: The observation that recurrent-state dynamics stabilize much earlier than downstream performance is interesting. Figure 1 provides fairly convincing evidence that this happens during the authors' training setup.

Practical improvements: The reported token savings are substantial. SEC reduces the training budget by 30%

Lastly, the context-parallel implementation seems practical and removes serial dependencies across devices. This could be valuable independently of the curriculum.

Critical weaknesses

Unclear justification for the state-equilibration metric: The authors use low variance in state similarity across positional distances as an indicator of readiness for longer-context training. However, it's unclear why this should correlate with useful long-context capabilities. A model that rapidly forgets distant tokens could also exhibit low variance. Additionally, the metric doesn't directly measure stabilization across training checkpoints, which is how the authors define equilibration.

Insufficient evidence for the proposed curriculum: The experiments show that shorter intermediate training stages work well, but don't establish that state equilibration is a useful stopping criterion. In particular, there's no comparison against fixed schedules with similarly short intermediate stages. It's unclear whether SEC provides any benefit over simply advancing after 0.15B or 0.25B tokens.

Limited generalization and weak evidence for context-length specialization: The curriculum is only evaluated on two Mamba-based hybrid models using a single 32K-to-128K transition. The authors suggest that the underlying phenomenon generalizes to other recurrent architectures, but don't validate SEC on them. Furthermore, Figure 3 shows near-zero performance at 40K throughout training, making it difficult to conclude that additional 32K training specifically fails to transfer to longer contexts.

Questions for authors

How much does the adaptive curriculum actually help? Could the authors compare SEC against fixed training schedules with similarly short intermediate stages (e.g., 0.15B or 0.25B tokens)? It would also be useful to demonstrate settings where equilibration occurs at different rates and SEC adapts accordingly.

3 comments

We thank the reviewer for recognizing the importance of efficient long-context adaptation, the observed early stabilization of recurrent-state dynamics, and our context-parallel implementation.

W1: Unclear justification for the state-equilibration metric and its relationship to stabilization across training checkpoints.

A: We thank the reviewer for this question. Our motivation is that, in a well-adapted hybrid SSM–Transformer model, individual recurrent heads should exhibit consistent retention behavior across positional distances. When a model is first exposed to longer contexts, its recurrent-state similarity profiles can exhibit substantial distance-dependent variation, potentially reflecting out-of-distribution state drift or unstable recurrent dynamics. As training progresses, this variation decreases, indicating that the recurrent-state behavior is becoming more stable across the extended context. The reviewer correctly observes that V(t)V(t) measures variance across positional distances instead of changes across training checkpoints. The distinction is that SEC tracks the evolution of V(t)V(t) throughout training, instead of interpreting a single measurement in isolation. As shown in Figure 3, V(t)V(t) initially decreases rapidly before reaching a plateau. This indicates that the measured state dynamics have entered a stable regime, which we refer to as state equilibration. Our key empirical observation is that this equilibration occurs substantially earlier than downstream task convergence. Further training at the intermediate context length provides diminishing benefits for subsequent context extension. SEC therefore uses the stabilization of V(t)V(t) as a practical signal to advance to the next context length. We will clarify the distinction between the distance-dependent variance measured at each checkpoint and its stabilization over training in the revised paper.

W2/Q1: Does SEC provide benefits over fixed short schedules of 0.15B or 0.25B tokens? Can it adapt to different equilibration rates?

We agree that distinguishing adaptive scheduling from simply shortening intermediate-stage training is important.

The key challenge is determining how long to train at an intermediate context length when adapting a new model or training configuration. A fixed schedule requires this budget to be selected in advance. Advancing too early may leave the recurrent-state dynamics insufficiently adapted, while continuing well beyond equilibration wastes computation. SEC instead monitors V(t)V(t) and advances when the recurrent-state behavior reaches a stable regime. Our experiments show that SEC-Early and SEC-Default advance after approximately 0.15B and 0.25B tokens, respectively, yet achieve similar final performance. This indicates that there is a range of effective transition points once state equilibration has occurred. A fixed schedule using the same transition points as SEC would naturally achieve equivalent results under identical training conditions. However, this assumes that the appropriate transition budgets are already known. SEC provides a model-internal criterion to identify these transition points during training, without requiring a predetermined token budget.

To further evaluate the stopping criterion, we are conducting an additional experiment that advances after only 0.05B tokens at 32K, before the observed state-equilibration point, followed by the same 128K adaptation procedure. This experiment will test whether advancing prematurely degrades final long-context performance. Unfortunately, we have encountered GPU cluster maintenance while running the long running training experiment. We will provide the results in a follow-up response once available.

W3: Limited generalization beyond Mamba-based hybrid models and insufficient evidence for context-length specialization.

A: Our main experiments evaluate SEC on two Mamba-based hybrid SSM–Transformer architectures, Nemotron-H-8B and Bamba-9B-v2. In Table 4, we additionally examine state-drift behavior in a hybrid Gated-DeltaNet Transformer model, providing evidence that similar state-dynamics phenomena can occur beyond Mamba-based architectures. We also conducted preliminary experiments on a 1B hybrid Gated-DeltaNet Transformer (GDN-H) checkpoint pretrained on 4K contexts for 100B tokens. We observed a similar recovery of state drift during long-context adaptation, but it fails to predict the advancing point for the stage, as the intermediate context length behavior is ~5 points off on average compared to training the full 3B tokens. These results suggest that state-drift recovery is not unique to Mamba-based models, but the relationship between state equilibration and optimal curriculum transitions may depend on the recurrent architecture. We will clarify that our demonstrated SEC efficiency gains apply to the evaluated Mamba-based hybrid models, while extending the stopping criterion to other recurrent architectures requires further investigation.

Figure 3 shows that, even after 3B tokens of continued training at 32K, performance on the 40K evaluation remains near zero. In contrast, the model acquires longer-context capability after training for only 0.15B tokens at 32K followed by 6B tokens at 128K. This comparison demonstrates that additional training at 32K does not necessarily transfer to longer contexts, whereas training at the extended context length enables the model to develop the missing capability. This supports SEC's central motivation: once recurrent-state dynamics have equilibrated, further training at the intermediate context length provides diminishing benefits for subsequent context extension, making earlier advancement more efficient.

AuthorsOpenReview ↗

Follow-up response to Reviewer JE73

As the rebuttal period draws to a close, we thank the reviewer again for their thoughtful comments, which helped us clarify the scope of our claims and identify an important comparison for SEC’s training schedule. Despite an unexpected GPU cluster outage prevented us from completing that comparison before the deadline, we would still like to further explain two points below.

W2/Q1. We agree that a fixed schedule with fewer intermediate-stage steps than SEC’s aggressive adaptive schedule would help clarify how much training the intermediate stage requires. GPU cluster maintenance may prevent us from completing this experiment before the rebuttal deadline. We apologize for the delay and will include the comparison in the camera-ready version.

Our one-stage baseline tests the limiting case of zero intermediate-stage steps. Training directly at the target length yielded weaker performance than SEC, indicating that intermediate-stage training was beneficial in our experiments. SEC uses a threshold on (V(t)) as an empirical criterion for state equilibration and advances to the next context length when (V(t)) falls at or below that threshold. The pending comparison asks whether a shorter, positive intermediate-stage budget can match the performance of SEC’s aggressive schedule.

W3. In Appendix C, we now report the weaker results on the GDN hybrid and limit our claim about SEC’s effectiveness to the Mamba2-based hybrid SSM–Transformer models evaluated in this work. One possible explanation concerns what (V(t)) measures: the variance in mean state cosine similarity across positional offsets. Each Mamba2 head applies a scalar decay to its previous state, while a GDN head can also selectively overwrite a stored association when a new key aligns with its key. Such an update can change what a later query retrieves while having little effect on the cosine similarity of the full state. A low (V(t)) may therefore be a less reliable guide to stage transitions in the GDN hybrid.

Reviewer_KPBWOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates long-context retraining of hybrid SSM-Transformer models. The authors characterize the cyclic states of chunk boundaries using state cosine similarity profiles and distance impact statistics, discovering a two-stage phenomenon in retraining: state equilibration and context-length specialization. Based on this, they propose SEC, saving approximately 30% of training tokens while maintaining sufficient performance.

Critical strengths
  1. The experiment is complete and relatively thorough. The observations of the two-stage training are comprehensive, and a propulsion signal that can be acquired online is derived.

  2. The theory study is thorough. The expansion of the affine dependence of the incident state proves that the chunk transfer summary can be independently computed. And the figures and tables are well presented.

Critical weaknesses
  1. Some critical ablation needed to be supplemented or explained. The average difference between SEC-Early and the finely selected SEC-Default is small across all models and lengths, so an accurate V(t) threshold is not necessary. Clearify the reason why using V(t) instead of a cheaper signal would enhance the paper.

  2. There are some critical typos. For example what I noticed: line 228 "Appendix ??". Appendix A.3 also have a missing "Figure ??". And figure 5 would be better if it is the same pdf form as other figures.

Questions for authors

The questions are the same as the weekness, thanks.

2 comments

We thank the reviewer for the positive assessment of our work and for the constructive questions.

W1: Why use V(t)V(t) instead of a cheaper signal, given the similar performance of SEC-Early and SEC-Default?

A: We thank the reviewer for this question. The similar performance of SEC-Early and SEC-Default suggests that precise threshold selection is unnecessary once state equilibration has occurred. However, this does not eliminate the need for a criterion to determine when equilibration occurs. For a new model or training configuration, it is unclear how to select an effective short intermediate-stage training budget in advance. Advancing too early may leave the recurrent-state dynamics insufficiently adapted, while continuing long after equilibration wastes computation. V(t)V(t) provides a model internal signal for identifying this transition instead of relying on a predefined token budget. As shown in Figure 3, V(t)V(t) decreases rapidly during early adaptation before reaching a stable regime, providing an observable indication of when to advance. We also considered cheaper alternatives, including forget/input gate statistics and fixed training-step counts. However, these alternatives did not provide a similarly clear and consistent signal of state equilibration in our preliminary analysis. Among signals that characterize recurrent states directly, V(t)V(t) is a straightforward and lightweight choice. Moreover, our context-parallel implementation already materializes intermediate recurrent states for cross-device communication. Computing V(t)V(t) reuses these states, introducing little additional overhead compared with context-parallel training without the metric. We will clarify the motivation for choosing V(t)V(t), its computational cost, and its distinction from alternative signals in the revised paper.

W2: Broken references and inconsistent figure formatting.

A: We apologize for the broken references and the inconsistency of figure forms and have updated them in the rebuttal revision.

As the rebuttal period draws to a close, we thank you again for your careful reading and thoughtful feedback. We hope our responses and revisions have addressed your concerns, and we appreciate the time you have devoted to reviewing our work.

Reviewer_AaUuOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

The paper identifies a 2-phase pattern in long-context adaptation of hybrid SSM–Transformer models and proposes a state-equilibration-based training curriculum with context parallelism. It achieves comparable or better performance with ~30% fewer training tokens and 42% less training time.

Critical strengths
  1. The paper identifies a two-phase behavior in hybrid SSM–Transformer models. An initial state equilibration phase, in which recurrent-state dynamics stabilize rapidly, followed by context-length specialization, in which additional training improves performance at the current length but transfers poorly to longer contexts.

  2. The proposed State-Equilibration Curriculum (SEC) is an appealing approach because it replaces a heuristic scheduling decision with a model-internal signal.

  3. The efficiency improvements are strong. SEC reduces training-token consumption by 30% and wall-clock training time by 42%. It matches the fully trained multi-stage baseline.

  4. The authors extensively evaluate and validate SEC on two distinct architectures - Nemotron-H-8B and Bamba-9B-v2. The evaluation includes RULER at 32K, 64K, and 128K, InfiniteBench, short-context language modeling and reasoning tasks.

Critical weaknesses
  1. The central hypothesis rests on the observation that recurrent-state cosine similarity becomes approximately constant across positional offsets. But, it does not does not necessarily imply that the model has learned to preserve, retrieve, or utilize long-range information. Authors do not establish why a small V(t) is a reliable indicator of readiness for further context extension.

  2. SEC uses a threshold of tau=5x10^-4 to determine the transition point, while SEC-Early uses 10^-3. A central selling point of SEC is adaptive scheduling. If thresholds must be calibrated separately for each training setting, the practical advantage over fixed schedules becomes less compelling.

Questions for authors
  1. The central hypothesis rests on the observation that recurrent-state cosine similarity becomes approximately constant across positional offsets. But, it does not does not necessarily imply that the model has learned to preserve, retrieve, or utilize long-range information. Authors do not establish why a small V(t) is a reliable indicator for further context extension. Can the authors provide analysis/theory on the reliabelity of V(t) as a reliable indicator for further context extension?

  2. SEC uses a threshold of tau=5x10^-4 to determine the transition point, while SEC-Early uses 10^-3. A central selling point of SEC is adaptive scheduling. If thresholds must be calibrated separately for each training setting, the practical advantage over fixed schedules becomes less compelling. Can the authors explain a mechanism for threshold selection or is it entirely experimental?

2 comments

We thank the reviewer for the positive assessment of our work and for the constructive questions.

W1/Q1: Why is V(t)V(t) a reliable indicator for further context extension?

A: Our motivation comes from how state-space models retain and retrieve information through recurrent states. The model produces outputs by projecting its internal recurrent state through an input-dependent readout. If information introduced at an earlier position remains available at a later position, the recurrent states should preserve components associated with that information. Cosine similarity therefore provides a scale-invariant way to characterize how recurrent-state representations change across positions.

Importantly, cosine similarity itself is not our stage-transition criterion. Instead, we measure its variance across relative positional distances, V(t)V(t), which characterizes the flatness of the state-similarity profile.Prior work, including LongMamba [1] and UPI [2], suggests that individual SSM heads exhibit different retention horizons: some preserve information over long distances, while others focus on shorter-range dependencies. We hypothesized that in a well-behaved hybrid SSM-Transformer model, each head should have inherent, constant and stable retention behaviors. A direct evidence is that, in the Nemotron-H checkpoint that is tuned to 128K, the state similarity remains stable across all position differences below that. This stable behavior corresponds to low cosine variance: head states drift over time, with varying degrees between heads, but in a bounded fashion within each head. When we observe a nonflat retention curve, particularly when cosine similarity decreases systematically with increasing positional distance, it may indicate that the head is experiencing out-of-distribution state drift or unstable recurrent dynamics. Such behavior suggests that the head has not yet established a consistent retention pattern across the extended context length. Thus, decreasing cosine variance indicates a return toward stable, head-specific retention behavior.

Based on this observation, SEC uses the stabilization of V(t)V(t) during training to determine when to advance to the next context length. V(t)V(t) measures retention stability, not retention capability. A low V(t)V(t) alone does not guarantee that the model can retrieve or utilize distant information. Instead, the evolution of V(t)V(t) during training indicates when the measured state dynamics have stabilized. Our key empirical observation is that this stabilization occurs substantially earlier than downstream task convergence, which can otherwise require a much longer intermediate training stage. Further training at the intermediate context length provides diminishing benefits for subsequent context extension. SEC therefore advances once state dynamics equilibrate, avoiding unnecessary intermediate-stage training while preserving final long-context performance in our evaluated models. We will clarify this intuition and distinguish retention stability from retrieval capability in the revised paper.

[1] LongMamba: Enhancing Mamba's Long Context Capabilities via Training-Free Receptive Field Enlargement, https://arxiv.org/abs/2504.16053 [2] From Collapse to Control: Understanding and Extending Context Length in Emerging Hybrid Models via Universal Position Interpolation, https://openreview.net/forum?id=MjmORKLHUI

W2/Q2: How should the threshold (\tau) be selected for different models?

A: In our current experiments, the absolute threshold is selected empirically. However, our sensitivity results suggest that SEC does not require a precisely calibrated threshold within the evaluated range. Specifically, as shown in Figure 3,, SEC-Early ((\tau=10^{-3})) and SEC-Default ((\tau=5\times10^{-4})) advance after approximately 0.15B and 0.25B tokens, respectively, yet achieve similar downstream performance. This suggests that there is a relatively broad region of effective transition points once the state dynamics have stabilized. Such robustness enables the following exploratory algorithm to apply our SEC method to a brand new model checkpoint: we may first kick off the intermediate length stages and track the retention score over time, and halt the stage once retention score stops decreasing. Figure 3 supports the motivation for this approach: V(t)V(t) decreases rapidly during early adaptation and subsequently stabilizes. The resulting implicit threshold would be relatively conservative compared to our chosen ones, but could still avoid substantial unnecessary intermediate stage training. We will clarify that the thresholds in the current experiments were selected empirically, while plateau-based stopping is a potential approach for reducing model-specific calibration that requires further validation.

As the rebuttal period draws to a close, we thank you again for your careful reading and thoughtful feedback. We hope our responses and revisions have addressed your concerns, and we appreciate the time you have devoted to reviewing our work.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.