- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors tackle a very practical unknown - we do not know how long to remain in a long context extension phase for each given extension. The authors observe that "recurrent state similarity" stabilizes but performance improves as they train a long-context phase for longer. The authors propose SEC to decide when to switch to the next long-context phase (longer sequence length). SEC simply compares a state similarity metric against a threshold. The authors also have a context parallel implementation for Mamba2.
Overall, I think the claims made are interesting and have some practical consequences. However, I don't think the results are game changing or trivially apply in all scenarios - authors do claim that this works in all recurrent language models, but I believe results are limited to Mamba based hybrid transformer architectures only. The authors provide state similary metric for GDN and pure Mamba2, but I do not see curriculum results for either of those.
Thus, the paper is a weak accept for me - it would be stronger if the authors: (1) showed that the curriculum applies for other architectures too and (2) helped explain why their metric is a good metric intuitively/mathematically.
Critical strengths
Interesting observation: The observation that recurrent-state dynamics stabilize much earlier than downstream performance is interesting. Figure 1 provides fairly convincing evidence that this happens during the authors' training setup.
Practical improvements: The reported token savings are substantial. SEC reduces the training budget by 30%
Lastly, the context-parallel implementation seems practical and removes serial dependencies across devices. This could be valuable independently of the curriculum.
Critical weaknesses
Unclear justification for the state-equilibration metric: The authors use low variance in state similarity across positional distances as an indicator of readiness for longer-context training. However, it's unclear why this should correlate with useful long-context capabilities. A model that rapidly forgets distant tokens could also exhibit low variance. Additionally, the metric doesn't directly measure stabilization across training checkpoints, which is how the authors define equilibration.
Insufficient evidence for the proposed curriculum: The experiments show that shorter intermediate training stages work well, but don't establish that state equilibration is a useful stopping criterion. In particular, there's no comparison against fixed schedules with similarly short intermediate stages. It's unclear whether SEC provides any benefit over simply advancing after 0.15B or 0.25B tokens.
Limited generalization and weak evidence for context-length specialization: The curriculum is only evaluated on two Mamba-based hybrid models using a single 32K-to-128K transition. The authors suggest that the underlying phenomenon generalizes to other recurrent architectures, but don't validate SEC on them. Furthermore, Figure 3 shows near-zero performance at 40K throughout training, making it difficult to conclude that additional 32K training specifically fails to transfer to longer contexts.
Questions for authors
How much does the adaptive curriculum actually help? Could the authors compare SEC against fixed training schedules with similarly short intermediate stages (e.g., 0.15B or 0.25B tokens)? It would also be useful to demonstrate settings where equilibration occurs at different rates and SEC adapts accordingly.
3 comments
W3: Limited generalization beyond Mamba-based hybrid models and insufficient evidence for context-length specialization.
A: Our main experiments evaluate SEC on two Mamba-based hybrid SSM–Transformer architectures, Nemotron-H-8B and Bamba-9B-v2. In Table 4, we additionally examine state-drift behavior in a hybrid Gated-DeltaNet Transformer model, providing evidence that similar state-dynamics phenomena can occur beyond Mamba-based architectures. We also conducted preliminary experiments on a 1B hybrid Gated-DeltaNet Transformer (GDN-H) checkpoint pretrained on 4K contexts for 100B tokens. We observed a similar recovery of state drift during long-context adaptation, but it fails to predict the advancing point for the stage, as the intermediate context length behavior is ~5 points off on average compared to training the full 3B tokens. These results suggest that state-drift recovery is not unique to Mamba-based models, but the relationship between state equilibration and optimal curriculum transitions may depend on the recurrent architecture. We will clarify that our demonstrated SEC efficiency gains apply to the evaluated Mamba-based hybrid models, while extending the stopping criterion to other recurrent architectures requires further investigation.
Figure 3 shows that, even after 3B tokens of continued training at 32K, performance on the 40K evaluation remains near zero. In contrast, the model acquires longer-context capability after training for only 0.15B tokens at 32K followed by 6B tokens at 128K. This comparison demonstrates that additional training at 32K does not necessarily transfer to longer contexts, whereas training at the extended context length enables the model to develop the missing capability. This supports SEC's central motivation: once recurrent-state dynamics have equilibrated, further training at the intermediate context length provides diminishing benefits for subsequent context extension, making earlier advancement more efficient.
Follow-up response to Reviewer JE73
As the rebuttal period draws to a close, we thank the reviewer again for their thoughtful comments, which helped us clarify the scope of our claims and identify an important comparison for SEC’s training schedule. Despite an unexpected GPU cluster outage prevented us from completing that comparison before the deadline, we would still like to further explain two points below.
W2/Q1. We agree that a fixed schedule with fewer intermediate-stage steps than SEC’s aggressive adaptive schedule would help clarify how much training the intermediate stage requires. GPU cluster maintenance may prevent us from completing this experiment before the rebuttal deadline. We apologize for the delay and will include the comparison in the camera-ready version.
Our one-stage baseline tests the limiting case of zero intermediate-stage steps. Training directly at the target length yielded weaker performance than SEC, indicating that intermediate-stage training was beneficial in our experiments. SEC uses a threshold on (V(t)) as an empirical criterion for state equilibration and advances to the next context length when (V(t)) falls at or below that threshold. The pending comparison asks whether a shorter, positive intermediate-stage budget can match the performance of SEC’s aggressive schedule.
W3. In Appendix C, we now report the weaker results on the GDN hybrid and limit our claim about SEC’s effectiveness to the Mamba2-based hybrid SSM–Transformer models evaluated in this work. One possible explanation concerns what (V(t)) measures: the variance in mean state cosine similarity across positional offsets. Each Mamba2 head applies a scalar decay to its previous state, while a GDN head can also selectively overwrite a stored association when a new key aligns with its key. Such an update can change what a later query retrieves while having little effect on the cosine similarity of the full state. A low (V(t)) may therefore be a less reliable guide to stage transitions in the GDN hybrid.
We thank the reviewer for recognizing the importance of efficient long-context adaptation, the observed early stabilization of recurrent-state dynamics, and our context-parallel implementation.
W1: Unclear justification for the state-equilibration metric and its relationship to stabilization across training checkpoints.
A: We thank the reviewer for this question. Our motivation is that, in a well-adapted hybrid SSM–Transformer model, individual recurrent heads should exhibit consistent retention behavior across positional distances. When a model is first exposed to longer contexts, its recurrent-state similarity profiles can exhibit substantial distance-dependent variation, potentially reflecting out-of-distribution state drift or unstable recurrent dynamics. As training progresses, this variation decreases, indicating that the recurrent-state behavior is becoming more stable across the extended context. The reviewer correctly observes that V(t) measures variance across positional distances instead of changes across training checkpoints. The distinction is that SEC tracks the evolution of V(t) throughout training, instead of interpreting a single measurement in isolation. As shown in Figure 3, V(t) initially decreases rapidly before reaching a plateau. This indicates that the measured state dynamics have entered a stable regime, which we refer to as state equilibration. Our key empirical observation is that this equilibration occurs substantially earlier than downstream task convergence. Further training at the intermediate context length provides diminishing benefits for subsequent context extension. SEC therefore uses the stabilization of V(t) as a practical signal to advance to the next context length. We will clarify the distinction between the distance-dependent variance measured at each checkpoint and its stabilization over training in the revised paper.
W2/Q1: Does SEC provide benefits over fixed short schedules of 0.15B or 0.25B tokens? Can it adapt to different equilibration rates?
We agree that distinguishing adaptive scheduling from simply shortening intermediate-stage training is important.
The key challenge is determining how long to train at an intermediate context length when adapting a new model or training configuration. A fixed schedule requires this budget to be selected in advance. Advancing too early may leave the recurrent-state dynamics insufficiently adapted, while continuing well beyond equilibration wastes computation. SEC instead monitors V(t) and advances when the recurrent-state behavior reaches a stable regime. Our experiments show that SEC-Early and SEC-Default advance after approximately 0.15B and 0.25B tokens, respectively, yet achieve similar final performance. This indicates that there is a range of effective transition points once state equilibration has occurred. A fixed schedule using the same transition points as SEC would naturally achieve equivalent results under identical training conditions. However, this assumes that the appropriate transition budgets are already known. SEC provides a model-internal criterion to identify these transition points during training, without requiring a predetermined token budget.
To further evaluate the stopping criterion, we are conducting an additional experiment that advances after only 0.05B tokens at 32K, before the observed state-equilibration point, followed by the same 128K adaptation procedure. This experiment will test whether advancing prematurely degrades final long-context performance. Unfortunately, we have encountered GPU cluster maintenance while running the long running training experiment. We will provide the results in a follow-up response once available.