acceptodds
Under review as a conference paper at ICLR 2027

TEST-TIME DILATION IN SEQUENCE MODELS

Abstract

A model can have access to earlier information yet fail to use it when a dependency becomes longer. We study test-time dilation: expanding the material between an earlier state and its use after training, while preserving the required answer. Controlled experiments separate three requirements: preserving one value, retrieving one of several bindings, and preserving an outer value while computing inside a scope. Attention can retrieve content directly, but positional scoring can make that route unreliable. Selective recurrence can preserve information, but repeated updates can alter it. Targeted interventions restore both forms of access, and scaling allows ordinary recurrence to learn more reliable carry. Neither successful retrieval nor accurate copying alone establishes robustness to the other requirements. A controlled scoped task makes the distinction concrete: models that master an order-sensitive program throughout the training range can still collapse when answer-preserving operations are inserted beyond that range. Dilation robustness is therefore a property of the learned computation, not a guarantee supplied by an attention edge or a long context window.

Primary area:
interpretability and explainable AI
Keywords:
test-time dilation, length generalization, long-range dependencies, associative retrieval, state-space models, positional encoding

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_gAAxOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies several different NLP architectures on the task of outputting xsx_s when the input is [Φ1,…,Φc−d−1,xs,Φc−d,…,Φc][\Phi_1,\ldots,\Phi_{c-d-1},x_s,\Phi_{c-d},\ldots,\Phi_c]. They fix all parameters but dd, which allows to study how robust the different models are to adding noise after the value of interest. By studying this task and a few variations of it, they conclude that no model is robust to longer dependencies in general, and all fail in some cases.

Critical strengths

The paper studies the capability of different NLP models in information retrieval tasks, and how adding noisy context modifies this capability. This is a relevant setting, of interest to many in the community.

Critical weaknesses

The paper has two main weaknesses.

First, it lacks clarity, a lot of notions are not formalized, and the motivation is not clearly exposed. Mostly, a lot of things are barely defined. Those include:

  • the specific models being studied,
  • how they are trained,
  • how xsx_s and Φ\Phi are chosen. This is especially problematic as those are central to the main message of the paper. As a result, it is quite hard to read and I believe it requires substantial rewriting before it can contribute to the community.

Second, the relevance with regards to prior work is barely discussed. As such, it is not clear to me what the contributions of this paper are. The main message seems to be that the architectures under study (which are not clearly defined) don't generalize on the specific tasks under study (not clearly defined either). However, the generalization limits of transformers are already a clearly established result [1,2,3]. Moreover, information retrieval tasks and copying as well as their limits have been widely studied [4,5,6].

[1] A Formal Framework for Understanding Length Generalization in Transformers, Huang et al, ICLR 2025

[2] Arithmetic Transformers Can Length-Generalize in Both Operand Length and Count, Cho et al, ICLR 2025

[3] Quantitative Bounds for Length Generalization in Transformers, Izzo et al, ICLR 2026

[4] RULER: What’s the Real Context Size of Your Long-Context Language Models?, Hsieh et al, COLM 2024

[5] Transformers need glasses! Information over-squashing in language tasks, Barbero et al, NeurIPS 2024

[6] Needle Threading: Can LLMs Follow Threads Through Near-Million-Scale Haystacks?, Roberts et al, ICLR 2025

Questions for authors

What are the specific architectures you are considering (saying simply RoPE or NoPE is not nearly enough to identify a precise architecture)?

And why did you choose those specifically?

How do your specific tasks and results compare to all the existing literature? (eg Why can your results not be derived from existing length generalization results?)

Reviewer_psXNOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper describes experiments on transformers and state-space models, in which the models have to recall key-value assignments across sequences of intermittent tokens whose lengths differ between training and testing (similar to existing work on length generalization). The paper shows that many models tend to fail at these tasks. The paper also shows that, even though there are fixes which can improve performance on some of the tasks, these can have adverse effects on other tasks, showing that these fixes do not constitute universal solutions to these tasks.

Critical strengths

The paper studies failure modes of common models (transformers and SSMs) on simple algorithmic tasks. Studying these failure modes is important and useful because it can reveal critical shortcomings of these models, especially in out-of-distribution scenarios.

Critical weaknesses

The paper is very hard to read. There is no flow or motivating sentences in any paragraph, just facts that are falling from the sky. There are also no intermittent conclusions, leaving it to the reader to guess what the point of each of the currently described experiments may be. I do not know if this is due to AI writing or not, but the fact that the paper uses the idiom “it is this, not that” in almost every paragraph is an additional aspect of the writing that makes the paper extremely unpleasant and hard to read.

The paper does not describe the details of the models that it evaluates on the tasks. It also does not evaluate any existing models, such as Llama or Qwen on these tasks.

Training runs use multiple different seeds but not multiple different learning rates. These would at least be equally important to using multiple seeds because transformers and SSMs can be very sensitive to learning rate variations.

The paper would be much easier to read if the tasks were described using illustrative examples instead of mere descriptions.

Questions for authors

The formula in line 151 is basically a repetition of the one line 081. Why?

Why is the experiment in Figure 3 using three instead of 6 seeds?

Reviewer_vK7XOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies if sequence models can still use earlier information when the information is moved farther from the point where it is needed. In the main experiments, the authors keep the input length, tokens, and correct answer unchanged, and change only the position of the relevant source. They test whether models can preserve a single value, retrieve a particular binding after other bindings have been added, and remember an outer value while performing an ordered computation inside a scope.

The results show that models fail for different reasons, for example in attention models, positional scoring can make an earlier token harder to access even though it remains available. In recurrent models, repeated updates can damage the stored information. Changing the positional scores or preventing those updates restores performance on the simple task. But a model that copies one value reliably may still struggle to retrieve an older binding or carry out a longer computation.

Critical strengths
  • A strength of the paper is that it aims at separating longer dependencies from longer inputs. In the main experiments, the authors keep the total sequence length, the tokens, and the correct answer fixed while moving the relevant source farther from the query. This makes it easier to study what happens when a dependency becomes longer without also increasing the amount of information the model must process.

  • The experiments are designed to tell apart skills of the model that a single long-context score can hide. For example, preserving one value, retrieving an older binding after later writes, and carrying out a computation while protecting another value require different behaviours. And testing them separately shows why success on a copying or retrieval task doesn't establish that a model can handle every kind of long dependency.

  • I think the results are handled properly: when a model already performs poorly at the training distance, the authors don't attribute all of the later errors to dilation, they reported both overall accuracy and the proportion of initially correct examples that remain correct, while acknowledging that this conditional measure compares different subsets across models.

Critical weaknesses
  • I think that the main experiment doesn't clearly separate a longer computation from an other kind of computation. During training each program contains only one A and one B, with identity operations elsewhere. So, a model could solve this task by recognising which operation comes first, instead of learning a general procedure to actually update the state. At test time the added identity blocks introduce repeated active operations and unseen patterns. Enev if the final answers stay the same, the computation needed to get them is different from what training needs. So, the results show that the learned solution fails on the expanded programs, but still leave open if it ever learned the general state transition rule.

  • I think that some comparisons are difficult to understand because the models don't start with the same abilities. The recurrent models already struggle to retrieve a target followed by other bindings at short distances, while NoPE doesn't do great at the ordered scoped task. The authors acknowledge these acquisition failures, but they limit what the longer distance results can tell us. Measuring how many initially correct examples remain correct is useful, even if each model is then evaluated on a different subset that may contain easier cases.

  • A problem of the recurrent intervention is that it identifies a source of failure but it doesn't fully explaining how it happens. Freezing the state after the source does restore the performance, and this show that the next updates make the stored information less usable. But this intervention prevents attenuation, overwriting, and interference together. The carry-decay calculation makes attenuation plausible, but the experiments don't establish how much each process contributes to the observed errors. I think a more plausible mechanistic conclusion should be that post-source updates cause the failure, rather than claiming that this is a particular form of memory decay.

  • It's a bit unclear if the findings apply also to bigger models: the paper doesn't test if the same failures and explanations happen in pretrained language models, for example. This matters because increasing the recurrent model’s size already removes its simple-dilation failure, showing that the learned behavior can change substantially with scale.

Questions for authors

did you tested the scoped task after training on more varied short programs containing multiple A and B operations? Since the current training examples contain only one of each, a model could perform well by learning their relative order rather than the general state-transition rule. T

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.