acceptodds
Under review as a conference paper at ICLR 2027

Teach Once, Execute Repeatedly: Test-Time Training in Looped Transformers

Abstract

Test-time training (TTT) changes a model for the current task; looped Transformers reuse the same block for more steps. We study their composition: an update learned from demonstrations is placed inside a tied loop and executed at every recurrent visit. This placement changes its finite- horizon effect. In a linear recurrence, a rank- update reused for visits can induce an operator difference of rank , and we show this is tight: exactly reproducing it with a single-use linear update requires rank . For nonlinear networks, we use complete visit gating to recover interaction orders one through . Test-Time Compiled Loops (TTCL) learns a low-rank adapter from demonstrations and shares it across the loop. On NumSeqBench, TTCL reaches 93.3% exact-match accuracy on held-out arithmetic rules, compared with 0% when the same update is used once. In a semantic-state audit, a rank-one update reaches executed rank 16 in a 17-dimensional state and remains exact for 32 visits. On Ouro-1.4B and Ouro-2.6B LoopLMs, shared execution improves answer NLL over the same adapter used once while approaching a depthwise adapter with four times as many task parameters. Visit-gating interventions at expose high-order hidden interactions and sensitivity to visit order. Random norm-matched adapters produce larger interactions without improving NLL, indicating that the reuse gain depends on the learned update.

Primary area:
learning theory
Keywords:
Test-Time Training, Looped Transformers, Recurrent Computation, Low-Rank Adaptation, In-Context Learning, Test-Time Adaptation, Neural Program Compilation, Rank–Depth Capacity

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_G95XOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper investigates how test-time training (TTT) interacts with recurrent computation in looped Transformers. It proposes Test-Time Compiled Loops (TTCL), which learns a task-specific low-rank adapter from demonstrations while keeping the pretrained backbone frozen, then reuses the same adapter at every recurrent step.

The main theoretical contribution is a rank–depth result for linear recurrent systems, showing that a rank-r update applied over K recurrent steps can induce an operator difference of rank up to Kr. For nonlinear models, the authors introduce a visit-gating analysis to characterize interactions between repeated adapter applications.

The approach is evaluated using controlled linear and nonlinear systems, NumSeqBench, and two pretrained recurrent language models (Ouro-1.4B and Ouro-2.6B). On held-out numerical sequence tasks, shared adapter execution substantially improves negative log-likelihood over applying the same adapter once and approaches the performance of independent depth-wise adapters with 1/4th of the task-specific parameters. Additional experiments examine effective-rank expansion, execution-order sensitivity, and the importance of learned versus random updates. Downstream evaluations on mathematical reasoning benchmarks show further, though less consistent, improvements.

Critical strengths
  1. A useful theoretical perspective on recurrent low-rank adaptation: The paper establishes a tight rank–depth bound showing that a rank-r update repeatedly applied over K recurrent steps can induce an operator difference of rank up to Kr. Although the proof relies on relatively standard linear-algebraic arguments and the result is limited to linear systems, it provides a clear formalization of how repeated execution can increase the expressive effect of a fixed-size adapter. This theoretical perspective helps motivate the proposed approach.

  2. Strong empirical results supported by controlled experiments: The experiments on Ouro-1.4B and Ouro-2.6B show substantial improvements in negative log-likelihood from repeated adapter execution, approaching depth-wise adaptation performance with 1/4th of the task-specific parameters. Visit-removal, execution-order, and random-adapter controls further demonstrate that both the learned update and its recurrent execution matter. These results provide meaningful empirical support for the proposed approach, although the evaluation is concentrated on structured numerical tasks.

Critical weaknesses
  1. Limited methodological novelty and an incomplete connection between theory and empirical results: The proposed method combines existing components—gradient-based test-time adaptation, low-rank adapters, and pretrained recurrent Transformers—with the main algorithmic distinction being repeated execution of the same adapter. The rank–depth theorem provides a useful formalization but follows from relatively straightforward linear-algebraic arguments and establishes an existence result for linear systems. The nonlinear interaction analysis does not directly establish that increased effective rank explains the observed performance gains. Consequently, it is unclear whether the paper provides sufficiently novel methodological or mechanistic insights beyond demonstrating the benefits of adapter reuse.

  2. Insufficient evaluation to establish the advantage and generality of the proposed approach: The main single-use comparison evaluates the same learned adapter under a restricted execution schedule, rather than clearly establishing an advantage over an independently optimized single-use adapter with matched adaptation compute. This distinction is important because the adapter is optimized for recurrent execution. Furthermore, the strongest results are concentrated on structured numerical sequence tasks, with only nine held-out formula clusters in the confirmatory Ouro evaluation. Improvements on downstream mathematical reasoning benchmarks are more modest and inconsistent. These limitations make it difficult to assess whether TTCL offers a robust advantage over competitive adaptation baselines across a broader range of tasks.

Questions for authors
  1. [Clarification / Additional evidence] Optimization-matched single-use baseline: Since the adapter is optimized through recurrent execution, has the single-use baseline been independently optimized for its own execution schedule under a matched adaptation budget? If not, could the authors provide this comparison?

  2. [Clarification] Connection between theory and empirical improvements: What evidence directly connects the rank-expansion mechanism established for linear systems to the performance gains observed on Ouro? Does the effective rank of the learned adapter's executed effect correlate with predictive improvements?

  3. [Clarification] Experimental setup and presentation: Could the authors provide a more self-contained description of the Ouro recurrent architecture, the adapter insertion and optimization procedure, and the evaluation datasets? In particular, the main text should explain the construction and meaning of the held-out formula clusters and clarify that the NumSeqBench experiment uses a separately trained, two-dimensional recurrent system rather than an Ouro checkpoint. These details are important for interpreting the results and understanding which claims each experiment supports.

  4. [Clarification] Evaluation generality: How were the nine held-out formula clusters selected, and how different are they from the exploratory tasks? Given the heterogeneous downstream results, what evidence supports generalization beyond structured numerical sequence prediction?

Reviewer_Tsk8OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

this paper introduces TTCL, a novel method that combines test-time training with looped transformers by learning a task-specific low-rank adapter and reusing it across recurrent steps. they provide a theorem on the expressivity of the linear case, suggesting that reusing updated low-rank adapters expands the model's capacity (higher-rank effect) compared to single-use models. empirical studies further validate TTCL in both a controlled environment (aligning with their linear setting) and standard downstream tasks using pre-trained looped transformers, where TTCL improves performance over applying the same adapter only once.

Critical strengths
  1. the idea is generally interesting and represents an important direction for research
  2. present controlled experiments to provide a proof of concept for their theory in the linear case
  3. demonstrate the approach on downstream tasks using pre-trained models.
Critical weaknesses

my main concerns relate to the paper's motivation, positioning relative to prior work, and presentation of its theoretical and empirical contributions. clarifying these aspects is essential for demonstrating the paper's importance, particularly for readers new to the area. my major concerns are as follows:

  1. here are almost no references throughout most of the paper (including the introduction), with only a very brief related works section providing some citations. It is important to show the position of this paper!
  2. very little discussion regarding the setting. For example, it is unclear when one should care about TTT versus Looped Transformers, and why you chose to combine these two approaches. You should elaborate on the specific scenarios you are addressing and provide the intuition behind why this combination is useful.
  3. if your main contribution is theoretical, you must provide the necessary background on related theory. Currently, there is no discussion of previous theoretical work on TTT, Looped Transformers/models, or other relevant areas (e.g., it appears you are applying typical control theory). It would be interesting to highlight what insights practitioners can gain from your theory and what theoretical gaps you are filling.
  4. the theory only demonstrates expressivity in the linear case. Furthermore, it seems possible that some cases may not even expand the rank space. However, we still cannot guarantee that the gradient-based update can reach the exact algorithm we want the model to learn or adapt to. I think it is also important to discuss whether better expressivity makes out-of-distribution generalisation harder, or in what situations this method might perform worse. In extreme cases regarding expressivity, one could simply update the entire model without a low-rank adapter.
  5. a significant amount of key experimental data (both results and figures) is only available in the appendix. Consequently, the main text feels poorly structured and difficult to read.
  6. it seems like the paper want to advise practitioners against using single-use adapters for looped models, a point that is also central to the theoretical discussion. While I agree this is a useful ablation and serves as a proof of concept, its practical motivation remains unclear. Specifically, it is uncertain why practitioners would utilise single-use adapters when the adaptation data consists of only one-step demonstrations. Further clarification on the intended use case for this approach would be valuable.
Questions for authors

In the weaknesses.

Reviewer_b9vROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TTCL fits a low-rank adapter from demonstrations and reuses it across recurrent visits of a looped Transformer. The paper establishes a tight linear rank–depth separation, analyzes nonlinear visit interactions, and evaluates synthetic tasks and Ouro models. Shared execution outperforms applying the identical adapter once and approaches depthwise adaptation with fewer task parameters.

Critical strengths
  1. The theory clearly distinguishes executed-operator rank from parameter degrees of freedom: reuse can increase the former without increasing the latter (Theorem 1, Proposition 2).

  2. The intervention suite provides useful mechanism evidence. Norm-matched random adapters produce strong interactions without reuse gains, while matched-dose permutations and visit-removal tests further characterize the learned update.

Critical weaknesses
  1. The main Ouro comparison is an execution ablation, not a comparison against separately optimized single-use adaptation. The adapter is trained with compilation horizon 4, then evaluated at every visit or only the first (§5.5 Table 9). This demonstrates the benefit of reuse for that adapter, but does not establish an advantage over an adapter trained specifically for first-use-only execution. The linear separation result remains valid.

  2. Evidence for rule compilation and practical advantage remains incomplete. Within-domain support swaps do not establish formula-specific adaptation, although the matched-format BBH controls support task sensitivity (B.3). Comparisons with same-support prompting are also mixed: TTCL wins on 2.6B, but prompting performs better on 1.4B GSM8K and GSM-Hard (Table 11). The main 256-query evaluation omits prompting, leaving the practical advantage unclear.

Questions for authors
  1. How does a separately optimized first-use-only adapter perform on Ouro, retaining all four backbone visits and matching rank, support, and optimization budget?

  2. Given B.3, what evidence supports formula-rule compilation beyond transferable domain calibration?

  3. On the main 256-query split, does TTCL outperform same-support prompting when answer formatting and parsing are controlled?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.