acceptodds
Withdrawn from ICLR 2027

Not All Recurrence Is Created Equal: Depth Sensitivity Differs Between Recurrence Dimensions of a Single Checkpoint

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Recurrent language models reuse computation across inference steps, so recurrence depth appears to offer an adjustable test-time compute budget. That interpretation assumes a checkpoint remains reliable when its depth is changed after training. We test it through a controlled within-checkpoint comparison: HRM-Text-1B couples two recurrence dimensions in one set of weights, so varying each in turn holds architecture, scale, data and training schedule fixed, a comparison no cross-model study can make. Its high- and low-level dimensions turn out to respond very differently. Reducing \(L\) from three steps to two retains most trained-depth accuracy while removing 25% of recurrent block passes; increasing \(H\) from two to four reduces GSM8K accuracy from 84.0% to 10.0% and causes substantial output-format failure. Two configurations with identical total block passes differ sharply, one scores zero on every benchmark while the other does not, so total recurrent compute does not characterise the effect of altered depth. A hidden-state intervention implicates the high-level representation: additional \(H\) steps displace it from its trained endpoint while preserving its norm, and interpolating back toward that endpoint recovers most of the lost accuracy before the trained state is fully restored. We situate the result against four single-loop checkpoints from two further families, none of which shows comparable degradation over the depths we evaluate. We distil the findings into a short protocol for validating recurrence depth as an inference-time compute dial.

Primary area:
foundation or frontier models, including LLMs
Keywords:
recurrent language models, recurrence depth, test-time compute, latent reasoning, hierarchical reasoning models, depth extrapolation, recurrent state dynamics, hidden-state intervention

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ymMcOpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

The paper evaluates the effects of modifying L and H from the L and H hyperparameters used during training.

Critical strengths

None

Critical weaknesses

The evaluation methodology is described poorly and it is not sound. The number of subsampled problems from each benchmark is very few (e.g. 50 problems for GSM8K, which contains total 1319 test problems). One of the experiments use 12 problems only: The paper itself notes "This analysis uses the smallest sample in the paper an we report no dispersion for it; the claims drawn from it should be read accordingly." The results are not statistically significant. The paper also does not identify the significance of the finding. It is not surprising that deviating from the hyperparameters used during training would negatively impact the performance.

Questions for authors
  • What are the results when evaluated over the entire problem set of the benchmarks?
  • What is the significance and impact of the findings?
Reviewer_sUVGOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper examines whether recurrence depth can reliably serve as an inference-time compute control. Its main experiment varies the high-level and low-level recurrence counts within a single HRM-Text checkpoint. The two dimensions show different sensitivity to depth changes, including different outcomes at the same recurrent block count. The paper also compares single-loop checkpoints and studies hidden-state trajectories. An interpolation intervention partially restores performance by moving the high-level state toward its trained-depth endpoint.

Critical strengths

The within-checkpoint comparison is useful because it holds model weights and training history fixed while examining different recurrence dimensions. The paper carefully validates the checkpoint-specific prompting interface and reports invalid outputs separately from accuracy. Paired behavioral comparisons and the hidden-state intervention provide complementary evidence. I also appreciate the explicit distinction between the main results and exploratory analyses, and the restraint in interpreting cross-family differences.

Critical weaknesses
  1. The manuscript is difficult to read and follow. For example, nothing is mentioned and introduced about what is H and L in the abstract. Similar issues happen a lot in the main manuscript content.

  2. It is difficult to capture the technical contribution. If the evaluation protocals is proposed by the author and is the main technical contribution, please motivate it clearly and distinguish with existing studies.

  3. The experimental results seem incomplete, e.g., the Table 1. And it is difficult to understand the comparison; no baseline is introduced and only providing the analysis of different values of hyperparameters is insufficient to demonstrate the original claim.

Questions for authors

Please see the weakness and address them as much as possible.

Reviewer_a7kfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In looped transformers, we repeatedly apply same set fo parameter / layers to the input in recursive manner. These models are trained with certain schedule of number of looping iterations with the hope model becomes adaptive and performs better with increased number of iterations. This comes with that performance will increases monotonously, and at worst the flat out i.e. it extra iteration won't damage the peformance -- which prior work has shown not to be the case. This paper studies reaffirms the prior work and study this phenomena closely to understand why it happens.

The main contributions are following:

  1. The model is sensitive to changes in number of iterations i.e. there persists a training-inference gap and the adaptivity of models they test is superficial. In Hierarchical model (HRM-Text-1B, which uses two level of looping, viz. outer and inner), they report that both increases or decreasing outer loop worsens performance. Also, under compute matched setting, different value of outer vs inner looping gets different performance.
  2. The paper tries to investigate where this failure happens.
  3. The paper goes onto show that this problem persists across different family of looped transformers.
Critical strengths
  1. Important and timely problem. Understanding how recurrent models behave when inference-time depth differs from training is an important problem, especially as recurrent architectures become increasingly relevant for scaling test-time compute.

  2. Reasonable coverage of recurrent models. The paper evaluates five checkpoints across three model families (HRM, Huginn, and Ouro), covering different recurrence architectures and training approaches. This provides a useful empirical comparison of depth sensitivity across existing models.

Critical weaknesses

See questions.

Questions for authors
  1. The H-state analysis (Section 4.3, p. 6, interpolation table) is largely self-confirming rather than explanatory. Additional H steps move representations away from the trained endpoint, and interpolating them back restores accuracy. This is expected and does not explain why performance degrades. A useful experiment would test whether task-relevant information is actually lost at H=4 or remains decodable but incompatible with the trained readout.

  2. The L-state trajectory analysis (Section 4.3, Tables 5–7 in Appendix E) is descriptive without establishing a causal connection to depth sensitivity (Figure 1a–b). Observing contraction in one phase and periodic statistics in another does not explain their computational roles or impact on accuracy. A controlled H-state swapping experiment, holding the L-state initialization fixed, could establish whether H conditioning actually causes these different dynamics.

  3. The original Ouro paper (Zhu et al., 2025, Tables 10–13) already evaluates recurrence extrapolation from 4 to 8 steps, reporting modest degradation on standard benchmarks but substantial degradation on harder reasoning tasks (e.g., Ouro-1.4B-Thinking drops from 65.0% to 38.67% on AIME 2024). Furthermore, the reported Ouro ARC-Challenge accuracies (90.7% and 92.7%) differ considerably from the original results (60.92% and 66.38%). Although evaluation setup differ, these discrepancies warrant validation against the official evaluation harness. The authors should reconcile these results and evaluate tasks where the original papers already report significant depth sensitivity.

  4. In general, I would like bit more evals and experiments for this paper that is stress testing the looping models under different eval setups. Its an interesting and important paper, but could use more eval ablations that in current draft.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.