acceptodds
Under review as a conference paper at ICLR 2027

Beyond the Loop: From Latent Dynamics to Performant Recurrent Models

Abstract

Recurrent Transformers offer a parameter-efficient way to increase latent computation through repeated application of shared modules. However, it remains unclear how to design better-performing recurrent architectures, train them effectively and efficiently, and assess how broadly their gains transfer. To address these questions, we examine HRM-Text as a representative structured recurrent model and find that repeated -module updates refine the state toward a narrow recurrent region, while the module applies a distinct finite transformation. This motivates , a simpler one-stream recurrent family that preserves this functional separation. Under the original training recipe, matches HRM-Text at matched trainable and executed depth; with our new recipe, its eight-benchmark mean improves by 4.8–5.7 points across both scales and surpasses recipe-matched HRM-Text. We further show that truncated backward windows preserve useful gradient directions, while shorter differentiable paths and parameter sharing reduce 60B-token training time by more than relative to matched-depth standard Transformers in our 8×H100 production setting. We finally examine how these gains extend beyond the training mixture. They persist on source-matched evaluations and controlled arithmetic shifts, while broader transfer remains mixed under the current breadth and scale of pretraining. As a complementary adaptability test, fine-tuning on a safety domain absent from pretraining substantially improves safety performance while preserving most general-task utility.

Primary area:
foundation or frontier models, including LLMs
Keywords:
Recurrent Transformers, Looped Transformers, Language Models, Efficient Training, Latent Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Jd5FOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper provides a thorough empirical and mechanistic audit of HRM-Text, a recently introduced structured recurrent language model. The authors conduct controlled interventions and latent-state analyses, revealing a functional asymmetry: repeated low-level LL-module calls act as a contraction operator pulling states toward an input-conditioned basin, while the high-level HH-module provides a distinct terminal transformation. Based on these insights, the paper advocates for a simpler, single-stream LnHL^n H architecture (closely related to the RINS formulation). Combined with a modernized training recipe (WSD50-Muon) and truncated backward windows, the proposed L7HL^7 H model matches or slightly outperforms HRM-Text while cutting 60B-token training time by over 50% relative to standard depth-matched Transformers on 8×H1008 \times \text{H100} hardware. The study concludes with a clear-eyed assessment of in-distribution versus out-of-distribution generalization.

Critical strengths
  1. Valuable Empirical Debunking & Simplification: The paper successfully untangles the over-engineered components of HRM-Text. By demonstrating that the two-stream, two-stage hierarchy can be simplified into a single-stream L7HL^7 H structure without performance loss, it offers a much cleaner baseline for future research in recurrent architectures.

  2. Rigorous Mechanistic and System Analysis: The latent-state dynamics (proving contraction into an input-conditioned basin via 256-start analyses) are meticulously executed. Furthermore, the production-level efficiency measurements (HBM consumption, training time reduction, and activation recomputation avoidance) are concrete and useful for the community.

  3. Scientific Transparency: The authors honestly report mixed out-of-distribution transfer results and limitations regarding pretraining corpus breadth, rather than hiding negative findings behind cherry-picked benchmarks.

Critical weaknesses
  1. Limited Architectural Novelty: The resulting LnHL^n H architecture is structurally equivalent to the existing RINS (ArBA^r B) paradigm (Alabdulmohsin & Zhai, 2025). While the paper's contribution lies in identifying the missing ingredients (terminal x0x_0 injection and truncation warmup) that made prior RINS baselines underperform, the architectural leap itself is incremental.

  2. Confounded Gains and Unexplained OOD Regression: The integration of the WSD50-Muon optimizer yields substantial improvements, but the paper lacks a standard Dense Transformer baseline under the exact same WSD50-Muon recipe to isolate what portion of the gain is uniquely tied to the recurrent architecture. Additionally, in Table 5, adopting WSD50-Muon leads to noticeable performance drops on broader reasoning/instruction benchmarks (e.g., BBH and IFEval). A deeper discussion on why this recipe exacerbates overfitting to source-matched patterns would strengthen the paper.

  3. Presentation and Organization Flaws:1) The mathematical derivation in Appendix D.1 (IFT and Neumann series expansion) spans multiple pages of standard textbook calculus, which feels bloated given that the probed states are not strict one--step fixed points (as shown in Table 20, dreld_{\text{rel}} remains high). 2) The safety fine-tuning experiment in Appendix G.4 is somewhat abrupt and evaluated exclusively on HRM-Text rather than the core proposed L7HL^7 H model, giving a slight impression of a tacked-on ablation.

Questions for authors
  1. OOD Degradation under Muon: Why does the WSD50-Muon recipe consistently degrade performance on certain broader transfer benchmarks like BBH and IFEval (Table 5)? Does the combination of Muon and recurrence accelerate overfitting to the source-matched training mixture?

  2. Safety SFT Inconsistency: The safety adaptation experiment is only evaluated on HRM-Text. Can the authors provide corresponding safety-utility trade-off results for the proposed L7HL^7 H architecture under the same fine-tuning protocol?

  3. Implicit Equilibrium Tightness: Given that Table 20 indicates the probed states deviate from exact fixed points, how sensitive are the directional alignment conclusions in Section 5.1 to the choice of the truncation window KK?

3 comments
AuthorsOpenReview ↗

Response to Reviewer Jd5F (Part 1)

Thank you for the constructive feedback and recognition of our contributions. We address the concerns below with clarifications and additional experimental results.

W1. Architectural novelty and the relationship to RINS

We acknowledge that LnHL^nH follows the ArBA^rB structure of RINS. However, our design is motivated by our analysis of L/H latent dynamics rather than simply adopting RINS. The interventions motivate a single-stream alternative, which reaches 68.40 versus HRM-Text's 68.22 at matched parameter and executed depth under the original recipe. We interpret this as comparable aggregate performance without the hierarchy. Thus, our contribution is to provide a mechanistic explanation for why this simple architecture can match HRM-Text performance. We also examine two important design and training choices: input injection and the truncated gradients used to train it (Appendix C.5 isolates terminal injection; Section 5.1 measures gradient agreement).

AuthorsOpenReview ↗

Response to Reviewer Jd5F (Part 2)

W2 / Q1. Recipe attribution and broader-transfer regressions

Recipe attribution.

We appreciate this point. Without a dense Transformer trained with WSD50–Muon, we cannot determine whether recurrent models benefit more from this recipe than dense models. But our claim is narrower: Section 5.2 reports improvements from changing the training recipe within each recurrent architecture, without attributing these gains specifically to recurrence. Separately, the architectural comparisons in Table 1 and Section 4 use the same original training recipe, so their conclusions do not depend on Muon.

OOD performance is task-dependent, not uniformly degraded.

Table 5 shows that WSD50–Muon improves some transfer benchmarks while reducing performance on others, with substantial differences across architectures and scales. The table below summarizes the changes relative to AdamATan2 (percentage points; IFEval averages 0-, 1-, and 3-shot results).

ModelScaleGSM-Symbolic p2MMLUMMLU-ProBBHIFEval
HRM-Text0.69B+6.2+0.8−0.6+0.0−4.0
L7HL^7H0.69B+9.9+4.3+0.6+4.2−2.8
HRM-Text1.0B+0.7−0.1−3.2−6.6−1.0
L7HL^7H1.0B+11.3+2.8+2.4−1.5−8.6

BBH: task trade-offs and answer-format sensitivity. The 0.69B models provide a clearer view of the variation within BBH. Both improve on logical deduction and multistep arithmetic, but regress on disambiguation and translation-error detection. Movie recommendation moves in opposite directions across the two architectures. Selected task accuracies are shown below (AdamATan2 → WSD50–Muon, %).

BBH taskHRM-Text, 0.69BL7HL^7H, 0.69B
Overall36.89 → 36.9136.61 → 40.81
Logical deduction, 3 objects56.8 → 71.663.6 → 76.0
Multistep arithmetic43.2 → 53.246.8 → 64.0
Pronoun disambiguation61.6 → 56.863.6 → 48.4
Translation-error detection22.4 → 13.642.8 → 34.0
Movie recommendation24.8 → 33.236.0 → 29.6

Inspection of the results also reveals that some apparent BBH degradation comes from changes in answer format rather than reasoning accuracy. On web_of_lies, HRM-Text 1.0B under WSD50–Muon scores 2.8%, largely because it outputs True/False rather than the expected Yes/No (239 of 250 predictions). Mapping these semantically equivalent labels raises accuracy to 52.4%, compared with 52.0% under AdamATan2. This diagnostic alone accounts for approximately 1.9 percentage points of the reported 6.6-point BBH decline. Other BBH tasks also exhibit changes in response format. For example, the 1.0B model's movie-recommendation outputs shift from option letters toward descriptive responses. Thus, the recipe changes task-response behavior, while some task-level accuracy losses remain genuine.

IFEval: constraint-specific changes. Unlike answer-label mismatches in BBH, violations of IFEval constraints are genuine instruction-following failures. However, the effects differ substantially by constraint: both 0.69B models become less reliable on JSON formatting, word counts, and title formatting, while improving on multiple-section and placeholder-count requirements. Selected loose accuracies are shown below (AdamATan2 → WSD50–Muon, %).

IFEval constraintHRM-Text, 0.69BL7HL^7H, 0.69B
JSON format47.1 → 33.341.2 → 23.5
Word count60.9 → 39.153.8 → 38.5
All lowercase52.1 → 26.539.3 → 42.7
Title format88.3 → 68.595.5 → 52.3
Multiple sections16.7 → 31.023.8 → 50.0
Number of placeholders35.8 → 45.730.9 → 44.4

Interpretation. These results suggest that WSD50–Muon affects not only task accuracy but also how models respond to different task requirements. The web_of_lies analysis shows that a change in answer format can substantially reduce measured accuracy without reducing the underlying judgment accuracy. Other BBH tasks exhibit genuine performance trade-offs, while IFEval shows that compliance with some constraints improves and others deteriorates. A plausible explanation is that changing the optimization recipe shifts the response patterns and task behaviors learned from the same instruction–response mixture. Such changes may benefit tasks aligned with those patterns while transferring less effectively to unfamiliar task formats or strict constraints. This points to task-dependent generalization trade-offs rather than a uniform loss of capability, and does not by itself indicate that Muon and recurrence accelerate overfitting.

AuthorsOpenReview ↗

Response to Reviewer Jd5F (Part 3)

W3 / Q2. Safety adaptation of L7HL^7H

The submitted appendix reports HRM-Text only, and we agree that those results cannot establish L7HL^7H-specific transfer. We have also conducted additional L7HL^7H experiments under the AdamATan2 safety-SFT protocol and included the results in our revision. These use the same deterministic data splits, three-epoch fine-tuning budget, batch size of 32,768 tokens, learning rate of 3×10−53\times10^{-5}, and final epoch-3 EMA evaluation rule. The model is the 0.69B L7HL^7H checkpoint; utility is the unweighted eight-benchmark mean, and safety is 100×(1−harmful-response rate)100\times(1-\text{harmful-response rate}) on 320 HarmBench direct requests classified by WildGuard.

L7HL^7H fine-tuning dataUtilityHarmBench safetyHarmful responses
No SFT68.4029.06227 / 320
WildJailbreak68.8496.8810 / 320
WildGuardMix69.0592.1925 / 320
Equal mixture68.8497.199 / 320

These results extend the domain-transfer observation to L7HL^7H: harmful responses decrease substantially while aggregate general-task utility is retained. The safety fine-tuning experiment is part of our domain-transfer analysis in Section 6, rather than a separate safety-focused study. We chose safety because safety-focused data are absent from our pretraining mixture, making it a suitable test of adaptation to an unseen domain; other unseen domains could serve the same purpose.

W3 / Q3. Non-equilibrium states and sensitivity to the backward window

We appreciate the reviewer's suggestions and agree that Appendix D.1 can be more concise. We also agree that Table 20 rules out treating the probed states as exact one-step fixed points. Section 5.1 contains two distinct comparisons:

  1. Exact finite reference: truncated gradients are compared directly with full backpropagation through the same executed finite program. This comparison requires no fixed-point assumption.
  2. IFT-style local reference: an implicit response is constructed at the observed states. This is an equilibrium-motivated diagnostic, not the exact gradient of our finite model.

The window sweeps directly address sensitivity. For HRM-Text's L parameters, Appendix Table 22 reports the following cosine ranges across checkpoints after epochs 1–3, with ten matched batches per checkpoint:

Backward window KKAgainst exact full finite BPAgainst IFT-style local reference
20.654–0.7230.634–0.748
30.654–0.7230.634–0.748
40.777–0.8350.708–0.831
50.824–0.8770.703–0.832

Here KK follows HRM-Text's actual backward policy, not an arbitrary contiguous tail: the retained calls are respectively {L6,H2}\{L_6,H_2\}, {H1,L6,H2}\{H_1,L_6,H_2\}, {H1,L5,L6,H2}\{H_1,L_5,L_6,H_2\}, and {H1,L4,L5,L6,H2}\{H_1,L_4,L_5,L_6,H_2\}. The identical L-gradient entries at K=2,3K=2,3 reflect this policy. Agreement with full finite BP improves as more L calls are retained, whereas agreement with the local implicit reference need not increase monotonically (which is reasonable because fixed points are not assumed). The L7HL^7H sweep in Tables 25–26 gives the same important distinction. All the truncation windows preserve the positive directional agreement with the exact finite reference and IFT-style local reference.

Reviewer_xLEvOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

The setting of the paper is investigating looped models on textual data. They focus on HRM-text and what differentiates it from a flat looped model. They show that the L-module in HRM-text is behaving like a looped model, with the changes in its hidden representation gradually getting smaller and the representation converging to a basin. However, they provide some evidence that the H-module is not doing the same thing, but rather changes the basin to which the L-module's representation converges to. Then, they argue that this provides some evidence for the source of improvement in HRM-text not being the hierarchy, and use this motivation to propose a scheme in which the H-module is being used as the "coda" section of a looped model, following [1]'s terminology. Finally, they look into the optimization and computational complexity of HRM-text and their proposed scheme.

Critical strengths
  • The paper addresses an important question in architecture design for looped models, which is understanding the utility of hierarchical reasoning.
  • Looking at the L-module and the H-module behavior from a convergence stand-point is interesting and informative.
  • The experiments involve large-scale training, and the training setup is following a well-established procedure.
  • The paper is easy to read and follow.
Critical weaknesses
  • The paper has very little novel contributions. The convergence perspective of the L-module and the related analysis is not new (present in the original paper [2], and TRM [3]), and the proposed method is similar to RINS [4], as the authors acknowledge. The use of the H-module at the end of the looping is the same as the coda step in [1]. Furthermore, replacing HRM's hierarchy with a flat looping has already been shown to work on HRM's original tasks [7, 8], with some explanation of the roles of the two modules [3, 8]. Truncated backpropagation and its relationship to the real gradient (through IFT) has been investigated before [5], and the implications of looping on scaling laws have been investigated extensively in [6]. The paper does not mention [3], [7], or [8] at all, and offers very minimal comparison to RINS.
  • I also have my qualms with the way the paper approaches the problem, how rigorous the experiments are, and whether the authors can claim from the results what they aim to claim. I'll name a few. The authors do not report any hyperparameter optimization for the baselines in Table 1 (Appendix E omits the sweep history and reports selected configurations only for the recurrent models), which is very important, especially considering the learning rate and optimization-related implications of looping. There are some confounding factors in the experiment design, such as the placement of the normalization layer. I'm especially referring to the standard looped Transformer, whose normalization boundaries (every 24 layers) differ from HRM-text's (every 12 layers). That brings the results in Table 1 and the claims made by the authors into question. Also, the improved WSD50–Muon recipe is applied only to the recurrent models, while the Table 1 baselines are trained only with the original recipe. Since the recipe effect (3.5–5.7 points) is larger than the gap between HRM-text and the monolithic standard Transformer (3.07), I'm not sure how you can claim the improvements are due to the L^n H structure, and not the optimizer.
  • The claim in line 67 is stronger than what is shown. The authors only show that the truncated gradient is positively aligned with the exact gradient (cosine as low as 0.51 for the embeddings in Table 3). They do not show that training with the truncated gradient reaches the same quality as full backpropagation (K=8), which is the experiment that would support the claim.
  • The two measures used for investigating the basin effect in lines 236-238 are redundant and the authors don't really talk about what differentiates them.
  • The entirety of section 5 either involves relevant results that are present in related works, or is related to hyperparameter optimization. Therefore, I believe most of the content belongs to the appendix section.
  • In 4 out of the 8 benchmarks in Figure 4 HRM-text performs better. And on MMLU the margin is 2.5 points. Without multiple seeds or any variance estimate, I'm not sure this supports the claim that a single stream is enough.

References

[1] Geiping, J., McLeish, S., Jain, N., Kirchenbauer, J., Singh, S., Bartoldson, B., Kailkhura, B., Bhatele, A. and Goldstein, T., 2026. Scaling up test-time compute with latent reasoning: A recurrent depth approach.

[2] Wang, G., Li, J., Sun, Y., Chen, X., Liu, C., Wu, Y., Lu, M., Song, S. and Yadkori, Y.A., Hierarchical reasoning model.

[3] Jolicoeur-Martineau, A., 2025. Less is more: Recursive reasoning with tiny networks.

[4] Alabdulmohsin, I. and Zhai, X., 2026. Recursive inference scaling: A winning path to scalable inference in language and multimodal systems.

[5] Geng, Z., Zhang, X.Y., Bai, S., Wang, Y. and Lin, Z., 2021. On training implicit models.

[6] Prairie, H., Novack, Z., Berg-Kirkpatrick, T. and Fu, D.Y., 2026. Parcae: Scaling laws for stable looped language models.

[7] Ge, R., Liao, Q. and Poggio, T., 2025. Hierarchical reasoning models: Perspectives and misconceptions.

[8] Rodrigues, L.R. and Kang, J., 2026. Dissecting Hierarchical Reasoning Models: A Mechanistic Study.

Questions for authors
  • Figure 2 shows the effect of extra L-steps and extra stages, and Table 10 shows the effect of extra H-steps after the final H2 call. So the authors use this to make the claim that the L-module is doing looping but not the H-module. It's somewhat convincing. But I want to also know what would happen with extra intermediate H-steps. So do 2 or more intermediate H-steps (between the two L stages) instead of the customary 1.
  • Table 9 suggests the remaining changes in the L-module output are bounded rather than converging: the cumulative drift saturates at 0.13 while the step residual stays at 0.016, and the cross-initialization spread slightly increases after 10 calls. Is this an oscillation within a region? If so, I'm not sure you can really claim the representation converging to a basin, as opposed to staying within one.
4 comments
AuthorsOpenReview ↗

Response to Reviewer xLEv (Part 1)

Thank you for the thoughtful review and detailed feedback. We address your concerns and questions below.

W1. Novelty and relationships to related work

The related works provided by the reviewer do propose some similar architectural and training ideas, but our contributions go beyond these existing studies. Our central contributions are what the L/H modules actually compute and which components are necessary; why truncated gradients are effective in recurrent language models and how they can be trained efficiently; how much of these gains depend on the training data and how well they generalize. These empirical findings and their connections are not established by the cited works. We clarify the differences below.

Huginn [1] and RINS [4]

We agree that Huginn's coda module and our terminal H module play a similar structural role. But our contribution is not this structural pattern, but the empirical study of its computational role in HRM-Text: does it continue the L-module's refinement, or transform the latent state in a different way? We test this through latent-state comparisons, fixed-conditioning continuation, and interventions on extra calls.

We explicitly acknowledge that LnHL^nH follows the ArBA^rB structure of RINS. Our design is motivated by our analysis of L/H latent dynamics rather than simply adopting RINS. By removing the unnecessary components in HRM-Text, we arrive at a RINS-style architecture. Our contribution is to provide a mechanistic explanation for why this simple architecture can match HRM-Text performance. We also examine two important design and training choices: input injection and the truncated gradients used to train it (Appendix C.5 isolates terminal injection; Section 5.1 measures gradient agreement).

HRM related work [2][3][7][8]

HRM-Text differs materially from the HRM setting in [2], [3], [7], and [8]. These works focus on symbolic or grid-based reasoning, whereas HRM-Text is a conditional language model trained with autoregressive response generation, without the cross-segment deep supervision or ACT used in original HRM. Beyond this difference, our work investigates distinct questions about recurrent dynamics and architectural design, as detailed below.

HRM [2] and TRM [3]. We agree that L-module convergence is not a new idea. HRM [2] studies local L fixed points under fixed H conditioning, as well as higher-level H fixed points (primarily to motivate its one-step gradient approximation). TRM [3] questions whether HRM's finite iterations actually reach these fixed points. However, HRM-Text differs from the original HRM in its training procedure without cross-segment deep supervision. Therefore, the higher-level H fixed-point analysis in HRM does not directly apply to this setting. Our novel observation is that the two rounds of L-module updates converge toward distinct input-conditioned basins, while the H-module performs a qualitatively different transformation rather than continuing the same iterative refinement. These distinct L/H dynamics motivate us to validate architectural simplification that is not addressed by either work.

Ge et al. [7] and Rodrigues and Kang [8]. Ge et al. [7] investigate whether HRM's hierarchical architecture is necessary by directly removing H and expanding L. Our objective is not simply to investigate flat looping as an alternative to HRM/HRM-Text. On the contrary, our results show that flat looping alone underperforms HRM-Text, which motivates us to further study which components need to be preserved.

Rodrigues and Kang [8] investigate the functional roles of H and L in the original HRM through causal interventions. They find that functional roles of HRM components vary across tasks since interventions applied to L and H modules show different behaviors in different tasks. However, their analysis focuses on symbolic reasoning with deep supervision across multiple segments, whereas HRM-Text uses a different training and execution setting. Unlike our findings in HRM-Text, their results lead to different conclusions about how the modules function. Thus, our work investigates different questions about H/L behavior and reaches different conclusions. We also note that Rodrigues and Kang [8] is a recent, contemporaneous study. It was presented at the ICML 2026 Mechanistic Interpretability Workshop and submitted to arXiv on August 30, 2026, around the completion of our work.

AuthorsOpenReview ↗

Response to Reviewer xLEv (Part 2)

Implicit-gradient work [5]

Geng et al. [5] study approximate gradients for implicit equilibrium models. We already ground our analysis in equilibrium and recurrent-backpropagation literature, including Bai et al. (2019) and Liao et al. (2018), and have added this closely related reference in Appendix D.1. Our setting is different: the recurrent models we study use a fixed number of forward steps, not solving for equilibrium states. We therefore compare truncated gradients directly with exact backpropagation. The IFT-based comparison is only used as an additional diagnostic. While gradient approximation itself is well studied, our experiments examine how well truncation works in these recurrent language models.

Parcae [6]

Parcae [6] studies how looped language models scale with recurrence depth, training data, and compute. It is already mentioned in Appendix B.2. Our work does not investigate scaling laws. Our comparisons focus on architectural differences and training ideas. Therefore, the reviewer's concern about scaling laws is not really relevant to our work.

We acknowledge the relevance of these prior works. We have expanded Appendix B.3 to clarify the relationship between HRM, TRM, and HRM-Text. However, our work goes beyond the existing ideas. We study what the L/H modules actually compute in HRM-Text, use these findings to motivate a simpler architecture, and evaluate its performance and training efficiency through controlled experiments. These questions and findings are not addressed by the cited works.

AuthorsOpenReview ↗

Response to Reviewer xLEv (Part 3)

W2. Baseline tuning, normalization boundaries, and optimizer confounding

Training configurations

We agree that different architectures may benefit from different hyperparameters, and independently tuning each baseline could change their relative performance. For Table 1, we used the original HRM-Text training recipe for all models to keep the comparison consistent and focus on architectural differences. We acknowledge that the results might differ if each model's hyperparameters are tuned individually. The common settings we used for models in Table 1 are:

SettingOriginal-recipe architecture comparisons
OptimizerAdamATan2
Learning rate2.5×10−42.5\times10^{-4}
LR warmup and subsequent schedule2,000 steps, then constant
Betas; weight decay(0.9,0.95)(0.9,0.95); 0.1
Global batch172,032 token positions
Training budgetFour epochs, approximately 60B processed positions
Precision; EMA; evaluated checkpointBF16; 0.9999; epoch-4 EMA

We would like to clarify that the reviewer's concern about the omitted optimizer sweep history in Appendix E is based on a misunderstanding. Appendix E discusses the WSD50–Muon optimizer exploration in Section 5.2, not the architecture comparisons in Table 1. The Table 1 baselines used the original fixed training recipe without optimizer sweeps.

Normalization boundaries

We follow HRM-Text's MagicNorm design across all architecture controls: each module contains PreNorm Transformer layers and ends with an additional post RMSNorm. Under this shared normalization rule, the standard looped Transformer has normalization boundaries every 24 layers and HRM-Text has them every 12 layers. We acknowledge that this fixed normalization rule may not be optimal for every architecture. Independently tuned normalization for each architecture might improve individual baselines.

For the standard Transformer controls, we explicitly test both normalization choices because standard Transformers do not necessarily benefit from the additional post normalization that can help stabilize repeated computations in looped models. In Table 1, the modular baseline retains MagicNorm at every 12-layer boundary, which is consistent with the HRM-Text design. The monolithic baseline uses only internal PreNorm without these post norms. The result shows that the monolithic model performs substantially better (65.15 vs. 58.16), so we report both rather than relying on the less favorable normalization choice.

Architecture versus optimization

We do not attribute the 3.5–5.7 point optimizer gains to the L7HL^7H architecture. Our paper separates architectural and optimization effects through two comparisons. In Section 4, we compare HRM-Text and L7HL^7H under the same original training recipe and show that L7HL^7H can simplify the architecture while achieving comparable performance. In Section 5.2, we separately evaluate the effect of changing the optimization recipe while keeping each architecture and its backward policy fixed.

These two findings are also explicitly distinguished in our abstract. The 3.07-point gap between HRM-Text and the monolithic Transformer in Table 1 was measured under the same original training recipe. The optimizer gains may be larger than the 3.07-point gap in Table 1, but we have never claimed that these gains come from the L7HL^7H architecture itself.

W3. What the gradient claim establishes

We do not claim that truncated backpropagation achieves the same final model quality as full backpropagation (K=8K=8). Our claim in Line 67 is that truncated gradients remain directionally informative relative to exact finite-depth gradients. Even the lowest 0.51 embedding-gradient cosine still shows positive alignment, which is consistent with what we claim. Achieving comparable final performance to full gradients is a different claim from the one we make.

W4. Redundancy of the two basin-distance measures

We agree that the two metrics are closely related. However, they measure different aspects of state differences: cosine distance measures differences in direction, while relative Euclidean distance also captures differences in magnitude.

For equal-norm states, the pairwise quantities satisfy

Drel2=2Dcos⁡.D_{\mathrm{rel}}^2 = 2D_{\cos}.

We report both metrics to show contraction in terms of direction and overall state displacement. We have clarified their relationship in Appendix C.1, with a cross-reference in Section 3.2.

AuthorsOpenReview ↗

Response to Reviewer xLEv (Part 4)

W5. Why retain training results in Section 5?

We respectfully disagree that Section 5 simply repeats findings from prior work or only reports hyperparameter optimization. It presents three sets of empirical findings:

  • Section 5.1: We investigate whether truncated gradients remain aligned with full finite-depth gradients, and why this alignment may persist. We first test this by measuring gradient cosine similarity in recurrent models, and then use the observed L-module contraction and the IFT analysis to explain why this alignment may persist. The differences from prior work are discussed in our response to W1.

  • Section 5.2: We train and evaluate HRM-Text and L7HL^7H under an improved optimization recipe, observing gains of 3.50–5.66 points. This is a controlled evaluation of training-recipe effects and resulting model performance. This is not merely the hyperparameter optimization described by the reviewer. Detailed optimizer configurations are provided in the appendix.

  • Section 5.3: We provide controlled measurements of memory and training time, isolating the effects of parameter sharing, retained-gradient depth, and activation recomputation. At matched executed depth, L7HL^7H takes 52.5 hours versus 106.2 hours for the monolithic standard Transformer. These experiments identify why shorter backward graphs reduce training cost in our production setting.

Therefore, these results show how recurrent computation can be trained effectively and efficiently. While related methods have been studied previously, the empirical findings and analyses presented here are not established by those works. We therefore believe Section 5 provides important evidence supporting the paper's contributions.

W6. Per-task trade-offs and the lack of training-seed uncertainty

The reviewer's concern about per-task trade-offs again assumes a stronger claim than we make. We do not claim that L7HL^7H consistently outperforms HRM-Text. Our claim is that removing the two-stream hierarchy and intermediate H updates can preserve comparable performance. In Figure 4, each architecture performs better on four of eight benchmarks, and their average scores are nearly identical. These results directly support our claim. Appendix C.4 further shows that L7HL^7H preserves the L-refinement and H-transformation behaviors motivating the simplification.

Q1. What happens when intermediate H calls are repeated?

We thank the reviewer for this suggestion. We conducted the requested L3HmL3HL^3H^mL^3H intervention with m=2,3,5,10m=2,3,5,10, holding zL3z_L^3 fixed during the repeated intermediate H calls. As a compute-matched control, we also evaluated Lm+2HL3HL^{m+2}HL^3H, which adds the same number of L calls instead. All experiments use fixed weights from the released HRM-Text-1B checkpoint and the same 200-example diagnostic subset (25 examples per benchmark).

Intermediate H calls mmH-intervention mean (%)Matched extra-L mean (%)
1 (native)68.7468.74
252.6968.14
342.8469.74
534.2869.64
1028.2068.24

Repeating intermediate H calls sharply degrades performance, while adding the same number of L calls preserves performance. This extends the asymmetry observed in Figure 2 and Appendix Table 10 to the intermediate H position. The results support our interpretation that L supports repeated refinement and H performs a distinct transformation rather than beneficial iterative refinement.

Q2. Does Table 9 show convergence, or bounded motion within a region?

Convergence toward a basin and continued motion within it are not contradictory. Our claim is that L trajectories from different initial states converge toward a common region (a basin), not a single fixed point. As shown in Table 9, cross-initialization spread decreases from 1.0001 to 0.0611 over the three trained L calls and remains small (0.0181) after 100 additional calls. The nonzero step residual (0.0160) indicates continued motion within this region, which is fully consistent with our basin interpretation. Thus, the reviewer's suggested oscillation is compatible with our findings.

Reviewer_bC58OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

The authors investigate a particular looped transformer architecture (HRM-Text) and find that a different way of scheduling of looping blocks with unified state works better. They also investigate truncating back propagation.

Critical strengths

None

Critical weaknesses

Even after careful reading I have a hard time understanding what the "message" of this work should be. I. The experiments seem very specific to a particular prior work (HRM-Text) that is not particularly well-known/widely used. Looping only some of the blocks (with unlooped prelude&coda) has been widely explored, in all its variants; same for truncated backpropagation. But this work is so hyperfitted to HRM-Text that it lacks generality. The takeaways in the text are also overly specific/not general (...so they are not really takeaways).

E.g.:

Takeaway: Under WSD50–Muon, L7H’s margin over HRM-Text grows with shift difficulty (58.7 vs. 47.1 on GSM-Sym. p2), but broader transfer is mixed, and the recipe lowers IFEval and BBH.

What kind of takeaway is this? What does the reader learn from this?

Generally this work makes a very AI-generated impression, it is overly bloated and lacks a central research hypothesis that is investigated.

Questions for authors

Please clarify what the main statement/contribution of this work is.

3 comments
AuthorsOpenReview ↗

Response to Reviewer bC58 (Part 1)

Thank you for your feedback and we would like to clarify the central statement, the contributions of our work, and their relevance beyond HRM-Text.

Motivation and Contributions

Looped transformers have attracted growing attention recently, and HRM-Text is a particularly interesting model to study. Beyond the discussion around it in the community, several interesting findings and surprising results motivate our investigation. Compared with standard looping, HRM-Text uses a more complex H/L architecture inherited from the Hierarchical Reasoning Model (HRM). More surprisingly, it achieves better performance with a relatively limited amount of instruction-response training data, without conventional large-scale language model pretraining. These observations leave multiple questions open and motivate us to take a closer look at what makes HRM-Text work so well.

Architecture. We first investigate why its architecture works and which components are actually necessary. While previous HRM/TRM studies have examined related fixed-point behaviors, they are focused on symbolic reasoning models and do not explain the computational roles of L-module and H-module in HRM-Text. Therefore, we conduct extensive experiments on the latent states to understand what these modules actually compute. We observe that repeated L-module calls contract the state into an input-conditioned basin, and the H-module performs a qualitatively different transformation rather than continuing the same iterative refinement. These findings naturally motivate us to remove the unnecessary components such as the two-stream hierarchy and intermediate H-module update, leading to the simpler RINS-style LnHL^nH architecture with matched performance.

Training. Given such a recurrent architecture, we ask how to train it more effectively and efficiently. Since looping creates a long backward computation graph, we ask: do we really need to backpropagate through every recurrent step, or can we truncate the gradients while still learning effectively? Why is it reasonable to use truncated BPTT? We investigate this by comparing truncated gradients with full backpropagation to show how well they align. Based on the recurrent dynamics we observed before, we provide a plausible derivation to show why they align. We then quantify how much training time can be saved through this approach compared with standard Transformers. Beyond training efficiency, we also explore how to further improve model performance and propose our WSD50-Muon training recipe.

Data. Finally, we also want to understand how much of HRM-Text’s success can be attributed to its training data. Its strong benchmark results motivate us to ask whether these gains reflect broader generalization abilities or are just closely related to the data it was trained on. We examine the training data mixture they released and conduct a more comprehensive evaluation. Specifically, we distinguish source-matched performance (in-distribution), controlled distribution shifts, and broader generalization (out-of-distribution). The results show where the gains persist and where transfer remains limited.

We believe this provides a clear and coherent research progression, from understanding the latent dynamics behind a representative recurrent language model to simplifying its architecture, improving training, and examining the role of training data in model performance. We therefore respectfully disagree with the reviewer's point that our work is "AI-generated" and lacking a central research hypothesis. We acknowledge that some parts of the presentation could be improved. However, we do not see why the breadth of our experiments and findings should be considered "overly bloated" and "AI-generated".

AuthorsOpenReview ↗

Response to Reviewer bC58 (Part 2)

Generality beyond HRM-Text

Although HRM-Text serves as our main testbed, the questions we investigate are not specific to HRM-Text. Its design choices, such as the L/H hierarchy, staged updates, and truncated backpropagation make it particularly suitable for investigating the roles of each component in recurrent language models, how recurrent models should be trained, and how their performance depends on training data.

We do not claim that the latent dynamics in HRM-Text hold universally across all recurrent architectures. However, questions about the computational roles of different modules, which design choices are necessary, and how recurrent models can be trained efficiently are not specific to HRM-Text. Using a representative architecture can help us get useful design insights without requiring exhaustive experiments across every existing recurrent model.

Presentation

Given the comprehensive experiments in our work, our intention in including "Takeaways" was to help the audience follow the results and conclusions. We have revised selected takeaways to emphasize broader insights rather than individual numerical results. We have also revised the Introduction to better explain our motivation for studying HRM-Text and how our research questions relate to broader challenges in recurrent language modeling.

After reading the rebuttal I still do not see the particular research hypothesis that is investigated by this work -- improving architecture, training and data is overly general, and doing it on top of HRM-Text is overly specific. I still think that it does not make sense to perform this study based on HRM-Text, in particular, since one contribution of this work is to remove the structure that distinguishes HRM-Text from other looped models. Many of the touched topics in this work, like truncated BP and having unlooped blocks (prelude/coda) are widely known/studied and I don't see what this work adds to the field. I therefore keep my score unchanged.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.