- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper provides a thorough empirical and mechanistic audit of HRM-Text, a recently introduced structured recurrent language model. The authors conduct controlled interventions and latent-state analyses, revealing a functional asymmetry: repeated low-level -module calls act as a contraction operator pulling states toward an input-conditioned basin, while the high-level -module provides a distinct terminal transformation. Based on these insights, the paper advocates for a simpler, single-stream architecture (closely related to the RINS formulation). Combined with a modernized training recipe (WSD50-Muon) and truncated backward windows, the proposed model matches or slightly outperforms HRM-Text while cutting 60B-token training time by over 50% relative to standard depth-matched Transformers on hardware. The study concludes with a clear-eyed assessment of in-distribution versus out-of-distribution generalization.
Critical strengths
-
Valuable Empirical Debunking & Simplification: The paper successfully untangles the over-engineered components of HRM-Text. By demonstrating that the two-stream, two-stage hierarchy can be simplified into a single-stream structure without performance loss, it offers a much cleaner baseline for future research in recurrent architectures.
-
Rigorous Mechanistic and System Analysis: The latent-state dynamics (proving contraction into an input-conditioned basin via 256-start analyses) are meticulously executed. Furthermore, the production-level efficiency measurements (HBM consumption, training time reduction, and activation recomputation avoidance) are concrete and useful for the community.
-
Scientific Transparency: The authors honestly report mixed out-of-distribution transfer results and limitations regarding pretraining corpus breadth, rather than hiding negative findings behind cherry-picked benchmarks.
Critical weaknesses
-
Limited Architectural Novelty: The resulting architecture is structurally equivalent to the existing RINS () paradigm (Alabdulmohsin & Zhai, 2025). While the paper's contribution lies in identifying the missing ingredients (terminal injection and truncation warmup) that made prior RINS baselines underperform, the architectural leap itself is incremental.
-
Confounded Gains and Unexplained OOD Regression: The integration of the WSD50-Muon optimizer yields substantial improvements, but the paper lacks a standard Dense Transformer baseline under the exact same WSD50-Muon recipe to isolate what portion of the gain is uniquely tied to the recurrent architecture. Additionally, in Table 5, adopting WSD50-Muon leads to noticeable performance drops on broader reasoning/instruction benchmarks (e.g., BBH and IFEval). A deeper discussion on why this recipe exacerbates overfitting to source-matched patterns would strengthen the paper.
-
Presentation and Organization Flaws:1) The mathematical derivation in Appendix D.1 (IFT and Neumann series expansion) spans multiple pages of standard textbook calculus, which feels bloated given that the probed states are not strict one--step fixed points (as shown in Table 20, remains high). 2) The safety fine-tuning experiment in Appendix G.4 is somewhat abrupt and evaluated exclusively on HRM-Text rather than the core proposed model, giving a slight impression of a tacked-on ablation.
Questions for authors
-
OOD Degradation under Muon: Why does the WSD50-Muon recipe consistently degrade performance on certain broader transfer benchmarks like BBH and IFEval (Table 5)? Does the combination of Muon and recurrence accelerate overfitting to the source-matched training mixture?
-
Safety SFT Inconsistency: The safety adaptation experiment is only evaluated on HRM-Text. Can the authors provide corresponding safety-utility trade-off results for the proposed architecture under the same fine-tuning protocol?
-
Implicit Equilibrium Tightness: Given that Table 20 indicates the probed states deviate from exact fixed points, how sensitive are the directional alignment conclusions in Section 5.1 to the choice of the truncation window ?
3 comments
Response to Reviewer Jd5F (Part 2)
W2 / Q1. Recipe attribution and broader-transfer regressions
Recipe attribution.
We appreciate this point. Without a dense Transformer trained with WSD50–Muon, we cannot determine whether recurrent models benefit more from this recipe than dense models. But our claim is narrower: Section 5.2 reports improvements from changing the training recipe within each recurrent architecture, without attributing these gains specifically to recurrence. Separately, the architectural comparisons in Table 1 and Section 4 use the same original training recipe, so their conclusions do not depend on Muon.
OOD performance is task-dependent, not uniformly degraded.
Table 5 shows that WSD50–Muon improves some transfer benchmarks while reducing performance on others, with substantial differences across architectures and scales. The table below summarizes the changes relative to AdamATan2 (percentage points; IFEval averages 0-, 1-, and 3-shot results).
| Model | Scale | GSM-Symbolic p2 | MMLU | MMLU-Pro | BBH | IFEval |
|---|---|---|---|---|---|---|
| HRM-Text | 0.69B | +6.2 | +0.8 | −0.6 | +0.0 | −4.0 |
| 0.69B | +9.9 | +4.3 | +0.6 | +4.2 | −2.8 | |
| HRM-Text | 1.0B | +0.7 | −0.1 | −3.2 | −6.6 | −1.0 |
| 1.0B | +11.3 | +2.8 | +2.4 | −1.5 | −8.6 |
BBH: task trade-offs and answer-format sensitivity. The 0.69B models provide a clearer view of the variation within BBH. Both improve on logical deduction and multistep arithmetic, but regress on disambiguation and translation-error detection. Movie recommendation moves in opposite directions across the two architectures. Selected task accuracies are shown below (AdamATan2 → WSD50–Muon, %).
| BBH task | HRM-Text, 0.69B | , 0.69B |
|---|---|---|
| Overall | 36.89 → 36.91 | 36.61 → 40.81 |
| Logical deduction, 3 objects | 56.8 → 71.6 | 63.6 → 76.0 |
| Multistep arithmetic | 43.2 → 53.2 | 46.8 → 64.0 |
| Pronoun disambiguation | 61.6 → 56.8 | 63.6 → 48.4 |
| Translation-error detection | 22.4 → 13.6 | 42.8 → 34.0 |
| Movie recommendation | 24.8 → 33.2 | 36.0 → 29.6 |
Inspection of the results also reveals that some apparent BBH degradation comes from changes in answer format rather than reasoning accuracy. On web_of_lies, HRM-Text 1.0B under WSD50–Muon scores 2.8%, largely because it outputs True/False rather than the expected Yes/No (239 of 250 predictions). Mapping these semantically equivalent labels raises accuracy to 52.4%, compared with 52.0% under AdamATan2. This diagnostic alone accounts for approximately 1.9 percentage points of the reported 6.6-point BBH decline. Other BBH tasks also exhibit changes in response format. For example, the 1.0B model's movie-recommendation outputs shift from option letters toward descriptive responses. Thus, the recipe changes task-response behavior, while some task-level accuracy losses remain genuine.
IFEval: constraint-specific changes. Unlike answer-label mismatches in BBH, violations of IFEval constraints are genuine instruction-following failures. However, the effects differ substantially by constraint: both 0.69B models become less reliable on JSON formatting, word counts, and title formatting, while improving on multiple-section and placeholder-count requirements. Selected loose accuracies are shown below (AdamATan2 → WSD50–Muon, %).
| IFEval constraint | HRM-Text, 0.69B | , 0.69B |
|---|---|---|
| JSON format | 47.1 → 33.3 | 41.2 → 23.5 |
| Word count | 60.9 → 39.1 | 53.8 → 38.5 |
| All lowercase | 52.1 → 26.5 | 39.3 → 42.7 |
| Title format | 88.3 → 68.5 | 95.5 → 52.3 |
| Multiple sections | 16.7 → 31.0 | 23.8 → 50.0 |
| Number of placeholders | 35.8 → 45.7 | 30.9 → 44.4 |
Interpretation. These results suggest that WSD50–Muon affects not only task accuracy but also how models respond to different task requirements. The web_of_lies analysis shows that a change in answer format can substantially reduce measured accuracy without reducing the underlying judgment accuracy. Other BBH tasks exhibit genuine performance trade-offs, while IFEval shows that compliance with some constraints improves and others deteriorates. A plausible explanation is that changing the optimization recipe shifts the response patterns and task behaviors learned from the same instruction–response mixture. Such changes may benefit tasks aligned with those patterns while transferring less effectively to unfamiliar task formats or strict constraints. This points to task-dependent generalization trade-offs rather than a uniform loss of capability, and does not by itself indicate that Muon and recurrence accelerate overfitting.
Response to Reviewer Jd5F (Part 3)
W3 / Q2. Safety adaptation of
The submitted appendix reports HRM-Text only, and we agree that those results cannot establish -specific transfer. We have also conducted additional experiments under the AdamATan2 safety-SFT protocol and included the results in our revision. These use the same deterministic data splits, three-epoch fine-tuning budget, batch size of 32,768 tokens, learning rate of , and final epoch-3 EMA evaluation rule. The model is the 0.69B checkpoint; utility is the unweighted eight-benchmark mean, and safety is on 320 HarmBench direct requests classified by WildGuard.
| fine-tuning data | Utility | HarmBench safety | Harmful responses |
|---|---|---|---|
| No SFT | 68.40 | 29.06 | 227 / 320 |
| WildJailbreak | 68.84 | 96.88 | 10 / 320 |
| WildGuardMix | 69.05 | 92.19 | 25 / 320 |
| Equal mixture | 68.84 | 97.19 | 9 / 320 |
These results extend the domain-transfer observation to : harmful responses decrease substantially while aggregate general-task utility is retained. The safety fine-tuning experiment is part of our domain-transfer analysis in Section 6, rather than a separate safety-focused study. We chose safety because safety-focused data are absent from our pretraining mixture, making it a suitable test of adaptation to an unseen domain; other unseen domains could serve the same purpose.
W3 / Q3. Non-equilibrium states and sensitivity to the backward window
We appreciate the reviewer's suggestions and agree that Appendix D.1 can be more concise. We also agree that Table 20 rules out treating the probed states as exact one-step fixed points. Section 5.1 contains two distinct comparisons:
- Exact finite reference: truncated gradients are compared directly with full backpropagation through the same executed finite program. This comparison requires no fixed-point assumption.
- IFT-style local reference: an implicit response is constructed at the observed states. This is an equilibrium-motivated diagnostic, not the exact gradient of our finite model.
The window sweeps directly address sensitivity. For HRM-Text's L parameters, Appendix Table 22 reports the following cosine ranges across checkpoints after epochs 1–3, with ten matched batches per checkpoint:
| Backward window | Against exact full finite BP | Against IFT-style local reference |
|---|---|---|
| 2 | 0.654–0.723 | 0.634–0.748 |
| 3 | 0.654–0.723 | 0.634–0.748 |
| 4 | 0.777–0.835 | 0.708–0.831 |
| 5 | 0.824–0.877 | 0.703–0.832 |
Here follows HRM-Text's actual backward policy, not an arbitrary contiguous tail: the retained calls are respectively , , , and . The identical L-gradient entries at reflect this policy. Agreement with full finite BP improves as more L calls are retained, whereas agreement with the local implicit reference need not increase monotonically (which is reasonable because fixed points are not assumed). The sweep in Tables 25–26 gives the same important distinction. All the truncation windows preserve the positive directional agreement with the exact finite reference and IFT-style local reference.
Response to Reviewer Jd5F (Part 1)
Thank you for the constructive feedback and recognition of our contributions. We address the concerns below with clarifications and additional experimental results.
W1. Architectural novelty and the relationship to RINS
We acknowledge that LnH follows the ArB structure of RINS. However, our design is motivated by our analysis of L/H latent dynamics rather than simply adopting RINS. The interventions motivate a single-stream alternative, which reaches 68.40 versus HRM-Text's 68.22 at matched parameter and executed depth under the original recipe. We interpret this as comparable aggregate performance without the hierarchy. Thus, our contribution is to provide a mechanistic explanation for why this simple architecture can match HRM-Text performance. We also examine two important design and training choices: input injection and the truncated gradients used to train it (Appendix C.5 isolates terminal injection; Section 5.1 measures gradient agreement).