acceptodds
Under review as a conference paper at ICLR 2027

LRFeed: Latent Residual Feedback for Efficient Reasoning in Looped Transformers

Abstract

Looped Transformers have emerged as a promising class of models for solving complex reasoning tasks. However, maintaining effective computation over long reasoning horizons remains challenging for these models. To tackle this problem, we introduce **Latent Residual Feedback (LRFeed)**, which captures how the model's internal latent state evolves and feeds this information back to guide subsequent computation. This feedback helps provide awareness of the reasoning process and enhance reasoning efficiency. We instantiate LRFeed on a single-state variant of the Tiny Recursive Model (TRM), retaining its shared Transformer backbone. Ablation studies show that LRFeed strengthens long-horizon reasoning, even beyond the training horizon. It also promotes more diverse internal reasoning dynamics, reflected in a higher effective rank of channel activity across iterations. Across Sudoku-Extreme, Maze-Hard, and ARC-AGI-1, LRFeed surpasses TRM using less than one-eighth of its inference compute. On Sudoku, LRFeed reaches 97.07% accuracy with extended inference, exceeding TRM by 16.22 percentage points at matched compute. LRFeed also offers a favorable accuracy–compute trade-off against other compact looped Transformers.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Looped Transformers, Latent Reasoning, Residual Feedback, Test-Time Computation, Recurrent Inference

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_QsdwOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LRFeed conditions a looped Transformer's shared computation on its latest latent update. On a single-state recurrent backbone, it pools residual energy across positions, normalizes the channel profile, and uses a small shared MLP to modulate subsequent Transformer blocks. The most detailed experiments are on Sudoku, with matched-backbone and constant-input controls, continuous long-horizon inference, and measured timing. They provide substantial evidence of improved accuracy and inference-time scaling beyond training depth. Maze and ARC broaden the evaluation, while prediction and dynamics analyses offer descriptive evidence about the feedback signal.

Critical strengths
  1. Matched Sudoku comparisons show a substantial benefit: at 128 steps, full-test accuracy rises from 79.05% without feedback to 93.61%. The constant-input control also separates informative feedback from added parameters alone.
  2. Continuous endpoint evaluation supports sustained gains beyond training depth, complemented by replication subsets. Extending inference to 2,048 steps yields 97.07% mean accuracy across three seeds.
  3. The module is compact, and measured timing exposes the practical trade-off: equal-depth inference has overhead, but fewer feedback steps can outperform deeper no-feedback inference.
Critical weaknesses
  1. W1 — Residual-specific conditioning is not fully distinguished from other informative inputs. Figure 9's constant-input control shows that modulation parameters alone do not explain the full gain. However, it does not compare the residual with state-energy or phase-based inputs using the same MLP and gates. Table 2's controlled prediction test establishes additional information about future update energy, rather than a task-level advantage over alternative conditioners. Thus, the value of input-dependent modulation is better established than the value specifically attributable to recent residual information (p.7, pp.23–24).
  2. W2 — The compressed feedback's relationship to reasoning progress remains indirect. Pooling removes position/sign information and normalization suppresses overall magnitude, so distinct local or directional updates can produce the same relative channel profile (Equation 4, p.5). The full state remains in the backbone, and near-zero attenuation retains some magnitude information; this restricts the added feedback channel. The authors acknowledge this boundary (p.9, §D.1). Their prediction test and descriptive state/output-profile rank analyses are useful, but do not directly examine when the actual residual profile and gates track useful progress rather than repeated or alternating activity. This limits the mechanistic interpretation of the long-horizon gains, without invalidating them or establishing that a richer feedback design would be better.
  3. W3 — The ARC accuracy–compute conclusion has weaker verification support. Table 7 compares 45.1% with a published TRM reference of 44.6%, without a reproduced TRM ARC curve or multi-seed uncertainty for LRFeed on ARC (p.21). The stated common augmentation/voting protocol and per-view accounting are helpful, but the small accuracy gap is less secure than the matched Sudoku result. The ARC checkpoint's training and selection recipe is also less explicit than the Sudoku recipe (§B.1, §C.3), making the comparison harder to verify. This narrows the support for an all-benchmark efficiency conclusion rather than challenging the core Sudoku gains.
Questions for authors
  1. Q1 — Task-level conditioning comparison (W1). With the same MLP, gates, training, and budget accounting, how does the residual input compare with an informative current-state RMS or phase-based input? Please also provide the exact constant-input values underlying Figure 9, so the incremental gain can be assessed without estimating points from the plot.
  2. Q2 — Feedback along actual trajectories (W2). On representative successful and unsuccessful long rollouts, how do the residual profile and resulting gates relate to answer progress and repeated or alternating updates? A trajectory-level diagnostic using the existing model would clarify the practical significance of the acknowledged compression boundary, without requiring a redesigned feedback architecture.
  3. Q3 — Evidence underpinning the ARC comparison (W3). What training/checkpoint provenance and uncertainty evidence support the 45.1% versus 44.6% comparison? Please identify the evaluated recipe, actual augmentation/view count, and the basis for assessing stability under the stated common protocol.
5 comments
AuthorsOpenReview ↗

Response to Reviewer Qsdw—Part 1/3

Dear Reviewer Qsdw,

Thank you for recognizing the Sudoku results. We added conditioning controls and trajectory diagnostics, clarified the ARC training and evaluation settings, and evaluated an open-source TRM checkpoint at multiple inference budgets.

W1 & Q1: The existing controls do not establish the task-level benefit of residuals over other informative inputs. How do residual, state-RMS, and phase inputs compare under matched controllers and budgets, and what are the exact constant-input results?

Thank you for this helpful suggestion. It helps us make the experimental evaluation more complete and strengthen the paper.

TL;DR: We added matched comparisons with current-state RMS and phase inputs and report exact accuracies. LRFeed outperforms all four controls at every tested budget, including both informative alternatives.

Detailed Response:

The conditioning variants change only the input to the feedback MLP. Let R(v)R(v) denote channel-wise RMS pooling over the SS positions, and let N\mathcal{N} denote the shared normalization and scaling:

R(v)c=1S∑i=1Svi,c2.R(v)_c = \sqrt{\frac{1}{S}\sum_{i=1}^{S}v_{i,c}^{2}}.

N(v)=12RMSNorm⁡(v).\mathcal{N}(v) = \frac{1}{\sqrt{2}}\operatorname{RMSNorm}(v).

The MLP inputs are:

  • Constant: N(1)\mathcal{N}(\mathbf{1}).
  • Phase: N(SinEnc⁡(k))\mathcal{N}(\operatorname{SinEnc}(k)).
  • State-RMS: N(R(zt))\mathcal{N}(R(z_t)).
  • LRFeed: N(R(zt−zt−1))\mathcal{N}(R(z_t-z_{t-1})).

For Phase, SinEnc⁡\operatorname{SinEnc} is sinusoidal encoding and k=0,…,9k=0,\ldots,9 is the atomic-transition position within each outer step, restarting at zero in the next outer step. All four variants retain the same MLP architecture, channel gates, modulation scale, and training policy.

Each variant is trained separately. The table below reproduces the full-test Sudoku accuracies (%) in Table 3 of the revised paper, including the constant-input results:

Conditioner16 steps3264128
LRFeed (w/o feedback)71.1674.7377.1579.05
Constant78.7283.3586.6689.16
Phase81.7586.2089.1691.23
State-RMS81.6886.1289.1091.20
LRFeed83.1888.2491.4893.61

At 128 steps, LRFeed exceeds State-RMS by 2.41 percentage points and Phase by 2.38 points. These results support the value of recent residual information beyond constant, current-state, or iteration-position conditioning.

We have replaced the original Figure 9 and its accompanying appendix discussion with this more complete comparison in Section 4.3 (Table 2), including the exact accuracies.

AuthorsOpenReview ↗

Response to Reviewer Qsdw—Part 2/3

W2 & Q2: How do residual profiles and gates relate to solving progress along successful and unsuccessful trajectories?

Thank you for this valuable suggestion. It helped us make the trajectory analysis more direct and informative.

TL;DR: We now summarize these observations in Section 4.4 of the revised paper and present the trajectories in Figure 11 (Appendix D.5). Their averages show a clear transition around the first correct answer, while unsolved trajectories remain comparatively stable in the observed window.

Detailed Response:

We recorded 128-step rollouts on 3,072 Sudoku puzzles. For the 2,861 solved trajectories, we examine eight steps before and after the first correct answer. As a reference, we use an equally long window at the end of the 211 unsolved trajectories (steps 112–128). Figure 11 in Appendix D.5 (p. 30) shows the corresponding group means. Missing observations are omitted.

  • Residual profiles: Changes between consecutive profiles peak around solving and then settle to a lower level. Channel participation briefly decreases and rebounds.
  • Gates: The mean gate decreases after the first correct answer and stabilizes. Changes between consecutive gates peak about two steps afterward.

In contrast, the unsolved group shows no comparable transition over steps 112–128. These observations provide a concrete view of how the residual profiles and resulting gates change around solving.

AuthorsOpenReview ↗

Response to Reviewer Qsdw—Part 3/3

W3 & Q3: The small ARC advantage needs clearer verification. What training recipe, augmentation count, and uncertainty evidence support the comparison with TRM?

Thank you for highlighting this gap. We agree that the original appendix described Sudoku in detail but did not present the Maze and ARC protocols as clearly.

TL;DR: In the revised paper, we have reorganized Appendix B.1 to describe the shared training procedure and the protocols for all three benchmarks, including ARC checkpoint selection and evaluation view counts. We also report a TRM checkpoint-based accuracy–compute comparison below. Additional ARC training runs were not feasible given their substantial computational cost.

Detailed Response:

Training and checkpoint selection. The reported LRFeed ARC results come from a model trained with residual feedback. It shares the core training procedure with our other benchmarks, with the following ARC settings:

ARC training settingValue
Training duration / global batch size100,000 epochs / 448
Backbone / puzzle-embedding learning rate10−410^{-4} / 10−210^{-2}
Weight decay0.1 for both parameter groups
Warmup / EMA decay2,000 updates / 0.999
Training horizon / exploration16 outer steps / original ACT exploration of 0.1

We use the final ARC training checkpoint and evaluate that same checkpoint across inference budgets.

Evaluation protocol and view counts. LRFeed follows TRM's ARC evaluation protocol: the same test tasks, test-time augmentation, inverse transformation, voting, and pass@2 scoring. Evaluation covers 400 tasks with 419 test inputs. We request 1,000 augmented views plus the original input; after deduplication, this gives 385,815 evaluated views in total, with at most 1,001 views per input. Both models use the same evaluated views. Compute is measured in matrix TFLOPs per augmented input.

Additional TRM comparison. We evaluated an open-source TRM reproduction checkpoint released by alphaXiv at multiple inference budgets, as shown below.

Model / metric1 outer step24816
LRFeed accuracy (%)43.8045.1046.0046.1046.90
LRFeed matrix TFLOPs0.1590.3180.6371.2742.548
Public TRM accuracy (%)41.1343.0043.1343.1343.13
Public TRM matrix TFLOPs0.2390.4780.9551.9113.822

LRFeed reaches 45.10% at 0.318 TFLOPs, while this TRM reproduction reaches 43.13% at 0.955 TFLOPs and remains at that accuracy with further computation. LRFeed reaches 46.90% with additional compute. This checkpoint-based comparison also supports LRFeed's accuracy–compute advantage.

Uncertainty. ARC results currently use one training seed; additional independent runs were not feasible given the substantial computational cost of ARC training.

Appendix B.1 now presents the shared training procedure followed by separate Sudoku, Maze, and ARC protocols. Appendix C.3 retains the sources and comparison conventions for external results.

Reviewer_87khOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a novel adaptation for Recurrent Reasoning Models called LRFeed. Previous Reasoning Models use a latent memory that persists across iterations but are unable to directly use the change in the memory per step as additional information. LRFeed tracks this change and uses it as an additional model input.

LRFeed is tested on the standard suite of Sudoku, Maze, and ARC-AGI. They show strong performance compared to other RRMs and performance gains in their ablations.

Critical strengths

RRMs are a currently highly researched domain and the authors provide a simple but effective tweak to the architecture. The paper is well-written and shows performance gains on Sudoku. The key motivation of the paper is sound.

Critical weaknesses

I find the paper to generally be strong. However, I have some concerns about the experiments and require clarifications for the setup.

The authors use a single latent state, which diverts from the commonly used formulation of zLz_L and zHz_H. Appendix A aims to motivate this choice, but I find the section to be unclear at several points. As I understand the first part of it, the authors simply restate, in unnecessary complexity, that zHz_H does not change during L-steps, which is obviously true and the whole point of the setup. Figure 7 provides a slightly better motivation, as the single-state setup has smoother update magnitudes, but I still find the motivation unconvincing. I would have liked to see a simple ablation of using two states in LRFeed.

Table 1 has multiple issues:

  • Six rows are spent on LLMs, which are completely uninformative for Sudoku and Maze
  • It contains several entries that were reruns by the authors, but these are not reported with standard deviations
  • The table omits explicit mentioning of outer step numbers, which obscures the compute-accuracy comparison
  • Columns are not well-aligned, which hurts readability

Lastly, I do not see the motivation for Table 2.Using a linear probe to predict the residual seems disconnected from the rest of the paper to me. While it does show that there is a relation between consecutive updates, I find this unrelated to the goal of good predictive performance.

Questions for authors
  1. Performance gains are much more unclear for Maze and ARC than for Sudoku. Can you identify why that is? You mention that the former two benefit less from additional depth but the connection to that is unclear to me.
  2. Several hyperparameters, like alpha, are reported without reasoning behind them. How did you optimise these?
  3. Can you provide an anonymous codebase to ensure reproducibility?

Experiments:

  1. Is it possible to conduct an experiment with two latent states, as in TRM?
10 comments
AuthorsOpenReview ↗

Response to Reviewer 87kh—Part 1/6

Dear Reviewer 87kh,

Thank you for your constructive review.

W1 & Experiment 1: Why use a single latent state? A two-state LRFeed ablation would help justify this choice.

Thank you for suggesting this direct comparison. We agree that this simple ablation is more convincing than the explanation in Appendix A.

TL;DR: We added a two-state LRFeed ablation to Appendix A.1 (Table 3) of the revised paper. Single-state LRFeed achieves higher accuracy across four inference budgets, supporting our choice of the simpler backbone.

Detailed Response:

LRFeed (two-state) adopts TRM's alternating updates between two latent states. Both variants retain residual feedback and follow our LRFeed training recipe. We compare Sudoku exact accuracy (%) on the full test set at matched matrix compute (TFLOPs per puzzle):

Model0.2860.5721.1441.716
LRFeed (single-state)85.4189.6992.4493.57
LRFeed (two-state)76.2082.1186.1888.07

This comparison supports our choice of a single-state backbone. We have replaced the previous explanation with these ablation results in Appendix A.1 (Table 4) of the revised paper.

AuthorsOpenReview ↗

Response to Reviewer 87kh—Part 2/6

W2: Table 1 needs a clearer presentation: the LLM results take up too much space, reproduced results lack standard deviations, inference step counts are missing, and the columns are difficult to read.

Thank you for these concrete suggestions. They help us make the comparison easier to read and interpret.

TL;DR: We have updated Table 1 of the revised paper following your suggestions.

Detailed Response:

(a) LLM rows. We included these results to illustrate that these tasks remain challenging for LLMs. We agree that devoting six rows to these uninformative comparisons takes too much space and have removed them from the main table.

(b) Standard deviations. We have completed the additional runs needed to report three-run statistics for all four HRM/TRM entries on Sudoku and Maze.

(c) Inference steps. We used FLOPs because a step can contain different amounts of computation across architectures. We now also list each method's native step cap, using outer steps for HRM, TRM, and LRFeed. Multi-candidate methods include the candidate count.

(d) Layout. We have aligned the numeric columns and clearly separated accuracy, step caps, and compute.

These changes are incorporated in the revised Table 1.

AuthorsOpenReview ↗

Response to Reviewer 87kh—Part 3/6

W3: Predicting residuals seems disconnected from task accuracy.

Thank you for pointing this out and helping us sharpen the experimental argument.

TL;DR: We agree that predicting the next update does not establish a benefit to task accuracy. We have removed this analysis and focus on ablations that directly evaluate LRFeed's contribution to task performance.

Detailed Response:

We used this probe to motivate our choice of feedback signal. Adding residual profiles improved prediction of the next update over using a state summary alone. This suggests that residuals provide useful information about how the model updates its state. We agree that this finding does not establish a benefit to task accuracy. We have therefore removed the original Table 2 and the related analysis from the revised paper and focus on ablations that directly measure the benefit of feedback to task accuracy.

Reviewer_zVqJOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes a method called LRFeed, which captures how the internal latent states of looped Transformers evolve and feeds this information back to guide subsequent computation. It computes the residual between two consecutive latent states to obtain a compact profile, which is then mapped to generate channel gates. These channel gates allow subsequent iterations to adapt their computation based on previous state changes. The authors evaluate LRFeed on several reasoning benchmarks, comparing it with compact looped Transformers under different inference budgets and conducting ablation studies to examine the contribution of residual feedback.

Critical strengths
  • By using latent-state changes as feedback to adapt computation, the paper addresses the lack of explicit feedback on previous latent-state updates in looped Transformers.
  • The paper presents a clear approach to explicitly incorporating feedback into looped Transformers.
  • LRFeed summarizes the full latent-state residual into a compact channel-wise representation, introducing only a small number of additional parameters.
Critical weaknesses
  • The paper does not compare its design with alternatives. For example, it is unclear whether zt+1=Fθ(zt;ht)+e(x)z_{t+1} = F_\theta(z_t;h_t) + e(x) or other methods could yield better results.
  • Differences in training and evaluation settings make it difficult to assess LRFeed's performance relative to existing methods.
  • The effective-rank analysis is conducted only on Sudoku-Extreme, without evaluating Maze-Hard or ARC-AGI-1. It is unclear whether the findings generalize to other tasks.
Questions for authors

Can LRFeed generalize to other architectures, such as TRM-MLP or other looped Transformer models?

6 comments
AuthorsOpenReview ↗

Response to Reviewer zVqJ—Part 1/4

Dear Reviewer zVqJ,

Thank you for your careful review and constructive suggestions. Your comments have helped us make the paper more rigorous and complete. We address each point below.

W1: Could zt+1=Fθ(zt;ht)+e(x)z_{t+1}=F_\theta(z_t;h_t)+e(x), or other designs, yield better results than the proposed zt+1=Fθ(zt+e(x);ht)z_{t+1}=F_\theta(z_t+e(x);h_t)?

Thank you for suggesting this direct comparison.

TL;DR: We trained two alternatives while retaining residual feedback: post-transition input addition and learned concatenation fusion. The original input addition achieves higher accuracy at all four evaluated depths.

Detailed Response:

We tested the suggested post-transition addition and a learned projection of the concatenated state and input. Both variants retain residual feedback. Here, FF is the complete two-block shared transition, and WW is a learned linear projection.

Exact accuracy (%) at each outer-step count:

Input designUpdate163264128
Original additionFθ(zt+e(x);ht)F_\theta(z_t+e(x);h_t)82.4987.6090.8993.13
Post-transition additionFθ(zt;ht)+e(x)F_\theta(z_t;h_t)+e(x)79.8584.8088.0990.69
Learned concatenation fusionFθ(W[zt;e(x)];ht)F_\theta(W[z_t;e(x)];h_t)74.3276.7378.6880.21

All variants use the same Transformer backbone, 80k-epoch training recipe, and halting policy. The learned fusion adds 0.52M parameters. We evaluate all three on the same 3,072 puzzles with continuous fixed-depth inference.

Following your suggestion, we have added these comparisons to Appendix C.5 (Table 13) of the revised paper, with a reference in Section 3.

AuthorsOpenReview ↗

Response to Reviewer zVqJ—Part 2/4

W2: Differences in training and evaluation obscure comparisons with existing methods.

Thank you for raising this concern. We agree that the comparison logic should be clearer in the main text.

TL;DR: We have clarified the comparison logic in Section 4.1. Training settings follow each method's original study, while evaluations generally use the same benchmarks and accuracy metrics. Figure 2 presents accuracy–compute curves to account for differences in inference cost.

Detailed Response:

In Table 1, HRM and TRM-Attention results on Sudoku and Maze come from our training reproductions. Results for SR², ItrSA++/C-voting, FPRM, SE-RRM, and Attractor, as well as HRM/TRM on ARC, come from the original publications or official model reports and are marked accordingly.

Thus, the training settings are those used in each original study, while the reported evaluations generally use the same benchmarks and accuracy definitions. Sudoku and Maze use exact-solution accuracy; ARC uses pass@2. These shared benchmarks and metrics provide a meaningful basis for comparison. The main exceptions are:

  • ItrSA++/C-voting uses a 1,000-puzzle Sudoku test sample rather than the full test set.
  • SR²'s reporting is inconsistent: its Table 1 labels the scores as pass@1, whereas its Section 3.1 specifies two trials and pass@2 for ARC.

Some methods use multiple trajectories or voting, which increases inference cost. Figure 2 therefore compares accuracy against matrix FLOPs. It combines evaluations of our trained models and publicly released checkpoints with separately labeled published results. These comparisons show LRFeed's strong accuracy–compute trade-off.

We also isolate feedback's contribution by holding the backbone and training settings fixed. Under this matched setup, adding feedback raises full-test Sudoku accuracy from 79.05% to 93.61% at 128 steps.

We have clarified this comparison logic in Section 4.1. Detailed sources and protocol exceptions are provided in Appendices C.1 and C.3.

AuthorsOpenReview ↗

Response to Reviewer zVqJ—Part 3/4

W3: Do the Sudoku effective-rank observations generalize to Maze and ARC?

Thank you for encouraging this broader check.

TL;DR: The added ARC analysis shows higher rank with residual feedback across all four windows, as on Sudoku. On Maze, this advantage is concentrated in the first few steps; we further examine why its rank pattern differs.

Detailed Response:

Following the paper, we compute effective rank separately in each eight-outer-step window. For each task, all windows use the same fixed set of puzzles that neither model solves during steps 1–33.

TaskModel1–89–1617–2425–32
Sudoku-ExtremeLRFeed (w/o feedback)4.383.393.383.38
LRFeed4.724.614.484.53
Maze-HardLRFeed (w/o feedback)1.156.216.176.19
LRFeed1.195.895.915.94
ARC-AGI-1LRFeed (w/o feedback)1.962.522.522.53
LRFeed2.672.552.993.17

Maze does not show the same rank pattern. We further examined its gains from additional inference, which may help explain this difference. Both models are already close to their final accuracy by step 4:

Outer steps248173467
LRFeed (w/o feedback)80.682.482.482.382.482.2
LRFeed86.586.786.786.786.786.7

The decoded answers also largely stop changing. Throughout steps 4–64, the complete answer remains unchanged for 98.5% of puzzles with LRFeed (w/o feedback) and 97.8% with LRFeed. On Maze, most effective reasoning occurs within the first four steps, with little additional improvement thereafter.

We therefore examine both rank and fluctuation size within each outer step, using its ten internal updates and the same fixed puzzle set. Relative fluctuation is the RMS variation around the mean channel-activity profile, divided by the profile's RMS magnitude:

Outer stepRank: w/o feedbackRank: LRFeedFluctuation (%): w/o feedbackFluctuation (%): LRFeed
12.392.7824.25619.094
22.041.811.8211.214
32.333.140.3380.158
44.905.020.0910.104
56.045.440.0630.071
66.545.460.0640.069
76.735.530.0480.067
86.645.550.0500.066

LRFeed has higher within-step rank at three of the first four steps, where most accuracy gains occur. By step 4, fluctuations have fallen to about 0.1% in both models. From step 5 onward, they remain below 0.08%, even as rank rises.

Effective rank measures how varied the directions of these changes are, not how large they are. Normalization removes their magnitude, so even tiny fluctuations can have a high rank. This helps explain why Maze's higher later rank accompanies little change in its answers. We therefore use rank as a descriptive measure of internal dynamics.

We have added these analyses to Appendix D.6 (Tables 20–22), with a brief discussion in Section 4.4.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Summary of Revisions to the Paper

We sincerely thank the reviewers for their constructive feedback. In response, we added experiments and analyses, clarified the protocols, and revised the paper as summarized below, in manuscript order.

  1. Sections 3.1–3.2: Method clarification. Specified the initial zero residual and linked the backbone choice to the new single-state versus two-state ablation.

  2. Sections 4–4.1: Experimental protocols. Clarified training and evaluation settings, comparison sources, and compute accounting, with details in Appendices B.1–B.2 and C.1–C.3.

  3. Table 1: Main-table revision. Removed LLM rows; added HRM/TRM training-run standard deviations, inference step caps, and candidate counts; improved the layout.

  4. Section 4.2, Table 2: Architecture transfer. Added with/without-LRFeed comparisons on TRM-Attention, TRM-MLP, and FPRM, with detailed results in Appendix C.8.

  5. Section 4.3, Table 3: Feedback-input controls. Added full-test Phase and State-RMS controls alongside Constant, replacing the original Figure 9. Removed the original Table 2 and residual-prediction analysis.

  6. Section 4.4: Dynamics analyses. Added Maze/ARC rank analyses and residual-profile and gate diagnostics for solved and unsolved trajectories (Appendices D.5–D.6).

  7. Sections 5–6: Discussion and conclusion. Updated these sections to reflect the architecture-transfer results, task differences, and dynamics analyses.

  8. Appendix A.1, Table 4: Backbone ablation. Replaced the earlier explanation with a full-test comparison of single-state and TRM-style two-state LRFeed at four matched compute budgets.

  9. Appendix C.5, Table 13: Input placement. Added post-transition input addition and learned concatenation fusion as alternative designs.

Substantive additions are marked in blue.

AuthorsOpenReview ↗

Summary for the Area Chair

Dear AC,

Thank you for your time and careful consideration. To reduce your workload, we provide a concise summary of key information regarding our submission.

1. One-sentence summary of the paper

We introduce LRFeed, a lightweight mechanism that uses recent latent-state changes to guide looped Transformers, enabling higher accuracy with less computation and sustained gains far beyond the training horizon.

2. Ratings and discussion status

All three reviewers recommended acceptance in their initial reviews (3/3/3, Weak Accept). We then added experiments and revised the paper in response to their feedback. All three reviewers continue to recommend acceptance following the discussion.

3. Strengths highlighted by the reviewers

  • Novel architectural contribution. Reviewer 87kh describes LRFeed as "a novel adaptation for Recurrent Reasoning Models" and states that "The key motivation of the paper is sound." Reviewer zVqJ highlights that it addresses the lack of explicit feedback on previous latent-state updates.
  • Strong reasoning performance. Reviewer 87kh notes "strong performance compared to other RRMs" and gains in the ablations. Reviewer Qsdw highlights "a substantial benefit" in matched Sudoku comparisons, supported by full-test ablations and conditioning controls.
  • Practical efficiency with a compact design. Reviewer Qsdw highlights the measured timing results, noting that "fewer feedback steps can outperform deeper no-feedback inference." Reviewer zVqJ emphasizes that the mechanism introduces "only a small number of additional parameters."
  • Strong test-time scaling performance. Reviewer Qsdw emphasizes that "Continuous endpoint evaluation supports sustained gains beyond training depth," complemented by replication subsets.

4. Main concerns and our responses

We have carefully responded to each reviewer's questions and concerns. To date, no new follow-up questions have been raised since our latest responses.

  • Reviewer zVqJ asked for alternative design comparisons, clearer evaluation protocols, and tests on other architectures and datasets. We added input-placement ablations that favor the original design, clarified the comparison protocols, and extended rank analysis to Maze and ARC. Transfer experiments show gains on TRM-Attention and FPRM (Section 4.2, Table 2). In the latest comment, zVqJ states that we have "addressed most of my concerns," with the remaining reservation concerning the limited TRM-MLP gains.

  • Reviewer 87kh asked for a direct justification of the single-state backbone and clearer experimental reporting. We added a full-test two-state ablation supporting the single-state design (Appendix A.1, Table 4), completed three-run HRM/TRM statistics on Sudoku and Maze, revised Table 1, and removed the prediction probe. We also supplied modulation-strength tests and discussed the different gains across tasks. The reviewer thanked us for the ablation and asked for an interpretation; we provided a possible explanation. We also invited any remaining concerns and offered further clarification.

  • Reviewer Qsdw asked whether residuals offer more than other conditioning signals, how feedback relates to solving progress, and how the ARC comparison can be verified. We added full-test controls showing that residual feedback outperforms phase, state-RMS, and constant inputs across all evaluated depths (Section 4.3, Table 3). We also added solved/unsolved trajectory diagnostics, clarified the ARC protocol, and supplied a public-checkpoint TRM ARC curve in the rebuttal. Qsdw noted that these additions partially addressed their concerns. We have asked which concerns remain and offered further clarification during the discussion.


Beyond these findings, we would like to emphasize a broader insight: the information generated by a model's own computation can itself become a resource for better reasoning. Internal states and their changes need not serve only as intermediate representations; they can also provide feedback for shaping subsequent computation.

LRFeed offers a concrete example of putting these internal signals to use without additional supervision or a larger reasoning backbone. We hope this work encourages the community to explore how information already available within a model can support more capable and efficient reasoning, alongside advances in model scale and inference budget.

Thank you again for your time and consideration.

Best regards,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.