- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
LRFeed conditions a looped Transformer's shared computation on its latest latent update. On a single-state recurrent backbone, it pools residual energy across positions, normalizes the channel profile, and uses a small shared MLP to modulate subsequent Transformer blocks. The most detailed experiments are on Sudoku, with matched-backbone and constant-input controls, continuous long-horizon inference, and measured timing. They provide substantial evidence of improved accuracy and inference-time scaling beyond training depth. Maze and ARC broaden the evaluation, while prediction and dynamics analyses offer descriptive evidence about the feedback signal.
Critical strengths
- Matched Sudoku comparisons show a substantial benefit: at 128 steps, full-test accuracy rises from 79.05% without feedback to 93.61%. The constant-input control also separates informative feedback from added parameters alone.
- Continuous endpoint evaluation supports sustained gains beyond training depth, complemented by replication subsets. Extending inference to 2,048 steps yields 97.07% mean accuracy across three seeds.
- The module is compact, and measured timing exposes the practical trade-off: equal-depth inference has overhead, but fewer feedback steps can outperform deeper no-feedback inference.
Critical weaknesses
- W1 — Residual-specific conditioning is not fully distinguished from other informative inputs. Figure 9's constant-input control shows that modulation parameters alone do not explain the full gain. However, it does not compare the residual with state-energy or phase-based inputs using the same MLP and gates. Table 2's controlled prediction test establishes additional information about future update energy, rather than a task-level advantage over alternative conditioners. Thus, the value of input-dependent modulation is better established than the value specifically attributable to recent residual information (p.7, pp.23–24).
- W2 — The compressed feedback's relationship to reasoning progress remains indirect. Pooling removes position/sign information and normalization suppresses overall magnitude, so distinct local or directional updates can produce the same relative channel profile (Equation 4, p.5). The full state remains in the backbone, and near-zero attenuation retains some magnitude information; this restricts the added feedback channel. The authors acknowledge this boundary (p.9, §D.1). Their prediction test and descriptive state/output-profile rank analyses are useful, but do not directly examine when the actual residual profile and gates track useful progress rather than repeated or alternating activity. This limits the mechanistic interpretation of the long-horizon gains, without invalidating them or establishing that a richer feedback design would be better.
- W3 — The ARC accuracy–compute conclusion has weaker verification support. Table 7 compares 45.1% with a published TRM reference of 44.6%, without a reproduced TRM ARC curve or multi-seed uncertainty for LRFeed on ARC (p.21). The stated common augmentation/voting protocol and per-view accounting are helpful, but the small accuracy gap is less secure than the matched Sudoku result. The ARC checkpoint's training and selection recipe is also less explicit than the Sudoku recipe (§B.1, §C.3), making the comparison harder to verify. This narrows the support for an all-benchmark efficiency conclusion rather than challenging the core Sudoku gains.
Questions for authors
- Q1 — Task-level conditioning comparison (W1). With the same MLP, gates, training, and budget accounting, how does the residual input compare with an informative current-state RMS or phase-based input? Please also provide the exact constant-input values underlying Figure 9, so the incremental gain can be assessed without estimating points from the plot.
- Q2 — Feedback along actual trajectories (W2). On representative successful and unsuccessful long rollouts, how do the residual profile and resulting gates relate to answer progress and repeated or alternating updates? A trajectory-level diagnostic using the existing model would clarify the practical significance of the acknowledged compression boundary, without requiring a redesigned feedback architecture.
- Q3 — Evidence underpinning the ARC comparison (W3). What training/checkpoint provenance and uncertainty evidence support the 45.1% versus 44.6% comparison? Please identify the evaluated recipe, actual augmentation/view count, and the basis for assessing stability under the stated common protocol.
5 comments
Response to Reviewer Qsdw—Part 2/3
W2 & Q2: How do residual profiles and gates relate to solving progress along successful and unsuccessful trajectories?
Thank you for this valuable suggestion. It helped us make the trajectory analysis more direct and informative.
TL;DR: We now summarize these observations in Section 4.4 of the revised paper and present the trajectories in Figure 11 (Appendix D.5). Their averages show a clear transition around the first correct answer, while unsolved trajectories remain comparatively stable in the observed window.
Detailed Response:
We recorded 128-step rollouts on 3,072 Sudoku puzzles. For the 2,861 solved trajectories, we examine eight steps before and after the first correct answer. As a reference, we use an equally long window at the end of the 211 unsolved trajectories (steps 112–128). Figure 11 in Appendix D.5 (p. 30) shows the corresponding group means. Missing observations are omitted.
- Residual profiles: Changes between consecutive profiles peak around solving and then settle to a lower level. Channel participation briefly decreases and rebounds.
- Gates: The mean gate decreases after the first correct answer and stabilizes. Changes between consecutive gates peak about two steps afterward.
In contrast, the unsolved group shows no comparable transition over steps 112–128. These observations provide a concrete view of how the residual profiles and resulting gates change around solving.
Response to Reviewer Qsdw—Part 3/3
W3 & Q3: The small ARC advantage needs clearer verification. What training recipe, augmentation count, and uncertainty evidence support the comparison with TRM?
Thank you for highlighting this gap. We agree that the original appendix described Sudoku in detail but did not present the Maze and ARC protocols as clearly.
TL;DR: In the revised paper, we have reorganized Appendix B.1 to describe the shared training procedure and the protocols for all three benchmarks, including ARC checkpoint selection and evaluation view counts. We also report a TRM checkpoint-based accuracy–compute comparison below. Additional ARC training runs were not feasible given their substantial computational cost.
Detailed Response:
Training and checkpoint selection. The reported LRFeed ARC results come from a model trained with residual feedback. It shares the core training procedure with our other benchmarks, with the following ARC settings:
| ARC training setting | Value |
|---|---|
| Training duration / global batch size | 100,000 epochs / 448 |
| Backbone / puzzle-embedding learning rate | / |
| Weight decay | 0.1 for both parameter groups |
| Warmup / EMA decay | 2,000 updates / 0.999 |
| Training horizon / exploration | 16 outer steps / original ACT exploration of 0.1 |
We use the final ARC training checkpoint and evaluate that same checkpoint across inference budgets.
Evaluation protocol and view counts. LRFeed follows TRM's ARC evaluation protocol: the same test tasks, test-time augmentation, inverse transformation, voting, and pass@2 scoring. Evaluation covers 400 tasks with 419 test inputs. We request 1,000 augmented views plus the original input; after deduplication, this gives 385,815 evaluated views in total, with at most 1,001 views per input. Both models use the same evaluated views. Compute is measured in matrix TFLOPs per augmented input.
Additional TRM comparison. We evaluated an open-source TRM reproduction checkpoint released by alphaXiv at multiple inference budgets, as shown below.
| Model / metric | 1 outer step | 2 | 4 | 8 | 16 |
|---|---|---|---|---|---|
| LRFeed accuracy (%) | 43.80 | 45.10 | 46.00 | 46.10 | 46.90 |
| LRFeed matrix TFLOPs | 0.159 | 0.318 | 0.637 | 1.274 | 2.548 |
| Public TRM accuracy (%) | 41.13 | 43.00 | 43.13 | 43.13 | 43.13 |
| Public TRM matrix TFLOPs | 0.239 | 0.478 | 0.955 | 1.911 | 3.822 |
LRFeed reaches 45.10% at 0.318 TFLOPs, while this TRM reproduction reaches 43.13% at 0.955 TFLOPs and remains at that accuracy with further computation. LRFeed reaches 46.90% with additional compute. This checkpoint-based comparison also supports LRFeed's accuracy–compute advantage.
Uncertainty. ARC results currently use one training seed; additional independent runs were not feasible given the substantial computational cost of ARC training.
Appendix B.1 now presents the shared training procedure followed by separate Sudoku, Maze, and ARC protocols. Appendix C.3 retains the sources and comparison conventions for external results.
Response to Reviewer Qsdw—Part 1/3
Dear Reviewer Qsdw,
Thank you for recognizing the Sudoku results. We added conditioning controls and trajectory diagnostics, clarified the ARC training and evaluation settings, and evaluated an open-source TRM checkpoint at multiple inference budgets.
Thank you for this helpful suggestion. It helps us make the experimental evaluation more complete and strengthen the paper.
TL;DR: We added matched comparisons with current-state RMS and phase inputs and report exact accuracies. LRFeed outperforms all four controls at every tested budget, including both informative alternatives.
Detailed Response:
The conditioning variants change only the input to the feedback MLP. Let R(v) denote channel-wise RMS pooling over the S positions, and let N denote the shared normalization and scaling:
R(v)c=S1∑i=1Svi,c2.
N(v)=21RMSNorm(v).
The MLP inputs are:
For Phase, SinEnc is sinusoidal encoding and k=0,…,9 is the atomic-transition position within each outer step, restarting at zero in the next outer step. All four variants retain the same MLP architecture, channel gates, modulation scale, and training policy.
Each variant is trained separately. The table below reproduces the full-test Sudoku accuracies (%) in Table 3 of the revised paper, including the constant-input results:
At 128 steps, LRFeed exceeds State-RMS by 2.41 percentage points and Phase by 2.38 points. These results support the value of recent residual information beyond constant, current-state, or iteration-position conditioning.
We have replaced the original Figure 9 and its accompanying appendix discussion with this more complete comparison in Section 4.3 (Table 2), including the exact accuracies.