acceptodds
Under review as a conference paper at ICLR 2027

Steering Recurrent Reasoners at Inference Time with Readout Feedback

Abstract

Recurrent models, which repeatedly update latent states with shared computation blocks, have emerged as powerful architectures for solving complex reasoning tasks. Existing inference-time methods scale computation by running more steps or sampling more trajectories, but ignore information revealed within each trajectory. Here we show that recurrent models can be improved at inference time by using their own readout probabilities to steer latent dynamics without retraining. We introduce Readout Feedback (RoFB), a test-time intervention that converts intermediate predictions into token-wise pairwise coupling forces injected into the latent dynamics. Across three recurrent models (AKOrN, ItrSA++, TRM) on Sudoku and Maze, RoFB yields clear gains in four of six model-task pairs and a small gain in one. The four positive pairs achieve performance unattainable by merely running more steps or selecting from multiple trajectories, at comparable or lower computational cost. These results suggest that closed-loop steering of latent dynamics can serve as a complementary inference-time control mechanism for recurrent reasoning models.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
recurrent neural networks, iterative reasoning, inference-time computation, latent dynamics, latent reasoning, looped transformer

Reviews

Among:
Shape:
1234this paper, Top 4%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.33 over 3 reviews, above 96% of the others.
Reviewer_mzvoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Recurrent models such as AKOrN and TRM achieve strong performance on complex reasoning tasks by applying shared modules recursively, showing that adding more depth or parallel runs gives further improvement at inference time. However, these gains mostly come from passive strategies such as repeating computation or running more rollouts. This paper introduces RoFB, a closed-loop approach that uses the readout signals as feedback to steer the reasoning paths toward organized, converged latent states. In experiments, RoFB shows a clear advantage in most setups on Sudoku and Maze tasks across three different recurrent models (AKOrN, ItrSA++, TRM).

Critical strengths
  • Clear and novel contribution. In recurrent models such as Looped Transformers, inference-time scaling behavior is one of the most important factors. Improving performance at inference time usually means running more loops or more rollouts, which is to say the model runs the recurrence without any intervention. RoFB instead takes a closed-loop approach that can actively intervene. Search during recurrence is another way to steer in a closed-loop manner, but it generally demands far more resources. RoFB instead steers the recurrence with a single feedback term added to each step, so it is likely to be more resource-efficient than search-based methods (although this comparison is not included in the paper).

  • Posing new potential in recurrent models. I found it compelling that AKOrN, ItrSA, and TRM are defined within the same recurrent framework. This is not just unifying them under the loose criterion of repeating a module recurrently. The synchrony behavior between latent tokens seen in AKOrN also arises from plain self-attention, and similar behavior shows up in ItrSA and TRM, demonstrating that the phenomenon is consistent enough to actually be exploited.

  • Thorough analysis on inter-token clusterization. RoFB is motivated by the observation that latent tokens form class-wise clusters when the model reaches a correct solution. However, the paper does not leave this as an empirical finding from AKOrN alone. It shows that the same behavior appears consistently in ItrSA++ and TRM, and compares it across models and tasks using measurable metrics such as the cosine gap and silhouette score. It also reports where this indicator becomes less reliable, for example TRM on Maze, and shows that RoFB increases the separation even on boards that remain unsolved. Together with the discussion of why inter-class separation can help the learned dynamics reach a solution, this gives solid support for the motivation behind the method.

Critical weaknesses

Limited scalability. Sudoku and maze are hard enough benchmarks, but the experiments remain narrow in scope, covering only two puzzle tasks with relatively small models, each trained once. Beyond these tasks, the only harder setting tested is ARC-AGI-1, where RoFB brings no improvement. The authors suggest that, as in TRM on Maze, confident errors cannot be distinguished from correct peers by the prediction-based coupling. However, with a single harder benchmark, it is difficult to tell whether RoFB fails because ARC-AGI-1 is simply harder, or because it falls into the same situation as TRM on Maze. I also still question whether the method is too task-specific. The coupling is computed from how much the readout distributions of two tokens overlap, which assumes a small, shared vocabulary. For tasks where outputs rarely share the same tokens, the coupling would become nearly uniform and lose its class-specific signal, reducing RoFB to a simple repulsion from the mean. I think this boundary should be discussed more explicitly in the limitations.

Questions for authors
  • Instead of injecting repulsion into the dynamics, what happens if one simply searches over trajectories using the same separation measure? This would clarify whether the gain comes from the force itself or from the separation objective.
  • The introduction states that the gains come with ×10–100 more efficient compute. How is this number derived? Based on the operating-point grids in the appendix, the ratio seems to fall in this range only for the two Maze pairs, while it is about 1/8 for AKOrN on Sudoku and about 1/1.3 for TRM on Sudoku. Could the authors report the ratio for each model–task pair?
  • Why is it not applied at training time? If it were applied during training and amortized, is there no improvement? If it can be internalized through post-training, what gains that would bring also seems like an important analysis.
6 comments

Thank you for your careful review and constructive feedback. We divide W1 into evaluation scope (W1a) and performance on harder tasks (W1b). Below, we respond to W1a, W2, Q1, and Q2. We will address W1b and Q3 in a follow-up response.

W1a. Limited coverage of tasks, model sizes, and training seeds

We expanded our evaluation to graph coloring, Nonogram, Futoshiki, and Mini-ARC. RoFB improved Futoshiki accuracy across all three models and training seeds, with smaller gains on graph coloring. Nonogram and Mini-ARC showed no clear benefit. These results broaden the evaluation while showing that benefits remain task- and model-dependent. Please see our Common Concerns comment for details.

W2. Dependence on a small, shared output vocabulary

Thank you very much for pointing out this important limitation. We admit tha t the current formulation of RoFB relies on a small, shared vocabulary to compute the inter-token coupling based on readout distributions. In tasks with large vocabularies, this coupling may become diluted, potentially reducing the effectiveness of RoFB. We have revised the limitations section to more explicitly discuss the potential challenges of applying RoFB to tasks with large vocabularies. We acknowledge that this boundary is an important consideration for future work and will explore potential modifications to RoFB that could address this issue in tasks with larger vocabularies.

Q1. Can separation-based trajectory selection explain RoFB’s gains?

Thank you for this suggestion. We compared entropy-based trajectory selection with cosine gap and cosine silhouette, computed from predicted labels and terminal hidden states without ground-truth labels. All selectors used the same trajectories from the unmodified dynamics. We evaluated AKOrN and ItrSA++ on 20,000 Sudoku and 1,000 Maze instances. Results with four candidates and 256 inference steps per trajectory are shown below.

Table. Exact accuracy (%) by trajectory selector, using four candidates and 256 internal updates per trajectory.

Model–taskEntropy selection (%)Cosine-gap selection (%)Cosine-silhouette selection (%)
ItrSA++ on Sudoku77.67577.67577.675
ItrSA++ on Maze81.10081.10081.100
AKOrN on Sudoku85.04585.04585.045
AKOrN on Maze74.40074.20074.300

Across seven candidate counts (1–64) and three update budgets (64, 128, 256), neither separation measure consistently outperformed entropy selection. The largest Sudoku improvement was +0.025 percentage points, while Maze differences ranged from −0.7 to +0.2 points. These results suggest that separation-based selection of completed trajectories alone is insufficient to explain RoFB’s gains in the tested settings.

Q2. Basis of the 10–100× efficiency claim and per-pair ratios

We agree that the original “10–100×” wording was too broad and have removed it from the Introduction. Comparing the cheapest baseline point attaining the maximum mean accuracy in the evaluated grid with the cheapest RoFB point exceeding it, the baseline-to-RoFB cost ratios are 7.89× for AKOrN on Sudoku, 1.32× for TRM on Sudoku, 58.31× for AKOrN on Maze, and 58.18× for ItrSA++ on Maze. ItrSA++ on Sudoku has a ratio of 0.96× under the same rule, providing no cost saving; TRM on Maze has no RoFB point exceeding the baseline maximum.

Q3. Training-time use and internalization of RoFB

Our focus is improving inference while keeping pretrained model weights fixed. To investigate training-time use, we tested learning only RoFB’s coefficients, jointly updating the pretrained model and RoFB, and training both from scratch. Learning only RoFB’s coefficients improved accuracy by 0.960 percentage points over fixed RoFB. However, incorporating RoFB into training did not consistently outperform comparable training without RoFB. After joint post-training, removing RoFB often reduced accuracy, indicating that its benefits were not fully internalized into the model weights. Training-time integration remains a promising extension, with benefits depending on the training configuration.

W1b. Failure on ARC-AGI-1: task difficulty or a shared failure mode?

Thank you for asking us to distinguish these explanations. To test whether RoFB works with a stronger reasoner on ARC-AGI-1, we trained a TRM variant following the recipe introduced by Rekursiv.ai (https://rekursiv.ai/blog/pushing-limits-arc-agi/), achieving 69.0% pass@2 on all 400 evaluation tasks. We then applied RoFB selected by a 30-trial Optuna search on 100 tasks. On the remaining 300 tasks, task-averaged pass@2 was 68.33% for both ordinary inference and selected RoFB. Extending ordinary inference from 16 to 18 outer steps to approximately match RoFB's runtime also gave 68.33%. The lack of improvement therefore persisted beyond the original TRM checkpoint and a single feedback setting within the tested search space.

Through failure analysis, we observed that high-confidence errors dominated ordinary-inference errors and were rarely corrected by RoFB. We examined 40 selected tasks from each of the validation and test splits, with four augmented inputs per task. High-confidence errors, defined as incorrect outputs for which every incorrectly predicted cell within the reference output had at least 99% softmax probability for its predicted class, constituted approximately 85–91% of ordinary-inference errors, and RoFB corrected almost none of them (Table). Thus, even with the stronger checkpoint and validation-selected feedback, failure to correct the dominant group of high-confidence errors remained the main observed limitation in this diagnostic sample. The smaller group below the confidence threshold was also never corrected in this comparison. These observations characterize the remaining errors; they do not isolate high confidence itself or task difficulty as the cause of resistance to correction.

Table: Ordinary-inference errors and corrections by RoFB, counted per augmented input before voting.

SplitHigh-confidence errors / all ordinary-inference errorsCorrected by RoFB: high-confidence errorsCorrected by RoFB: errors below this threshold
Validation66/78 (84.6%)0/660/12
Test64/70 (91.4%)1/640/6
Reviewer_dJ3SOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Readout Feedback (RoFB), a training-free, inference-time intervention for recurrent reasoning models. The motivation (Sec. 3.1, Fig. 2/4, App. B) is that when these models reach a correct solution, latent tokens form class-wise clusters. RoFB adds a tangent-projected pairwise coupling term to the latent update (Eq. 8). The coupling weight comes from the distance between per-token readout probabilities, J = h(1 − pᵢᵀpⱼ) with h(d) = −d (Eq. 9). An entropy-based gate with warm-up (Eq. 10) controls when it is active. RoFB is tested on AKOrN, ItrSA++ and TRM on Sudoku-Extreme and Maze-Hard, with an extra TRM run on ARC-AGI-1 (App. M). It gives clear gains in 4 of 6 model–task pairs, for example AKOrN/Sudoku 85.1→91.5% and TRM/Sudoku 68.4→74.2% (Table 7). In those four pairs it reaches accuracies the baseline does not reach with longer rollouts or confidence-based voting, at similar or lower FLOPs (Fig. 3). Mechanistic studies (App. I–K) remove the separation-related component, compare attractive, mixed and constant couplings, and test entropy-gradient controls.

Critical strengths
  • [Importance of the research problem] Training-free inference-time improvement of recurrent reasoners (AKOrN, ItrSA++, TRM), beyond more steps or voting, is timely for test-time compute scaling.
  • [Novelty of the approach] Readout Feedback is a fresh, elegant idea. A tangent-projected pairwise coupling is weighted by readout disagreement, J = h(1 − pᵢᵀpⱼ) (Eqs. 8–9), and gated by entropy (Eq. 10). It is motivated by class-wise latent clustering in solved states (Sec. 3.1).
  • [Soundness and feasibility] The mechanistic ablations are strong. Removing the separation-related component costs 2.3–3.3 pp on Sudoku for all three models and 8.9/11.0 pp on Maze for AKOrN/ItrSA++ (Table 11).
  • [Soundness and feasibility] The reach-vs-retain analysis gives readout dependence a specific role: 11.4% vs. 3.1% of AKOrN trajectories later lose a found solution under constant vs. readout-dependent coupling (Sec. 5.2, App. J).
  • [Soundness and feasibility] Per-step overhead is low: r_tot is 1.008–1.098 (App. F).
  • [Experiments and practical impact] The statistics are careful: paired starts, 8 inference seeds, bootstrap CIs and Holm-corrected tests.
  • [Experiments and practical impact] There are clear gains in 4 of 6 pairs, e.g., AKOrN Sudoku 85.1 → 91.5% (+6.38, CI [6.05, 6.70]) and TRM Sudoku 68.4 → 74.2% (Table 7). ItrSA++ Maze reaches 83.4% versus the best baseline's 82.1% at about 1/62 of its compute (Fig. 3).
  • [Experiments and practical impact] Negative results are reported honestly.
  • [Theoretical and application-level implications] The paper gives a concrete, causally tested account of how recurrent reasoners succeed (latent class separation) and fail (collapse or insufficient separation).
  • [Theoretical and application-level implications] RoFB is a cheap, training-free test-time compute knob that can beat extra steps or voting on a FLOPs axis.
Critical weaknesses
  • [Importance of the research problem] Impact is bounded by the scope: small models on grid puzzles, with no language models (as the paper states).
  • [Novelty of the approach] On Sudoku, constant repulsion (h ≡ −1) matches RoFB for all three models (AKOrN 91.6 vs. 91.5%, TRM 74.7 vs. 74.2%, ItrSA++ 77.0 vs. 77.0%; Table 13). This weakens the "readout-guided" framing, because readout dependence matters mainly on Maze (e.g., AKOrN 73.8 vs. 78.8%).
  • [Soundness and feasibility] Four hyperparameters (λ, t_min, α, τ) are tuned with 30 Optuna trials for every model–task pair, on augmented training boards (N_val = 100 for Maze; App. E).
  • [Soundness and feasibility] Transferring a configuration across architectures cost 31.7–48.2 pp (App. L).
  • [Soundness and feasibility] There is no stability or convergence analysis of the modified dynamics (Eq. 8).
  • [Experiments and practical impact] There is no gain for TRM on Maze and no consistent pass@1 improvement on ARC-AGI-1 (App. M).
  • [Experiments and practical impact] Robustness across training seeds was checked on Sudoku only. Without natural frequencies, the gains were mixed in the modified architecture and absent in the original-architecture reproduction (App. L, Table 17).
  • [Theoretical and application-level implications] The hypothesis has limits: 45 of 97 confident terminal predictions on ARC-AGI-1 were wrong (App. M), so separation does not guarantee correctness.
  • [Theoretical and application-level implications] Adoption needs a default or self-calibrating configuration, robustness across training seeds, and evidence beyond grid puzzles. The one-time tuning sweep can be costly (up to 18.879× the baseline test cost for TRM × Maze).
Questions for authors
  1. Can you report RoFB gains across ≥3 training seeds for every model–task pair, not only AKOrN/Sudoku?
  2. Why does constant repulsion match RoFB on Sudoku? Does a gate-only plus constant-repulsion variant also need the readout at all on Sudoku?
  3. Table 21 lists the attention variant (mlp_t = False) trained for 5×10⁴ epochs. How does the 68.4% TRM Sudoku baseline compare with published TRM numbers, and what explains any gap?
  4. How do results change if hyperparameters are tuned on a held-out validation split instead of augmented training boards? Is there a single shared configuration that works across pairs?
  5. Can RoFB use task constraints (e.g., Sudoku row/column structure) instead of only predicted-class agreement, to fix confident errors like those in TRM/Maze and ARC (App. M)?
  6. Is there any evidence, even small-scale, of transfer to a looped or recurrent-depth language model or to non-grid tasks?
4 comments

Thank you for your careful review and constructive feedback. Below, we respond point by point to the weaknesses (W) and questions (Q) raised in your review.

W1. Limited scope: small models on grid puzzles, without language-model evaluation

Thank you for your comments on the generalizability of RoFB. We expanded the evaluation to graph coloring, Nonogram, Futoshiki, and Mini-ARC (see Generalizability in our Official Comment). RoFB yields small gains on some tasks, including non-grid graph coloring, but benefits remain task- and model-dependent. We will include these results and limitations in the revised manuscript. Meanwhile, further investigation and experiments are needed to establish the effectiveness of RoFB on language models, and for now we leave this as future work.

W2 & Q2. Why does constant repulsion match RoFB on Sudoku, and is readout dependence necessary?

Thank you for this question. We agree that improvements on Sudoku do not always require intermediate-readout-dependent feedback. As shown below, constant repulsion without the entropy gate, a setting where no readout information is fed to the model, increased accuracy over the baseline in all three models. The TRM improvement of 6.2 percentage points was statistically significant, while the increases for AKOrN and ItrSA++ were numerical. This shows that improvements on Sudoku can arise without intermediate-readout-dependent feedback.

**Table: Accuracy (%) on 500 shared test instances per task. **

Model–task pairBaselineRoFB + gateConstant + gateRoFB, no gateConstant, no gate
AKOrN on Sudoku86.093.2†92.889.089.2
ItrSA++ on Sudoku75.877.077.077.077.0
TRM on Sudoku65.271.073.069.471.4*
AKOrN on Maze75.078.8†73.875.064.4*
ItrSA++ on Maze81.484.683.484.683.6*
TRM on Maze93.092.893.092.893.0

* Constant, no gate differs significantly from Baseline (lower on AKOrN–Maze); † RoFB + gate significantly exceeds Constant, no gate. Two-sided exact McNemar tests; Holm-adjusted p < 0.05 across 12 comparisons.

The force similarities help explain why constant repulsion can reproduce these benefits. The appendix reports cosine similarities of 0.997–0.999 on Sudoku before projection; a preliminary check of four AKOrN–Sudoku states also found similarities of 0.9991–0.9995 after tangent projection.

Nevertheless, removing readout dependence does not preserve performance in every setting. Full RoFB with the entropy gate outperformed constant repulsion without the gate by 4.0 points on AKOrN–Sudoku and 14.4 points on AKOrN–Maze. These results support an additional, setting-dependent benefit of readout-based modulation. Thus, its dispensability in some Sudoku settings does not imply that it can be removed without loss across model–task settings. We will clarify this scope in the revised manuscript.

W3 & W4. Per-pair tuning on training-derived puzzles and poor transfer across architectures

We acknowledge that RoFB needs a laborious hyperparameter search for each settings, and we have not established a universal default. However, settings transferred across training seeds outperform the baseline in most transfers within the five pairs that benefit from per-checkpoint tuning (Tables 3a–b of our Official Comment). This supports reusing settings for new checkpoints of the same model–task pair without repeating the search.

W5. Missing stability and convergence analysis

We agree that stability and convergence of the modified dynamics require explicit examination. We have now measured consecutive latent-state changes and prediction correctness across six model–task pairs, using 500 instances per pair and extending inference to four times the standard horizon.

For some model–task pairs, latent states continued to change while the predicted boards remained correct, showing that correct predictions did not necessarily correspond to a fixed latent state within the observed horizon. We measured relative state change as rt=∥Zt+1−Zt∥F/(∥Zt∥Fr_t=\|Z_{t+1}-Z_t\|_F/(\|Z_t\|_F, where ZtZ_t denotes the latent state at step tt. For AKOrN, the median relative state change over the final quarter was 0.04461 versus 0.04476 on Sudoku and 0.03367 versus 0.03847 on Maze, without versus with RoFB. Meanwhile, the numbers of instances remaining correct throughout that interval increased from 417 to 450 on Sudoku and from 362 to 382 on Maze, out of 500 instances each. Thus, RoFB can improve sustained prediction correctness without inducing latent-state stationarity over the observed horizon.

W6. No gain on TRM on Maze and no consistent pass@1 gain on ARC-AGI-1

We agree that these null results limit the scope of RoFB. Additional evaluations across three training seeds confirmed no meaningful gain for TRM on Maze. Our diagnostics nevertheless show that RoFB increases repulsion and class separation in the low-level states in all three checkpoints. Thus, changing the latent geometry does not necessarily translate into improved predictions. Further hyperparameter search on the original ARC-AGI-1 checkpoint also yielded no consistent pass@1 improvement. These results show that RoFB’s benefits depend on the model and task; predicting these benefits before application remains an open problem.

W7 & Q1. Robustness across ≥3 training seeds for all pairs

Thank you for your important comment. We have conducted additional experiments with three different training seeds and found that RoFB outperforms the baseline methods in five out of the six successful pairs (i.e., all pairs except TRM on Maze) using the hyperparameter choice on each training checkpoint. We also found that the hyperparameter setting is transferable across checkpoints, i.e., RoFB outperforms the baseline methods in the same five pairs using the hyperparameter choice on a different seed. Please refer to the Common Concerns among Reviewer note on details.

W8. Separation and confidence do not guarantee correctness

We agree that separation is not a sufficient condition for correctness. The 45 incorrect predictions among 97 confident terminal predictions demonstrate that confidence alone cannot certify a correct solution. Our additional analysis also found that increasing reference-color separation did not increase accuracy on the selected ARC diagnostic subset. These observations suggest a limitation of prediction-based feedback in RoFB, and a complete causal explanation of the failures are left to further analysis.

W9. Practical adoption: default settings, robustness across training seeds, non-grid evidence, and tuning cost

Thank you for your valuable comment. For default settings and robustness across training seeds, please refer to the answer for W3 & W4.

On evidence beyond grid puzzles, we applied RoFB to graph coloring, whose input and outpus are graphs rather than grids, and found that RoFB can provide a gain on the task, although the performance boost is small. We acknowledge that further investigation and experiments are needed to establish the effectiveness of RoFB on language models and other non-grid tasks, and for now we leave this as future work.

Q3. Why does the TRM Sudoku baseline fall below published results?

Thank you for your detailed questions. We have investigated the discrepancy between our TRM Sudoku baseline and published results. The published TRM numbers report a success rate of 75%, while our implementation achieved 68.4%.

Our reported checkpoint was trained on a single GH200 GPU, whereas retraining with the same code and seed settings on a single H100 reached approximately 75%. Together with the variation across training seeds reported in our Common Concerns, this suggests that differences in the computational environment and training randomness may partly explain the gap. We retain the original GH200 results for consistency.

Q4. Held-out validation and a shared configuration across model–task pairs

Thank you for raising these points. We evaluated RoFB with hyperparameters on held-out puzzles disjoint from training and found that RoFB outperformed the matched baseline for all three seeds in the same five pairs at the evaluated operating points. Please see Tables 1 and 2 in our Common Concerns among Reviewers comment for details.

We have not established a single configuration that consistently improves performance across all model–task pairs. However, within the same model–task pair, hyperparameters selected on held-out validation for one checkpoint could be reused across training seeds: the same configuration yielded positive gains over the baseline in all three checkpoints for five of the six pairs. These results support configuration reuse within a model–task pair, although a universal default across pairs is not established.

Q6. Evidence on looped or recurrent-depth language models or non-grid tasks

Thank you very much for pointing out this important limitation. We currently do not have evidence of transfer to looped or recurrent-depth language models. Instead, on non-grid graph coloring, we observed a small gain (see Generalizability in Common Concerns). Broader transfer to language models and other non-grid tasks remains future work.

Q5. Can task constraints help RoFB correct confident errors?

Thank you for this suggestion. We ran a preliminary Maze experiment. For Maze, many errors already satisfied path-validity constraints: for TRM, 255/256 outputs were valid paths (i.e., connected paths from start to goal without hitting the wall), but only 240 were shortest. Thus, validity checks alone would miss 15 of its 16 errors.

We therefore tested a brief inference-time correction to hidden states that encourages fewer cells to be predicted as part of the path, without using the ground-truth solution. We selected its strength on 64 separate puzzles, then evaluated the 256 puzzles and continued ordinary inference after removing the correction. This length bias corrected no additional errors in any model, and combining it with RoFB provided no consistent improvement over RoFB alone.

These initial results suggest that a simple preference for shorter paths is insufficient to correct the observed Maze errors. We will provide a more detailed update as further results become available.

Following up on our preliminary Maze experiment, we tested whether sustained length bias could correct trajectories that otherwise ended in valid but non-shortest paths: 42 for AKOrN, 29 for ItrSA++, and 15 for TRM, from the same 256 puzzles. We applied the bias at every internal update for one standard inference duration, starting either midway through inference or after the predicted path had stopped changing. We tested three strengths, without using ground-truth solutions to compute the correction.

For TRM, the strongest bias produced a shortest path at least once in 8/15 trajectories with the earlier start and 9/15 with the later start. However, none were shortest at the intervention endpoint, and 14/15 and 15/15 outputs, respectively, were invalid by then. At the intermediate strength, one earlier-start trajectory was shortest at the endpoint, although its output fluctuated during the intervention. AKOrN and ItrSA++ showed no shortest-path corrections under the tested settings.

These results refine our initial observation: the length bias can transiently correct some Maze errors, including after predictions have stopped changing, but sustained application yielded limited final correction and could destroy path validity. We therefore conclude that, under the tested settings, a simple length bias was insufficient to consistently guide Maze inference toward valid shortest-path solutions.

Reviewer_qxqxOpenReview ↗
Rating
4
Confidence
2
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

Recurrent reasoning models repeatedly update latent token states through a recurrent module. This paper proposes Readout Feedback (RoFB), a training-free inference-time intervention that steers frozen recurrent reasoning models by feeding intermediate predictions back into their latent-state dynamics. At each recurrent step, RoFB uses per-token class-probability vectors to introduce repulsive interactions between latent tokens, with stronger repulsion between more dissimilar predictions. A warmup delays the intervention, while an entropy-based gate modulates its strength according to predictive uncertainty. The method is motivated by the observation that latent tokens exhibit class-dependent clustering when trajectories reach correct solutions. Unlike the evaluated baselines of longer rollouts and confidence-based voting, RoFB directly modifies the trajectory of recurrent computation. The paper evaluates RoFB on Sudoku Extreme and Maze Hard using three architectures, AKOrN, ItrSA++, and TRM, with computational cost measured in FLOPs. RoFB achieves clear gains in four of six model–task pairs, a small gain in one, and no accuracy gain for TRM on Maze. In the four strongest cases, it reaches accuracy beyond the evaluated baseline grid at lower inference cost; on Sudoku, TRM accuracy increases from 68.4% to 74.2%. Ablations support a contribution from separation-related feedback directions to the gains. Constant repulsion can be sufficient to reach solutions in some settings, while trajectory comparisons suggest that readout-dependent weighting may help retain them. In a separate longer-horizon evaluation, norm-matched random directions underperform RoFB in three pairs, while a fixed-scale entropy-gradient control changes accuracy by at most 0.7 percentage points in the four pairs where RoFB gains 3.2–7.2 points. For TRM on Maze, under ordinary inference without RoFB, 71 of 72 incorrect outputs have maximum normalized entropy at most 0.10, and all 72 have silhouette scores of at least 0.5. The authors suggest that this, together with the coupling’s inability to distinguish wrong assignments from correct ones, may limit the benefit of feedback, but do not establish it as the cause. An additional TRM experiment on ARC-AGI-1 shows no consistent pass@1 improvement. The paper acknowledges that it has no validated diagnostic for predicting when RoFB will help and that its main evaluation is limited to puzzle benchmarks and compact recurrent models.

Critical strengths
  1. In four of six model–task pairs, RoFB reaches accuracy beyond the evaluated baseline grid at equal or lower inference FLOPs. For AKOrN and ItrSA++, baseline and RoFB runs use matched initial states across eight inference seeds; TRM is evaluated with a deterministic initial state. The per-step overhead is at most 20%, and the reported Pareto improvement persists when the one-time hyperparameter-sweep cost is included. The paper demonstrates an accuracy–compute advantage over simply increasing the evaluated rollout lengths or vote counts, including on TRM–Sudoku, where voting is unavailable.
  2. A magnitude-controlled ablation supports a contribution from separation-related feedback directions in the four model–task pairs with clear accuracy gains. Constant repulsion performs similarly to RoFB on Sudoku but falls below it on Maze for AKOrN and ItrSA++; norm-matched random forces underperform in three pairs (in a separate longer-horizon evaluation), and a fixed-scale entropy-gradient control changes accuracy by at most 0.7 percentage points in the four pairs where RoFB gains 3.2–7.2 points. Trajectory analyses further suggest that repulsion helps trajectories reach solutions, and readout-dependent weighting helps them retain solutions. The TRM–Maze failure also demonstrates that confidence and latent separation do not guarantee correctness, although the authors’ explanation for this failure remains unconfirmed.

The paper tests its proposed mechanism and does not rely on accuracy improvements alone. The results provide evidence that feedback structure contributes to the gains, but do not establish that magnitude is unimportant.

Critical weaknesses

The main limitations concern robustness and generalizability, not the validity of the reported results. In the AKOrN–Sudoku analysis, gains vary with training seed (under a fixed RoFB configuration) and with hyperparameter choice. Comparable training-seed evaluations are not provided for the other five model–task pairs. The main evaluation covers two puzzle benchmarks and three compact recurrent architectures. The paper does not provide a way to predict when RoFB will help; it tunes each model–task pair separately, and robustness across checkpoints remains uncertain. The validation boards are augmented from the training set, so robustness to hyperparameter selection on puzzles disjoint from the underlying training puzzles remains unestablished. Finally, comparisons with alternative separation interventions use a finite tuning search and do not fully isolate force direction from magnitude. These limitations leave uncertainty about broader applicability and mechanism, but do not undermine the reported accuracy–compute gains in the tested settings.

Questions for authors

The main evaluation uses one checkpoint per model–task pair. Appendix L provides the only multi-checkpoint analysis, limited to AKOrN on Sudoku, and reports mixed gains across three checkpoints for the modified architecture without natural frequencies (the architecture used in the main text), using one fixed RoFB configuration. For this pair, validation accuracy reaches 100% (Table 3), the selected λ\lambda is near the upper boundary of its search range, and test gains are sensitive to hyperparameter changes (Appendix H). Could the authors clarify how many of the 30 Optuna trials attained 100% validation accuracy and how ties were resolved? More importantly, do RoFB’s gains replicate across independently trained checkpoints when hyperparameters are re-selected for each checkpoint, ideally using validation puzzles disjoint from the underlying training puzzles?

4 comments

Thank you for your careful review and constructive feedback. Below, we respond point by point to the weaknesses (W) and questions (Q) raised in your review.

W1 & Q1. Robustness across training seeds and hyperparameter choices with held-out validation

Thank you for raising these points. We evaluated all six model–task pairs with three training seeds and reselected RoFB hyperparameters for each checkpoint using (i) training-derived validation puzzles and (ii) held-out puzzles disjoint from training. Under both settings, RoFB outperformed the matched baseline for all three seeds in the same five pairs at the evaluated operating points; TRM on Maze showed no improvement. Please see Tables 1 and 2 in our Common Concerns among Reviewers comment for details.

W2. Limited coverage of tasks and architectures

Thank you very much for pointing out this important limitation. We conducted additional experiments on puzzle tasks, including graph coloring, Nonogram, and Futoshiki on all three models, and Mini-ARC on TRM (Kim et al., 2022). The results show that although the benefit remains task- and model-dependent, RoFB can provide small gains on some of these tasks. Please refer to our Common Concerns among Reviewers on further details. We will add these results and a discussion of robustness to the revised manuscript.

W3. Predicting RoFB’s benefits

We agree that we do not yet have a validated criterion for predicting RoFB gains on a new model–task pair. Our mechanistic analyses characterize observed effects and failure cases, but do not establish a predictor. We leave the predictive diagnostic as a future work.

W4. Limited tuning of alternatives and incomplete separation of force direction and magnitude

Thank you for pointing this out. We agree that a finite validation search cannot establish the best achievable performance of alternative separation interventions. Our comparisons are limited to the evaluated search spaces and budgets.

Feedback-magnitude sensitivity is examined in Appendix H (Hyperparameter Sensitivity, Fig. 13). We evaluate the coupling strength at 0.5, 1, and 1.5 times its selected value while retaining the RoFB directional rule. The six model–task pairs showed mixed results: three maintained positive gains across all three strengths, whereas the remaining pairs showed degradation or no clear improvement.

To examine direction, we compared RoFB with sign-flipped forces and norm-matched random directions (Appendix J) across all six model–task pairs. Sign flip negates the force while preserving its norm at a given state. For random directions, we matched the force norm per candidate to the RoFB force computed at that control’s current state, including the model-specific projection. Both comparisons for ItrSA++–Maze use the updated hyperparameters.

Model–taskSign-flipRandom direction
AKOrN–Sudoku91.5 → 69.8*96.6 → 88.3*
AKOrN–Maze77.2 → 38.4*7*8.4 → 76.3
ItrSA++–Sudoku78.3 → 76.9*78.6 → 79.5
ItrSA++–Maze84.5 → 52.0*84.0 → 81.5*
TRM–Sudoku74.2 → 59.0*81.4 → 70.3*
TRM–Maze91.9 → 92.092.8 → 93.0

Table 1. Accuracy (%), shown as RoFB → direction control. Sign-flip results use standard-length inference (256 updates for AKOrN/ItrSA++; 16 outer steps for TRM) on 20,000 Sudoku or 1,000 Maze instances. Random-direction results use four times this inference length on 500 instances. * indicates that RoFB outperformed random directions after Holm correction over 18 comparisons.

Sign flip reduced reported accuracy in five of the six pairs, and RoFB outperformed norm-matched random directions in three pairs after correction. Together with the strength-sensitivity analysis, these findings support the view that both feedback magnitude and direction contribute to RoFB’s effectiveness, with their contributions depending on the model and task.

Q-2. Number of perfect-validation trials and tie-breaking procedure

Thank you for asking us to clarify the hyperparameter selection procedure for AKOrN on Sudoku. Of the 30 Optuna trials, 10 completed evaluation on all 500 validation puzzles and 20 were pruned. Five completed trials—12, 15, 17, 25, and 28—attained 100% validation accuracy. We resolved ties by selecting the smallest trial index among those achieving the maximum validation accuracy, yielding trial 12. Neither test accuracy nor runtime was used for tie-breaking.

To assess sensitivity to this choice, we additionally evaluated all five tied configurations on the same 20,000 test puzzles. The table reports mean test accuracy (%) across the three inference seeds, with gains over the baseline in parentheses (percentage points).

ConfigurationTest accuracy
Baseline85.1
Trial 12 (originally selected)91.5 (+6.4)
Trial 1591.3 (+6.2)
Trial 1791.3 (+6.2)
Trial 2591.3 (+6.2)
Trial 2891.7 (+6.6)

All five tied configurations improved over the baseline in each of the three inference seeds, with mean gains of +6.2 to +6.6 percentage points. Thus, the gain at this operating point was not specific to the originally selected trial 12. We retain trial 12 and have not reselected hyperparameters based on these test results.

Thank you for the response and the new appendices. My main concern was generality across training checkpoints and the validity of the AKOrN–Sudoku selection, which are now resolved. For the final version, please: (1) state why the ItrSA++–Maze sweep was redone and confirm the new setting was chosen without reference to test results; (2) reflect Appendix J.1 in the main-text mechanism summary, since readout-free repulsion reproduces the Sudoku gains.

Thank you for the follow-up and helpful suggestions. We have revised the manuscript accordingly, with the following changes highlighted in blue:

(i) Added a note at the beginning of the appendix explaining why the ItrSA++ on Maze sweep was repeated and confirming that the new setting was selected without reference to test results. (ii) Updated the main-text “Two roles” discussion to reflect Appendix J.1, clarifying that constant repulsion without the entropy gate can improve Sudoku accuracy.

We would appreciate it if you could take a look at these updates. Again, thank you very much for your constructive feedback!

Comments on the paper

3 comments
AuthorsOpenReview ↗

Correction and Common Concerns among Reviewers 1/2

We sincerely thank the reviewers for their feedback. Below we report a correction and additional experiments. A revised manuscript with changes marked in red will follow.

Correction: ItrSA++ on Maze.

We audited the hyperparameter-search implementation and confirmed that ItrSA++ on Maze had used 500 validation boards, rather than the reported 100. The validation sizes for the other model–task pairs matched the reported protocol. We sincerely apologize for this error and the resulting confusion.

We repeated the search with 100 boards and re-evaluated the performance and re-performed relevant analyses. At the original training checkpoint, RoFB retains a +3.01 pp gain over baseline, with slightly lower accuracy than the original configuration. Inter-class separation remains supported as a contributing factor, while claims about hyperparameter robustness and entropy-gradient intervention are narrowed.

Robustness on Training Seeds.

We conducted additional experiments using three training seeds per model–task pair, including the original checkpoint, followed by the RoFB hyperparameter search for each seed. We report that RoFB outperforms the baseline for all three training seeds in five of the six model–task pairs, supporting robustness across the tested training seeds in the five successful pairs.

Table 1. Performance with hyperparameters selected separately for each training seed using training-derived validation puzzles.

Model–task pairBaseline → RoFB accuracy (%)Gain (pp), mean ± SDSeeds with positive gain
AKOrN on Sudoku84.13 → 90.39+6.25 ± 1.493/3
AKOrN on Maze43.75 → 45.51+1.77 ± 1.543/3
ItrSA++ on Sudoku77.35 → 78.29+0.94 ± 0.463/3
ItrSA++ on Maze82.01 → 83.54+1.53 ± 1.353/3
TRM on Sudoku64.47 → 71.00+6.53 ± 3.023/3
TRM on Maze91.23 → 91.20−0.03 ± 0.060/3

Robustness on Held-out Validation Sets.

We next conducted experiments of RoFB with held-out validation sets and found that RoFB outperforms the baseline for all three training seeds in the same five model–task pairs, showing that the gains in the five successful pairs persist after hyperparameter search on disjoint validation sets (Table 2).

For Sudoku, we performed hyperparameter search on 500 boards taken from the original Sudoku Extreme training set that were not used for model training and disjoint from the 20,000 test boards. For Maze, we performed hyperparameter search on 100 boards taken from the Maze Hard test set and evaluated performance using the remaining 900 test boards.

Table 2. Accuracy (%) with hyperparameters selected separately for each training seed using held-out validation puzzles.

Model–task pairBaseline → RoFB accuracyGain (pp), mean ± SDSeeds with positive gain
AKOrN on Sudoku84.10 → 90.19+6.08 ± 1.713/3
AKOrN on Maze43.17 → 45.40+2.23 ± 1.553/3
ItrSA++ on Sudoku77.34 → 78.51+1.16 ± 0.773/3
ItrSA++ on Maze81.65 → 83.92+2.27 ± 1.543/3
TRM on Sudoku64.47 → 70.24+5.77 ± 2.283/3
TRM on Maze90.96 → 90.960.00 ± 0.000/3
AuthorsOpenReview ↗

Common Concerns among Reviewers 2/2

Hyperparameter Transferability.

We also conducted experiments to evaluate the transferability of hyperparameters across different training seeds and found that RoFB still outperforms the baseline in most transfers for the five model–task pairs that benefited from per-checkpoint tuning (Tables 3a & b).

To evaluate transferability, we performed hyperparameter search on one seed and then evaluated RoFB using the selected hyperparameters on another seed within the same model–task pair. Some seeds in the same model-task pair contain identical hyperparameter values for some checkpoints due to shared initial candidates and tied validation scores. We excluded transfers between checkpoints with identical selected settings from the summaries below.

Table 3a. Accuracy (%) after transferring hyperparameters selected using training-derived validation puzzles, excluding identical-setting transfers.

Model–task pairBaselineCheckpoint-specific RoFBTransferred RoFB
AKOrN on Sudoku84.3890.26 (+5.88)90.46 (+6.08)
AKOrN on Maze43.7545.51 (+1.77)44.99 (+1.24)
ItrSA++ on Sudoku77.3578.29 (+0.94)78.22 (+0.87)
ItrSA++ on Maze82.0183.54 (+1.53)83.38 (+1.37)
TRM on Sudoku64.4771.00 (+6.53)69.69 (+5.22)
TRM on Maze90.9390.90 (−0.03)90.90 (−0.03)

Table 3b. Accuracy (%) after transferring hyperparameters selected using held-out validation puzzles, excluding identical-setting transfers.

Model–task pairBaselineCheckpoint-specific RoFBTransferred RoFB
AKOrN on Sudoku84.1090.19 (+6.08)90.24 (+6.14)
AKOrN on Maze43.1745.40 (+2.23)44.53 (+1.37)
ItrSA++ on Sudoku77.3478.51 (+1.16)78.32 (+0.98)
ItrSA++ on Maze81.6583.92 (+2.27)83.74 (+2.09)
TRM on Sudoku64.4770.24 (+5.77)69.45 (+4.99)
TRM on Maze91.1791.17 (0.00)91.08 (−0.08)

Generalizability.

To address the generalizability of RoFB, we conducted additional experiments on graph coloring (30 vertices, four colors), Nonogram (15×15), and Futoshiki (9×9). We evaluated 2,000 test problems per task across three training seeds. Entries are mean baseline → RoFB accuracy (%), averaging eight inference seeds for AKOrN/ItrSA++; TRM is deterministic. Hyperparameters were selected on separate validation sets.

Table 4. Accuracy (%) of each model on each additional task .

TaskAKOrNItrSA++TRM
Graph coloring99.36 → 99.3792.65 → 93.0455.62 → 56.08
Nonogram99.58 → 99.5499.65 → 99.6599.80 → 99.78
Futoshiki83.00 → 86.6593.25 → 93.5393.22 → 93.85

Futoshiki improves in every training seed for all three models; graph-coloring gains are small, and the TRM average includes a 0%-accuracy baseline run. Nonogram shows no improvement.

On 100 Mini-ARC tasks, a public TRM checkpoint trained on Mini-ARC improved pass@1 from 36% to 37%, but a runtime-matched longer baseline solved the same 37 tasks. The gain was not statistically significant. Benefits therefore remain task- and model-dependent.

AuthorsOpenReview ↗

Manuscript Updated

We thank again all reviewers for their comments. We have submitted our revised manuscript with changes marked in red; the main text still ends on page 10. Reviewer IDs indicate related concerns. Details are provided in our Common Concerns comment and reviewer-specific responses.

Added

  • Evaluation across three training seeds for all six model–task pairs, with both training-derived and held-out validation — Apps. O–P, Tables 23–28. [qxqx, dJ3S, mzvo]
  • Hyperparameter transfer across training checkpoints within each model–task pair, excluding identical-setting transfers — App. Q, Tables 29–32. [qxqx, dJ3S]
  • Additional tasks: Graph coloring, Nonogram, and Futoshiki with all three models, plus Mini-ARC with TRM; a short summary is also added to the main text — Sec. 5.1, App. R, Tables 34–35. [qxqx, dJ3S, mzvo]
  • Constant and readout-dependent repulsion, each with and without the entropy gate, compared on 500 shared test boards per task — App. J.1, Table 15. [dJ3S]

Corrected

  • ItrSA++ on Maze: the submitted search used 500 validation boards rather than the reported 100. We repeated the search with 100 boards and reran affected analyses; RoFB still gains 3.01 percentage points at the standard operating point — Sec. 5, Fig. 3, Apps. E–K. [qxqx, dJ3S]
  • Search cost: replaced the uniform 0.5 pruning estimate with trial-wise accounting of evaluated boards and feedback onset, and corrected TRM's inner/outer update units. The main accuracy–compute conclusion, as well as accuracy and selected settings, is unchanged; the search-inclusive wording is corrected from “all six” to "five of six" pairs over part of the interpolated cost range — App. F, Eq. 26, Table 4, Fig. 9. [dJ3S, mzvo]
  • Wall-clock results: updated ItrSA++ on Maze using GH200 measurements with linear scaling to other horizons and the search time updated for the repeated search — App. F, Figs. 10–11. [dJ3S, mzvo]
  • Methods: corrected the fixed Sudoku test-subset description and TRM optimizer, and specified the earliest-trial tie-break for equal full-validation accuracies — Sec. 5, Apps. D–E, N. [qxqx, dJ3S]

Clarified

  • Appendix L used separately trained AKOrN checkpoints. A new comparison separates checkpoint and RoFB-setting differences; training-environment differences remain a possible explanation, not an established cause — App. Q.2.1, Table 33, with a cross-reference from App. L.
  • Removed the broad “10–100×” efficiency wording and narrowed the sensitivity, alternative-intervention, and repulsion-role claims to the corrected evidence — Secs. 1 and 5, App. H. [mzvo, qxqx, dJ3S]
  • Readout dependence is not necessary for every Sudoku gain, and benefits remain model- and task-dependent. The limit on predicting RoFB's benefits remains explicit, and the language-model limitation is made more specific: readouts over a large vocabulary may overlap little across positions, making the coupling nearly uniform — App. J.1, Sec. 6. [qxqx, dJ3S, mzvo]

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.