acceptodds
Under review as a conference paper at ICLR 2027

When Recursive Models Finish Computing

Abstract

Recursive models can continue updating their latent states beyond their nominal inference budget, so an incorrect output at that budget does not show whether computation is unfinished or has entered a persistently unsuccessful regime. We study the dynamics of completion in attention and MLP-based Tiny Recursive Models (TRMs) on 1,000 hard Sudoku puzzles. Extending recurrence from the nominal 16 steps to 512 steps increases cumulative exact-solve accuracy from 59.2% to 87.5% for the attention model and from 74.4% to 91.9% for the MLP model, solving more than two-thirds of the puzzles unsolved in the nominal budget. Across both architectures, latent-state motion drops sharply after the first exact solution. Completed states are typically locally contractive along the trajectory direction, even though the same local Jacobian retains strongly expanding directions. We characterize this phenomenon as trajectory-conditioned anisotropic stability. Perturbation experiments confirm this directional stability across both models. The multi-step fate of the maximally expanding direction differs: it is absorbed within 16 steps in the attention model but persists longer in the MLP model. The anisotropic-stability pattern also holds for a second attention checkpoint. Together, these results distinguish nominal-budget failure from completed computation and identify a common dynamical signature of completion across two recurrent architectures.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
Recurrent computation, Recursive models, latent dynamics, Stability analysis

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_neNbOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

Recent papers have proposed recurrent models with the ability of leveraging more recurrent steps at inference than used during training to improve task performance. This paper highlights that when these models output an incorrect answer, it may be because the model has not carried out enough recurrent steps yet or it has entered a regime where it is persistently incorrect. To gain more insight into how these models work, the authors study the hidden state dynamics. They show that the accuracy increases when the recurrent steps are increased, and they perform various analyses, such as showing that the latent settling aligns with computational completion and that the completed computation is directionally stable.

Critical strengths

The paper does a good job of describing the analysis and the paper is clear and easy to follow. The plots are particularly nice, and the alignment plots in Figure 2 really help emphasize that early and late solvers share similar trajectories once aligned.

Critical weaknesses

My main concern is with the impactfulness of this paper. In particular, I believe some of the results are already known and the authors struggle to motivate the novel analysis.

The paper highlights that more recurrent steps can improve the performance of recurrent models like the TRM. This is already known, for example in [1], where they show increasing the number of recurrent steps improves performance in Sudoku, albeit they only increase from 16 to 48 recurrent steps and not 512 as is done here. Similarly, both [1] and [2] also acknowledge the existence of trajectories where there is quick success, delayed success and failure, similar to this paper's definition of early, late and persistent trajectories. Additionally, the analysis in Section 3.2 is similar to the analysis of the latents in [1] figure 2, which already suggests that there is a large reorganization at the puzzle solving step, followed by much less movement.

It is not clear to me how the novel findings, such as the finding that the trajectories are locally contracting along their direction of motion will have an impact in the field or for improving recurrent models. The authors do a good job describing the analysis, but struggle to provide a concrete motivation for doing so, with the short discussion and conclusion sections not clearly describing avenues for follow up work.

A minor weakness is that the authors use a very limited number of model weights in their analysis.

[1] Probabilistic Tiny Recursive Model, Sghaier et al., 2026

[2] Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models, Ren and Lie, 2026

Questions for authors

What do the authors see as the main motivation for studying aspects like the directional stability of trajectories? How is understanding these dynamics useful for improving recurrent models?

A clear answer here would affect my assessment.

3 comments

Among the PERSISTENT unsolved puzzles, 62 (14 in Attention-A and 48 in Attention-B) settle on an incorrect board. Targeted kicks,

s~t=st±α∥st∥v1,∥v1∥=1,\tilde{s}_t=s_t\pm\alpha\lVert s_t\rVert v_1,\qquad \lVert v_1\rVert=1,

using the leading right singular vector of the recurrence Jacobian, rescue 6–11 puzzles (10–18%) at every tested strength (new Figure 15c). At α=0.03\alpha=0.03, they reach an exact solution within 96 updates in 10/62 cases, compared with 0/62 for random kicks of equal size. Random kicks solve none at α≤0.03\alpha\leq0.03, showing that direction matters at small strengths. At α=0.3\alpha=0.3, rescue rates are 18% for v1v_1, 6% for isotropic random kicks and 13% for random kicks restricted to ZHZ_H. These interventions also carry a cost: kicks disrupt 2 of 24 completed attention states, though none in the MLP. This motivates applying them only to settled states having a wrong solution.

The choice of direction distinguishes this result from [1, 2]. While [2] shows that arbitrary latent perturbations can destabilize dynamics, [1] explores inference-time diversity through isotropic Gaussian noise across 100 rollouts. Its directed alternative uses Langevin sampling guided by the gradient of an energy defined through the Q-head. Removing this gradient while keeping the noise gives essentially the same accuracy [1, App. F], indicating no measurable benefit from that direction. Both comparisons isolate the value of direction: [1] finds no benefit from the Q-head gradient, whereas our experiment finds a benefit from a direction derived from the recurrence Jacobian.

Our EARLY/LATE/PERSISTENT groups correspond to the trajectory modes in [1, 2], but we define them operationally for all 1,000 puzzles and align trajectories to their completion time. Figure 2 of [1] presents separate two-dimensional PCA projections of the answer latent for PPBench puzzles. Their axes differ and cannot be compared directly. Our Section 3.2 measures full-state motion across all late solvers. Low motion alone cannot distinguish an attractor from slow neutral drift or proximity to a saddle. Sections 3.3–3.5 examine these possibilities and identify the v1v_1 direction used above.

Completion coincides with a drop of more than 10×10\times in latent motion, aligned to each puzzle’s solve step (Figure 2), and solutions usually persist once found (859/875, 919/919 and 719/727). On Maze-Hard, valid alternative paths settle like reference-matching paths, while invalid paths remain about 1000×1000\times more active (Appendix K). Motion therefore provides a training-free signal of settling across tasks.

These diagnostics suggest how to allocate compute. About 69% of nominal-budget failures are resolved by additional steps alone. A trajectory that is still moving can receive more steps; one that has settled but fails a correctness check can receive a v1v_1 kick; one that has settled and passes can stop. This motivates allocating compute by trajectory state instead of assigning KK rollouts to every puzzle as in [1].

The stability results also suggest what to regularize. Completion lowers the estimated local spectral radius ρ\rho from 3–5 to at most 1, while σmax⁡\sigma_{\max} remains 6–27, revealing strong non-normality. Perturbations along v1v_1 initially grow, then decay in the attention models or remain bounded in the MLP (Figure 15b). Jacobian-norm penalties stabilize fixed-point solving in equilibrium models [3]. For recurrent reasoners, our results motivate studying ρ\rho or the trajectory gain γnat\gamma_{\mathrm{nat}} as regularization targets.

The main analysis covers four checkpoints: Attention-A, Attention-B, an MLP, and a Maze model. All exhibit trajectory-conditioned anisotropic stability. Attention-B has 25 saved training snapshots; the later snapshots show the same qualitative behavior as the reported terminal checkpoint, so we omitted these repetitive results. We also add a public MLP checkpoint and a separate DT-Recall maze study, where 954/1,000 trajectories first become valid after 30 updates. Further training runs are underway to assess generality across more checkpoints.

[1] A. Sghaier, A. Parviz, and A. Jolicoeur-Martineau. Probabilistic Tiny Recursive Model.

[2] Z. Ren and Z. Liu. Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models.

[3] S. Bai, V. Koltun, and J. Z. Kolter. Stabilizing Equilibrium Models by Jacobian Regularization.

Thank you for the timely response.

Regarding the usefulness of analyzing the directional stability of trajectories, the authors propose to use targeted kicks to the states using the recurrent Jacobian. The authors claim that these target kicks can rescue 10 to 18% of persistently unsolved puzzles, although this also disrupts ~8% of correct solutions. The authors then suggest that this intervention should only be applied when we know the model has reached an incorrect solution. While this suggestion is sensible, I do believe it limits the applicability of this method, as there are many cases where the correct solution may not be known or verifiable at test time.

While I am still not fully convinced of the novelty of this work, the authors have however provided a way in which this method can be used for allocating compute and I am therefore inclined to raise my score.

Thanks for pointing this out. In our test, a targeted perturbation of 3% of the state norm recovered the correct answer in 10 of 62 cases where an Attention model had settled on a wrong answer.

Reviewer_EM7dOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the dynamics of Tiny Recursive Models (TRMs) beyond their nominal inference budget. Using an attention-based and an MLP-based TRM checkpoint on 1,000 hard Sudoku puzzles, the authors extend the outer recurrence from 16 to 512 steps with unchanged weights. Cumulative exact-solve accuracy rises from 59.2% to 87.5% (attention) and from 74.4% to 91.9% (MLP), so roughly two-thirds of nominal-budget failures are in fact unfinished computations.

Critical strengths
  • Distinguishing unfinished computation from failed computation is important for understanding test-time compute in recurrent and recursive reasoners. The 16-to-512-step result is simple, striking, and well documented.

  • The paper is in good-writing and easy to follow.

Critical weaknesses
  • Gains from increasing recurrent steps at inference time are not entirely new. To my knowledge, the HRM paper reports inference-time scaling with a larger M_max, and the logical-extrapolation literature (Schwarzschild et al., 2021; Bansal et al., 2022) studies “thinking longer.” The absence of overthinking here (solutions are almost never lost) contrasts interestingly with Bansal et al. and deserves discussion.

  • The full dynamical analysis is concentrated on a single task and a small number of checkpoints. The additional tasks provide limited evidence about late-solving behavior. Moreover, the perturbation analysis follows the expanding direction for only 16 additional steps, leaving its longer-term fate, particularly in the MLP model, unresolved.

Questions for authors
  • Since contraction along the natural trajectory direction is closely related to decreasing latent-state updates, can the authors demonstrate a setting where the proposed characterization provides insights beyond the observed settling pattern?

  • Can the authors extend the full analysis to additional independently trained checkpoints or tasks with substantial numbers of late solvers, and track expanding-direction perturbations beyond 16 steps to clarify their long-term behavior?

2 comments

We thank the reviewer for the valuable comments.

W1 (thinking longer and overthinking). Accuracy gains from more inference-time recurrence are known (HRM; Schwarzschild et al., 2021; Bansal et al., 2022). We added the overthinking discussion (§3.2). On Sudoku, TRM shows almost no overthinking. First solutions are kept for 859/875 (Attention-A), 919/919 (MLP), 719/727 (Attention-B) and 944/944 (a fourth, public MLP checkpoint) puzzles. Dynamically, completion makes the local Jacobian stable, and perturbations along the most-expanding direction v1v_1 at completed states decay or stay bounded for 256 steps without losing a solution. A plausible cause is that TRM has analogues of both of their fixes. The first is recall. TRM adds the input embedding to every zLz_L update, so the puzzle is always available to the update, much as their recall connection feeds the input into every iteration. The second is progressive training. TRM's deep supervision starts each step from the detached state of the previous step and applies the same loss at every step. Like their progressive training, this encourages an update that improves the answer at any depth rather than at one particular step. We have not ablated these analyses, so we state them as hypotheses.

Q1 (insight beyond settling). We agree: γnat\gamma_{\text{nat}} approximates the ratio of successive update norms (Eq. 5), so γnat<1\gamma_{\text{nat}} < 1 and shrinking motion are the same observation, which is why a motion-based stopping rule works. What goes beyond settling is the rest of the Jacobian (ρ\rho, σmax⁡\sigma_{\max}, v1v_1), and it matters most where puzzles settle on a wrong answer. Most persistent-unsolved puzzles never settle, but 62 do (14 of 125 in Attention-A, 48 of 273 in Attention-B). The Jacobian shows that they are trapped, ρ<1\rho < 1 at 86–92% of them, γnat<1\gamma_{\text{nat}} < 1, and σmax⁡\sigma_{\max} matches that of correct completions. The same analysis also points a way out. A kick s~t=st±α∥st∥v1\tilde s_t = s_t \pm \alpha \lVert s_t \rVert v_1 with α=0.03\alpha = 0.03 reaches the exact solution for 10 of the 62 puzzles (16%), while random kicks of the same norm solve none. At α=0.3\alpha = 0.3, v1v_1 kicks rescue 18% against 6% of random control. Since these kicks can remove the solution at 2 of 24 completed attention states, they are applied only to settled states that failed. The resulting compute policy is simple: keep stepping while a trajectory moves, kick it along v1v_1 if it settles on a wrong answer, and stop if it settles on a correct one.

W2 (scope). Every experiment in the paper (512-step trajectories, Jacobian gains, perturbations, etc.) is run in full on three checkpoints (Attention A/B, MLP). Results agree across all three. On mazes, a different task, we find the same settling. In TRM on Maze-Hard, valid paths settle like exact matches, while invalid paths stay about 1000× more active (App. K). We also trained three independent MLP runs with the same recipe and evaluated their final checkpoints on the same 1,000 Sudoku puzzles. Across all six unperturbed evaluations, 78.5–84.8% of puzzles unsolved by step 16 reached an exact answer by step 512. We have added appendix N that gives the full breakdown.

Q2 (longer horizon). We now follow v1v_1 perturbations for 256 steps instead of 16, at 24 completed states per checkpoint. In Attention-A and Attention-B, the perturbation grows 26× and 16×, falls back below its injected size within 7-8 steps, and has decayed to the float32 numerical floor (about 11\\% of its injected size) by k≈32k \approx 32. In the MLP, the perturbation grows 6.6×, falls back below its injected size after about 20 steps, and then levels off at about half its injected size up to k=256k = 256. The long-term fate is therefore transient amplification followed by decay in the attention models, and by a plateau at about half the injected size in the MLP. No solution is lost: all 72 perturbed states decode the exact answer at k=256k = 256.

Also thanks again for asking whether the perturbation results extend to independently trained checkpoints. Appendix N (Table 13) reports Jacobian and perturbation tests for all three new MLP seeds. All 56 sampled states that were correct at step 512 kept the exact Sudoku answer at every one of the next 128 updates across all 672 perturbation trials.

Reviewer_6gKbOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
3(+1)
Read review
Paper summary

This paper looks at what happens inside a Tiny Recursive Model when it finishes solving a puzzle. The authors take released TRM checkpoints, keep running the same recurrence far past the nominal 16-step budget on hard Sudoku puzzles, and align each trajectory to the step at which that puzzle is first solved. Running longer solves many more puzzles. For the main attention checkpoint (Attention-A) the exact-solve rate goes from 59.2% to 87.5%, so most of the nominal-horizon failures were computations that had not finished. Aligned to the solve step, latent motion is flat before the solution, spikes at the solving step and then drops to a low level. The authors then probe the local Jacobian at visited states and report that completed states contract along the direction the trajectory is moving while the largest singular value stays well above one, which they call trajectory-conditioned anisotropic stability. The analysis is repeated on a TRM checkpoint that replaces attention with MLP mixing and on a second attention checkpoint, with more limited checks on easy Sudoku and Maze-Hard.

Critical strengths

The solve-aligned view is the most useful thing in the paper. Fig. 1(b,c) shows that at step 16 the puzzles that will later be solved are still moving about as much as the ones that never are, while the puzzles solved early have already settled. Together with Fig. 1(a), lining every solved trajectory up at its own solve step (Fig. 2) puts the early and late solvers on the same curve, which is a clean way to show that a large share of the nominal-horizon failures is a budget problem rather than a model failure. The check in Appendix D, where the persistently unsolved puzzles if anything need fewer solver backtracks, also argues against the obvious explanation that late and persistent puzzles are simply harder, at least by that measure. This part of the paper is convincing and is the main reason I lean toward acceptance.

Critical weaknesses

I am less sure what the stability analysis adds. I don't work directly on the dynamics of recurrent reasoners, so the authors should correct me if I'm misreading this. The perturbation experiments do show that completed states contract along the trajectory direction, but in Fig. 4(a) random directions contract at least as fast, and in the MLP model (Fig. 11(a)) the trajectory direction is the slowest of the three. So completed states contract in essentially every direction tested except v1v_1, which in the attention model expands for a few steps and is then rotated away. My understanding is that this is what one would expect near a stable fixed point of a non-normal map. Since the largest singular value is above one at every probed state (Fig. 3(c)), the second condition in Eq. 9 never separates one probed state from another, and "trajectory-conditioned anisotropic stability" reads to me as ordinary local convergence plus transient amplification. The paper mentions non-normal transient amplification as a possible mechanism but does not test it; it would help to say what the name adds beyond that. I would also like a numerical check of the low-motion regime. In Fig. 1(b,c) completed trajectories settle to a level and then stay flat for hundreds of steps, while in the Maze-Hard appendix, where the paper says differences in settling cannot be resolved reliably and replays the recurrence in float64, the updates keep falling through all 16 steps (Fig. 13(b,c)). The Sudoku trajectories run in bfloat16 and the Jacobians in float32, so a matched-precision replay would show whether the plateau is real.

The second issue is scope. The full analysis covers one task with puzzles from a single source list, and the main claims rest on one checkpoint for each of the two architectures in the abstract's "common dynamical signature", with more limited checks on easy Sudoku and Maze-Hard. The two-thirds figure holds for the two primary models, but a second checkpoint of the same attention architecture gives 43% (Table 5), so the share seems to depend on the training run, and the architecture difference in the fate of v1v_1 rests on one MLP checkpoint and about a hundred completed states per model, with an MLP interval that includes one. That extra iterations let recurrent models solve harder instances has been shown before (Schwarzschild et al., 2021; Bansal et al., 2022), and Equilibrium Reasoners (Huang et al., 2026) ties test-time gains on Sudoku-Extreme to convergence toward solution-aligned attractors. The sudden solving step also resembles the critical reasoning step Ren & Liu (2026) describe for HRM. The paper cites all of these and calls its question complementary to the dynamics work; a sentence on what its results add would help. I may not know every paper in this fast-moving line, but as far as I can tell what it adds is mainly the solve-aligned measurement, which is useful on its own.

References:

Arpit Bansal, Avi Schwarzschild, Eitan Borgnia, Zeyad Emam, Furong Huang, Micah Goldblum, and Tom Goldstein. End-to-End Algorithm Synthesis with Recurrent Networks: Extrapolation without Overthinking. In Advances in Neural Information Processing Systems (NeurIPS), volume 35, pages 20232–20242, 2022. https://doi.org/10.52202/068431-1471

Benhao Huang, Zhengyang Geng, and Zico Kolter. Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. In International Conference on Machine Learning (ICML), 2026. https://arxiv.org/abs/2605.21488

Zirui Ren and Ziming Liu. Are Your Reasoning Models Reasoning or Guessing? A Mechanistic Analysis of Hierarchical Reasoning Models. arXiv:2601.10679, 2026. https://arxiv.org/abs/2601.10679

Avi Schwarzschild, Eitan Borgnia, Arjun Gupta, Furong Huang, Uzi Vishkin, Micah Goldblum, and Tom Goldstein. Can You Learn an Algorithm? Generalizing from Easy to Hard Problems with Recurrent Networks. In Advances in Neural Information Processing Systems (NeurIPS), volume 34, pages 6695–6706, 2021. https://proceedings.neurips.cc/paper/2021/hash/3501672ebc68a5524629080e3ef60aef-Abstract.html

Questions for authors
  1. Clarification: If the Sudoku trajectories are replayed in float32 or float64, as in the Maze appendix, do the completed-state plateaus in Fig. 1(b,c) stay at the same level, and does any puzzle change group? Were the perturbed trajectories evolved in bfloat16 or float32? If bfloat16, how large is the perturbation after casting, since a 10−410^{-4} perturbation relative to the state norm may be partly rounded away?
  2. Clarification: At completed states, how do γnat\gamma^{\text{nat}} and the natural-direction perturbation compare with random directions state by state? What is the spectral radius of JtJ_t there?
  3. Suggestion: Can latent motion or γnat\gamma^{\text{nat}} detect completion without the ground truth, for example as a halting rule? That would show what the signature is useful for, and how it differs from the convergence criteria in Huang et al. (2026).

References:

Benhao Huang, Zhengyang Geng, and Zico Kolter. Equilibrium Reasoners: Learning Attractors Enables Scalable Reasoning. In International Conference on Machine Learning (ICML), 2026. https://arxiv.org/abs/2605.21488

5 comments

Thank you for the careful review.

Switching from BF16 to FP32 changed the solve-time group for 33/150 Attention-A and 27/150 MLP puzzles. At 32 updates after each run's first exact answer, the median change in the high-level hidden state (as a fraction of its size before the update) fell from 0.00609 to 0.00000926 in Attention-A (42 paired cases) and from 0.0335 to 0.0325 in the MLP (46 paired cases). With modified FP64, these changes became smaller in five selected Attention-A cases and remained measurable in four selected MLP cases. Relative to FP32, 0/6 Attention-A and 1/6 MLP puzzles changed groups. In the original tests, we added a one-time perturbation to one copy of the hidden state, equal to 0.01% of its magnitude. We applied the perturbation and continued both copies in FP32, then compared their states and answers.

Perturbations along the model's usual update direction generally had a larger effect after one update than the median effect across four random directions at the same state. At update 512, we took states that gave the correct Sudoku answer. We estimated the one-step gain (the factor by which one update changes the size of a small perturbation) along the model's usual update direction. We divided it by the median gain for four random directions at the same state. The median ratios were 1.71, 3.73, and 1.47 for Attention-A, the original MLP, and Attention-B. In direct tests at both sizes, the usual direction left a larger gap from the unperturbed run after one update than the median random gap in 15/17, 16/16, and 16/16 states for those checkpoints, and 20/20 for the public MLP.

We also estimated the spectral radius (the largest absolute eigenvalue of the local update). The median estimates were 0.794, 0.966, and 0.672 for Attention-A, the original MLP, and Attention-B. An estimate above one indicates a growing mode in the local linear model. This occurred in 0/17, 6/16, and 0/16 states, respectively. The Attention results are consistent with non-normal transient amplification (short-lived growth even when the estimated spectral radius is below one).

All 69 selected states kept the correct Sudoku answer at every step for 128 updates in all four tests. We applied perturbations along the model's usual update direction and the direction estimated to amplify a small change most in one step, each at 0.01% and 0.1% of the state's magnitude. In the separate test, all 72 selected states gave the correct answer after 256 updates. Whether the perturbed states converge as updates continue indefinitely remains open.

The Sudoku analysis draws on a fixed set of 1,000 Sudoku-Extreme puzzles. We provide the exact rows, and the original selection rule is unknown. Attention-B recovered 43% of nominal-budget failures. Three new MLP seeds reproduced delayed solving and smaller hidden-state changes after solving in BF16 and FP32. The original and rerun MLP intervals at 16 updates both included the initial perturbation size. In the separate 256-update test, median separation was 0.48 times that size.

On puzzles reserved for testing, a rule that stops when the hidden state changes very little solved 168/200 for Attention-A and 186/200 for the MLP, one fewer per model than at 512 updates. It averaged 106.8 and 56.2 updates, respectively. Our rule compares consecutive hidden states within one run. Equilibrium Reasoners (Huang et al., 2026) are trained so repeated updates tend to settle on correct solutions. They use a learned stopping rule and separately rank restarts by the average hidden-state change over the last three updates.

Thank you for the detailed response. The float32 and float64 reruns (Appendix M), the state-by-state comparison with random directions, the spectral radius estimates (Appendix L) and the halting rule answer most of my questions. My concern now is that the main text does not reflect these results, and in places contradicts the appendices and your replies. This matters more for my assessment than any missing experiment.

  1. Appendix M shows that the Attention-A plateau after solving is a bfloat16 floor (Figure 16), and that precision changes the completion group of 33 of 150 puzzles. The drop in motion after solving still holds, but Figures 1 and 2 and Sections 3.1–3.2 present the bfloat16 level without saying so, and the main text never refers to Appendix M.

  2. Your reply says the stability label and the MLP persistence claim were removed, but both remain in the abstract, and Section 3.5 still uses the label. The claim in Section 3.2 that completed states have spectral radius at most one also does not hold for the MLP: 10 of 24 correct states are above one (Appendix L), and so is the median for all three new seeds (Table 13). I would ask you to align the abstract and main text with your reply and the appendices.

  3. On the halting rule you added in response to my question, how were the 0.05 threshold and five-step window chosen, and are the 200 test puzzles separate from the 1,000 used elsewhere?

Thanks for the follow-up. We've clarified these points in the revised draft.

Clarifications to our first comment: We removed the statements on the "stability" and "persistence" claims in our previous comments. We apologize if they were confusing.

Figures 1 and 2 now label the Attention-A runs as BF16, and Section 3.2 points to Appendix M. Hidden-state updates shrink after solving in BF16 and FP32. The flat BF16 level is consistent with a numerical floor. Switching to FP32 changed the solve-time group for 33 of 150 Attention-A puzzles.

We have removed "stability" and "persistence" in our previous reply. Now the abstract and main body of the paper reflects the new and old experiments including the comment.

We fixed the three-consecutive-update for the halting rule. We used 600 puzzles to propose thresholds and 200 to pick the thresholds that saved the most updates while staying within one percentage point of the accuracy at 512 updates. The thresholds were 0.240 for Attention-A and 0.405 for the MLP. The remaining 200 of the 1,000 puzzles were reserved for testing both models.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of changes in the revised manuscript.

We thank the reviewers for their feedback and discussion. We have revised the manuscript to include additional experiments and clarify the interpretation and novelty of our findings. The main changes in the revised PDF are:

  1. Introduction -- Directional stability. We expanded “Completion has a directional stability signature” to highlight how Jacobian geometry guides interventions when trajectories settle on incorrect answers. Targeted perturbations recover solutions in 10 of 62 selected cases within 96 updates, while continued recurrence and random perturbations of the same norm recover none.

  2. Figure 1. We revised the caption for clarity.

  3. Section 3.2 --Solution stability and overthinking. We expanded the discussion of whether exact solutions persist under extended recurrence and how this relates to overthinking.

  4. Section 3.4 -- Fate of perturbations. We added results from tracking perturbations for 256 updates, expanding the discussion of how the fate of a v1v_1 perturbation depends on computational completion.

  5. Section 3.4 -- Additional insights and halting. We added two paragraphs, “Insight beyond settling” and “Halting rule,” discussing the additional insights from our analyses and their implications for halting.

  6. Section 3.5 -- Comparison across checkpoints. We updated “The crucial difference lies in the expanding direction” to incorporate the results of the 256-update perturbation experiments.

  7. Discussion. We expanded the discussion beginning with “Beyond describing settling” to explain the implications of the additional findings.

  8. Table 5. We added a row reporting local spectral-radius estimates alongside the existing measurements.

  9. Appendix A -- Reproducibility. We updated the reproducibility statement to include information about the additional experiments.

  10. Appendices L onward. We added appendices documenting the additional analyses, experimental procedures, and results.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.