- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper aim to establish the relation between fixed point convergence of a implcicit deep learning model and its actual performance over tasks. To achieve so, a continuous residual dynamics is formulated and analyzed under local linearization. In my opinion the established mismatch is true and well supported. Following this the paper tries to establish a new training algorithm that retrains the fixed point to prediction part of the model in the hope to address the mismatch, they do so through noisy perturbation near the fixed point, in my understanding essentially readjusting the sensitivity of the loss so that it decreases along the converging dynamics. The method is evaluated on several different datasets and compared against some baselines, one thing it does seem to establish quite well is the monotonicity of the task performance as the number of loop increases. The rest of the results is a bit hard to parse for me.
Critical strengths
I think the paper identified a real issue that is rather underdiscussed in looped models or DEQ style models in general, that is whether the contraction of the dynamics alone means anything to the actual prediction of the model, and I think the paper did a good job establishing their claim, both in theory and numerically, and this is where I do think the linearization model is acceptable.
Under the papers own assumptions and setup the theory claims and derivations seem correct.
Critical weaknesses
I am in general very skeptical of the linearization assumptions made throughout the paper, where this is a common technique used in dynamical systems, but mostly on low dimensional problem, here the same assumptions are made willing to large neural network models with millions of parameters with high dimensional hidden state sizes, I do not consider it to be legal until some evidence can be shown, whether theoretical or empirically. this is particularly troubling when you are making a Lyapunov argument solely based on these assumptions, to which I remain skeptical.
The numerical results section reads weak to me. First, the main table is poor in presentation, with models of different origins, sizes, and numbers mixed together. Details on these models and how they are trained are largely missing from the draft. I also think the datasets used in the work are on the smaller and simpler side, where it is very easy to saturate. I find the claims of the paper numerically unconvincing when multiple rows in the table are hitting extremely high accuracy at the same time as it will be hard to actually separate the reason behind these results. Table 2 is also a bit hard to get, for the Maze-Hard example B=1 essentially is all you need, doesn't that go against the idea of the paper? looks like the learned perturbation is just degenerate.
Lastly, there is some language that I’m not sure is well substantiated. For instance, the paper uses a lot of wording such as “reasoning” or “overthinking,” but I’m not sure there is evidence to suggest anything meaningful along those lines. Similarly, the depth-scaling results are also mainly based on one of the maze examples. The paper mainly targets a DEQ problem and aims to determine the fixed point convergence and corresponding prediction loss behavior. The consistent upselling in my opinion dilutes the actual contribution and focus of the work and makes it very hard to read.
Questions for authors
The theories are derived in continuous time, while the actual model uses a discrete recurrence that the paper interprets as an Euler discretization of the dynamics, this is fine but not without details taken care of, contraction properties and Lyapunov conditions also can break if time stepping is not chosen carefully, do you consider anything other than dt = 1? should i trust dt = 1? this would be a very large step size for most dynamical systems.
The second half of the theory is mostly on the output part of the full model with the dynamics fixed and not updated, i don't think you are assuming a new fixed point when deriving it, but in the actual implementation and algorithm the dynamics are also updated? why? seems like a pretty major discrepancy? does the method actually not work if you do it the way the theory suggest? I am overall skeptical in the assumptions made regarding the method and this major difference is certainly not helping.
The selection of K is not well discussed in my opinion? since everything in the paper is suppose to be "local" getting K wrong can be catastrophic under the setup right? How do you make sure K is properly determined? and what happens if I set it too small?
5 comments
Q2. How does the continuous-time analysis relate to the finite-step recurrence?
Our continuous-time formulation is used specifically to characterize the local dynamics near an attracting fixed point, as stated in Lines 192–200, where late recurrent states are already close to equilibrium. Similar continuous-time dynamical formulations have been used to analyze recursive reasoning models [1, 2]. The implemented NDLM itself remains the native finite-step recurrence used by the original looped model.
To address the gap between this local continuous-time analysis and the implemented finite-step model, we directly examine whether the actual recurrent states enter the regime assumed by the theory. As shown in Table R4.1, the late recurrent states exhibit a relative residual of only about 2.2%, while approximately 99.2% of the eigenvalues of the symmetric residual Jacobian are negative, indicating that the actual finite-step trajectories enter a predominantly contractive near-fixed-point regime. These results support the native finite-step recurrence as a reasonable practical realization of the local dynamics characterized by our continuous-time analysis.
[1] Es'kin, Vasiliy A., and Mikhail E. Smorkalov. “Dynamical Systems Theory Behind a Hierarchical Reasoning Model.” arXiv preprint arXiv:2603.22871 (2026).
[2] Chen, Lizhang, Jonathan Li, Chen Liang, Ni Lao, and Qiang Liu. “Training-Free Looped Transformers.” arXiv preprint arXiv:2605.23872 (2026).
Q3. Why are the recurrent dynamics updated during training if the theory conditions on fixed dynamics?
The practical training procedure can be viewed as a coordinate-style alternating optimization between the task model and the noise network. Given the current recurrent dynamics and residual Jacobian J, the noise network is optimized through to adapt the perturbation covariance to the current local dynamics. The task model is then optimized through under the resulting perturbations. Importantly, does not directly update the backbone. This procedure repeatedly applies the conditional covariance-design principle as the task model evolves, providing a practical counterpart to fixed-dynamics analysis.
We further examine whether the theoretical predictions remain valid after this joint optimization.
Table R4.2: Validation of the covariance-design predictions after joint training.
| Diagnostic | Result |
|---|---|
| Normalized Eq. (14) residual, EqR | 0.5267 |
| Normalized Eq. (14) residual, NDLM | 0.081 |
| : mean negative eigenvalue ratio | 91.49% |
| : mean near-zero eigenvalue ratio | 5.01% |
| Deterministic one-step | <0 at all evaluated depths |
After joint training, NDLM substantially reduces the normalized Eq. (14) residual from 0.5267 to 0.081, showing that the learned covariance better satisfies the theoretically derived matrix condition. Meanwhile, the spectrum of is strongly dominated by negative directions, with an average of 91.49% negative eigenvalues across the evaluated depths. The deterministic one-step task-loss change is also consistently negative. These results show that the jointly trained model retains both the covariance-design condition and the task-loss descent behavior predicted by the conditional analysis. The results are now included in Appendix E of the revised version.
Q4. How sensitive is NDLM to the choice of K?
The role of K is precisely to restrict structured perturbations to recurrent states closer to the local regime analyzed theoretically. In the submitted implementation, structured perturbations are applied only during the final two recurrent steps. We additionally evaluate the sensitivity to the injection schedule:
Table R4.3: Ablation on the noise-injection schedule.
| Noise Injection Schedule | Acc@16 ↑ | Acc@32 ↑ | Acc@64 ↑ | Acc@128 ↑ |
|---|---|---|---|---|
| All recurrent steps | 97.3 | 98.5 | 98.7 | 99.1 |
| Last 4 steps | 98.7 | 99.2 | 99.3 | 99.5 |
| Last 2 steps (default) | 98.7 | 99.2 | 99.4 | 99.5 |
| Last 1 step | 98.7 | 99.2 | 99.3 | 99.4 |
The results show that NDLM is not sensitive to the exact choice of the late-step injection schedule. Applying structured perturbations during the last 1, 2, or 4 recurrent steps yields very similar performance across all evaluated depths, with the default last-2-step setting achieving the best or tied-best accuracy throughout. In contrast, injecting structured noise at all recurrent steps consistently performs slightly worse. These results support restricting structured perturbations to late recurrent states, where the dynamics are closer to the local near-attractor regime analyzed in our theory, while also showing that performance is robust to moderate changes in K. The results are now included in Appendix E of the revised version.
Q5. How should the experimental results and the Maze-Hard breadth result be interpreted?
We agree that Table 1 does not clearly distinguish controlled comparisons from broader contextual baselines. Our primary controlled comparison is with EqR, while the remaining methods provide context across recent looped models. Detailed training configurations and hyperparameters for NDLM, EqR, and TRM are already provided in Appendix D.
We also agree that endpoint accuracy is close to saturation on some benchmarks. For this reason, we kindly note that our main empirical claim is not uniform superiority in peak accuracy, but improved test-time depth efficiency. Rather than relying on a single saturated endpoint, we compare the full accuracy–depth trajectory. We additionally evaluate three independent runs and report mean and standard deviation across three datasets:
Table R4.4: Test-time depth scaling across three independent runs.
| Dataset | Method | Acc@16 ↑ | Acc@32 ↑ | Acc@64 ↑ | Acc@128 ↑ |
|---|---|---|---|---|---|
| Maze-Unique | EqR | 97.7 | 98.6 | 98.5 | 99.2 |
| Maze-Unique | NDLM | 98.73 ± 0.49 | 99.23 ± 0.45 | 99.37 ± 0.33 | 99.50 ± 0.29 |
| Maze-Hard | EqR | 82.9 | 82.6 | 83.1 | 83.6 |
| Maze-Hard | NDLM | 85.10 ± 0.08 | 84.97 ± 0.25 | 85.0 ± 0.29 | 85.37 ± 0.19 |
| Sudoku-Lite | EqR | 83.8 | 88.4 | 91.9 | 93.2 |
| Sudoku-Lite | NDLM | 84.11 ± 0.36 | 88.96 ± 0.38 | 92.07 ± 0.26 | 94.22 ± 0.10 |
Across all three datasets, NDLM consistently outperforms EqR at every evaluated recurrent depth, showing that the improvement extends beyond the Maze-Unique setting. Moreover, the small standard deviations across three independent runs indicate that the observed improvements are stable across random seeds. These results provide additional evidence that NDLM improves test-time depth scaling consistently across different structured reasoning tasks. The results are now included in Appendix E of the revised version.
We further clarify that breadth B and the learned perturbation are distinct. At B=1, NDLM still uses the learned structured covariance and samples one stochastic trajectory; increasing B only evaluates additional trajectories. Therefore, the strong Maze-Hard result at B=1 does not indicate a degenerate perturbation. Instead, it shows that the learned covariance is already effective with a single trajectory, while additional breadth provides only a modest gain on this dataset. Larger breadth gains are observed on Maze-Unique and Sudoku-Lite.
We thank the reviewer for the detailed feedback and for recognizing the attractor–task mismatch as a meaningful issue, as well as the correctness of our derivations under the stated assumptions. We address the concerns regarding the local approximation, continuous-time formulation, training procedure, choice of K, experimental evidence, and scope of the claims below.
Q1. Is the local linearization meaningful for large neural network models?
Understanding why additional recurrent computation remains beneficial near convergence is still limited. Following prior analyses of looped and equilibrium models through local dynamics and Jacobian-based characterizations [1–3], we study the neighborhood of an attracting fixed point using a local linearization. Although this setting simplifies the full nonlinear dynamics, it reveals an important mechanism: residual contraction and task-loss descent are governed by different local geometries, and appropriately designed noise covariance can improve their alignment. This analysis directly motivates NDLM and leads to a practical covariance-learning algorithm that improves test-time depth efficiency in our experiments.
For practical deployment and fair comparison, NDLM uses the standard backbone and training protocol shared by TRM, EqR, and GRAM, with no additional contraction regularization imposed by the theory. To address the gap between the theoretical regime and the actual trained model, we directly examine whether the assumptions underlying the local analysis are reflected in practice. Specifically, we evaluate the residual magnitude, the spectrum of the symmetric residual Jacobian, and the local linearization error on Maze-Unique at late recurrent states. We measure the relative linearization error as
$ E_{\mathrm{lin}} = \frac{|R(z+\delta)-R(z)-J(z)\delta|_2} {|J(z)\delta|_2+\varepsilon}. $
using random perturbations δ.
Table R4.1: Empirical validation of the local theoretical regime on Maze-Unique.
Across the evaluated late iterations, the relative residual remains around 2.2%, indicating that the recurrent states operate close to the fixed-point regime considered in our analysis. Approximately 99.2% of the eigenvalues of the symmetric residual Jacobian are negative, indicating predominantly contractive local dynamics. The relative linearization error is also only about 0.5%, showing that the first-order approximation accurately describes the nonlinear residual map at the tested perturbation scale. These diagnostics provide direct empirical support that the local regime underlying our analysis is realized in the actual trained model. The results are now included in Appendix E of the revised version.
[1] Yang, Xiao-Wen, Ziyu Han, Xi-Hua Zhang, Wen-Da Wei, Jie-Jing Shao, Lan-Zhe Guo, and Yu-Feng Li. “Stabilizing Recurrent Dynamics for Test-Time Scalable Latent Reasoning in Looped Language Models.” Proceedings of the 43rd International Conference on Machine Learning (2026).
[2] Winston, Ezra, and J. Zico Kolter. “Monotone Operator Equilibrium Networks.” Advances in Neural Information Processing Systems 33 (2020).
[3] Bai, Shaojie, Vladlen Koltun, and J. Zico Kolter. “Stabilizing Equilibrium Models by Jacobian Regularization.” Proceedings of the 38th International Conference on Machine Learning (2021).