- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper examines whether wrong answers produced by weight-shared recurrent reasoners have settled into stable states or continue to change when the models run beyond their training budget. Most of the analysis uses an HRM trained by the authors on Sudoku-Extreme. The authors continue the model from its terminal states, with and without perturbations, and analyse the endpoints using Jacobian spectra and linear models of the updates. They also compare perturb-and-reselect methods from prior work with compute-matched alternatives, and repeat the diagnostics with a second seed, TRM, GRAM-guided TRM, and two additional puzzle tasks, Maze and N-Queens. Across these settings, wrong answers are less persistent than correct ones. The spectral and timescale patterns observed in HRM, however, do not carry over to the other settings.
Critical strengths
-
The paper asks a useful question about how recurrent-depth models fail as they are given more test-time compute. It is worth understanding whether a wrong terminal answer is stable or simply a state the model passes through. As the authors point out, neither a flat loss curve nor clusters in a PCA plot can establish this.
-
The unperturbed continuation control is well chosen. It separates the effect of running the model longer from the effect of adding noise, and shows that noise contributes little. The paired accounting of gained and lost solutions, together with the compute-matched baselines in Sec. 8, tells us more than net accuracy alone. I also appreciate that the authors report negative results, including the failure of several HRM findings to transfer to other settings (Fig. 2).
Critical weaknesses
1. No main conclusion that is both supported and new. The only finding that holds in all six settings (Fig. 2) is that correct answers persist more than wrong ones. For models trained with a halting head to stop on correct solutions, this is expected. The paper's framing question, whether wrong answers sit in attractors, is not resolved by its measurement, which compares the decoded answer at only two readouts. A trajectory cycling among wrong answers would also look "non-persistent", and the long-horizon data in App. N fit this reading: wrong-answer persistence stays at about 0.43 through 2048 segments with no trend, and the solved fraction of the wrong class levels off at 0.28–0.30. The authors acknowledge that periodic dynamics and attractors are not excluded (App. N, Conclusion). The remaining findings (endpoint spectra, the fast/slow split, injection sites) do not transfer beyond HRM by the paper's own tally.
2. Limited novelty. Perturb-and-reselect is acknowledged as "established prior work" (Sec. 1), and the analysis tools (endpoint spectra, linear fits to updates, perturb-and-continue tests) are standard. The practical conclusions are also familiar: extra depth fixes some errors (+5.6/+6.0 points), and task symmetry augmentation dominates random noise (+34.6 vs. +4.8). The paper ends up recommending against the procedure it studies, without proposing an alternative.
3. The experiments are too small and too narrow for the general claims. All main-text analyses use one 27M-parameter HRM checkpoint trained for 20000 epochs on 1000 base puzzles. HRM is known to overfit in this regime (the TRM paper, which the submission cites, notes that HRM "tends to overfit quickly and then diverge" on Sudoku-Extreme and Maze-Hard), and no training accuracy is reported. The checkpoint reaches 41.76%, against 55.0% reported for HRM on this benchmark, and the Maze model reaches 10.7% against 74.5% (Table 4 of Jolicoeur-Martineau, 2025). Model and task are never crossed (TRM only on Sudoku, GRAM-TRM only on N-Queens), so failures of transfer cannot be attributed. With essentially two small architectures on small puzzles, the claims in the title and abstract about "recurrent reasoners" and transfer "across recurrent models" are not supported. I also note that the headline 0.42 for mild wrong endpoints drops to 0.178 under replay-consistent labels (App. H), but the abstract keeps 0.42.
Other Concerns: Presentation and Writing
The main text is a dense run of numbers and appendix pointers, and the content needed to follow the argument is spread over appendices A to V. Several sentences describe the authors' pipeline rather than the science (e.g. "Concordance is now emitted by every basin run", "Thirteen of the 63 job scripts behind these carry an interpretation rule committed before execution"). I was also confused that the paper states, in Sec. 2, App. U and its bibliography, that the full text of Efstathiou & Balwani (2026), the main work it argues against, was "unavailable", so that everything attributed to it comes from one sentence of the abstract. That cited work is also about TRM, whereas the headline evidence here is from HRM. Finally, almost every result is followed by a sentence limiting what it means (e.g. "does not certify a fixed point", "not an attractor definition", "We therefore do not claim noise injection as the best use of a test-time budget", "without excluding asymptotic attractors"). Each caveat may be fair, but together they leave little that the paper actually claims.
Questions for authors
- What is the single main conclusion of the paper, beyond "correct answers persist more than wrong ones"? Which finding holds in all six settings and, in your view, changes how these models should be built or used?
- Can you separate oscillation from exploration? For each wrong trajectory over the 2048-segment continuation, how many distinct decoded answers are visited, how often does an answer recur, and is there periodicity in the decoded answer or in ? If wrong trajectories cycle among a small set of answers, how is that different from the attractor description the paper argues against?
- What are the training and test accuracies of the analysed checkpoint, and how large is its generalization gap? Do the persistence and spectral results hold on a well-trained checkpoint (the released HRM checkpoint, or TRM at its reported accuracy), on a larger model, and on at least one non-puzzle task or another family of looped models?
- Why do the abstract and Sec. 4 keep the dump-labelled 0.42 when App. H gives a replay-consistent estimate of 0.178 for the same quantity?
- I’m very confused by the statement in your paper that you have only read the abstract of the work you are arguing against and have not read the full text. That's really weird.