- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 3
Read review
Paper summary
This paper studies the training of recurrent reasoning models (RRMs) and proposes a new framework to improve their generalization. Rather than directly supervising model predictions with a cross-entropy loss against ground-truth solutions, the framework introduces a critic network that scores both ground-truth and model-generated solutions. The critic is trained to assign higher scores to ground-truth solutions and lower scores to model-generated ones, while the RRM is trained to maximize the critic’s scores for its outputs. This yields an adversarial training procedure similar to that of generative adversarial networks (GANs). To further improve generalization, the framework adds Gaussian noise to ground-truth solutions and encourages the critic to assign high scores to these perturbed solutions as well. Experiments on Maze and Sudoku demonstrate the empirical effectiveness of the proposed framework.
Critical strengths
This paper considers a timely problem. There has been quite a bit of interest in recurrent reasoning models (RRMs) recently, and how to make them generalize better is an important question in this area. What I find useful about this work is that it offers a general training framework that can be used with different RRM architectures. The experiments also show that the framework does help improve performance in practice, which gives some support to the proposed approach. Based on these results, I think this work could be useful to other researchers and may influence how people train RRMs in the future.
Critical weaknesses
My main concern is with the writing of the paper. Some parts of the paper feel a bit weird to read. For example, in lines 59–63, the authors say “we hypothesize…,” but this hypothesis does not seem to be tested later in the paper. The next sentence states, “In this paper, we use candidate-space expansion and Sudoku candidate entropy as operational stressors rather than direct estimates of the valid/invalid solution ratio.” The notion of a “valid/invalid solution ratio” appears quite abruptly here, and it is not clear why the authors draw this distinction or why readers should expect these measures to estimate that ratio in the first place. I think the introduction would benefit from clearer explanations of how these ideas connect.
The experimental section is also somewhat difficult to navigate. Although the authors identify two primary analyses, there are many supporting figures and tables. Organizing the section into subsections, each explaining what a particular set of experiments is intended to show, would make it easier to follow. Section 6 contains only a single paragraph, so it might fit better as a subsection within the experimental section. By the way, one minor point: in line 227, appears in before it has been defined.
Questions for authors
None
2 comments
Thanks the authors for their response and for the revisions to the paper’s organization.
Thank you for the positive assessment and for the very concrete writing suggestions. We agree with all of them and will revise the paper accordingly. Here is what we plan to change.
(1) Lines 59–63: the hypothesis and the "valid/invalid solution ratio". You are right that this passage is unclear. The intended hypothesis is simple: training a recurrent model against a single one-hot solution per instance becomes brittle as the output space grows, and replacing the single point with a re-sampled neighborhood around it makes training more robust. The phrase about the valid/invalid ratio was meant as a caveat (we do not count valid solutions). Placed in the introduction, however, it introduces a concept the reader has no reason to expect. We will remove it from the introduction and keep the caveat only in the Limitations paragraph. Proposed revision:
We will also make the experimental section refer back to this hypothesis, so that it is clear where it is tested.
(2) Organization of the experimental section. Thanks for this suggestion; we agree the section is hard to navigate. In the revision we will reorganize it as you suggest:
(3) Line 227: (x, s) used before definition. Good catch, thank you. We will define it at first use:
On generality. We were glad that you found DART useful as a general framework across RRM architectures. To support this further, we added the same objective, with no other change, to an ELF-style embedding-flow planner (Hu et al., 2026) on Sokoban. This is a different model family from the RRMs in the paper. Over 4 seeds, it raises the solve rate on held-out boards from 18.4% to 39.0%, and the optimal-plan rate from 6.8% to 36.3%. We will include this in the revised experimental section.
Thanks again for your careful reading. Your suggestions will make the paper much easier to follow, and we are happy to clarify anything else.