acceptodds
Under review as a conference paper at ICLR 2027

DART: Distributional Adversarial Recurrent Training for Algorithm Learning

Abstract

Recurrent reasoning models (RRMs) can solve structured problems, achieving *easy-to-hard* generalization through iterative computation in hidden space. These models are typically trained with instance-level supervision, which becomes increasingly problematic as task difficulty grows: valid solutions occupy a tiny region of the solution space, while invalid solutions proliferate rapidly. We propose **Distributional Adversarial Recurrent Training (DART)**, a training framework that replaces single-point supervision with a local target distribution around the ground-truth solution and aligns model outputs with this distribution through an adversarial objective. DART provides a richer learning signal and encourages more stable iterative trajectories toward valid solutions. When evaluated on Maze, Chess, and masked Sudoku with multiple RRMs, including Deep Thinking Systems and Tiny Recursive Models, DART improves solution quality, stability, and robustness under the evaluated distribution shifts. Comparisons with label smoothing, Gaussian softened targets, and progressive training show that DART is not explained by target softening alone and is complementary to training schemes that stabilize long-horizon recurrence. These results identify DART as a promising approach for improving robustness across the evaluated recurrent reasoning models.

Primary area:
transfer learning, meta learning, and lifelong learning
Keywords:
recurrent reasoning, algorithm learning, easy-to-hard generalization, adversarial training, iterative inference

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 4 reviews, above 68% of the others.
Reviewer_uuhJOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

This paper studies the training of recurrent reasoning models (RRMs) and proposes a new framework to improve their generalization. Rather than directly supervising model predictions with a cross-entropy loss against ground-truth solutions, the framework introduces a critic network that scores both ground-truth and model-generated solutions. The critic is trained to assign higher scores to ground-truth solutions and lower scores to model-generated ones, while the RRM is trained to maximize the critic’s scores for its outputs. This yields an adversarial training procedure similar to that of generative adversarial networks (GANs). To further improve generalization, the framework adds Gaussian noise to ground-truth solutions and encourages the critic to assign high scores to these perturbed solutions as well. Experiments on Maze and Sudoku demonstrate the empirical effectiveness of the proposed framework.

Critical strengths

This paper considers a timely problem. There has been quite a bit of interest in recurrent reasoning models (RRMs) recently, and how to make them generalize better is an important question in this area. What I find useful about this work is that it offers a general training framework that can be used with different RRM architectures. The experiments also show that the framework does help improve performance in practice, which gives some support to the proposed approach. Based on these results, I think this work could be useful to other researchers and may influence how people train RRMs in the future.

Critical weaknesses

My main concern is with the writing of the paper. Some parts of the paper feel a bit weird to read. For example, in lines 59–63, the authors say “we hypothesize…,” but this hypothesis does not seem to be tested later in the paper. The next sentence states, “In this paper, we use candidate-space expansion and Sudoku candidate entropy as operational stressors rather than direct estimates of the valid/invalid solution ratio.” The notion of a “valid/invalid solution ratio” appears quite abruptly here, and it is not clear why the authors draw this distinction or why readers should expect these measures to estimate that ratio in the first place. I think the introduction would benefit from clearer explanations of how these ideas connect.

The experimental section is also somewhat difficult to navigate. Although the authors identify two primary analyses, there are many supporting figures and tables. Organizing the section into subsections, each explaining what a particular set of experiments is intended to show, would make it easier to follow. Section 6 contains only a single paragraph, so it might fit better as a subsection within the experimental section. By the way, one minor point: in line 227, ss appears in (x,s)(x,s) before it has been defined.

Questions for authors

None

2 comments

Thank you for the positive assessment and for the very concrete writing suggestions. We agree with all of them and will revise the paper accordingly. Here is what we plan to change.

(1) Lines 59–63: the hypothesis and the "valid/invalid solution ratio". You are right that this passage is unclear. The intended hypothesis is simple: training a recurrent model against a single one-hot solution per instance becomes brittle as the output space grows, and replacing the single point with a re-sampled neighborhood around it makes training more robust. The phrase about the valid/invalid ratio was meant as a caveat (we do not count valid solutions). Placed in the introduction, however, it introduces a concept the reader has no reason to expect. We will remove it from the introduction and keep the caveat only in the Limitations paragraph. Proposed revision:

"We hypothesize that this creates a challenge for easy-to-hard generalization: as problems grow harder, the output space grows, yet the model still sees only one reference solution per instance. We test this by increasing difficulty in a controlled way (larger mazes, more masked Sudoku cells) and comparing point-wise training with DART."

We will also make the experimental section refer back to this hypothesis, so that it is clear where it is tested.

(2) Organization of the experimental section. Thanks for this suggestion; we agree the section is hard to navigate. In the revision we will reorganize it as you suggest:

  • Split it into subsections, each opening with a sentence on what that set of experiments is meant to show.
  • Merge the current Sec. 6 into the experimental section as a subsection.
  • Move supporting figures that are not essential to the main argument to the appendix.

(3) Line 227: (x, s) used before definition. Good catch, thank you. We will define it at first use:

"…using a critic C_ψ(x, s) that scores a candidate solution s for input x, where s is either a soft reference sample v̂ ~ P_ref(· | x, y) or a model prediction ŷ ~ P_gen(· | x)."*

On generality. We were glad that you found DART useful as a general framework across RRM architectures. To support this further, we added the same objective, with no other change, to an ELF-style embedding-flow planner (Hu et al., 2026) on Sokoban. This is a different model family from the RRMs in the paper. Over 4 seeds, it raises the solve rate on held-out boards from 18.4% to 39.0%, and the optimal-plan rate from 6.8% to 36.3%. We will include this in the revised experimental section.

Thanks again for your careful reading. Your suggestions will make the paper much easier to follow, and we are happy to clarify anything else.

Thanks the authors for their response and for the revisions to the paper’s organization.

Reviewer_6AxyOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Distributional Adversarial Recurrent Training (DART) for recurrent reasoning models. The stated method combines the original supervised loss with a conditional Wasserstein-style critic that distinguishes model predictions from bounded-noise, softmax-transformed reference targets. The critic is applied to the final training output and removed at inference. Experiments use DTS on spatially extrapolated mazes and chess puzzles, and a reduced-width TRM trained on 20%-masked Sudoku and evaluated at higher masking ratios. DART substantially improves the reported hard-Maze and Sudoku results; smoothing, hard-target-critic, progressive-training, and extra-cycle controls show that the useful components and stability effects depend on the task.

Critical strengths
  1. The strongest empirical results are substantial and evaluated beyond the training difficulty. On 31x31 Maze, final accuracy improves from 79.55 +/- 21.60% to 99.88 +/- 0.17% over five seeds. On 50% masked Sudoku, valid-and-clue-consistent solutions improve from 51.8 +/- 3.8% to 64.8 +/- 3.4%. Using validity plus clue preservation, rather than reference exact match alone, is appropriate when masking can admit alternative completions. Checkpoint selection at the training masking ratio also makes the Sudoku transfer evaluation more convincing.

  2. The paper includes useful controls and reports important limitations. It compares against label smoothing, softened targets without a critic, CE plus soft-target MSE, a hard-target critic, progressive training, and additional TRM cycles. In particular, it acknowledges that the hard-target critic already matches DART on Maze, that softened targets without a critic retain Chess accuracy better than non-progressive DART, and that DART does not improve every Sudoku structural-validity diagnostic. The critic adds training cost but no inference network, which makes the successful results practically relevant.

Critical weaknesses

The experiments establish the usefulness of the combined objective more clearly than the proposed distributional or conditional mechanism. Even zero noise in the stated Eq. (2) yields softmax(onehot), a deterministic smoothed target. The hard-target-critic comparison therefore changes both target confidence and stochastic width; the no-critic comparisons also remove the learned compatibility function. A conditional critic using a deterministic softened target with matched mean, and an output-only critic, would distinguish the claimed roles of the local distribution and input conditioning from more general adversarial regularization. This matters because the hard-target critic already matches DART on Maze (Table 8), while the necessity of the full construction is primarily supported by Sudoku. The paper correctly acknowledges that the target distribution does not enumerate alternative valid solutions, but the stronger mechanistic interpretation remains partially isolated.

Questions for authors

How does full DART compare with the same conditional critic trained against a deterministic softened reference, such as the mean of its target distribution? Is there evidence that conditioning the critic on the puzzle is necessary beyond an output-only critic?

The supplied trajectory script selects the first N dataset examples, which clarifies its sampling rule. Which checkpoints, seeds, and dataset ordering produced Table 6, and was this script used unchanged? This would explain how its 43.8% final Maze CE rate relates to the five-seed 79.55% in Table 3.

3 comments

Thank you for the careful review and the constructive questions. We agree with your main point: the original experiments show that the combined objective helps more clearly than they isolate why. We ran exactly the two controls you suggested. Both use the same critic, architecture and hyper-parameters as DART and change only one component.

Q1(a) — Conditional critic with a deterministic, mean-matched soft target. You are right that even zero noise gives softmax(onehot), a fixed soft target. With σ = 0.3, the mean of DART's noisy target is almost the same as softmax(onehot): the target-class probability is 0.212 vs. 0.214 for Sudoku's 11 classes, computed by Monte-Carlo. So this control keeps the input-conditioned critic and the average target confidence, and removes only the re-sampling of the noise.

Q1(b) — Output-only critic. The critic never sees the puzzle. On Sudoku, all clue embeddings are replaced by the blank token; on Maze, the input channels are zeroed. Everything else is unchanged.

TRM-Sudoku (trained at 20% masking; 10k test puzzles per level; valid + clue-consistent / exact match; mean ± std over 3 seeds):

Variant30%40%50%
CE99.2 ± 0.0 / 95.6 ± 0.190.6 ± 0.7 / 78.1 ± 0.751.8 ± 3.1 / 32.0 ± 1.9
Hard-target critic99.3 ± 0.3 / 95.8 ± 0.290.3 ± 2.2 / 78.0 ± 2.036.4 ± 4.7 / 22.2 ± 2.7
Fixed mean soft target + conditional critic98.9 ± 0.8 / 95.3 ± 0.785.4 ± 3.3 / 73.7 ± 2.731.2 ± 7.6 / 19.0 ± 4.5
Output-only critic + re-sampled target99.2 ± 0.3 / 95.5 ± 0.485.3 ± 4.1 / 73.6 ± 3.534.7 ± 7.0 / 21.4 ± 4.4
DART (both)99.8 ± 0.1 / 96.2 ± 0.295.3 ± 0.5 / 82.4 ± 0.664.8 ± 2.8 / 40.4 ± 1.9

On Sudoku, both components are necessary. Removing either one lowers accuracy below CE at every masking level, and at 50% masking costs 30–34 points relative to DART.

The drop is not caused by a vanishing adversarial signal. We tracked the critic's Wasserstein estimate over the last 20% of training:

  • Hard targets: the estimate collapses to ≈0 (std ≈ 3·10⁻⁴).
  • Fixed mean target: it stays informative (mean 1.05, std 0.76).
  • Output-only critic: it stays informative (mean 1.63, std 0.17).

So both critics keep learning. What they lose is a useful signal:

  • The fixed target gives the generator a single point it can match exactly, rather than a neighborhood of the ground truth.
  • The output-only critic can separate real from generated outputs without checking whether a completion is compatible with the given clues.

On Maze, as you anticipated, the controls do not separate. The output-only critic is on par with DART (31×31: 99.91 ± 0.10 vs. 99.87 ± 0.16; 59×59: 99.60 ± 0.32 vs. 99.43 ± 0.72; 3 vs. 5 seeds), as is the hard-target critic (App. B.6). A valid maze path is recognizable from the output alone: a connected, non-branching road from start to goal. Any reasonable critic therefore gives a strong signal there, whereas on Sudoku correctness depends on the clues.

We will state this explicitly in the revision. Maze shows that an adversarial critic helps; the evidence for the full distributional and conditional construction comes from Sudoku.

Q2 — Checkpoints, seeds and ordering behind Table 6. Thank you for checking the script. Table 6 was produced with the supplied script unchanged, using the first 128 test samples and a single checkpoint per method. For Maze CE, this checkpoint was the weakest of the five CE seeds (47.9% final accuracy), which explains the gap to the five-seed mean of 79.55% in Table 3. We have recomputed the statistics on the full test sets and all seeds, and will replace Table 6:

Maze 31×31 (10k test, 300 iters)SeedsSolved at least onceFinal exactRetention after first exactFraction of iterations exact
CE584.9 ± 17.479.5 ± 19.392.9 ± 5.756.6 ± 14.1
Label smoothing399.2 ± 0.698.8 ± 1.099.3 ± 0.570.5 ± 1.1
Hard-target critic399.96 ± 0.0599.95 ± 0.0699.3 ± 0.871.5 ± 0.8
DART599.94 ± 0.0899.87 ± 0.1699.8 ± 0.171.1 ± 1.2
Chess (100k test, 50 iters)SeedsPeak acc.Solved at least onceRetention after first exactFinal exact
CE280.8 ± 0.484.3 ± 0.711.3 ± 0.50.9 ± 0.1
Soft target, no critic282.2 ± 0.187.8 ± 0.294.0 ± 0.379.3 ± 0.5
Hard-target critic384.7 ± 0.187.6 ± 0.446.4 ± 19.06.7 ± 6.2
DART284.8 ± 0.087.9 ± 0.259.6 ± 4.411.1 ± 4.6
CE + progressive loss283.5 ± 0.089.1 ± 0.094.1 ± 0.083.5 ± 0.0
DART + progressive loss283.6 ± 0.189.4 ± 0.293.8 ± 0.183.5 ± 0.2

With all seeds, the qualitative conclusions of Table 6 hold on Maze. DART solves almost every maze and keeps the solution once found (retention 99.8% vs. 92.9% for CE), with much lower variance across seeds.

On Chess, consistent with what the paper already acknowledges:

  • Without progressive loss, DART reaches the highest peak accuracy and solve rate, and improves retention over CE (59.6% vs. 11.3%), but does not fully retain solutions to iteration 50.
  • The soft-target model without a critic is the most stable in that setting.
  • With progressive loss, CE and DART are on par.

The revision will report these full tables, the exact checkpoint and seed list for every run, and the evaluation script with a fixed, documented sample order.

Thank you again for the precise questions. They led to the two controls above, which make the mechanistic claim of the paper much more concrete.

Dear Reviewer 6Axy,

We have addressed your comments and questions in our responses. As the discussion period is coming to a close, we would greatly appreciate it if you could review our replies and share any remaining feedback within the next few hours, if possible.

Thank you very much for your time and consideration.

Best regards, The Authors

Reviewer_rgmSOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper proposes DART, an adversarial training framework designed to stabilize the solution trajectories of RRMs and achieve easy-to-hard generalization. DART constructs soft targets by adding Gaussian noise to the ground-truth one-hot labels. It then adds a WGAN-style adversarial loss to the cross-entropy loss, using a critic that takes problem-solution pairs as input. The experiments mainly focus on Maze, Sudoku, and Chess. The results show improved generalization in OOD settings where the model is trained on easy problems and evaluated on harder ones.

Critical strengths
  • The problem statement that providing only a single ground-truth label per instance might be insufficient for easy-to-hard generalization is interesting, and the motivation is clear.
  • The authors construct stress tests for easy-to-hard generalization using mask ratios and candidate entropy in Sudoku.
  • The paper introduces adversarial training, which has rarely been combined with RRMs. It exhibits the smallest performance degradation under stress testing.
Critical weaknesses
  • The evaluation is limited to artificial tasks, making the broader significance of the performance gains unclear. While scaling Maze sizes or increasing Sudoku masking ratios are valid ways to test generalization, the practical value of succeeding in these synthetic setups is not well justified.

  • Related to the point above, prior works such as Artificial Kuramoto Oscillatory Neurons (AKOrN, Miyato et al., ICLR 2025) and other recurrent models have already demonstrated easy-to-hard generalization on Sudoku by testing on harder instances without using adversarial training. This weakens the argument for the necessity of the proposed approach.

  • The paper claims that the method stabilizes solution trajectories, but the evidence supporting this mechanism is insufficient.

  • Improvements in Chess are mentioned, but the task is barely discussed in the main text.

  • The actual performance gains on Sudoku appear marginal.

Questions for authors
  • Why exactly does combining distributed targets with adversarial training improve easy-to-hard generalization, and why do other regularization methods fail to achieve this?

  • What is the specific mechanism that allows this distributed adversarial approach to stabilize solution trajectories?

  • Can you provide a theoretical explanation for why the proposed method is highly effective on certain tasks but less effective on others?

9 comments

Thank you for the detailed review and for finding the motivation clear and the Sudoku stress tests interesting. We address your concerns below and have run several new experiments to do so.

First, a clarification of the paper's goal, which we will also make explicit in the revision. The objective of DART is easy-to-hard generalization of recurrent reasoning models. Solving a harder instance requires running more recurrent iterations than were seen during training, so stable trajectories are an observable consequence of successful extrapolation (and a necessary condition for it), not a separate design target. We will tone down the stability language in the abstract and Sec. 4 accordingly.

W1 — Evaluation limited to artificial tasks. We agree that Maze, Sudoku and Chess are synthetic. We chose them because they are the standard easy-to-hard benchmarks for recurrent reasoners (Schwarzschild et al., 2021; Bansal et al., 2022; Jolicoeur-Martineau, 2025). Their difficulty can be controlled precisely (maze size, masking ratio), so a gain can be attributed to generalization to harder instances rather than to a change in data distribution.

To show that DART is not tied to these benchmarks or to RRMs, we added the same objective to a different model family on a different task: an ELF-style embedding-flow planner (Hu et al., 2026) on Sokoban planning. Architecture, data and schedule are unchanged; only the DART term is added. Over 4 seeds, on held-out 1-box boards:

Sokoban (1 box)Plain ELFELF + DART
Solve rate18.4 ± 3.339.0 ± 2.2
Optimal-plan rate6.836.3
Mean no-op actions ↓2.561.47

The gain holds for every seed (+16.4 to +26.7 points) and in every difficulty bucket by solution length. We do not claim easy-to-hard transfer on Sokoban (e.g., to more boxes). The point is that the objective works as a general regularizer beyond the RRMs in the paper.

W2 — AKOrN and other recurrent models already generalize on Sudoku without adversarial training. Thank you for the pointer; we will discuss AKOrN (Miyato et al., 2025) in the related work. We see the two as complementary rather than competing:

  • AKOrN changes the architecture (oscillatory neurons).
  • DART changes the training objective and leaves the architecture untouched.

DART therefore does not claim that adversarial training is the only route to easy-to-hard generalization. It claims that, for a given recurrent architecture, replacing point-wise supervision with a distributional adversarial objective improves it. We now show this on three different architectures:

ArchitectureTaskBaseline+ DART
DTS (recurrent CNN)Maze 59×59 (trained 9×9)56.9 ± 33.899.4 ± 0.7
TRM (recursive transformer)Sudoku, 50% masking (trained 20%)51.8 ± 3.164.8 ± 2.8
ELF (embedding flow)Sokoban, 1 box18.4 ± 3.339.0 ± 2.2

The gain also persists when the model is scaled up. With hidden size 256 for both the TRM solver and the critic, DART reaches 62.1 ± 6.7 vs. 43.9 ± 4.9 for CE at 50% masking (2 seeds). In other words, a larger model alone does not close the gap. Combining DART with architectures such as AKOrN is a natural direction for future work.

W5 — Gains on Sudoku appear marginal. The gain grows with difficulty, which is the regime the paper targets (valid + clue-consistent, 3 seeds, 10k puzzles per level):

Sudoku30%40%50%60% (new)
CE99.2 ± 0.090.6 ± 0.751.8 ± 3.110.6 ± 2.0
DART99.8 ± 0.195.3 ± 0.564.8 ± 2.814.7 ± 1.7
Gain+0.6+4.7+13.0+4.1 (+39% rel.)

At 50% masking, the gain is +13.0 points (+25% relative) in valid completions and +8.4 points in exact match (40.4 vs. 32.0). Every alternative regularizer we tried falls below CE at this level (see Q1). On this task, beating CE is therefore not easy.

W3 / Q2 — Evidence and mechanism for stable trajectories. We replaced the trajectory statistics of Table 6 (a single checkpoint per method and 128 samples) with statistics on the full test sets and all seeds. Models are trained on 9×9 mazes.

Maze 59×59 (1,000 iterations)SeedsPeakFinalRetention after first exact
CE557.3 ± 33.956.9 ± 33.882.8 ± 13.5
Label smoothing394.6 ± 3.994.4 ± 4.099.0 ± 0.7
Soft target, no critic365.5 ± 43.365.4 ± 43.497.2 ± 3.5
DART599.5 ± 0.699.4 ± 0.799.8 ± 0.3

On 31×31 mazes (full 10k test set, 300 iterations), DART solves 99.94 ± 0.08% of mazes at least once and retains the solution in 99.8 ± 0.1% of the subsequent iterations, compared with 84.9 ± 17.4% and 92.9 ± 5.7% for CE.

Mechanism. The critic sees the input together with either a soft target sampled around the ground truth, or the model's output. It is trained to tell them apart, and the reasoning model is trained to make its output indistinguishable from the sampled targets. Because the target is re-sampled at every step, the model is pulled toward a neighborhood of the solution for that specific input, not toward a single point. Once the recurrent state enters this region, further iterations that move it away are penalized, so the solution is retained.

Two new controls on TRM-Sudoku support this reading. Removing either the re-sampling or the input conditioning destroys the gain, even though the critic keeps learning in both cases. We tracked the critic's Wasserstein estimate over the last 20% of training:

  • Fixed mean target: mean 1.05, std 0.76.
  • Output-only critic: mean 1.63, std 0.17.
  • One-hot (hard) targets: by contrast, the estimate collapses to ≈0.

Details are in Q1. We present this as an empirically supported interpretation, not a formal proof.

W4 — Chess is barely discussed. We will move Chess into the main text. Full 100k test set, 50 iterations:

ChessSeedsPeak acc.Solved at least onceRetention after first exactFinal exact
CE280.8 ± 0.484.3 ± 0.711.3 ± 0.50.9 ± 0.1
Soft target, no critic282.2 ± 0.187.8 ± 0.294.0 ± 0.379.3 ± 0.5
DART284.8 ± 0.087.9 ± 0.259.6 ± 4.411.1 ± 4.6
CE + progressive loss283.5 ± 0.089.1 ± 0.094.1 ± 0.083.5 ± 0.0
DART + progressive loss283.6 ± 0.189.4 ± 0.293.8 ± 0.183.5 ± 0.2

We report this transparently:

  • DART reaches the highest peak accuracy and raises retention from 11.3% to 59.6%.
  • On its own, it does not keep solutions until iteration 50. Chess is dominated by late-iteration overthinking, which progressive training targets directly.
  • With progressive loss, CE and DART are on par.

Chess therefore illustrates that DART addresses a different failure mode than progressive training. We discuss this further in Q3.

Q1 — Why does combining distributional targets with adversarial training help, and why do other regularizers not? We isolated each component on TRM-Sudoku: 3 seeds, 10k puzzles, 50% masking, valid + clue-consistent / exact. The two variants marked "new" were run for this rebuttal.

VariantRe-sampled targetCriticCritic sees input50% masking
CE–––51.8 ± 3.1 / 32.0 ± 1.9
Label smoothingfixed soft––45.8 ± 5.0 / 28.9 ± 3.1
Soft target, no critic✓––0.0 / 0.0
Hard-target criticone-hot✓✓36.4 ± 4.7 / 22.2 ± 2.7
Fixed mean soft target (new)fixed soft✓✓31.2 ± 7.6 / 19.0 ± 4.5
Output-only critic (new)✓✓–34.7 ± 7.0 / 21.4 ± 4.4
DART✓✓✓64.8 ± 2.8 / 40.4 ± 1.9

Every partial variant falls below CE, and only the full combination improves over it. The "fixed mean soft target" has the same average softness as DART's target (target-class probability 0.212 vs. 0.214), so softening alone does not explain the gain. As discussed in W3, the critic does not collapse in these variants, so the drop does not come from a vanishing adversarial signal.

Our interpretation of why other regularizers fail:

  • Label smoothing and MSE to a soft target regularize each output cell independently. They do not check whether a completion is consistent with the clues.
  • The conditional critic scores the whole (puzzle, solution) pair. Re-sampling prevents the model from simply matching one fixed point.

Q3 — Why is DART more effective on some tasks than others? We do not have a formal theory, and we will state this as a limitation. Our experiments do suggest a consistent empirical picture: which component matters depends on the dominant failure mode.

TaskDominant failure modeWhat helpsEvidence
MazeLonger paths in larger mazesAny critic. A valid path is recognizable from the output aloneOutput-only critic ≈ hard-target critic ≈ DART on 59×59 (99.6 / 99.7 / 99.4)
SudokuCorrectness depends on the cluesRe-sampled target and input-conditioned criticTable in Q1
ChessLate-iteration overthinkingProgressive trainingTable in W4

We will add this analysis to the revised discussion.

Thank you again for the thoughtful questions. We hope these new results address your concerns, and we are happy to clarify anything further.

Reviewer_rLNgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

DART studies how recurrent reasoning models generalize from easy training examples to harder test cases. It adds a critic that compares the model’s soft predictions with softened reference solutions. This lets gradients flow back to the solver without passing through discrete sampling. Compared with standard cross-entropy, DART improves final accuracy on 31×31 mazes from 79.55% to 99.88%. On Sudoku with 50% masking, it raises the rate of valid solutions that preserve all clues from 51.8% to 64.8%.

Critical strengths
  • The Sudoku ablations provide the clearest evidence for the full method. At 50% masking, combining the critic with soft targets outperforms either component alone.
  • The critic network supervise the soft network output veruse the sampled tokens, therefore bypass the backprop through sampling.
Critical weaknesses
  • On Maze, label smoothing already reaches 98.80% accuracy versus 99.88% for DART, leaving little room to assess the full method’s added value. Would larger mazes separate these methods at a matched inference budget?
  • Critic stability. The default critic is recurrent, and the paper compares it with a feedforward variant. How stable is critic optimization? What if we increase both critic and the reasoning model's capacity?
  • On Sudoku, how much does resampling Gaussian noise add beyond label softening? A comparison using the same critic and fixed soft targets matched to the mean of the noisy targets would isolate that effect.
Questions for authors

Please read the feedback above.

5 comments

We thank the reviewer for the precise and constructive questions. All new experiments use a single code version, explicit seeds, and one evaluation script. We will release the script and the checkpoint list with the revision.

W1 — Label smoothing vs. DART on larger mazes at a matched inference budget.

Yes, larger mazes separate the two methods. We evaluated label smoothing and DART on 59×59 mazes. Both are trained on 9×9 mazes and run for the same 1,000 recurrent iterations on 2,000 fixed test mazes (31×31: full 10k test set, 300 iterations):

MethodSeeds31×31 final (iter. 300)59×59 peak59×59 final (iter. 1000)
Label smoothing398.8 ± 1.094.6 ± 3.994.4 ± 4.0
DART599.9 ± 0.299.5 ± 0.699.4 ± 0.7

At the larger size, the gap grows from 1.1 points (31×31) to 5.0 points (59×59). Label smoothing also becomes much less consistent across seeds (std 4.0 vs. 0.7), while DART stays above 99%.

W2 — Critic stability; increasing both critic and reasoning-model capacity.

Stability of critic optimization. We tracked the critic's Wasserstein estimate E[C(x,v̂)] − E[C(x,ŷ)] during training on TRM-Sudoku:

  • DART: the estimate stays bounded throughout training (46.9k steps), with no divergence or NaNs.
  • Critic with one-hot (hard) targets: the estimate collapses to ≈0 (std ≈ 3·10⁻⁴ over the last 20% of training). The critic then gives almost no learning signal, which matches the weaker accuracy of this variant (36.4 vs. 64.8 at 50% masking).
  • Critic with a fixed mean soft target (see W3): the estimate stays informative (mean 1.05, std 0.76 over the last 20% of training), yet accuracy drops. A stable critic is therefore necessary but not sufficient for DART's gain.

All new runs now log the critic (Wasserstein estimate, critic scores, gradient norms) every 10 steps. We will add the training curves to the appendix.

Critic capacity. On DTS-Maze (generator width 128; up to 60 epochs with early stopping at 100% validation accuracy), we varied the critic width. Final exact accuracy:

Critic widthSeeds31×3159×59
64299.6 ± 0.498.1 ± 1.9
128 (default)599.9 ± 0.299.4 ± 0.7
256299.7 ± 0.399.3 ± 0.7
CE (no critic)579.5 ± 19.356.9 ± 33.8

Across a 4× range of critic width (0.5× to 2× the generator width), DART stays at ≥98% on 59×59 mazes, far above CE. Critic optimization is therefore not sensitive to critic size in this range.

Increasing the reasoning model's capacity. Prior work shows that recurrent reasoners at large width extrapolate well (Bansal et al., 2022). Overthinking and unstable recurrence mainly arise in the compact regime, where Bear et al. (2024) had to redesign normalization to recover stable extrapolation. DART targets this compact regime, where training is most fragile. Our capacity results agree with this picture. On DTS-Maze:

Maze, final exactParamsSeeds31×3159×59
CE, width 1280.78M579.5 ± 19.356.9 ± 33.8
CE, width 2563.04M199.798.0
DART, width 1280.78M599.9 ± 0.299.4 ± 0.7
DART, width 256 (solver and critic)3.04M2100.0 ± 0.099.98 ± 0.03

Two observations:

  • With 3.9× fewer parameters, DART at width 128 matches or exceeds CE at width 256, with much lower variance than CE at width 128.
  • Scaling both the reasoning model and the critic to width 256 keeps DART's advantage over CE at the same width: 100.0% vs. 99.7% on 31×31, and 99.98% vs. 98.0% on 59×59.

On TRM-Sudoku, we scaled the hidden size of both the solver and the critic from 128 to 256 (trained at 20% masking; 10k test puzzles per level; valid + clue-consistent / exact match):

SudokuSeeds30%40%50%
CE, hidden 128399.2 ± 0.0 / 95.6 ± 0.190.6 ± 0.7 / 78.1 ± 0.751.8 ± 3.1 / 32.0 ± 1.9
CE, hidden 256198.2 / 95.286.0 / 76.648.8 / 32.7
DART, hidden 128399.8 ± 0.1 / 96.2 ± 0.295.3 ± 0.5 / 82.4 ± 0.664.8 ± 2.8 / 40.4 ± 1.9
DART, hidden 256199.95 / 96.297.1 / 84.068.8 / 43.4

Unlike on Maze, doubling the hidden size does not help CE on Sudoku. DART's advantage persists at the larger scale: +20.0 points (valid + clue-consistent) and +10.7 points (exact) over CE at 50% masking. Width-256 Maze results use 2 DART seeds and 1 CE seed; the hidden-256 Sudoku results use 1 seed each. Additional seeds are running and we will update these tables.

W3 — How much does re-sampling the Gaussian noise add beyond label softening?

We ran exactly the suggested control. The critic, architecture and hyper-parameters are unchanged, but the critic is trained against a fixed soft target equal to the mean of DART's noisy target, E[softmax(onehot + ε)] (computed by Monte-Carlo), instead of a freshly sampled target at every step.

With σ = 0.3, this mean target is almost identical to softmax(onehot): the target-class probability is 0.212 vs. 0.214 for Sudoku's 11 classes. The comparison therefore keeps the average softness fixed and removes only the re-sampling.

TRM-Sudoku (trained at 20% masking; 10k test puzzles per masking level). Each cell: valid + clue-consistent / exact match.

VariantSeeds30%40%50%
CE399.2 ± 0.0 / 95.6 ± 0.190.6 ± 0.7 / 78.1 ± 0.751.8 ± 3.1 / 32.0 ± 1.9
Fixed mean soft target + critic398.9 ± 0.8 / 95.3 ± 0.785.4 ± 3.3 / 73.7 ± 2.731.2 ± 7.6 / 19.0 ± 4.5
DART (re-sampled noise)399.8 ± 0.1 / 96.2 ± 0.295.3 ± 0.5 / 82.4 ± 0.664.8 ± 2.8 / 40.4 ± 1.9

Removing the re-sampling while keeping the same average softness makes the method worse than CE at every masking level. At 50% masking it loses 34 points relative to DART. Label softening alone therefore does not explain DART's gain; the gain comes from re-sampling.

The fixed target does not make the critic collapse (W2), so the drop is not caused by a vanishing adversarial signal. Our interpretation: with re-sampling, the critic compares the generator's output against a distribution of plausible soft targets around the ground truth, rather than against a single point that the generator can match exactly.

Several additional runs (more seeds for the width-256 / hidden-256 models) are still in progress. We will post the latest results in this thread as soon as they finish, and include all of them in the revised paper. We hope these new experiments address the reviewer's concerns, and we are happy to provide further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.