- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies latent diffusion planning in autoregressive models trained from scratch. It builds on an existing joint diffusion and language modeling architecture, adds separate feed-forward experts for text and diffusion tokens, and evaluates star-graph traversal, Sudoku, algorithmic creativity, and GSM8K. The model first samples a continuous plan and then generates tokens conditioned on it. The results show large gains on the synthetic reasoning tasks and improved accuracy and coverage on GSM8K.
Critical strengths
Clear diagnostic evidence on the synthetic tasks. The star-graph and Sudoku tasks are well chosen, because next-token prediction fails on them for an identifiable reason. LDP reaches 100% on all star-graph configurations, where AR, AR_Large, pause-token and looped baselines stay near chance (Figure 4a). On Sudoku it reaches 65.8% against 49.2% for the strongest baseline (Figure 6). The analysis separating the first mistake from the first constraint violation is particularly informative: only 14 of 2,571 AR failures conflict with the partial solution. This supports the claim that errors come from missing look-ahead rather than poor constraint checking, and it makes the Sudoku result more than a score comparison.
A useful observation about where diversity enters the model. On GSM8K, decoding greedily from sampled plans reaches higher pass@20 than any temperature/top-p setting of the AR baselines (Figure 10a), and programs stay valid at diversity levels where AR programs degrade (Figure 10c, d). Combined with the ablation in Figure 10b, this is a precise and practically relevant finding. Plans supply coverage, and joint training improves the decoder. The latency analysis (Appendix A) shows that this comes at modest cost: with 10 denoising steps LDP is 13% faster than AR_Large at the evaluated scale.
Critical weaknesses
1.On GSM8K, the gains appear to come from joint training rather than from planning. Pass@1 is essentially flat across plan noise levels from σ²=0 to 0.9 and across 1 to 100 denoising steps (Figure 12). With no plan it is 8.8 (Figure 10b), against 5.3 for AR_Base. So the plan mainly adds coverage (diversity), not single-shot accuracy. The paper acknowledges that joint training improves the decoder, but the title and abstract frame the result as planning. The comparison also gives LDP extra supervision: a frozen encoder and autoencoder (~150M parameters, uncounted) trained for up to 1,000 epochs on the same data, and up to 100 training epochs against 20 for AR. The paper lacks the controls that would separate planning from these effects. These are an AR model with the same auxiliary latent-prediction loss but no plan conditioning, and a seed-conditioned AR baseline (Nagarajan et al., 2025), which is cited but not compared.
2.The creativity comparison is not capacity- or decoding-matched, and the claims outrun the scale. In the creativity tasks, AR has 14.2M parameters against 22.0M/19.9M active (32.6M/30.5M total) for LDP (Table 6). AR is sampled only at T=1 (Appendix G), while GSM8K baselines get a 30-setting sweep. At the hardest Triangle setting the creative share is about 2% for AR and 3% for LDP (Figure 8). Discrete-diffusion planning methods (e.g., Ye et al., 2024) and teacherless or multi-token objectives are not compared. With GSM8K pass@1 at 9.7% and Sudoku at 65.8%, the title and abstract claim to "address fundamental limitations" of AR models. I would be glad to raise my score if the authors could clarify these points.
Questions for authors
1.Can you add (a) an AR model with the same auxiliary latent-prediction loss but no plan conditioning, and (b) a seed-conditioned AR baseline, on GSM8K and on one discovery setting? This is the question most likely to affect my assessment.
2.For the creativity tasks, how does LDP compare with an AR model of at least 32M parameters and tuned decoding? Which data were used to select the best checkpoints and the GSM8K decoding settings?
3.What are the actual stopping epochs and total training cost for each GSM8K model, including the encoder and autoencoder? How many independent training runs support the main results? The reported standard errors reflect variation across evaluation examples only.
5 comments
Triangle Discovery. We agree with the reviewer. At the hardest Triangle setting all models struggle similarly (2.2% for both AR and AR w/ LDP under the new protocol), so we will drop our claim of an advantage there and revise our wording. At n = 1000, AR w/ LDP reaches 18.2% against 13.2 / 12.8 / 14.2 for AR at 14M / 33M / 113M parameters, demonstrating its advantage.
Teacherless and multi-token objectives. We trained teacherless models (Bachmann & Nagarajan, 2024) with the hybrid loss of Nagarajan et al. (2025), and multi-token prediction (Gloeckle et al., 2024; n = 2 and 4). All are at 0.0% on star graph G10,5.
| % creative | Sibling, middle | Sibling, hardest | Triangle (n = 1000) |
|---|---|---|---|
| AR | 32.0 | 9.3 | 13.2 |
| Teacherless (hybrid loss) | 30.8 | 8.3 | 11.2 |
| Multi-token, n = 2 / 4 | 33.4 / 30.0 | 8.4 / 8.3 | 10.8 / 14.2 |
| AR w/ LDP | 59.0 | 56.1 | 18.2 |
Discrete diffusion (Ye et al., 2024). We trained a masked discrete diffusion model on the AR_Base backbone for every task:
| star G10,5 | Sibling, middle / hardest | Triangle (n = 1000) | GSM8K pass@1 / @20 / maj@20 | Sudoku | |
|---|---|---|---|---|---|
| Masked diffusion | 0.0 | 18.4 / 0.0 | 9.4 | 2.9 / 25.2 / 7.5 | 98.8 |
| AR w/ LDP | 100.0 | 59.0 / 56.1 | 18.2 | 9.7 / 33.6 / 14.3 | 70.4 |
We observe that AR w/ LDP is stronger on every task except Sudoku.
- Star graph. Masking appears to worsen the shortcut problem: training reveals random subsets of the path, so each node can be inferred from its revealed neighbours. Masked diffusion falls below chance even on G2,5 (41.8%).
- Discovery and GSM8K. It does not share the diversity benefit of a sampled plan, and on GSM8K it falls below AR_Base on every metric.
- Sudoku. Masked diffusion is very strong, in line with Ye et al.: bidirectional, any-order decoding suits filling a fixed grid under global constraints. This advantage is specific to Sudoku and does not carry over to the other tasks, which illustrates the value of evaluating on synthetic tasks that target different capabilities alongside an end-to-end check on GSM8K.
We will add this comparison to the paper and discuss when each approach is preferable.
Scale of the claims. We agree that "address fundamental limitations" is stronger than results at this scale support, and we will scope the abstract and conclusion to the settings we test.
Re: Q2 — which data were used to select checkpoints and GSM8K decoding settings?
Checkpoints. Every model is early-stopped on 500 held-out validation problems (TinyGSM for GSM8K; held-out puzzles for Sudoku and the star graph). GSM8K is used only for final evaluation.
GSM8K decoding settings. We sweep 30 temperature / top-p settings for each AR baseline and 12 for AR w/ LDP (guidance weight × denoising steps). Every model is reported at the point on its pass@1–pass@20 frontier with the highest maj@20. Fig. 10(a) plots every setting for every model, so the effect of the decoding parameters can be inspected directly. The table summarizes the range over reasonable settings:
| GSM8K (pass@1 / pass@20 / maj@20) | reported | T ∈ [0.7, 1.0], top-p ∈ [0.9, 0.99] (9 settings) |
|---|---|---|
| AR_Base | 5.0 / 27.1 / 8.5 | 4.3–5.2 / 24.4–27.5 / 7.7–9.6 |
| AR_Large | 6.9 / 30.4 / 11.9 | 6.4–7.4 / 26.5–31.2 / 9.9–11.9 |
| AR w/ LDP | 9.7 / 33.6 / 14.3 |
We observe that the reported points are representative of this range, and the top of each range stays below AR w/ LDP on all three metrics.
Creativity. The creativity metric scores each generation against the oracle source graph, so these tasks have no natural held-out split, and the submission reported the best creativity achieved during training. We agree this can be optimistic, so we adopted a stricter protocol for every model: the checkpoint and temperature are chosen on one draw of generations and the result is reported on a fresh draw. All creativity numbers in this response use this protocol.
Re: Q3 — training schedules, training cost, and independent runs
Training schedules (GSM8K). With early stopping, the AR baselines select checkpoints around epoch 11–12 and then overfit, while AR w/ LDP selects a checkpoint around epoch 25. AR w/ LDP was still improving at the end of a 20-epoch schedule, so we extended it. When we retrain the AR baselines with the same 100-epoch schedule, they do not improve upon the shorter schedule.
Training cost. Per training example, AR w/ LDP costs 1.54x the FLOPs of AR_Base on GSM8K (1.35x on Sudoku, 1.07x on the star graph), and AR_Large costs 3.3x (4.5x on Sudoku). The encoder and autoencoder are used only on GSM8K. They are trained once, from scratch, on the same TinyGSM training set. During AR w/ LDP they are frozen, produce training targets only, and are not used at inference. The reviewer is right that they add compute. The latent controls above are trained on exactly the same latents, so they are fully compute-controlled with our method, and they do not match AR w/ LDP.
Independent runs. We agree that the standard errors in the paper reflect variation across evaluation examples only. The comparisons span 18 task settings and around 20 model variants, so we selected representative settings for repeated training runs and found the results to be stable. On Sudoku, four runs of AR w/ LDP give 69.5 ± 2.5, each well above the best baseline (AR_Large, 49.2%). On the hardest Sibling Discovery setting, three runs give 56.1 ± 4.4, against 19.0 ± 2.1 for the strongest baseline (seed-conditioned AR).
We hope these controls address the reviewer's main question. In light of them, we would like to ask whether the reviewer would be willing to reconsider their score, and we are happy to discuss further.
We thank the reviewer for the detailed review, and for telling us which question matters most to their assessment. We ran both requested controls across tasks and respond to each point below.
Re: Do the GSM8K gains come from joint training rather than planning? (W1, Q1)
We ran both requested controls, along with two closely related variants, to test directly whether training without diffusion recovers the gains. The first three predict exactly the same latent target as AR w/ LDP, with the same architecture and output head; the fourth is the seed-conditioning baseline the reviewer requested:
Synthetic and discovery tasks (discovery entries are % creative):
GSM8K:
We observe that no control matches AR w/ LDP on any task:
So we find that joint training only accounts for part of the GSM8K gain, and the full diffusion process adds the rest. On the tasks that require lookahead or diverse solutions, joint training alone consistently struggles compared to diffusion modeling. Conditioning on the latent plan enables the lookahead behavior, and diffusion is needed for accurate and diverse plans.
We will add these controls to the paper for every task, and make the roles of the experiments explicit in the abstract and introduction: the synthetic tasks isolate planning, and GSM8K tests transfer to real data with a learned latent.
Correction. While preparing these experiments, we found that the Sudoku number in the submission (65.8%) came from a model trained without diffusion experts and modestly understates our performance with experts (70.4%). We apologize for the oversight and will report both in the final version.
Capacity. We trained AR models with 33M parameters (8 layers, width 512; AR w/ LDP has 33M in total) and 113M parameters (12 layers, width 768), and added Pause Token and Looped AR. Every model now receives a temperature sweep (T ∈ {0, 0.5, 1, 1.5, 2}).
We observe that neither capacity nor tuned decoding explains the gap: an AR model with 3.5x AR w/ LDP's total parameters is within a point of the 14M model, and tuning the temperature does not change the ordering. Looped AR comes close at the middle setting but is significantly worse at the hardest.