acceptodds
Under review as a conference paper at ICLR 2027

Think Before You Paint: Recursive Latent Reasoning for Diffusion Models

11 views

Abstract

Diffusion models generate realistic images but often fail on visual reasoning tasks, such as filling in a Sudoku or drawing the path through a maze. In a diffusion model an intermediate conclusion can only be carried forward by drawing it into the image. We argue that this is what impedes reasoning, because a wrong guess that has been partly drawn is hard to undo. We propose Painter-Thinker (PaTh), which performs additional reasoning steps before constructing each denoised image: a small recursive network (the Thinker) refines a latent state that is never rendered, and its result conditions the next step of a frozen denoising model (the Painter). The Thinker is trained with the standard reconstruction loss, with no solver or verifier. PaTh solves 92.5% of hard MNIST Sudoku puzzles (prior best 75%) and 71.2% of extreme ones (prior best 4.1%), with 10M parameters against 82M for a standard diffusion model. It also improves on mazes, Queens, and CLEVR scenes with specified spatial relations, and its advantage grows with problem size. Diagnostic experiments show that the diffusion model corrects only mistakes that are directly visible in the image, while PaTh also corrects mistakes that have to be inferred. Together, these results suggest that what diffusion models lack on rule-governed data is not capacity but a place to think, and that supplying one opens a path toward generating data under increasingly complex constraints.

Primary area:
generative models
Keywords:
diffusion models, latent reasoning, recursive reasoning, constrained generation, inference-time compute, visual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_UxB3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper demonstrates that a small, modality-decoupled recursive reasoner can serve as a latent scratchpad for a frozen image diffusion model, substantially improving exact constraint satisfaction without requiring an explicit symbolic decomposition, verifier, or task-specific solver. Rather than running recursive thinking steps in the complex and costly latent space of the diffusion model, the additional reasoner acts on a much smaller latent space. Experiments demonstrate the working of the approach on 3 different visual tasks.

Critical strengths

Although there have been very similar concepts in literature (see weaknesses below), the implementation in this paper is particularly simple but still effective. It is interesting to see that the diffusion model can be frozen and that they need not train a shared encoder.

In comparison to previous symbolic approaches, the method does not require additional components like solvers, verifiers, or symbolic intermediate targets, which hamper general applicability on image tasks. This strength is not entirely unique (see weaknesses), but very recent.

The results across four quite different visual constraint settings are convincing. I also liked the mechanistic experiments demonstrating the advantage of an uncommitted latent scratchpad.

Critical weaknesses

The paper should mention Latent Imagination Thinking (LIT), which is very close in spirit to the paper's contribution, even though implemented differently. Like the present paper, LIT argues that reasoning should be separated from modeling the observational space, trains recurrent latent states, and evaluates both linguistic and visual MNIST Sudoku. LIT reports 54.5% on visual Sudoku Extreme, whereas the abstract of the present paper claims that the prior best on Extreme is 4.1%. This claim must be revised. Also LIT should be discussed including the technical differences and their effect in experiments. K Farid, J Bratulić, S Mittal, C Schmid, T Brox, Latent Imagination Thinking: Beyond Recursive Models for Reasoning, ICLR 2026 2nd Workshop on World Models.

Also the statement that PaTh would fill the gap that continuous diffusion has no way to mark part of a sample as pending is not taking into account that other works like LIT, Thinking Pixel, recurrent diffusion, FRMs and looped flows all introduce internal state or recurrence precisely to avoid treating the observable generation state as the whole computation. Clearly, this limits the scope of the contribution of the present paper, although not in a critical manner. However, the paper's statements must be more honest in terms of these related works. I would also change the abstract to reflect the real strengths of the paper (see strengths above) rather than a generic statement, which could be regarded as overclaiming, since other recent works argued the same way already.

Another major weakness seems to be on the technical side of the concept. While simplicity is an advantage, there could be also a drawback here: because there is no way for the Thinker to use the internal representation of the diffusion model, the Thinker has to learn its own perceptual representation from scratch through an indirect control signal. The Thinker's small convolutional encoder has to rediscover whatever visual semantics it needs. I expect this to work only for visually simplistic settings used for the experiments in this paper, but I cannot imagine this is scalable to more complex visual settings.

Related to this is another overclaiming statement. While it is mentioned in the limitations that the method has a severe spatial-grounding dependency, the main text still says that Thinker token resolution need not coincide with Sudoku cells, which sounds like it has escaped symbolic/spatial alignment. But the appendix shows that it has only escaped exact 9×9 discretization. It still relies heavily on a fixed image-coordinate correspondence. The Thinker's tokens must remain spatially aligned with the conditioning; randomly translating/scaling the output while leaving the conditioning untransformed makes Sudoku cell accuracy collapse to chance. CLEVR makes this problem particularly visible. Relations/object tokens alone achieve only 50% spatial accuracy and 8% attribute binding. Giving the model part of the finished image does not help much. Even learned visual attribute embeddings do not solve spatial reasoning. Performance jumps only when the conditioning supplies a centroid mask that explicitly localizes the objects. This contradicts the headline notion of learning visual reasoning “without symbolic representation.” It only reasons well once the symbolic constraints have been anchored into image coordinates.

Questions for authors

Can you comment on the issue of the encoder being decoupled from the diffusion model and what this means for more complex visual settings that require a more sophisticated encoder?

How does this affect the claim of "visual reasoning without symbolic representation"?

Is the Thinker informed of diffusion timestep t?

How much of the performance comes from reasoning over the current visual state x_t, as opposed to solving the conditioning independently of x_t?

4 comments

We thank the reviewer for the careful and constructive review. We address each point below.

W1 / W2. LIT, related work, and scope.

We thank the reviewer for highlighting LIT and apologize for overlooking this closely related work. LIT provides valuable evidence that separating latent reasoning from observation modeling benefits visual reasoning, and we agree that this motivation is shared with PaTh. We have added LIT to Table 1, corrected the prior-best accuracy, and revised the related work and abstract to acknowledge this shared foundation.

The main distinction is how this separation is achieved. LIT trains recurrent representations through a teacher–student imagination objective, with the teacher receiving additional views of the training solution. PaTh instead places a separate, compact reasoner alongside a frozen diffusion model and trains it through the standard reconstruction objective, without auxiliary latent targets. As the reviewer notes, this simple, modular coupling is a strength of our approach. PaTh reaches 71.2% on visual Sudoku Extreme versus LIT’s reported 54.5%, with approximately 10M versus 81M parameters, and demonstrates benefits across four visual constraint settings.

We have therefore removed the broad “gap we fill” claim and sharpened the contribution around this architecture and its empirical benefits, while expanding the discussion of LIT and the other recurrent approaches identified by the reviewer (Sec. 2; App. U).

W3 / Q1. The encoder is decoupled from the diffusion model; will it scale?

We treat decoupling and training the encoder from scratch as a hyperparameter that we hold fixed for the visually simple settings we study. Nothing prevents the Thinker from reading features of a pretrained model. We test this on Sudoku Extreme with an 8-block MM-DiT Painter (App. V): its first two blocks serve as the Thinker's encoder, so the Thinker reads their hidden states instead of x_t, and steers blocks 3 to 8.

Thinker inputPuzzle acc.
x_t, separate encoder52.5
Painter features after block 258.9

Reading the Painter's features outperforms learning a separate encoder, so the Thinker can build on the Painter's representation.

We agree that this, like CLEVR, is a proof of concept of scalability rather than a demonstration on complex imagery, and we now list scaling to complex visual settings as an explicit future direction (Sec. 5, App. L.2).

W4 / Q2. Spatial alignment and "without symbolic representation".

We thank the reviewer for prompting us to investigate the source of this limitation. New experiments established that the spatial-alignment limitation is actually much weaker than the paper stated, and it comes from the UNet Painter rather than from PaTh. We hypothesise that it comes from the UNet's strong spatial inductive bias: its computation is mostly convolutional, so a residual that the Thinker adds at one position mainly affects the corresponding image region, and so the Thinker must know where each constraint belongs. A Painter based on SOTA transformer with ROtary Position Embeddings, whose every block mixes all positions through attention, should be largely free of this limitation. We test this on CLEVR with an MM-DiT Painter, keeping the Thinker's architecture, with no centroid mask for either network (App. L):

PainterCentroid maskSpatial acc.
UNet aloneno50.8
UNet + PaThno51.2
UNet + PaThyes89.4
MM-DiT aloneno69.5
MM-DiT + PaThno87.1

Without a mask, PaTh adds nothing to the UNet Painter, but with the MM-DiT Painter it raises spatial accuracy from 69% to 87% without any spatial anchoring, close to the 89.4% that the UNet PaTh reaches only with centroid masks. We updated the Limitations accordingly: the alignment requirement applies to Painters with strong spatial inductive biases, such as UNets, not to PaTh itself.

Q3. Is the Thinker informed of the timestep?

The recursion itself is not. The translator, which maps the Thinker's output to the Painter's residuals, receives a timestep embedding, as does the encoder.

Q4. How much comes from reasoning over x_t?

We ablate this on Sudoku Extreme and Hard (App. T.3):

2 states + x_t (PaTh)2 states, no x_t1 state + x_t1 state, no x_t
EXTREME71.257.757.346.3
HARD92.50.091.20.0

On HARD, where most puzzles have many valid solutions, x_t is essential: without it, the Thinker cannot tell which solution the Painter is drawing, and solves no puzzle. On EXTREME, whose solution is unique, it still adds 13.5 points, acting as an additional state (about as much as the second latent state).

Thank you for the rebuttal, which answers my questions and resolves some of my concerns. The exchange of UNet in favor of a transformer architecture brings some interesting new evidence, which, however, should not be over-interpreted, since multiple things were changed at a time and the cause of the better numbers has not been identified uniquely. One question that came up with the modified architecture. You write that the Painter's first two blocks serve as the Thinker's encoder, so the Thinker reads their hidden states instead of xtx_t, and steers blocks 3 to 8. Are blocks 1–2 still frozen Painter blocks, whether their output is branched into both Painter and Thinker, and does the Thinker then influences blocks 3–8 while the entire MM-DiT remains frozen?

Thank you for your reply. We are glad that our rebuttal answered your questions and that you find the new results interesting.

UNet vs. MM-DiT Painter. We believe the MM-DiT result addresses the main concern of your review: that PaTh relies on "a fixed image-coordinate correspondence" and only reasons well once the constraints are "anchored into image coordinates". Without spatially aligned conditioning, PaTh with the MM-DiT Painter reaches 87.1% spatial accuracy on CLEVR (69.5% for the MM-DiT Painter alone), compared to 51.2% for the UNet Painter version. The dependence on spatial grounding therefore comes from the UNet Painter, not from PaTh, and the claim that PaTh learns visual reasoning without symbolic representations stands.

We agree that the rebuttal does not fully explain why the UNet-based PaTh fails without spatially aligned conditioning. Attributing it to the UNet's strong spatial inductive bias remains a hypothesis, although we believe the results provide strong evidence for it. Either way, our claim above does not depend on it.

Painter features as the Thinker's input (App. V). Yes, to all three questions. The entire MM-DiT Painter, including blocks 1–2, is trained in the first stage and stays frozen afterwards. At every Thinker call, blocks 1–2 run once on x_t, and their output is shared: the Painter continues from it through blocks 3–8, and the Thinker reads it in place of x_t. The Thinker's output then steers blocks 3–8 through residuals added after each block. No part of the Painter is trained or changed during the Thinker's training stage.

We would be happy to answer any further questions. If our response has addressed your concerns, we would be grateful if you would consider updating your score.

Sounds good. I raised my score to 3.

Reviewer_QrSqOpenReview ↗
Rating
3(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

The paper introduces PaTh, a framework for augmenting the diffusion process with a thinking step which provides a conditional signal through controlnet. PaTh studies visual reasoning problems with constraints in continuous domains and shows a significant improvement against existing works for Sudoku, and competitive performance against standard diffusion for maze and CLEVR dataset.

Critical strengths

The performance reported is strong on Sudoku, PaTh significantly improves over existing methods on MNIST sudoku extreme.

The method is intuitive: adding a reasoning step for each diffusion step can improve performance.

Critical weaknesses

The baselines used are weak, the paper primarily uses the simplest base diffusion as the baseline for tasks other than Sudoku. One additional simple baseline would be a predictor-corrector style diffusion process which introduces additional corrector steps at each diffusion timestep which could in some way be interpreted to serve a similar role as the thinker. IPR which the authors states is the most closely related method, is only included as a baseline for Sudoku. Similarly, TRM is not included besides sudoku. Some other related works/ potential baselines include: [1] Yang et al. Learning to Solve Constraint Satisfaction Problems with Recurrent Transformer . ICLR 2023 [2] Xu et al. Self-Supervised Transformers as Iterative Solution Improvers for Constraint Satisfaction. ICML 2025 [3] IRED which the paper already cites

The methodology is described in text without mathematical formalism. It would be better to define the input output of the different modules as well as how they interact. Details of controlnet’s implementation are also missing. More importantly, the paper does not discuss how the diffusion process is affected by the augmented thinker. Does the resulting method still sample from the learned data distribution?

It is not clear what exactly is causing the improvement in performance. Section 4.2 observes different subtasks that PaTh does better than regular diffusion, and that the thinker’s projected internal state changes. These observations however, does not justify the need for a TRM style thinker. Could the performance gain come from more iterations or from the controlnet signals? Ablations on the number of iterations (including with no iterations, ie implement a non-recurrent transformer as the thinker) is therefore critical.

Questions for authors

Does the method sample from the same data distribution as standard diffusion?

Do you have any insights on whether a recurrent model is effective as a thinker for producing conditional signals?

Minor points:

In figure 2, guidance is done by a “Translator” which is not defined in the text.

MNIST Sudoku Extreme’s previous SOTA is said to be 4.1% in the abstract but table 1 shows TRM at 8.3%

4 comments

We thank the reviewer for the careful review. We ran every requested experiment and addressed all points below.

We respectfully note our setting: PaTh generates the solution as an image, in pixel space (not discrete symbols), trained from pixels alone (no labels, rules, solver, or verifier). The rules we check must hold in the output image itself, and most such tasks, like our CLEVR scenes, have no symbolic form to solve and render back.

W1. Weak baselines.

We respectfully disagree with the statement that "the baselines used are weak, the paper primarily uses the simplest base diffusion as the baseline for tasks other than Sudoku". We would like to point out that on AMAZE, we compare our approach against recent unified models. Bagel (14B) and Janus-Pro (7B) are trained on our training set until convergence, and proprietary editors pass at most 5.4% of mazes. Bagel solves 43.4% of mazes and 13.7% of Queens exactly, against 75.1% and 100% for PaTh with 10M parameters.

To further strengthen the evaluation, we have added three suggested diffusion-family baselines on every task (App. S; Fig. 11):

EXTREMEHARDMaze (Exact@1)Queens (Exact@1)CLEVR (spatial)
DM2.98.319.090.968.8
DM + Langevin corrector (best K ≤ 8)3.840.224.396.070.1
DM + Restart corrector (best K ≤ 8)4.456.322.994.670.8
TRM-DM (TRM as the denoiser)8.376.768.898.961.7
SRM3.251.656.880.062.8
SRM + IPR4.175.067.980.762.3
PaTh71.292.575.1100.089.4
  • Correctors saturate far below PaTh, even at 5× its FLOPs on Sudoku (Fig. 11, Sec. 4.3; Tab. 27, App. S).
  • TRM-DM (recursion inside the denoiser, same depth and cost) is close only on the visually simplest Maze and Queens.

Proposed baselines (Sec. 2).

We are thankful for additional benchmark suggestions, but we'd like to point out that they work in a different setting, assuming symbolic outputs, symbolic solution supervision, or hand-written constraints, which makes a fair comparison impossible (clear advantage for MNIST-SUDOKU, with no possible application in CLEVR):

  • Yang et al. (2023), Sec. 3.2: "l ∈ {na, 1, …, 9}^81 is the 'ungrounded' solution for the Sudoku puzzle".
  • ConsFormer, Sec. 3: "the Transformer outputs a one-hot vector over the domain", trained on "approximations to the amount of constraint violations".
  • IRED, Sec. 3.2, 4.2: supervised with the "corrupted ground truth label"; on visual Sudoku, it "fuse[s] the image embeddings with the noisy answer".

PaTh outputs the solved puzzle as an image, trained only to reconstruct pixels.

W2. Mathematical formalism.

We thank the reviewer for this suggestion. We have added Appendix Q where we formally define the inputs and outputs of each module and their interactions, the loss, and the sampler.

ControlNet details. The translator (Fig. 2) arranges the Thinker's read-out on its token grid, upsamples it to the Painter's resolution and passes it through a small convolutional network mirroring the resolutions of the Painter's encoder. Zero-initialised convolutions output one residual per UNet skip connection and one for the mid-block, so PaTh initially reproduces the Painter. Unlike the original ControlNet, it is not a trainable copy of the Painter's encoder, but a small adapter. (App. Q; Sec. 3.2).

W2 / Q1. Same data distribution as standard diffusion?

Yes. PaTh changes only the architecture of the x0-predictor, trained with the standard denoising loss alone. Its minimiser is E[x_0 | x_t, c], exactly what the unchanged sampler needs (App. R).

W3 / Q2. What causes the gain?

Below, we present the results for three suggested additional ablations focusing on the Thinker architecture (Also added to App. T). We evaluate those setups on EXTREME, puzzle accuracy:

  1. "Implement a non-recurrent transformer as the thinker" - We compare our original TRM-based thinker with a non-recurrent transformer with different size:
No recursion, layers24816PaTh (2 layers, recursive)
Params10M15M29M56M10M
Puzzle acc.3.89.79.13.271.2
  1. Ablations on the number of iterations - We compare the performance of PaTH under different numbers of recursive iterations during training:
Trained N_sup124816
Puzzle acc.29.753.163.767.971.2
  1. Justification for a TRM-style thinker - We evaluate whether additional states are required to solve hard puzzles.
2 states + x_t (PaTh)2 states, no x_t1 state + x_t1 state, no x_t
EXTREME71.257.757.346.3
HARD92.50.091.20.0
  • The gain comes from recursion, not from the added module: depth without recursion fails, and accuracy grows with the trained recursion budget.
  • Recursion must interact with the Painter: without x_t, the Thinker solves no HARD puzzles, which have many valid solutions.
  • Recursion must be separate from rendering: TRM-DM reaches 8.3%, correctors at most 4.4%.

Minor. The Translator is now defined (App. Q). The abstract now cites LIT (54.5% on EXTREME) as the prior best instead of SRM + IPR (4.1%). Table 1's "TRM" row (8.3%) is our own baseline, not prior work; we renamed it to TRM-DM.

Thank you for the rebuttal and additional experiments. Most of my concerns have been addressed. My main remaining concern is whether the benefit of recurrence has been adequately isolated from the benefit of increased computational depth. While the non-recurrent Transformer ablations are helpful, they do not control for the effective computational depth. Nevertheless, the additional experiments and clarifications significantly strengthen the paper, and I will raise my score to 3.

Thank you for your reply. We are glad that our rebuttal answered your questions, and thank you for raising the score.

Benefit of recurrence isolated from increased computational depth

Training a non-recurrent transformer of depth matching that of the full recursive Thinker would be computationally prohibitive. Therefore, to separate recursion from depth, we trained the PaTh Thinker (2 layers) with recursion set to apply exactly 4, 8 or 16 layers per denoising step, and compared it with non-recursive Thinkers of the same depth (MNIST Sudoku EXTREME, puzzle accuracy):

Layers per denoising step4816
No recursion9.79.13.2
Recursion (PaTh Thinker)11.412.429.0

Recursion wins at every depth, and the gap grows with depth.

Once more, thank you for the fruitful discussion and suggestions that strengthen our paper!

Reviewer_hyQSOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Painter–Thinker (PaTh), a framework that couples a recursive latent reasoner with a frozen diffusion model through ControlNet-style adapters. The Thinker repeatedly updates latent states within each denoising step and provides guidance to the Painter. Training uses target-image reconstruction without symbolic solution labels or an external solver/verifier.

PaTh reports substantial improvements on MNIST Sudoku, maze solving, Queens, and constrained CLEVR scene generation. In particular, it achieves 92.5% puzzle accuracy on HARD Sudoku and 71.2% on EXTREME Sudoku.

The results provide evidence that recursive latent computation can improve constraint satisfaction in image generation. However, important details of the data and evaluation protocols are missing, and the evidence for generalization and the proposed reasoning mechanism requires more careful qualification.

Critical strengths

Interesting architectural contribution. Separating recursive latent computation from image rendering is a sensible design. Coupling the two through reconstruction-based training provides a practical way to apply recursive reasoners to visual outputs without requiring explicit intermediate reasoning annotations.

Detailed analysis on Sudoku, with selected diagnostics on other tasks. The paper provides substantial architectural and computational ablations on Sudoku, covering coupling mechanisms, training strategies, recursion budgets, and inference FLOPs. Error-recovery experiments on CLEVR and mazes provide additional evidence beyond Sudoku.

Critical weaknesses

The training and evaluation protocols are insufficiently specified. The manuscript reports training steps, but does not clearly state the number of distinct training examples, batch sizes, or actual evaluation-set sizes for the main tasks. For example, the 100,000 Painter steps and 160,000 Thinker steps on Sudoku do not establish how many independent puzzles or completed grids were observed. The 384 samples mentioned in Section 4.2 concern specific diagnostic analyses and do not establish the sample sizes underlying Tables 1–3. The distinction between underlying solution grids, masked puzzles, and rendered or augmented images is particularly important for Sudoku. The paper should document the split and augmentation procedure for both training stages, including whether Painter pretraining uses any completed grids associated with the test puzzles.

Generalization beyond the benchmark layouts is not clearly stated. Three out of four tasks have exact visual patterns for training and testing examples. For the remaining maze task, results are broken down by geometry and size, but the training/test allocation of those geometries and sizes is unclear. Strong performance across several geometries does not establish transfer to an unseen geometry if all were represented during training. Likewise, visual similarity between training and test images is not itself evidence of memorization. An explicit structural holdout protocol would make the generalization claim more informative. Appendix L provides a hint regarding this concern, that, translating or scaling the Sudoku target causes performance to collapse, and applying the same transformation to the conditioning recovers only 3.4% puzzle accuracy.

The mechanistic interpretation is stronger than the evidence. The probes show non-monotonic changes in decoded constraint violations, which are consistent with candidate revision. However, this observation does not uniquely establish search or backtracking. In Appendix K, the probe is trained to predict digits from the trajectory’s own final generated image. Its agreement with that output varies across denoising stages. Consequently, changes in decoded violations may reflect both actual changes in latent candidates and variation in probe reliability. This is especially relevant because the claimed rethink behavior is concentrated early in sampling. The recursion ablation also requires careful interpretation. Figure 10 evaluates a model trained with extensive recursion using Nsup=H=L=1N_{sup}=H=L=1. Its poor performance demonstrates sensitivity to inference truncation, but does not by itself establish how a separately trained shallow guidance network would perform. A control trained for its intended inference budget would better isolate the contribution of recursion.

The respective contributions of the Thinker and Painter remain insufficiently disentangled. The answer state y already carries information about the predicted solution before it is passed to the Painter. The Sudoku probing results suggest that substantial solution computation occurs within the Thinker, with the Painter potentially serving primarily to render the inferred answer. This division of labor is consistent with the architecture, but its extent is not established across the four tasks. The computational contribution of the Thinker is also substantial: the default configuration performs 336 applications of the shared reasoning network per denoising step, or 6,720 across a 20-step sampling trajectory. Although Figure 10 provides useful FLOP comparisons on Sudoku, the small parameter count alone does not characterize the reasoning budget. A clearer decomposition is needed to establish what iterative interaction with the diffusion model contributes beyond a task-trained recursive predictor followed by an image decoder. The standalone TRM baseline is informative, but does not directly isolate the contribution of the Painter when paired with the same learned Thinker representation.

Questions for authors

Splitting and processing of training and evaluation data. For each of the four tasks, please report the number of distinct training, validation, and test instances; batch sizes; augmentation procedures; and data exposure during both Painter and Thinker training. How is independence between training and testing ensured for each task, including Painter pretraining? Please describe checks for duplicate or equivalent instances, such as Sudoku puzzles and solution grids under symmetry, maze topologies under rotation/reflection, Queens board configurations, and CLEVR scene graphs and layouts. Is splitting performed before augmentation, and can different input instances share the same underlying solution?

Generalization beyond the training distribution. For each task, which problem sizes, structural configurations, and visual layouts are represented in training versus testing? Do the reported results measure generalization to unseen instances within the same distribution, or also to held-out structures and rendering conditions? Can you provide an explicit structural or visual generalization evaluation for each task?

Architectural choices and computational trade-offs. Do the benefits of recursion, state persistence, early stopping, and the chosen coupling mechanism hold across all four tasks? Could you report representative comparisons across tasks against guidance modules trained for their intended inference budgets, alongside parameter counts, inference FLOPs, and latency?

Mechanism across tasks. What evidence supports a common mechanism of latent candidate revision across Sudoku, Maze, Queens, and CLEVR? Can task-appropriate probes or controlled interventions distinguish useful intermediate revision from probe error or generic recurrent refinement? Please also clarify whether early concentration of reasoning and recovery from indirect errors occur consistently across tasks and difficulty levels.

3 comments

We thank the reviewer for the careful review. We address each point below.

W1 / Q1. Data and evaluation protocols.

We added a full description (App. A.1, Tab. 4) and the training-set sizes and batch sizes to the hyperparameter table (Tab. 7). For every task, the data is split before any augmentation, augmentation is applied to the training set only, and the Painter and the Thinker are trained on the same training set. No test instance is seen in either stage, including Painter pretraining.

  • Sudoku EXTREME: following TRM, we use its Sudoku-Extreme-1k train and test splits (1,000 and 422,786 puzzles), which contain different puzzles, also under augmentation. We augment exactly as TRM does: 1,000 precomputed variants per training puzzle, using validity-preserving symmetries (digit relabelling, transposition, and permutations of bands, rows within bands, stacks, and columns within stacks). Each drawn training puzzle is rendered with random MNIST training digits. Test puzzles are rendered with MNIST test digits, so their handwriting is unseen.
  • Sudoku HARD: for a fair comparison, we use exactly SRM's boards (1M solution grids, the last 10,000 are test) and render them as SRM does, without further augmentation. The revealed cells are resampled each time a training grid is drawn, and fixed per test grid.
  • Maze / Queens: the AMAZE benchmark, with 29,988 / 30,000 training boards and held-out test sets distinct from them: 100 mazes per geometry and size (2,800) and 50 Queens boards per size (350). No augmentation.
  • CLEVR: the official split, training on the 70,000 training scenes and testing on the validation split. Images are resized to 256×256. The conditioning is augmented: each time a scene is drawn, its sparse relation set is one of 16 transitive reductions of the full relation graph, with each relation stated in a random direction.
TrainEvaluated test
EXTREME1k puzzles (1.001M with augmentation)7,680 puzzles
HARD990k grids7,680 grids
Maze29,9882,800 (× 5 samples)
Queens30,000350 (× 5 samples)
CLEVR70,000 scenes1,500 scenes

All models are trained until convergence and evaluated on these test sets. The 384 samples of Sec. 4.2 are used only for the diagnostic analyses.

W2 / Q2. Generalization.

PaTh, like our DM baselines, is trained from scratch on each task, so the main benchmarks measure generalization to unseen instances (on EXTREME, also unseen handwriting). We test broader generalization in two ways:

  • Size (App. D): trained on the smallest instances of the geometric tasks of Goren et al. (2026), PaTh raises validity on the largest Simple Polygons from 0.620 to 0.869 (optimality rate 0.062 → 0.264), and is on par on Steiner Tree and Inscribed Square.
  • Unseen maze structures (App. W, new): By probing the inner state of the Thinker, we notice that it performs dead-end filling, a classic maze algorithm: it marks dead ends, then removes them tip-first in the algorithm's order (76% of junctions, against 34% at random). On mazes with a loop or extra starts, never seen in training, it gives that algorithm's answer: it keeps or removes extra branches whole (94%) and keeps both routes around a loop (49–88%). Those results show that our approach learns how to apply a valid procedure, not memorised layouts.

W3. Mechanistic interpretation.

The probe is indeed less reliable at the very start of sampling, but the rethinks are not confined to those steps: from t = 5 (25% of sampling) the probe is above 95% accurate, while the rethink magnitude peaks around t = 7 (35% of sampling, about 6.4 additional violated constraints) and the rethink rate stays above 0.11 until t = 10 (Fig. 9).

In Fig. 10 (Fig. 11 in the revision), we added the requested controls, each trained for its intended budget (App. T.1, T.2, EXTREME). Thinkers trained with fewer supervision steps reach 29.7 / 53.1 / 63.7 / 67.9% at N_sup = 1 / 2 / 4 / 8, and deeper non-recursive Thinkers (up to 56M parameters) reach at most 9.7%, against 71.2% for PaTh.

W4. Thinker vs. Painter.

We agree with the reviewer: by design, the Thinker does most of the solving and the Painter mostly renders. We added the comparison the reviewer describes, which shows this directly: the same Thinker, trained in the same way, but never seeing the current sample x_t (App. T.3). It reads only the puzzle, so it is a task-trained recursive predictor whose output the Painter only renders, and it already reaches 57.7% on EXTREME, far above every diffusion baseline. The iterative interaction with the diffusion model adds the remaining 13.5 points (71.2% for PaTh). On HARD, where most puzzles have many valid solutions, it solves none (0.0% against 92.5%): without x_t, it cannot tell which solution the Painter is drawing. For the reasoning budget, we now report inference FLOPs and latency on all four tasks (App. X, see Q3).

Q3. Architectural choices and compute.

We report inference FLOPs and latency for PaTh, the DM, TRM-DM, and the predictor-correctors on every task (App. X; parameters in Tab. 6). Recursion without the Painter (TRM-DM) and extra sampling compute for the DM (correctors) are evaluated on all tasks (App. S). Early stopping also transfers (App. N): reasoning only during the first part of sampling and reusing the Thinker's last output afterwards gives

Reasoning for the first5%20%50%100%
Maze (Exact@1)46.359.972.575.4
Queens (Exact@1)99.1100.0100.0100.0
CLEVR (spatial)74.981.287.988.8

so, as on Sudoku, reasoning during the first half of sampling recovers nearly all of the accuracy, at about half of the Thinker's cost.

Q4. Mechanism across tasks.

We repeated the diagnostics on the other tasks (App. Y):

  • Constraints are resolved early. Scoring the predicted clean sample along the trajectory, PaTh resolves mazes and Queens early: on Maze, its Exact rate reaches 66.8% after 10% of sampling (72.7% at the end), while the DM stays below 20% throughout; on Queens, PaTh is exact from the first recorded step. On CLEVR, objects are placed in the middle of sampling, matching the early-stopping results above (Fig. 22).
  • Rethinks grow with difficulty. With the same probe analysis as on Sudoku, the Thinker's rethink rate on Maze grows with maze size:
Maze size5789111316
Rethink rate (%)1.21.93.35.412.817.520.5

On Queens, which the DM already solves in 91% of cases, rethinks are rare (0.5%) and rise only on the largest boards.

  • Recovery from indirect errors is not specific to Sudoku: on mazes, the DM retains mistakes that are only indirectly visible far more often than directly visible ones, while PaTh treats both alike (Fig. 8), and on CLEVR, PaTh recovers from corrupted scenes throughout the part of the trajectory where change remains possible (Fig. 7).
Reviewer_hyQSOpenReview ↗

Responce to authors

Thank you to the authors for your response and for providing comprehensive additional experiments and clarifications. These have adequately addressed my previous concerns and questions. I appreciate the authors' efforts and will maintain my original positive score of 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.