acceptodds
Under review as a conference paper at ICLR 2027

Relational Abstractions for Spatial Reasoning with Diffusion Models

Abstract

Diffusion models excel at image synthesis, but they remain limited in their ability to reliably satisfy structured spatial reasoning constraints. In conditional data distribution modeling tasks with implicit logical structure, such as puzzles defined by visible clues paired with consistent solutions, state-of-the-art generative models tend to approximate pixel-space distributions without learning the underlying logical rules required for inference. To address this limitation, we present a novel framework for spatial reasoning with diffusion models that leverages unsupervised object discovery and abstractions of object relations. We show that the relational knowledge derived from object-centric representations enriches diffusion models with structural primitives, allowing them to effectively guide the generative representation space during both training and inference, and enabling conditional image generation that satisfies reasoning constraints. Additionally, we introduce a large-scale generative spatial reasoning benchmark with four datasets inspired by human-solvable puzzles. Our results show that relational abstractions significantly improve reasoning capabilities of diffusion models on a variety of complex reasoning tasks, while enabling robust generalization in out-of-distribution settings.

Primary area:
generative models
Keywords:
diffusion models, abstract visual reasoning, object-centric learning, image generation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_KobhOpenReview ↗
Rating
4(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces Relational Abstractions for Diffusion Models (RDM), a framework for improving spatial reasoning in conditional image generation. The method first learns object-centric representations using a slot-based autoencoder trained with a reconstruction objective, then learns relational abstractions over these representations. The resulting relational features guide diffusion-based generation, with the goal of helping generative models capture the structural relationships and logical constraints underlying a task. The authors evaluate RDM on several reasoning tasks, including Akari, Coldoku, Tangram, and LogicFace, and compare it against standard diffusion baselines and Spatial Reasoning with Denoising Models (SRM). They also provide ablations examining the contribution of relational abstractions and experiments on additional benchmarks, including MNIST-Sudoku and Counting Polygons.

Critical strengths
  • Promising approach to incorporating structure into generative reasoning: The paper addresses an important limitation of generative models: they may learn visual patterns without explicitly capturing the underlying relations and constraints that define a valid solution. Combining object-centric representations with relational abstractions is a compelling approach to this problem.
  • Evaluation across diverse reasoning tasks: The experiments span grid-based puzzles, geometric compositions, and facial attributes, providing evidence across different forms of structured generation. The inclusion of additional benchmarks and ablations helps assess the applicability of the proposed approach.
Critical weaknesses
  • Insufficient clarity regarding the SRM comparison and its task-specific configurations: The comparison with SRM is potentially informative, but the implementation and configuration of the baseline are not sufficiently clear across the different tasks. In particular, it is unclear how the patch-based formulation and uncertainty-guided sequential sampling are adapted to tasks such as LogicFace, where the variables are semantic facial attributes rather than grid cells. More details would make the comparison easier to interpret.
  • Motivation for the reconstruction objective: The Discoverer is trained using an image reconstruction objective, which encourages the slots to collectively encode the information needed to reconstruct the input, but does not explicitly encourage object-level decomposition or semantic disentanglement. The supplementary results show that some slots remain inactive or capture multiple Tangram pieces, while some facial attributes are not cleanly disentangled in LogicFace. This raises the question of why reconstruction alone is an appropriate objective for learning representations intended for relational reasoning, and whether the authors could better motivate this design choice.

Minor suggestion for extending the related work:

  • The LogicFace setting is related to prior work on representing and controlling disentangled facial semantics . For example, Nested Attention: Semantic-aware Attention Values for Concept Personalization, Omni-ID: Holistic Identity Representation Designed for Generative Tasks, and Latent-Identity Tuning in Text-to-Image Personalization Models investigate disentangled semantic representations of personalized subjects using learned queries.
Questions for authors

I would be open to raising my recommendation to a Clear Accept if the authors' responses adequately address the following questions:

  • SRM configuration: Could the authors clarify the exact SRM configuration used for each benchmark? In particular, how are patches defined, and how is uncertainty-based sequential sampling implemented for non-grid tasks such as LogicFace?
  • Motivation for the Discoverer objective: Why is image reconstruction alone sufficient for learning representations suitable for relational reasoning, given that it does not explicitly enforce object-level correspondence or semantic disentanglement? Could the authors better motivate this choice?
4 comments
AuthorsOpenReview ↗

Response to Reviewer Kobh

Q1: SRM configuration: Could the authors clarify the exact SRM configuration used for each benchmark? In particular, how are patches defined, and how is uncertainty-based sequential sampling implemented for non-grid tasks such as LogicFace?

W1: Insufficient clarity regarding the SRM comparison and its task-specific configurations: The comparison with SRM is potentially informative, but the implementation and configuration of the baseline are not sufficiently clear across the different tasks. [...] More details would make the comparison easier to interpret.

We thank the reviewer for pointing out this ambiguity. We now added a new subsection with details on "Implementation of SRM'' under Sec. B.4.

We follow the official SRM implementation. In SRM, uncertainty is learned per square patches during training, and we use the predicted-order sampling without overlap (their best setting on MNIST Sudoku), which denoises the lowest-uncertainty patch first. In LogicFace, indeed SRM has no notion of semantic attributes as its variables are always spatial patches, so attributes can span several patches. This mismatch, which is also present in Tangram and Tildoku, explains why SRM's sequential sampling only helps on aligned grids (MNIST-Sudoku, Coldoku). This is an inherent limitation of SRM for generalized reasoning settings, since it is restricted to grid-aligned reasoning problems.

Our implementations were adapted to align with SRM's inductive bias, by matching the patch size to the grid dimensions in Akari (patch size fixed to 18) and Coldoku (patch size fixed to 20). For unstructured puzzles (Tangram and LogicFace), we set the patch size to 8, which was proposed for real-world images by [Wewer et al., 2025] in Counting Polygons. Alternative values did not yield any improvements.

Q2: Motivation for the Discoverer objective: Why is image reconstruction alone sufficient for learning representations suitable for relational reasoning, given that it does not explicitly enforce object-level correspondence or semantic disentanglement? Could the authors better motivate this choice?

W2: Motivation for the reconstruction objective: The Discoverer is trained using an image reconstruction objective, which encourages the slots to collectively encode the information needed to reconstruct the input, but does not explicitly encourage object-level decomposition or semantic disentanglement. The supplementary results show that some slots remain inactive or capture multiple Tangram pieces, while some facial attributes are not cleanly disentangled in LogicFace. This raises the question of why reconstruction alone is an appropriate objective for learning representations intended for relational reasoning, and whether the authors could better motivate this design choice.

Object-centric representation learning methods, such as Slot Attention [Locatello et al., 2020], are designed to learn disentangled object representations through image reconstruction objectives. By design of the training algorithm, the iterative refinement loop in the bottleneck forces slots to compete for distinct regions in images, resulting in object disentanglement (see Figure 13). Our motivation to use this approach is two-fold. First, we argue that generalized spatial reasoning requires abstraction of object semantics rather than generating a plausible solution from raw pixels. Second, this abstraction should be task-agnostic and independent of puzzle labels. Representations learned via Slot Attention satisfies both, since its reconstruction objective requires no object masks or labels, and it does not impose a fixed spatial (grid) structure.

We agree that slot decomposition is not always perfect, but previous work has shown that disentanglement does not need to be perfect to help downstream tasks (e.g., OCRA [Webb et al., 2023] for RPM puzzles). Our results also support this, where learned relational abstractions improve reasoning performance across all datasets. Since slots are high-dimensional continuous representations, their encoded information is relatively robust against noise, and the Abstractor can still exploit and interpret the disentangled space to learn patterns and relations that effectively guide the diffusion model for reasoning.

Minor suggestion: The LogicFace setting is related to prior work on representing and controlling disentangled facial semantics. For example, Nested Attention: Semantic-aware Attention Values for Concept Personalization, Omni-ID: Holistic Identity Representation Designed for Generative Tasks, and Latent-Identity Tuning in Text-to-Image Personalization Models investigate disentangled semantic representations of personalized subjects using learned queries.

We thank the reviewer for these references. We now mention them in context of our work under Sec. A.4, with the details of LogicFace.

Dear Reviewer Kobh,

Thank you again for taking the time to review our work.

We have provided detailed responses to your comments and questions, and revised our manuscript with additional experiments and clarifications. We would appreciate your feedback on our rebuttal, as the discussion period is approaching the end.

We would also be happy to address any remaining concerns about our submission.

Best regards, Authors

Reviewer_KobhOpenReview ↗

Response to authors

I thank the authors for their response which resolved my concerns. I will raise my score.

We thank Reviewer Kobh once again for their time and thoughtful suggestions.

We are pleased to hear that all concerns are now fully addressed, and also thank the reviewer for supporting the acceptance of our submission.

Best regards, Authors

Reviewer_Ni4VOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces RDM (Relational Abstractions for Diffusion Models), a framework that improves spatial reasoning in diffusion models. Standard diffusion models tend to fit pixel-level distributions without learning the logical rules behind puzzles. RDM addresses this by using Slot Attention to discover objects without supervision, then applying Abstractors to extract object relations, which guide a flow-matching U-Net through cross-attention during training and inference. The authors also release a benchmark of four datasets: Akari, Coldoku, Tangram, and LogicFace. RDM outperforms baselines on most tasks, including up to 3× gains on Tangram, and generalizes better out-of-distribution. SRM remains stronger only on the hardest Coldoku level.

Critical strengths
  • Instead of tweaking the sampling like SRM does, they change what the model conditions on: object-level relations from Slot Attention plus Abstractors. It's a fresh way to push diffusion models toward following rules.
  • The solution's relational embedding is shown half the time and hidden the other half, CFG-style. At test time you only have the clue, so only that branch runs. Weirdly, even the "solution" setting, which uses nothing extra at inference, still helps.
  • The new decoder handles 81 or 121 slots, while most prior work stays around 10 or fewer.
Critical weaknesses
  • Slot Attention, Abstractors, CFG-style dropout, cross-attention, and flow matching all already exist. The novelty is the combination, not a new technique.
  • The best setting (clue, solution, or full) changes by task. More slots isn't always better, and the 81-slot full model on Coldoku actually drops. SRM still wins on the hardest Coldoku level.
  • Coldoku Hard is about 12.6% and Tangram tops out under 47%. The "3×" is measured against a pretty weak baseline.
  • Evaluation relies on neural evaluators, Tangram comes from only 1,013 base silhouettes, slot decomposition on Tangram is admittedly poor, and some table entries look like typos (e.g., a SHAPE score of 4.31).
  • It's mostly Base versus SRM, with no search-based or neuro-symbolic methods to show where this really stands.
Questions for authors
  • What is the total cost, including pretraining the discoverer, relative to Base and SRM, at both τ=10 and τ=100? Does the advantage hold at equal compute?
  • Your analysis suggests the main difficulty is linking arbitrary colors in the piece view to the final arrangement. Does RDM address this specifically, or only help indirectly? And how do you expect it to behave on scenes with many more objects or less clean visual structure?
5 comments
AuthorsOpenReview ↗

Response to Reviewer Ni4V (Part 1/2)

Q1: What is the total cost, including pretraining the discoverer, relative to Base and SRM, at both τ\tau=10 and τ\tau=100? Does the advantage hold at equal compute?

For training, relative to Base and SRM, the only additional cost of RDM is the discoverer pretraining. Since that is a small convolutional network (1.4-7.6M params), it trains more than 10$\times$ faster than the diffusion models, and it is trained only once for all RDM variants. The diffusion models of Base and SRM are capacity-matched to RDM in parameters, and trained for the same number of steps, so the current comparison is at an equal compute.

At inference, RDM additionally requires a forward pass through the slot encoder and the Abstractor stack. This cost is also negligible compared to the U-Net diffusion backbone, and in practice we observed no major difference in sampling time, at both τ=10\tau=10 and τ=100\tau=100. However SRM (sequential) is significantly slower, since it requires 81×81\times, 121×121\times and 256×256\times more evaluations on Sudoku, Akari and Tangram/LogicFace, respectively, as patches are denoised one at a time.

Q2: Your analysis suggests the main difficulty is linking arbitrary colors in the piece view to the final arrangement. Does RDM address this specifically, or only help indirectly? And how do you expect it to behave on scenes with many more objects or less clean visual structure?

We thank the reviewer for bringing up this very interesting point. We would not claim that RDM addresses this problem specifically, since it was not designed for this specific task (linking piece colors in the Tangram view), but rather that its mechanism (i.e., disentangled relational representations to guide the diffusion process) indirectly helps. In Tangram, RDM (clue) leaves shape accuracy almost unchanged with respect to Base (57.72 →\to 58.41) while doubling the overall accuracy (15.46 →\to 34.37), which indicates that the gain comes from assigning the correct colors to pieces.

We would expect all diffusion models (Base, SRM and RDM) to deteriorate with more objects or noise since the visual scenes become denser, and extracting the patterns that solve the puzzle becomes more difficult. Currently, RDM shows the largest gains on real images (LogicFace), suggesting some robustness to less clean visual structures than other baselines. Still, the number of objects a cognitive limitation also in human reasoning, and noisier scenes overall harm neural models when abstracting visual information. More powerful object representation learning modules may enable better scalability to natural scenes, making open-world purely-visual reasoning an important direction for future research.

We now also discuss this under Sec. 7 "Limitations".

W1: Slot Attention, Abstractors, CFG-style dropout, cross-attention, and flow matching all already exist. The novelty is the combination, not a new technique.

Yes, several individual components are existing techniques. Our primary contribution is in demonstrating how learning relational abstractions based on object-centric representations can effectively guide spatial reasoning in diffusion models task-agnostically. Current approach by SRM, which primarily harnesses diffusion forcing [Chen et al., 2024] to improve reasoning beyond vanilla diffusion models, relies on grid-aligned training and sampling strategies to solve Sudoku. Our experiments on Tildoku demonstrate the limitations of this view (see Figure 6).

We present a generalized solution for the task-agnostic case, which is not restricted to structured grids, with several innovative aspects that were required for effective integration of the individual components.

(1) We show that relational abstractions provide benefits beyond existing slot-based diffusion methods (see ablations in Table 11), which supports our novel perspective. Our observations on the evidence of rule learning further strengthen these findings.

(2) We introduce a scalable slot decoder (Algorithm 1), which is necessary to scale this framework to complex reasoning problems with large number of objects (e.g., Akari).

(3) We introduce a new large-scale benchmark inspired by real spatial reasoning puzzles comprising four tasks to study reasoning problems beyond MNIST-Sudoku.

We now added a dedicated subsection on "Scientific Novelty" under Sec. 7, with the above discussions.

AuthorsOpenReview ↗

Response to Reviewer Ni4V (Part 2/2)

W2: The best setting (clue, solution, or full) changes by task. More slots isn't always better, and the 81-slot full model on Coldoku actually drops. SRM still wins on the hardest Coldoku level.

Given a single unified protocol (i.e., added as Table 13 in the revised paper), RDM matches or outperforms Base on all datasets, therefore the gains do not depend on selecting the best variant per task. The ranking among variants reflects the nature of each puzzle. In LogicFace, the rules depend on attributes of the clues, thus conditioning on the clues (clue, full) is essential, while solution does not help. In Tangram, the difficulty lies in partitioning the silhouette, which the solution embeddings expose directly, so solution outperforms clue, and full performs best. In grid-based puzzles, the performance is also influenced by the amount of slots: conciliating a large number of slots is harder for the diffusion model, which is why overall RDM benefits more from a lower number of slots. With configuration [S], we observed that RDM outperforms SRM also on Coldoku Hard (17.08 vs 15.14 in Table 13), which implies SRM's dependency on a larger architecture for performance.

We note that our work does not aim to position a single variant (clue, solution, or full) as the preferred recipe, but instead demonstrates for the first time how object-centric relational abstractions can guide generative spatial reasoning task-agnostically. We now added a statement regarding this in Sec. 7 as well.

W3: Coldoku Hard is about 12.6% and Tangram tops out under 47%. The "3×" is measured against a pretty weak baseline.

Our claim, "by up to 3$\times$ on the challenging multi-context reasoning problems", directly refers to the most challenging multi-context image generation tasks: Tangram and LogicFace, as presented in Sec. 4.2. The low baseline performance in fact reflects the difficulty of task-agnostic reasoning in these two problems, rather than a weak baseline choice. All diffusion models were trained under the same conditions with capacity-matched backbones. Nevertheless, we agree that our benchmark leaves substantial room for future work, considering the remaining gap between RDM and the ground truth.

W4: Evaluation relies on neural evaluators, Tangram comes from only 1,013 base silhouettes, slot decomposition on Tangram is admittedly poor, and some table entries look like typos (e.g., a SHAPE score of 4.31).

We use neural evaluators only for Coldoku and LogicFace. Both are highly accurate (>99% and >98% test accuracy in A.2 and A.4), and are applied identically to all models. Akari and Tangram use deterministic evaluators.

The 1,013 silhouettes are the complete set of annotated Kilogram solutions, as we described under Sec. A.3. Each is augmented with shuffled piece views, rotations and random colorings, and test silhouettes are disjoint from training (which is now explicitly stated in Sec. A.3).

We agree that slot decomposition in Tangram is imperfect, but this is rather common for traditional object-centric models trained only with a reconstruction loss. Nevertheless, RDM shows its largest gain on Tangram, suggesting that the relational module is currently robust to imperfect slots.

We thank the reviewer for spotting the typo, which we corrected: 4.31→43.14.31 \to 43.1.

W5: It's mostly Base versus SRM, with no search-based or neuro-symbolic methods to show where this really stands.

We thank the reviewer for this suggestion. Our benchmark targets reasoning in pixel space, where rules must be inferred from data. Symbolic or search-based solvers generally require pre-specified rules and operate on symbolic inputs, so they would trivially solve grid puzzles given a perfect perception module. That comparison would evaluate perception rather than unsupervised rule learning. Most neuro-symbolic approaches likewise operate on symbolic representations or rely on fixed grid structures, and do not extend to generative problems like Tangram and LogicFace.

We now discuss this in Section 2, and better position our work with respect to such methods, also with reference to recent works.

Dear Reviewer Ni4V,

Thank you again for taking the time to review our work.

We have provided detailed responses to your comments and questions, and revised our manuscript with additional experiments and clarifications. We would appreciate your feedback on our rebuttal, as the discussion period is approaching the end.

We would also be happy to address any remaining concerns about our submission.

Best regards, Authors

Reviewer_jmGLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper frames spatial reasoning as conditional image generation, in which a diffusion model must produce a solution image that satisfies implicit logical constraints given a clue image. The proposed RDM extracts object-centric slots from clues and solutions with a pretrained Slot Attention autoencoder and converts them into relational embeddings with Slot Abstractors. These embeddings are injected into a flow-matching U-Net through cross-attention, and the solution-derived embeddings are used only during training under CFG-style dropout. The paper also introduces four puzzle datasets, Akari, Coldoku, Tangram and LogicFace, and compares RDM with a capacity-matched U-Net and SRM, including out-of-distribution settings such as extreme wall densities and tilted Sudoku grids.

Critical strengths

The benchmark is a genuine contribution. Tangram, LogicFace and Tildoku push generative reasoning beyond fixed-grid Sudoku inpainting. They require piece-level geometric matching, logical composition over real face attributes, and robustness to geometric distortion. This matters because the community gets a harder and more diverse testbed whatever one concludes about the method. On the multi-context and out-of-distribution settings the reported gains are large. Tangram accuracy rises from 15.46 to 46.76, and accuracy at low Akari wall density rises from 39.48 to above 72. The slot-only ablation in Table 11 also suggests that the relational module adds something beyond plain slot conditioning, which is the main empirical support for the central claim.

Critical weaknesses

The empirical evidence is not reliable enough in its current form. Numbers for the same configuration disagree across tables. The Akari BLB values differ between Table 1 and Table 7. The LogicFace Base accuracy is 11.24 in Table 2 but 25.13 or 26.03 in Table 10. The discoverer accuracies in Table 5 contradict Section C.1. The reproduced SRM is also far below the numbers reported by Wewer et al. on MNIST-Sudoku (0.01 versus 51.6 on hard in Table 12). Model size, sampling steps and conditioning variant change from dataset to dataset, and no variance over seeds is reported. Because several RDM variants fall below Base (Akari in-distribution, MNIST-Sudoku medium, LogicFace solution), the headline gains cannot yet be separated from configuration selection. This is fixable in principle, but it requires rerunning the baselines under a single protocol.

The claim that the relational module learns the underlying rules is weakly supported, and the method is mostly a combination of existing components, namely Slot Attention, Slot Abstractors and cross-attention conditioning with CFG-style dropout. The strongest rule-learning evidence in Section C.2.1 conditions on solution slots at inference, which leaks the answer. On LogicFace, under the uniform configuration sampling described in Section A.4, a constant-output strategy already reaches about 21 percent accuracy, a level that Base, RDM solution and RDM[L] clue barely exceed. The positioning also omits closely related diffusion-based reasoning work, namely Learning Iterative Reasoning through Energy Diffusion (Du, Mao and Tenenbaum, ICML 2024) and Beyond Autoregression, Discrete Diffusion for Complex Reasoning and Planning (Ye et al., ICLR 2025).

Questions for authors

1.Explain the discrepancies between Table 1 and Table 7 for Akari, between Table 2 and Table 10 for the LogicFace Base, and between Table 5 and Section C.1 for the discoverer, and state which values are correct. 2.Explain why the reproduced SRM falls so far below the results of Wewer et al. on MNIST-Sudoku and on the other tasks, and state whether the official implementation and training budget were used. 3.Report the LogicFace accuracy of a constant-output baseline. Please also clarify whether the Tangram test silhouettes and the LogicFace test images are disjoint from those used in training. 4.Report the mean and standard deviation over several seeds for the main results, using one fixed model size, number of sampling steps and conditioning variant across all datasets. 5.Suggest for future work. Please analyze why solution-only guidance, which uses the same inputs as Base at inference, improves performance on some tasks and hurts it on others.

4 comments
AuthorsOpenReview ↗

Response to Reviewer jmGL (Part 1/3)

Q1: Explain the discrepancies between Table 1 and Table 7 for Akari, between Table 2 and Table 10 for the LogicFace Base, and between Table 5 and Section C.1 for the discoverer, and state which values are correct.

W1: The empirical evidence is not reliable enough in its current form. Numbers for the same configuration disagree across tables. The Akari BLB values differ between Table 1 and Table 7. The LogicFace Base accuracy is 11.24 in Table 2 but 25.13 or 26.03 in Table 10. The discoverer accuracies in Table 5 contradict Section C.1.

We thank the reviewer for the careful reading and apologize for these errors, which arose from manual transcription when formatting the tables. These typos are now all corrected, and they did not modify any of the claims of our paper.

  • Akari (Table 1 to match Table 7): 88.30 →\to 85.98 (low, RDM clue, 16 slots).
  • Akari (Table 7 to match Table 1): 82.20 →\to 82.21, 75.34 →\to 81.12, 98.76 →\to 92.76, 91.14 →\to 92.37, 93.33 →\to 92.51.
  • LogicFace Base (Table 2 and 10): Table 10 is the correct one (ACC: 25.13), the Base row in Table 2 was incorrectly transcribed from Table 10.
  • Discoverer (Table 5 and C.1): Both are correct but refer to different configurations. Table 5 reports configurations [S] and [M], while Sec. C.1 reports [L]. We now state this explicitly in Sec. C.1.

Q2: Explain why the reproduced SRM falls so far below the results of Wewer et al. on MNIST-Sudoku and on the other tasks, and state whether the official implementation and training budget were used.

W2: The reproduced SRM is also far below the numbers reported by Wewer et al. on MNIST-Sudoku (0.01 versus 51.6 on hard in Table 12).

SRM performs very low in our proposed benchmark due to its reasoning paradigm not being task-agnostic. This is one of the novel contributions of RDM. SRM's patch-based sequential sampling is tailored to grid-structured puzzles and yields high gains on MNIST-Sudoku and Coldoku, but not on more generalized spatial reasoning problems like Tangram or LogicFace. This finding is also consistent with the modest gains reported by Wewer et al. on Counting Polygons. We discuss this further for each task in Sec. 6.

We use the official SRM implementation of [Wewer et al., 2025], with only minor changes in channel dimensions of the U-Net backbone to match model capacity across tasks. Specifically, for Coldoku and Tildoku we use the [L] configuration in our main results (see Table 6), similar to what SRM requires for MNIST-Sudoku. Our configuration [S], which we used for all other datasets in our main results, was trained for 300K steps, while [L] could only be trained for 45K-80K steps (depending on the dataset as outlined in Table 6) due to our compute limitations. On the other hand, SRM reports training of [L] models for 250K steps on MNIST-Sudoku, which we cannot perform for all our experiments. For a unified comparison, we now retrained SRM [S] on Coldoku, Tildoku and MNIST-Sudoku as well instead, under the same training budget, which we report in Table R1.

Regarding MNIST-Sudoku, reproduced comparisons were performed via the original implementation checkpoints. We have indeed discovered a naming mismatch when loading the original SRM checkpoint for Table 12. With the correctly loaded checkpoint and our evaluation setup, SRM could now obtain 46.7 (hard) and 82.94 (medium), which is now revised in Table 12.

Q3: Report the LogicFace accuracy of a constant-output baseline. Please also clarify whether the Tangram test silhouettes and the LogicFace test images are disjoint from those used in training.

W5: On LogicFace, under the uniform configuration sampling described in Section A.4, a constant-output strategy already reaches about 21 percent accuracy, a level that Base, RDM solution and RDM[L] clue barely exceed.

The LogicFace accuracy on the test set given a constant answer varies between 0.78% (for [0,1,0,0]) at the lowest, and 21.0% (for [1,0,1,1]) at the highest. We now added this information in Sec. A.4.

Our models in fact largely outperform the best constant-output baseline (21%): RDM [S] (clue) reaches 74.74 and (full) reaches 60.76, and Base [S] only reaches 25.13. We reported RDM [L] in Table 10 of the appendix only for completeness, which indeed performs worse than RDM [S]. The reason was due to training compute limits, where RDM [L] was only trained for 40K steps on LogicFace, and did not reach the same level of convergence.

Indeed, Tangram test silhouettes are disjoint, which is now explicitly stated in Sec. A.3. In LogicFace, no test triplet appears in training either, which is also now explicitly stated in Sec. A.4.

AuthorsOpenReview ↗

Response to Reviewer jmGL (Part 2/3)

Q4: Report the mean and standard deviation over several seeds for the main results, using one fixed model size, number of sampling steps and conditioning variant across all datasets.

W3: Model size, sampling steps and conditioning variant change from dataset to dataset, and no variance over seeds is reported. Because several RDM variants fall below Base (Akari in-distribution, MNIST-Sudoku medium, LogicFace solution), the headline gains cannot yet be separated from configuration selection. This is fixable in principle, but it requires rerunning the baselines under a single protocol.

We thank the reviewer for this suggestion. We agree that different model sizes and sampling steps might make it difficult to follow, although we tried to keep it transparent with details in the appendix. We had to adjust some configurations to align with SRM, which required the [L] backbone for MNIST Sudoku, but used smaller configurations for other tasks.

As suggested, for a clearer comparison, we provide the new Table R1 that uniformly presents the performance of Base, SRM (sequential) and RDM (full) with the [S] configuration trained on 300K steps, τ=100\tau=100, and five sampling seeds across all datasets (standard deviation in parentheses, which were considerably small). The results maintain the central claims of the paper, as SRM shows clear gains in Sudoku and Coldoku (hard) against the baseline, but fails in all remaining datasets with more complex puzzles and spatially generalized reasoning rules. Notably, on Coldoku (hard), RDM now also outperforms SRM with a smaller architecture. We now included this comparison under Section C.2.1 as Table 13, with a direct reference in the main text.

Table R1: Unified comparisons of methods across all datasets with the [S] configuration trained for 300K steps, τ=100\tau=100 sampling steps, and five seeds (standard deviation in parentheses).

MethodAkari IDAkari lowAkari highTangramLogicFaceColdoku mediumColdoku hardTildoku mediumTildoku hardMNIST-Sudoku mediumMNIST-Sudoku hard
Base79.04 (0.61)41.23 (1.08)87.04 (0.61)21.54 (2.80)25.29 (1.29)55.93 (0.46)10.09 (0.76)2.95 (0.07)0.08 (0.00)45.31 (0.39)0.00 (0.00)
SRM (seq.)----5.58 (0.10)63.13 (0.73)15.14 (0.62)0.00 (0.00)0.00 (0.00)78.15 (0.88)33.24 (1.14)
RDM (full)80.06 (0.29)74.20 (0.42)87.71 (0.63)46.55 (0.87)58.96 (1.34)69.05 (0.21)17.08 (0.38)5.21 (0.09)0.78 (0.00)50.88 (0.43)0.00 (0.00)

Q5: Suggest for future work. Please analyze why solution-only guidance, which uses the same inputs as Base at inference, improves performance on some tasks and hurts it on others.

We thank the reviewer for this suggestion. Although it seems that RDM (solution) varies in performance with respect to the baselines, we observed a task-driven pattern that might explain its behavior. During training, R(z0)\mathcal{R}(\mathbf{z}^0) provides a disentangled signal of where objects lie in the solution, which shapes the reasoner's weights even though it is dropped at inference. This helps most when the task difficulty is spatial and the model must generalize, e.g., on Akari low (solution: 70.73 vs Base: 39.48), Tildoku (28.40 vs 12.48), and Tangram (38.57 vs 15.46). On ID grid puzzles however, it yields no gain or slightly hurts performance (e.g., Akari ID, MNIST Sudoku), possibly due to training the unguided branch with pdrop=0.5p_\text{drop}=0.5 costs some in-distribution fit. In LogicFace, the solution depends on combining semantic attributes rather than positions, thus the positional signal from only the solution SS carries little useful information.

We now added this discussion also under Sec. 7 of our revised paper.

AuthorsOpenReview ↗

Response to Reviewer jmGL (Part 3/3)

W4: The claim that the relational module learns the underlying rules is weakly supported, and the method is mostly a combination of existing components, namely Slot Attention, Slot Abstractors and cross-attention conditioning with CFG-style dropout. The strongest rule-learning evidence in Section C.2.1 conditions on solution slots at inference, which leaks the answer.

We apologize for the ambiguity of our rule learning investigation narrative with solution-conditioned sampling. In that experiment, the purpose of granting RDM (solution) access to the solution relational embeddings also at inference (by sampling from pθ(x0∣xc,R(z0))p_\theta(\mathbf{x}_0 | \mathbf{x}_c,\mathcal{R}(\mathbf{z}^0))) was to purposefully leak this information to assess its predictive utility, which indeed led to near-optimal solution-conditioned inference. We now revised our narrative to make it explicit, by referring to these experiments that indicate an oracle upper bound.

On the other hand, our Tangram experiment with training RDM on pθ(x0∣R(zc))p_\theta(\mathbf{x}_0|\mathcal{R}(\mathbf{z}^c)) without conditioning on xc\mathbf{x}_c also demonstrated that relational embeddings alone are sufficient to improve Base performance (15.46 →\to 37.55) without leaking the answer, hence encoding rule-relevant information.

To additionally evaluate the relational embeddings from the clue (with no leakage of the solution), we now performed another experiment on LogicFace. We froze RDM (clue) and trained two probes to compare the relational embeddings R(zc)\mathcal{R}(\mathbf{z}^c) against the inputs zc\mathbf{z}^c. We use the same probe architecture, which was a single attention layer with a [CLS] token, followed by a classification head. The network receives the sequence of embeddings, appends the [CLS] token, passes it through the attention layer, and feeds the contextualized [CLS] token to the classification head. Since the probe is identical for both inputs, the comparison isolates the signal learned by the Abstractor stack. We trained the probe to predict the binary vector s\boldsymbol{s} of the solution from the relational embeddings of the clue, which only encoded the information of AA and BB. With R(zc)\mathcal{R}(\mathbf{z}^c), it reached 92-95% accuracy on AND, OR and IMPLIES, and 70% on XOR. With zc\mathbf{z}^c, it reached 82-86% and 51%. This gap between R(zc)\mathcal{R}(\mathbf{z}^c) and zc\mathbf{z}^c demonstrates that relational embeddings computed from the clues alone, encode rule-relevant information that is much more readily accessible than object-centric representations.

We added these new experiments in Sec. C.2.1 of our paper as additional empirical evidence on rule learning, and also revised that subsection as well as its main text reference in Sec. 6 for better clarity. Regarding methodological novelty, we also added a discussion of the novel aspects of our work that progresses beyond the state-of-the-art methods in this area, under a new dedicated section on "Scientific Novelty" under Sec. 7.

W6: The positioning also omits closely related diffusion-based reasoning work, namely Learning Iterative Reasoning through Energy Diffusion (Du, Mao and Tenenbaum, ICML 2024) and Beyond Autoregression, Discrete Diffusion for Complex Reasoning and Planning (Ye et al., ICLR 2025).

We thank the reviewer for pointing out these works, which we now incorporated under Sec. 2. Both study diffusion as an iterative reasoning process on tasks such as Sudoku and planning, by operating on symbolic representations (e.g., digit labels on a board). In contrast, RDM and SRM reason directly in pixel space, where abstracting the objects and their relations from images is itself part of the challenge.

Dear Reviewer jmGL,

Thank you again for taking the time to review our work.

We have provided detailed responses to your comments and questions, and revised our manuscript with additional experiments and clarifications. We would appreciate your feedback on our rebuttal, as the discussion period is approaching the end.

We would also be happy to address any remaining concerns about our submission.

Best regards, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Comment

We thank all reviewers for their time and very valuable feedback.

We are pleased that the reviewers recognized the novelty of our approach for generative spatial reasoning (Ni4V, Kobh), its strong empirical performance supported by extensive evaluations and ablation studies (jmGL, Kobh) across diverse reasoning tasks and challenging generalization settings, and the scalability of our approach to larger numbers of object slots (Ni4V). We are also pleased that the reviewers highlighted our novel and challenging benchmarks as an additional valuable contribution to the community (jmGL, Kobh).

We provide our detailed responses to all comments raised by the reviewers, alongside our revised manuscript (changes marked in color).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.