acceptodds
Under review as a conference paper at ICLR 2027

TOSCA: Transport Operators with Set-valued Certified Assembly for Scene-Graph-to-3D Generation

Abstract

Generating a 3D scene from a scene graph requires selecting plausible arrangements that satisfy its relations. Graph-conditioned generators can produce plausible layouts that leave requested relations unsatisfied, and high average relation accuracy can coexist with many scenes that fail to satisfy the complete graph. To address this problem, we propose Transport Operators with Set-valued Certified Assembly, TOSCA, which integrates geometric checks and corrections into generation through explicit constraint sets called transport sets. These sets retain the range of configurations permitted by each relation, such as the region of relative object positions allowed by close-by. Projection residuals measure deviations from the allowed sets and guide graph attention and geometric corrections in a flow-matching generator, followed by final refinement. Because individually valid relations may conflict when combined, the same formulation also provides infeasibility certificates for difference constraints and objective bounds for fixed affine convex subproblems. Across SG-FRONT room types, TOSCA increases the fraction of requested relations satisfied by generated geometry and lowers rendered-image FID relative to the evaluated full-split baselines. Ablations support retaining transport sets and final refinement, while conflict and editing experiments assess infeasibility detection and preservation of surrounding objects.

Primary area:
generative models
Keywords:
3D Scene Generation, Scene Graphs, Constraint-Guided Generative Modeling, Set-Valued Geometric Constraints, Flow Matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_G4sTOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper addresses scene-graph-to-3D layout generation, motivated by the observation that existing graph-conditioned generators reach high edge-level relation accuracy while many scenes still violate at least one requested relation. TOSCA represents each relation as an explicit set of allowed feature values (a "transport set"), e.g., a half-space for directional relations, a box for close-by, and two reflection branches for symmetrical-to, with inverse relations sharing parameters. For each edge, the projection residual onto its set conditions a graph-attention network inside a flow-matching generator, and is also used for a geometric correction term. After sampling, 500 residual-gradient refinement steps further reduce violations. For fixed affine convex subproblems, the paper derives infeasibility certificates (negative cycles for difference constraints), primal-dual bounds on the minimum violation, and a decay bound on how relation edits affect anchored optima.

On SG-FRONT, TOSCA reports higher relation satisfaction (e.g., bedroom Micro .997 vs. about .975 for baselines, with large gains on close-by and symmetrical-to) and lower FID/KID than retrained EchoLayout, MMGDreamer, and InstructScene. Further results cover Inf3DLayout, the InstructScene benchmark, component ablations, conflict detection on synthetic contradictions, relation editing, footprint overlap, and a user study with 30 participants.

Critical strengths
  1. The evaluation is very careful. The authors retrain baselines on the same splits with a shared renderer, average over seeds and sampling runs, and audit things that are usually ignored: training overlap in the released EchoScene checkpoint, agreement between annotations and the benchmark predicates, the frame dependence of the symmetrical-to rule, and layout twins between train and test. They also report Joint (scene-level) satisfaction in addition to edge-level scores. The user study uses matched rendering and reports confidence intervals. Even where I disagree with the conclusions, I could trust that the numbers were measured carefully, and several of these diagnostics are useful to the community on their own.

  2. Making relation constraints explicit, instead of relying only on learned relation embeddings, is a reasonable and practical direction. It enables per-edge checking and correction, and it makes it possible to distinguish an infeasible request from incomplete optimization. The fact that the feasibility check found three contradictory scenes in the released SG-FRONT annotations is a nice concrete demonstration of this.

Critical weaknesses
  1. The relation-satisfaction gains seem to come mostly from post-hoc refinement rather than from the proposed learned components. In Table 18, removing final refinement drops Micro to 0.829 and close-by to 0.576, which is clearly below all baselines in Table 2 (around 0.95–0.975 Micro). With singleton sets, Micro is 0.661. So the residual-conditioned flow by itself does not satisfy relations better than existing methods. Most of the improvement comes from 500 steps of projected gradient descent toward sets that were fitted to approximate the benchmark predicates. This makes it essential to know what the baselines achieve on the relation metrics when given the same refinement. The paper only reports FID for "EchoScene + our final refinement" (Table 16), not relation satisfaction. Without this comparison, I cannot tell whether the contribution is the generator or a constraint-projection post-processing step that could be applied to any method.

    A related concern is symmetrical-to. Table 13 shows that this predicate is tied to the scene origin, to the point that translating ground-truth layouts by 0.5 m drops satisfaction from 1.0 to 0. TOSCA explicitly aligns layouts to that origin, and it is not clear whether the baselines received the same alignment. Part of the large Sym. gain, and therefore of the Macro gain, may reflect this frame alignment rather than better generation.

  2. The FID improvement is not clearly attributable to the main contribution, and the "certified" framing claims more than is delivered. According to Table 20, the transport-set-only configuration has FID 36.7, close to EchoLayout's 38.4. The drop to 25.1 comes from orientation projection (a per-category frame choice with a quadrant classifier) and category-specific size sets, which are not part of the transport-set or residual-conditioning idea. The baselines do not have these components. It is also unclear why final refinement improves TOSCA's FID (38.2 to 25.1) but makes EchoScene's FID worse.

    On the theory side, the guarantees cover only fixed affine convex subproblems. They rely on standard tools (Farkas' lemma, Bellman-Ford negative cycles, primal-dual bounds for squared-distance objectives, decay of sparse SPD inverses), and Theorem 1 is close to immediate. The certificates do not cover size-dependent bounds, branch choices, or collisions. In practice the pipeline's conflict detection misses some cases (bedroom TPR 0.914), even though exact negative-cycle checking is complete for these synthetic conflicts. Finally, the paper is very dense and heavily hedged, which made it hard to identify the actual contribution of each component.

Questions for authors
  1. Could you report the full relation-satisfaction table (Micro/Macro/Joint and per-relation) for the baselines after applying the same final refinement, and with the same reference-frame alignment? This is the main factor in my assessment.
  2. Table 20 suggests that most of the FID gain comes from orientation projection and size sets. Could these components be added to a baseline for a fairer comparison? Also, why does refinement improve TOSCA's FID but worsen EchoScene's?
  3. Why does the complete pipeline miss some injected conflicts (bedroom TPR 0.914) when exact negative-cycle checking is complete for these inputs?

I am very willing to raise my score if all the questions are answered adequately.

Reviewer_aQv2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

TOSCA generates 3D scene layouts from scene graphs using explicit sets of admissible geometric relations. Projection residuals guide graph attention and geometric updates within a flow-matching generator, followed by final refinement. The method also provides infeasibility certificates and objective bounds for specified constraint subproblems. Experiments show improved relation satisfaction and rendered-image fidelity on the evaluated benchmarks.

Critical strengths

Meaningful integration of generation and geometric constraints. The method incorporates explicit relation constraints through projection residuals and a differentiable violation energy within the trained generation architecture and subsequent refinement. This combines learned layout plausibility with interpretable geometric correction, while retaining multiple valid configurations for each relation. The reported results and ablations support the practical value of this design.

Critical weaknesses
  1. Incomplete evaluation of inference efficiency. The default pipeline uses 50 integration steps followed by 500 refinement steps, with additional backtracking evaluations during refinement. Although the paper analyzes negative-cycle detection complexity and ablates refinement length, it does not report end-to-end latency, peak memory, or scaling with graph size and density. Matched-hardware comparisons and quality-versus-runtime measurements are needed to assess whether the gains justify the additional computation.

  2. Limited evidence of scalability beyond closed-vocabulary indoor scenes. Despite the additional Inf3DLayout experiments, evaluation remains confined to bedrooms, living rooms, and dining rooms with predefined object categories and relation types. Category-dependent constraints and asset retrieval may limit transfer to unfamiliar objects, richer open-vocabulary requests, or outdoor scenes. Although this limitation is shared by related methods, the paper does not establish how its constraint representation and generation pipeline generalize beyond these restricted settings.

  3. Additional geometric and physical validity checks would be useful. Satisfying the requested relations does not necessarily ensure a geometrically or physically reasonable arrangement. Although the paper includes footprint-overlap analysis and a user study, the pipeline lacks explicit checks of full 3D collisions, physical support, or accessibility. Such checks would complement the relation metrics and strengthen the evidence for practical scene quality.

Questions for authors
  1. Could you report inference latency and peak memory under shared hardware settings, separating learned generation, final refinement, and constraint checking? How do these costs scale with object and relation counts, and what quality-runtime trade-off is achievable by reducing refinement?

  2. How does TOSCA perform on held-out object categories or room types without retraining? What changes to the transport-set catalog and asset retrieval would be required for open-vocabulary or outdoor scene generation?

Reviewer_cnH5OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents TOSCA, a method for generating 3D scene layouts from scene graphs. Instead of assigning one fixed pose to each spatial relation, TOSCA represents the relation as a set of valid arrangements. During generation, it checks whether the current layout satisfies these arrangements. If not, it uses the errors to adjust the objects. A final refinement step further reduces any remaining violations. The paper also studies cases where the input relations conflict with each other. For some impossible layouts, TOSCA can provide a certificate showing that no valid solution exists. Experiments on several indoor-scene datasets show that TOSCA satisfies spatial relations better and produces higher-quality rendered images than the tested baselines. The paper also includes ablation studies, scene-editing experiments, conflict-detection tests, and a user study.

Critical strengths
  1. The paper focuses on an important problem: a generated scene can look realistic but still fail to satisfy the relations in the input graph. The distinction between edge-level accuracy and whole-graph satisfaction is especially helpful.
  2. The idea of transport sets is simple and easy to understand. A relation such as close-by can have many valid layouts instead of one fixed solution. The projection residuals also give a clear signal for correcting violations.
  3. The evaluation is thorough. It covers several room types and datasets. The paper also includes ablation studies, conflict detection, relation editing, and a user study. Overall, TOSCA performs consistently better than the evaluated baselines.
Critical weaknesses
  1. In my understanding, TOSCA relies on manually specified transport sets for a predefined relation vocabulary. Extending the method to new relations requires additional redesign like new features or constraint sets, limiting its scalability to open-vocabulary scene generation.
  2. The quantitative evaluation of visual quality relies primarily on FID and KID. Complementary semantic metrics (as adopted in DirectLayout, FlowScene, SAGE), such as VLM-based evaluation or CLIPScore computed between the rendered images and the corresponding textual instructions in the InstructScene benchmark, would help show whether the generated scenes preserve the required objects and spatial relations while remaining semantically coherent.

DirectLayout: Direct Numerical Layout Generation for 3D Indoor Scene Synthesis via Spatial Reasoning

FlowScene: Style-Consistent Indoor Scene Generation with Multimodal Graph Rectified Flow

SAGE: Scalable Agentic 3D Scene Generation for Embodied AI

Questions for authors
  1. Please See Critical Weaknesses.
  2. The paper does not report end-to-end inference time. Since TOSCA adds extra refinement steps after sampling, it would be helpful to compare its runtime with the baselines under the same hardware and settings.
  3. I am curious about how TOSCA scales to larger scenes with more objects and relations. Since projection residuals and refinement are computed over all graph edges for hundreds of steps, it would be helpful to report performance and inference time across scenes of different sizes.
  4. Since TOSCA mainly focuses on scene layout prediction, it would also be useful to compare it with recent agentic indoor scene generation methods, such as Holodeck (CVPR24) , LayoutVLM (CVPR25), SceneWeaver (NeurIPS25), and SceneSmith (ICML26).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.