acceptodds
Withdrawn from ICLR 2027

Align3D: Image-Structure Co-Aligned 3D Scene Generation with Observability-Aware Fidelity-Preserving Guidance

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

We present Align3D, an image-conditioned diffusion model that generates 3D indoor layouts that are both faithful to a single input photograph and physically plausible beyond the observed view. From the photograph we infer a confidencecalibrated scene graph and train a set-diffusion layout denoiser with imagestructure co-alignment where confidence-weighted modality fusion balances image and scene-graph alignment using predicted confidences, while observabilityaware fidelity-preserving guidance projects conflicting scene-quality gradients and reweights them with an object-level confidence mask so under-observed degrees of freedom absorb physical priors without undoing the image-graph match. On 3D-FRONT, Align3D improves image-alignment and scene-quality metrics over recent baselines; once trained, it maps previously unseen photographs to layouts in a single feed-forward pass.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D scene generation, image-to-scene, layout diffusion, scene graphs, confidence-aware learning, multi-task optimization

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mcAQOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Align3D generates indoor 3D layouts from a single image by combining image features with an inferred scene graph. The method introduces confidence-weighted image-graph alignment and observability-aware fidelity-preserving guidance (OFG), which projects and reweights collision-loss gradients. A diffusion model predicts object layouts and retrieved assets from the final scene. Experiments on 3D-FRONT report improvements in several alignment and reconstruction metrics, with component ablations and qualitative results on real photographs.

Critical strengths
  1. The problem formulation is useful. Reconciling image fidelity with plausible furniture placement is a meaningful challenge. Table 2 shows complementary benefits from image and relational supervision.

  2. The reported inference efficiency is attractive. The approximately 10-second runtime includes graph prediction, diffusion sampling, and scene composition, making the system potentially useful for applications that accept layouts assembled from retrieved assets.

Critical weaknesses
  1. The central OFG update is mathematically inconsistent, and the training procedure leaves essential parts of the algorithm undefined. Masking invalidates the fidelity-preserving argument. Eq. (14) projects the quality gradient to remove its component that conflicts with the alignment gradient. Eq. (17) then applies a nonuniform diagonal mask CC, which generally destroys that orthogonality:
\quad\not\Rightarrow\quad g_{\mathrm{co\text{-}align}}^\top C\tilde g_{\mathrm{quality}}\geq0.$$ The masked gradient can therefore conflict with alignment again, and the combined update can increase the alignment loss even to first order. The central claim of preserving observational fidelity does not follow from the actual update. The mask itself can become negative. Eq. (7) normalizes matching probabilities across generated objects, while Eq. (16) sums over observed nodes. Multiple observations can match the same generated object, producing $o_k^{\mathrm{gen}}>1$ and hence $C_{kk}<0$. This reverses the corresponding quality-gradient block. The same many-to-one matching also allows distinct observed nodes to contribute self-relation terms in Eq. (8). The gradient space is ambiguous. The mask is defined over generated-object layout blocks, whereas training updates shared denoiser parameters with Adam. If the gradients are taken with respect to network parameters, the object-wise mask has no specified correspondence to those parameters. If they are taken with respect to predicted layouts, orthogonality in layout space does not automatically survive backpropagation through the shared network. The paper does not explain this distinction or the actual implementation of the update. The diffusion training objective is incomplete. Eq. (1) and Table 2 present alignment and collision penalties as the training objectives but omit the diffusion denoising loss and its interaction with OFG. This makes the "quality loss only" ablation particularly difficult to interpret: collision penalties admit trivial nonoverlapping or shrunken layouts and do not themselves define learning a distribution of furnished rooms. If a standard denoising objective is retained throughout, it must be included explicitly. The paper also leaves unclear whether geometric losses act on a clean-layout estimate or an intermediate noisy state. Finally, the graph encoder and the token correspondence needed for the additive fusion in Eq. (6) are unspecified. These omissions concern the central algorithm, rather than peripheral implementation details. 2. The geometric and visual surrogates are too crude for the claimed observation-aware physical completion, and the experiments overstate what they demonstrate. The inferred graph contains systematic geometric approximations that confidence weighting does not repair. Eq. (2) treats the depth at a detection-box center as the 3D object center, although this depth describes a visible surface and can even fall on the background. Eq. (3) then assigns confidence using detector scores and a fixed-distance heuristic, while object thickness is set to half of its projected height regardless of category or orientation. The metric scale is also inadequately specified: "Depth Anything (base)" does not identify whether a relative or metric checkpoint is used, yet the method applies thresholds in meters. Both relative and metric Base variants exist. The alignment between the single-view camera reconstruction and the layout's world-coordinate convention is similarly unexplained. Confidence is conflated with geometric observability. A confidently detected object can still have ambiguous depth, size, and hidden extent. One scalar per object cannot distinguish changes that preserve the object's image projection from those that disrupt it. Moreover, individual relation confidences are averaged into a graph-wide scalar, while Eq. (8) gives every observed relation equal weight. Consequently, the loss does not selectively suppress unreliable relations as the motivation suggests. The objectives do not model the properties they are claimed to preserve or improve. The image proxy in Eqs. (9)-(11) consists of additive, class-colored Gaussian blobs. It omits actual shapes, yaw, materials, and depth-dependent occlusion. Protecting its global CLIP/DINO similarity provides a weak connection to preserving the rendered furniture's silhouettes and appearance. Meanwhile, the quality loss is only the pairwise box overlap: an isolated floating object incurs no penalty. This objective provides no direct mechanism for correcting unsupported furniture or completing missing structure. The evaluation does not directly test the central physical claims. The reported box IoU measures agreement with ground-truth boxes, not inter-object collision. Interpreting the reversed-projection variant's higher IoU as better "generic plausibility" therefore confuses reconstruction accuracy with physical validity. The tables report neither collision statistics nor support errors. CLIP and DINO are also optimized during training, so improvements in those scores provide limited additional evidence for the broader geometric claims. Most of the CLIP improvement in Table 3 comes from ordinary projection, from 0.901 to 0.921. The observability mask adds only 0.004. The comparison protocol further weakens the conclusions. Known-rendering cameras with $c_{\mathrm{cam}}=1$ remove the camera uncertainty that motivates part of CWMF, and the optional post-hoc yaw correction is not identified in individual results. Align3D retrieves 3D-FUTURE assets, but the paper does not clearly explain whether each baseline retains its native shapes or uses the same retrieval pipeline. Surface metrics and runtime, therefore, cannot be attributed specifically to layout prediction. The existing box-IoU comparison is useful, but its improvement over SceneGen is modest: 0.558 versus 0.541.
Questions for authors
  1. Please provide the complete training objective and concise update pseudocode, including the denoising loss, the diffusion state receiving geometric supervision, the gradient space, and the order of projection and masking. Does the implementation normalize or clip Eq. (16), and what exactly remains active in the "quality loss only" ablation?

  2. Please identify the depth checkpoint, metric-scale conversion, and camera-to-layout coordinate alignment. For the main comparisons, specify each method's asset source, camera information, and whether it uses the optional yaw correction.

  3. Using existing test outputs, please report collision statistics for the Sum, Projection, and OFG variants, together with the frequency of okgen>1o_k^{\mathrm{gen}}>1. These checks would directly address the proposed mechanism without requiring new training.

Reviewer_wakqOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Align3D addresses single-image indoor scene layout generation. It constructs a scene graph using object detection, depth estimation, and camera estimation, and combines image and graph features to condition a layout diffusion model. Image alignment is supervised through a differentiable Gaussian proxy renderer, while graph consistency is enforced through soft object matching and relational losses. The proposed observability-aware fidelity-preserving guidance (OFG) projects conflicting collision gradients and weights them by object observability. Experiments on 3D-FRONT report competitive geometry and improved image-alignment scores, supported by component ablations, a human preference study, and qualitative examples on real images.

Presentation: 2 — Generally understandable, but requires substantial effort in places.

The overall pipeline is understandable, but the training objective and gradient implementation need a more precise description.

Informativeness: 2 — I learned something interesting or useful.

Soundness: 1 — I have significant concerns about the paper’s soundness.

Critical Strengths

  1. The paper addresses a relevant tension in image-conditioned scene generation: improving an inferred layout’s physical plausibility can move visible objects away from the input observation. Using observational reliability to determine where layout adjustments are permitted is a useful direction, particularly when the scene graph itself is estimated from imperfect single-view evidence.

  2. The experiments include comparisons with several recent methods and separate ablations for image alignment, graph alignment, confidence weighting, and gradient projection. The reported results suggest that image and graph supervision provide complementary benefits. Align3D improves CLIP and DINO similarity over SceneGen while remaining competitive on geometric metrics, although SceneGen retains better Chamfer distance and FID.

Critical Weaknesses

  1. The fidelity-preservation argument does not extend to the final OFG update. Equation 14 removes the conflicting component of the quality gradient, making the projected gradient approximately orthogonal to the alignment gradient. Equation 17 then multiplies that projected gradient by the nonuniform object mask CC. This operation generally destroys orthogonality: gco-align⊤g~quality=0g_{\mathrm{co\text{-}align}}^\top \tilde g_{\mathrm{quality}}=0 does not imply gco-align⊤Cg~quality≥0g_{\mathrm{co\text{-}align}}^\top C\tilde g_{\mathrm{quality}}\geq 0. The masked correction can therefore conflict with alignment again. The first-order argument following Eq. 14 is insufficient to justify the final update, which is central to the claimed fidelity-preserving behavior.

  2. The transported observability score is not bounded as intended. The matching matrix in Eq. 7 is normalized over generated objects for each observed node. However, Eq. 16 sums over observed nodes without normalizing the incoming weights. Several observed nodes can assign substantial probability to the same generated object, allowing its observability score to exceed 1. For example, two observed nodes with observability 0.78 and matching weight 0.9 to the same generated object give a score of 1.404. The corresponding entry of CC becomes negative, reversing that object’s projected quality update rather than suppressing it. No normalization or clipping is specified in the paper or Appendix A.

  3. The diffusion training objective and gradient domain are underspecified. Equation 1 presents the objective as alignment plus a modified quality term, and Table 2 similarly lists these terms as the training objectives. A standard diffusion denoising loss is not explicitly included, although Appendix A specifies a diffusion schedule and DDIM sampling. It is also unclear whether the gradients in Eqs. 14–17 are computed with respect to predicted clean layouts, noisy layouts, or network parameters. An object-block mask has a natural interpretation in layout space, but its application to network optimization requires an explicit description. These omissions make it difficult to determine the actual training procedure and interpret the ablations.

  4. The confidence-calibration claim lacks supporting evidence. Node confidence combines detection confidence with a hand-designed distance decay, while edge confidence additionally decreases with inter-object distance. These are reasonable heuristics, but the paper does not show that they are calibrated against localization error or relation correctness. The construction of the global camera confidence is also unspecified. Furthermore, Eq. 8 weights observed relations uniformly; edge confidence enters the alignment objective through the global scalar csgc^{\mathrm{sg}}, rather than directly reducing the contribution of an unreliable relation. This provides weaker local uncertainty handling than the description suggests.

  5. The evaluation does not directly establish physical plausibility beyond the observed view. Chamfer distance, F-Score, and matched ground-truth box IoU measure reconstruction agreement rather than collision avoidance or support validity. The human study uses matched-view renderings, which also limit the assessment of hidden interpenetrations and clearances. Direct collision and support measurements, together with an evaluation of occluded regions, would better support the central claim. The benefit of the observability mask over projection alone is relatively small in Table 3, and no uncertainty estimates are provided to establish its consistency.

Questions For Authors

  1. Clarification of the implemented OFG update would help resolve the two mathematical concerns above, particularly any normalization or clipping after Eq. 16 and any conflict check after applying CC. Measurements of alignment-gradient conflict before and after masking would help establish the behavior of the final update.

  2. The complete training objective, including the diffusion denoising term and its weighting relative to the auxiliary losses, is needed to interpret the method. Pseudocode specifying the gradient variables, timestep handling, and how the modified layout gradients reach the network parameters would make the procedure reproducible.

  3. Details of the depth checkpoint, conversion to metric depth, world-coordinate alignment, and construction of camera confidence would clarify the single-image geometry pipeline. An analysis relating the proposed confidence scores to actual node and relation errors would help substantiate the calibration claim.

  4. A precise account of the evaluation settings would help assess the comparisons. Appendix A describes known-camera substitution and global-yaw correction as optional; identifying which results use these options, and whether equivalent processing is applied to baselines, would remove this ambiguity. The treatment of retrieved assets versus directly generated meshes also needs clarification.

  5. Collision rates, penetration severity, and support violations would provide a more direct assessment of the intended physical improvements. Reporting these alongside image alignment for the sum, projection-only, and full OFG variants would help establish the contribution of observability weighting.

Flag For Ethics Review: No.

Details Of Ethics Concerns

No specific concern requiring a separate ethics review was identified from the submission.

Rating: 2 — Weak rejection.

The motivation is useful and the reported results are promising. However, the final OFG update is not covered by the stated fidelity-preservation argument, and the observability transport can produce negative correction weights as written. Together with the incomplete training description, these issues prevent a reliable assessment of the main contribution. Clarification that the implementation addresses these problems, supported by corresponding experimental evidence, could change my assessment.

Critical strengths
  1. The paper addresses a relevant tension in image-conditioned scene generation: improving an inferred layout’s physical plausibility can move visible objects away from the input observation. Using observational reliability to determine where layout adjustments are permitted is a useful direction, particularly when the scene graph itself is estimated from imperfect single-view evidence.
  2. The experiments include comparisons with several recent methods and separate ablations for image alignment, graph alignment, confidence weighting, and gradient projection. The reported results suggest that image and graph supervision provide complementary benefits. Align3D improves CLIP and DINO similarity over SceneGen while remaining competitive on geometric metrics, although SceneGen retains better Chamfer distance and FID.
Critical weaknesses
  1. Equation 14 removes the conflicting component of the quality gradient, making the projected gradient approximately orthogonal to the alignment gradient. Equation 17 then multiplies that projected gradient by the nonuniform object mask CC. This operation generally destroys orthogonality: gco-align⊤g~quality=0g_{\mathrm{co\text{-}align}}^\top \tilde g_{\mathrm{quality}}=0 does not imply gco-align⊤Cg~quality≥0g_{\mathrm{co\text{-}align}}^\top C\tilde g_{\mathrm{quality}}\geq 0. The masked correction can therefore conflict with alignment again. The first-order argument following Eq. 14 is insufficient to justify the final update, which is central to the claimed fidelity-preserving behavior.
  2. The matching matrix in Eq. 7 is normalized over generated objects for each observed node. However, Eq. 16 sums over observed nodes without normalizing the incoming weights. Several observed nodes can assign substantial probability to the same generated object, allowing its observability score to exceed 1. For example, two observed nodes with observability 0.78 and matching weight 0.9 to the same generated object give a score of 1.404. The corresponding entry of CC becomes negative, reversing that object’s projected quality update rather than suppressing it. No normalization or clipping is specified in the paper or Appendix A.
  3. Equation 1 presents the objective as alignment plus a modified quality term, and Table 2 similarly lists these terms as the training objectives. A standard diffusion denoising loss is not explicitly included, although Appendix A specifies a diffusion schedule and DDIM sampling. It is also unclear whether the gradients in Eqs. 14–17 are computed with respect to predicted clean layouts, noisy layouts, or network parameters. An object-block mask has a natural interpretation in layout space, but its application to network optimization requires an explicit description. These omissions make it difficult to determine the actual training procedure and interpret the ablations.
  4. Node confidence combines detection confidence with a hand-designed distance decay, while edge confidence additionally decreases with inter-object distance. These are reasonable heuristics, but the paper does not show that they are calibrated against localization error or relation correctness. The construction of the global camera confidence is also unspecified. Furthermore, Eq. 8 weights observed relations uniformly; edge confidence enters the alignment objective through the global scalar csgc^{\mathrm{sg}}, rather than directly reducing the contribution of an unreliable relation. This provides weaker local uncertainty handling than the description suggests.
  5. Chamfer distance, F-Score, and matched ground-truth box IoU measure reconstruction agreement rather than collision avoidance or support validity. The human study uses matched-view renderings, which also limit the assessment of hidden interpenetrations and clearances. Direct collision and support measurements, together with an evaluation of occluded regions, would better support the central claim. The benefit of the observability mask over projection alone is relatively small in Table 3, and no uncertainty estimates are provided to establish its consistency.
Questions for authors
  1. Clarification of the implemented OFG update would help resolve the two mathematical concerns above, particularly any normalization or clipping after Eq. 16 and any conflict check after applying (C). Measurements of alignment-gradient conflict before and after masking would help establish the behavior of the final update.

  2. The complete training objective, including the diffusion denoising term and its weighting relative to the auxiliary losses, is needed to interpret the method. Pseudocode specifying the gradient variables, timestep handling, and how the modified layout gradients reach the network parameters would make the procedure reproducible.

  3. Details of the depth checkpoint, conversion to metric depth, world-coordinate alignment, and construction of camera confidence would clarify the single-image geometry pipeline. An analysis relating the proposed confidence scores to actual node and relation errors would help substantiate the calibration claim.

Reviewer_GGpgOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Align3D, a system for generating indoor 3D layouts from a single image, followed by furniture retrieval to construct the scene. It infers a scene graph using pretrained detection, depth, and camera-estimation models, and assigns confidence scores to the inferred observations. An image-conditioned layout diffusion model combines image and graph information through confidence-weighted conditioning and alignment losses. Its central component, observability-aware fidelity-preserving guidance (OFG), projects scene quality gradients against the alignment gradient and then applies an object specific confidence mask, aiming to reduce collisions while preserving well observed content. Validation includes comparisons with five baselines on 3D-FRONT, component ablations, real image qualitative examples, and a human preference study.

Critical strengths
  1. The paper addresses a meaningful problem, and the ablations support the complementary value of image and graph supervision. Single-image layout generation must balance fidelity to visible evidence with plausible placement of under constrained objects. Table 2 demonstrates that image alignment and graph alignment contribute differently: image only supervision provides stronger visual similarity, while graph only supervision provides stronger geometric agreement. Combining them improves both. This establishes a useful design result and supports the overall approach.

  2. The complete system offers a competitive combination of layout fidelity and runtime. In Table 1, Align3D improves over SceneGen in F Score (88.7 versus 87.8), box IoU (0.558 versus 0.541), CLIP similarity (0.925 versus 0.909), and DINO similarity (0.843 versus 0.824), although SceneGen remains better on Chamfer distance and FID. The reported 10 second inference time is attractive for a layout&retrieval pipeline. Table 3 also meaningfully separates ordinary summation, projection, reversed projection, and the full OFG method.

Critical weaknesses
  1. The central fidelity-preserving property does not follow for the OFG update as written. Equation (14) removes the conflicting component of the quality gradient, giving approximately

gco-align⊤g~quality=0g_{\text{co-align}}^\top \widetilde{g}_{\text{quality}} = 0

when projection is active. However, Equation (17) subsequently applies a nonuniform object mask CC. In general,

gco-align⊤Cg~qualityg_{\text{co-align}}^\top C \widetilde{g}_{\text{quality}}

can be negative, even when CC is diagonal and nonnegative. Therefore, the first order protection argument for the unmasked projection does not extend to the final update.

There is also a problem with the mask’s bounds. Equation (7) normalizes MikM_{ik} over generated objects kk, whereas Equation (16) sums over observed nodes ii. Consequently, okgeno_k^{\text{gen}} can exceed one when several observations match the same generated object, making entries of CC negative and reversing the corresponding quality update. Neither column normalization nor clipping is specified.

  1. The evaluation supports overall alignment improvements more strongly than the distinctive confidence claims. The reported IoU measures agreement between generated and GT boxes (Section 4.2); it does not measure the interobject overlap penalized by Equation (13). Table 3 provides useful evidence that OFG improves aggregate metrics over projection alone, but does not directly establish this selective behavior. The confidence scores are also constructed from fixed heuristics in Equations (3) & (4), without evidence that they are calibrated estimates of reliability. Moreover, the quantitative protocol permits known rendering cameras with camera confidence set to one, limiting what these results establish about handling uncertain camera estimates. Appendix A’s optional global yaw alignment is not tied explicitly to the reported tables or baseline treatment. These ambiguities matter because confidence handling and observation preservation constitute the main contribution beyond the existing components.
Questions for authors
  1. Clarification of the implemented OFG algorithm. What exact update produced Tables 1–3? Please clarify how Equation (16) is kept within ([0,1]), and how fidelity preservation is maintained after applying (C). Please also state the diffusion denoising objective and how it combines with Equation (1).

  2. Clarification of the evaluation protocol. For Tables 1–3 and the human preference study, which results use known vs estimated cameras, and which use the optional global yaw alignment? Were equivalent camera information and global alignment procedures available to the compared methods?

  3. Focused evidence for the central mechanism. Using the existing outputs for the “Projection” and “OFG” variants, can the authors show whether the confidence mask reduces interobject overlap while preserving alignment of well observed objects, with improvements concentrated in less observed objects? This directly tests the claimed contribution and should not require a new dataset or a broad set of retraining experiments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.