- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies how to complete hierarchical 3D scene graphs from observed 3D wall planes. In the setting considered here, a robot has already mapped a static environment and extracted wall planes. To build a full scene graph, the model must infer high-level concepts such as walls, rooms, floors, buildings, and cities.
To solve this, the authors propose a generative pipeline combining two existing methods (FLAGG and MiDi). FLAGG provides the autoregressive framework for adding nodes in small blocks and predicting when to stop. Within each step, a MiDi-based module uses a Graph Transformer with mixed categorical and Gaussian diffusion. This module jointly denoises node labels, edge connections, and 3D centroid coordinates. The authors also use noise augmentation on conditioning graphs to deal with exposure bias, as well as Min-SNR loss weighting for 3D coordinates.
For evaluation, the paper introduces a benchmark with synthetic layouts, floor plans from the MSD dataset, and few real-world robot scans. The authors use the Fused Gromov-Wasserstein (FGW) distance to measure differences in both graph connectivity and node attributes. The authors report lower FGW scores and better stability on deeper hierarchies, although the one-shot baseline performs better on M-F.
Critical strengths
I found two main strengths in this paper:
First, I agree that inferring high-level abstractions like rooms and floors directly from low-level geometric primitives is an important task in robotics and spatial understanding. Learning these concepts within a unified generative framework, rather than using a separate heuristic pipeline for each concept, is an appealing and timely direction.
Second, the MiDi-based joint formulation for graph topology and 3D geometry is sensible for this task. Denoising discrete edge connections, semantic labels, and continuous 3D centroid coordinates within the same reverse diffusion process is a natural way to handle these coupled outputs. I also find the choice of equivariance in distribution under horizontal translations and rotations about the vertical axis reasonable for indoor multi-floor environments.
Critical weaknesses
I found two critical weaknesses that make it hard to support acceptance in its current form:
First, the comparison between the autoregressive method and the one-shot baseline (MiDi+Or) seems like it's not a fully controlled experiment. The paper claims this comparison isolates the benefit of autoregressive generation, but Table 5 shows that several key training settings differ at the same time. For example, the loss weights are different ( vs. ), Min-SNR loss weighting is only listed for the proposed method, and the conditioning modules are implemented differently. Because of these confounding factors, it is difficult to tell whether the performance gains come from the autoregressive scheduling itself or from these additional design choices.
Second, the mathematical definition of the FGW metric in Section 5.2 seems incomplete. The text states that dummy nodes are penalized in both the and terms by setting a large shortest-path distance. However, Eq. 5 defines using only 3D centroids and one-hot labels, where shortest-path distance does not appear. Without specifying the coordinates, labels, or exact feature penalty assigned to dummy nodes, this metric cannot be mathematically verified (or used). It is also unclear whether the correspondences for the observed input planes are explicitly fixed during optimal transport.
Questions for authors
Could the authors clarify the following points during the rebuttal?
-
Controlled comparison on the autoregressive schedule:
Could you compare a single-block baseline against the proposed autoregressive model with matched settings (the same Fill architecture, conditioning encoder, loss weights, and Min-SNR weighting) on a representative multi-level dataset? This would directly show how much the generation schedule itself contributes to the performance gains.
-
Clarification of the FGW implementation:
How are dummy nodes handled in the Wasserstein term ? Since Eq. 5 only depends on 3D centroids and one-hot labels, what coordinates, labels, or feature penalties are assigned to dummy nodes? Also, how are infinite shortest-path distances between disconnected components handled in ?
-
Fixing observed plane correspondences in FGW evaluation:
When solving for the transport plan , are the correspondences for the observed input planes fixed between the generated graph and the ground truth? I understand that already includes coordinates, but does additionally fixing the known plane correspondences change the results or method ranking? Alternatively, could you report a grouping metric that explicitly preserves input-plane identities?
-
Ablations on the effect of node ordering:
Section 4.2 and Appendix B.1 describe hierarchical and hierarchical proximity orderings. Do you have results comparing the chosen ordering with hierarchical ordering without the proximity component, random ordering, or purely spatial ordering? This would help show how much the ordering contributes.
I am open to revising my assessment based on the authors' response, especially clarification of the evaluation and evidence from a controlled comparison. The above-mentioned points are currently limiting my confidence in the conclusions.
We thank the reviewer for the very thorough review and constructive criticisms, asking for clarifications and useful comparisons. We address them point by point.
CW1, Q1 - Controlled comparison: We agree that our comparison of MiDi+Or and our method does not provide a precise isolation of autoregressive vs. oneshot scheduling. The choice of hyperparameters of MiDi+Or comes from the set used in the original paper. In fact, oneshot MiDi was not built to receive input nodes, so we adapted its original implementation to have a comparison, and added the oracle to account for MiDi's requirement of taking as input the number of nodes. Thus, the two architectures differ. To correctly isolate the effect of autoregressive generation as the reviewer suggests, we implemented a one-shot variant of FLAGG + MiDi, where non-planes nodes are generated as one single block (thus simulating the original MiDi behavior). The autoregressive and one-shot variants share Fill architecture, conditioning encoder, loss weights and Min-SNR weighting, together with other hyperparameters. We are currently running experiments on the M-C dataset, and due to their computational cost we will report results in the coming days.
CW2, Q2 - Dummy nodes in FGW: We agree that the original explanation was incomplete, so we integrate our explanation in the following. We pad the smaller graph with isolated dummy nodes until both graphs have N=max(n,n′) nodes. For W, feature distances between real nodes are normalized to [0,1]; every real-dummy distance is then explicitly set to 2, after normalization, overriding the feature-distance formula in Eq. 5. Dummy centroids and label vectors are stored as zeros, but due to the overriding, they are effectively ignored. For GW, shortest-path distances between distinct disconnected nodes, including dummy-real pairs, are set to N, before the normalization in Eq. 6. Thus, W explicitly penalizes graph-size mismatch, while GW compares the resulting finite structural distances. We have clarified these conventions below Eqs. 5-6 and in Appendix J.
CW2, Q3 - Fixing planes identities in FGW: Planes are in fact not fixed when solving the transport plan for FGW, but we agree that this might be a sensible choice. Thus, we compute "FGW fixed" with its GW and W distances, fixing the identity of planes, and solving with this constraint. The following table compares the original values with new values on the challenging M-C dataset, comprising concepts up to city level.
It can be seen that the differences between original and fixed are negligible for RAND and MiDi+Or, and small for Ours, where we notice that GW differs by ~1.2. Still, the aggregated FGW remains very close to our original definition, and the ranking remains unchanged on M-C. Thus, we can conclude that, although desirable, this new definition is empirically consistent with the original. We included this new comparison and the table above in Appendix J of the revised manuscript.
Q4 - Ablations on node ordering: We thank the reviewer for this suggestion. We are running the following ablations on the M-C dataset: random ordering and hierarchical ordering without proximity. Because these training runs are computationally expensive, we will report final results in the following days.
Statements relevant to Q1 and Q4 will be adjusted in the revised manuscript after all experiments have concluded and we have clear evidence.