- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
The author formalizes Scene Retargeting: given a reference scene, the author's framework aim to transfer its spatial layout to another empty room (walls, doors and windows only) using a new set of objects. In other words, the output is a layout of the target objects in the empty room that follows the spatial structure of the reference. The author's main motivation is to create a framework that exhibits flexibility, diversity, and transferability, such that it could benefit numerous downstream applications.
The framework consists of three stages.
-
Stage 1 (Cluster Placement) partitions the reference scene into functional clusters via agglomerative clustering using positions and OpenShape features. Each cluster is moved into the empty room by using a network to predict their poses, while layouts inside the clusters are unchanged.
-
Stage 2 (Object Placement) replaces the moved reference objects in the new room with target objects. Specifically, the moved reference objects and the floor of the empty room are combined into one point cloud and encoded via Concerto. Then, a second network predicts the pose of each target object given the scene-level Concerto features and a bunch of object-level features of the target object (OpenShape, category, size, etc). The aim is for each target object to take the place of a matching reference object in the new room.
-
Stage 3 (Analogical Refinement) is training-free. It uses reweighted random walk graph matching followed by Hungarian algorithm to match replaced target objects and their corresponding reference objects. It then use gradient descent to optimize for relational and orientation consistency with the reference scene, as well as wall-clearance and opening-clearance (keep objects out of doorways). Finally, a projection step moves objects back inside the room and pushes overlapping objects apart.
Both networks in Stage 1 and Stage 2 are trained on 3D-FRONT rooms via layout recovery. Evaluation uses collision-free and in-boundary rates, LLM-judged positions and rotation scores, and PSA following LayoutVLM, as well as iRecall following InstructScene.
Critical strengths
- The task is clearly defined. The use cases and applications discussed are concrete and real. For example, the real-to-sim application is quite sound, and I can see how the author's Concerto features can help.
- The system works end-to-end and scores high on physical metrics.
- The appendix is detailed and thorough, including lots of analysis such as what the layout decoder attention attends to. The authors clearly put lots of effort into this work.
Critical weaknesses
1. Major Weakness: The paper seems to overclaim its contributions
The paper calls Scene Retargeting "a flexible, scalable way to generate diverse 3D scene layouts" (Sec. 1). The authors claimed that the method transfers the reference scene's "spatial organization" and "draws its structure from a single exemplar rather than a learned generative prior". The paper also promised that "varying the floor plan and object instances yields diverse scenes that serve a similar function as the reference".
Compared to the claims, I find that the three stages shows much narrower contributions. However, I am willing to update my scores and comments if the authors are able to provide strong justifications or narrower their claims.
Here's a detailed breakdown:
-
Room-level arrangement does not transfer (Inter-cluster relations are generative priors): The author claims the method transfers the reference scene's "spatial organization" and "draws its structure from a single exemplar rather than learned generative prior". However, cluster placements does not seem utilize the reference arragement much. For each cluster, the cluster placement network receives three inputs: averaged categories of its objects, sizes, and current poses (Eq.2 and Appendix B). It also inputs the boundary of the empty room. However, it never receives inter-cluster information from the reference scene. The only cluster relation information the target scene receive are the cluster centroids used as initial position of denoising, which I believe is too weak to be meaningful. Specifically, Appendix C states that inference adds positional noise with 0.5 scale in normalized room coordinates, while training perturbation use a 0.15 scale. So from my understanding the reference centroids are perturbed about 3x more than anything the network was trained to correct (or if 0.15 means variance, than 1.3x, which is still larger). Stage 3 doesn't receive said information either. They concern objects within a single cluster, and the wall-clearance term only matches walls next to where Stage 1 already placed each cluster (Appendix D.1). Therefore, it seems to me that room-level organization is not transferred. The inter-cluster relations are largely decided by the network's learned generative prior, which appears to be the opposite of the author's claims.
-
The method is not diverse (Intra-cluster relations are static): The author claims that the model generates diverse layouts. However, objects inside each clusters are basically direct replicas of the reference scene. Stage 1 moves each cluster as a rigid unit. Stage 2 introduces some variation in object-level placement through denoising, but this variation seems to be immediately removed in stage 3, where the relational and orientation terms push the objects to replicate the offsets and headings relative to the cluster anchor as their reference counterparts. From my understanding, whatever the layout decoder predicts inside a cluster is pulled back towards a direct copy of the reference by the relational and orientation terms. The wall term or clearance term only makes small, local changes that doesn't seem to be meaningful enough to be called "flexible" or "diverse". Therefore, the intra-cluster relations seems static and I don't see how it is diverse. The qualitative results provided by the author further substantiates this, where most of the intra-cluster arrangements are exactly the same as their references (Fig 4, and especially Fig 12). While Section 1 says "varying the floor plan and object instances yields diverse scenes", this variation is supplied from outside the method; the method itself adds no variation inside the cluster. Cluster placement adds variety, but as mentioned in the last point, it comes from learned prior rather than scene reference. So it contradicts with the abstract's claim of "flexibly...adapts to distinctive floor plans." Essentially, the paper's framework sits on two extremes:
- you either have strong transfer but weak variety (a static replica of the reference scene) as seen within each cluster, or
- you have strong variety but weak transfer (pure generative priors) as seen between clusters.
-
The method is not flexible (Clustering seems like an overly strong prior): Clustering basically assumes that a room naturally splits into compact groups that can be placed independently (Stage 1 did not transfer inter-cluster relations). I think this is roughly true for living rooms and bedrooms where you have strong localities. But for many other scene types such as classrooms, restaurants, or gyms, the layout is more about repeated units and locality is secondary. Kitchen and bathroom fixtures run along the walls. Libraries, supermarkets, and gyms are laid out around aisles and clearance spaces. I would appreciate if the authors can provide some insights that proves otherwise, or if the authors acknowledge that their inductive bias limits certain scene types.
Metrics such as iRecall does not reveal any of the above. I hypothesize the reason is that it extracts relation with short-range rules (e.g., adjacency needs gap below 0.22, facing and alignment needs distance below 0.6 etc.). Pairs this close are usually in the same cluster, so iRecall counts the very relation that the framework replicates (see the 2nd point above). Thus, iRecall mostly measures how closely the transfer copies the reference. It tends to ignore inter-cluster relations which tends to be further apart. It would also penalize a sensible change such as moving chairs around a table to a two rows along the long edge, etc.
Medium Weakness: If the final layout is a replica that satisfies constraint, why neural nets?
As mentioned earlier, two of the main issues are:
- Intra-cluster layout is static: Stage 3 seems to overwrite any diversity the intra-cluster decoder may have introduced. So the intra-cluster layout decoder's output barely survives into the final layout.
- Inter-cluster arrangement in Stage 1 comes from generative prior
For intra-cluster layout, since the matching in stage 3 comes from graph matching not the decoder, I don't understand the necessity of a neural net. The motivation is unclear to me. Also, the decoder also had no way to learn adaption anyways: in layout recovery (diffusion training), the objects in its scene context token are exactly the same objects it will place, so the model never sees a mismatch in size, count, or conflict with the wall. So I find it hard to imagine how the model can adapt to novel scene, or "diversity" and "flexibility" the paper promised in any way.
For inter-cluster layout, while generative prior offers some diversity, it is unclear how much better it is to some procedural algorithms (e.g., DFS, MCMC etc). I was unable to tell from the author's provided qualitative results whether the model truly learned scene statistics, or is simply finding arbitrary clearance that fits. For example, in Figure 4, I can easily write a ~30 line algorithm that achieves similar cluster arrangement without using your cluster placement network.
Collectively, these issues makes the motivation of the pipeline's complexity unjustified. If cluster placements are basically direct replica (intra-cluster) plus some generative prior not conditioned on inter-cluster arrangements from the reference scene, why not use a simpler approach (e.g., a procedural framework)? For example, one can extract each cluster's wall contacts, offsets between clusters, and simply solve a Constraint Satisfaction Problem (given you already fixed intra-cluster layout and that the cases in the paper are rather simple, I presume the computational complexity is bounded).
An ablation that keeps only graph matching, refinement, and projection may help, but I don't think it can settle the question. Simply arguing that directly pasting the reference coordinates would leave furniture outside boundaries or inside doorways would be even less convincing. So instead, I would prefer if the authors could explain, ideally from first principles, what the networks provide that these terms do not. An ablation cannot settle the question because you simply can't rule out the posibility that Stage 3 is simply not tuned for such task. Constraint problems like this can be solved in many ways: SA with Metropolis-Hasting acceptance criterion in Infinigen (Raistrick et al., CVPR 2024), Cluster-reparameterized gradient in R3L (Gu et al., ICML 2026), or the simplest, grid-based DFS in HSM (Pun et al., 3DV 2026).
Again, I don't expect the authors to actually reproduce and tune one of such algorithms for the sake of this question, because I believe this is a fundamental concern that should be addressed through first principles analysis rather than more experiments.
Minor Weakness: Potentially inaccurate explanations.
The paper attributed Concerto's benefit to its features being richer than category lables or texts. Correct me if I am wrong, but I believe a different property matters more here.
The layout decoder is trained by layout recovery, so during training the reference objects in the Concerto input are exactly those objects being placed. But at test time, they are different objects. Thus, for the decoder to still find the right counterpart, it must not learn to match objects by exact identity, but rather by invariance. My theory is that two design choices helped here (neither is about information richness):
- Concerto features likely stay similar for similar but different objects.
- Feature misalignment bewteen Concerto and OpenShape prevents shortcut learning.
Table 2(e) cannot tell apart these explanations. I think a more direct test might be to replace Concerto input with OpenShape token per transferred reference object, plus their positions, and compare layout recovery (training) against retargeting (inference). Both explanations expect retargeting to get worse, since per-object tokens lose the scene-level context the authors find important (Sec. 4.4); so what tells them apart is the gap between the two settings. If information richness is truly the reason, performance should degrade on both layout recovery and retargeting; if my explanation is right, layout recovery performance should stay as good or even improve, while retargeting performance degrades. If neither layout recovery nor retargeting performance degrades, Concerto may not be necessary at all.
But as suggested by the title, this is just a minor weakness. I am pointing this out simply to give some opinions that may help authors improve their paper. This weakness does not affect my review score, and the author can leave this question if time doesn't permit.
Questions for authors
-
The method generates layout at two extremes. The inter-cluster relations come from a generative prior unconditional to inter-cluster arrangement in the reference, and the intra-cluster layout is a direct replica of the reference. Given this, how does the method transfer the reference’s “spatial organization” or “flexibly handle mismatched object instances”? If it cannot, could the authors narrow these claims, including which scene types the clustering prior suits? (see major weakness above).
-
The paper argued against language-based methods, given that their verbalization "discards precise spatial metrics, retaining only coarse relational concepts" (Sec 2.2). However, formulating the author's scene retargeting problem as a constraint satisfaction problem directly from the reference scene loses none of these metrics. It might even be possible to use the same anchor offsets, orietnations, and wall clearances that stage 3 already uses. Stage 3 is where those metrtics entered the final layout in the existing framework anyways. Plus, as mentioned in the weaknesses, the layout decoder's intra-cluster output is pulled back towards these offsets, so the necessity of the object placement network is unclear. The cluster placement network does contribute varied cluster placement, but comes from learned generative prior and not from reference. So my question is: What information, if any, does the two neural nets contribute that a regular constraint solver formulation with the same metric constraints used in Stage 3 cannot recover? (see medium weakness for more details).
-
Appendix C says that the layout decoder "begins from target inventorty poses", but Sec 3.1 defines the target inventory as unarranged. Am I missing something here? What do these poses refer to? Otherwise, this could be an evaluation concern.
-
The paper's attribution of Concerto feature's benefits isn't very convincing. I think there exists a stronger hypothesis (See minor weakness above).