acceptodds
Withdrawn from ICLR 2027

Scene Retargeting: Learning Object Placement with Analogical Transfer

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive simulations of embodied AI or spatial computing applications build on realistic 3D scenes that support daily activities. However, sparse, irregular layout structures impose scene-specific physical constraints, making it hard to define a generalizable framework for generating similar functional context. We formalize Scene Retargeting as stably transferring the semantically coherent spatial organization across layouts, rather than relying on textual descriptions or pairwise relationships. Our cluster-wise transfer flexibly handles mismatched object instances and adapts to distinctive floor plans. We optimize to preserve the rich semantic context of individual clusters by respecting the spatial distribution of foundation features. We can then impose physical constraints to refine wall contacts, pairwise alignment, or clear passageways and openings. Our framework outperforms state-of-the-art methods on layout generation on the 3D-FRONT dataset, and demonstrates downstream applications including real-to-sim transfer, analogical trajectory transfer, and multi-reference composition.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D indoor scene synthesis, Scene layout generation, Scene retargeting, Exemplar-based generation, Object arrangement

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_XbUtOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The author formalizes Scene Retargeting: given a reference scene, the author's framework aim to transfer its spatial layout to another empty room (walls, doors and windows only) using a new set of objects. In other words, the output is a layout of the target objects in the empty room that follows the spatial structure of the reference. The author's main motivation is to create a framework that exhibits flexibility, diversity, and transferability, such that it could benefit numerous downstream applications.

The framework consists of three stages.

  1. Stage 1 (Cluster Placement) partitions the reference scene into functional clusters via agglomerative clustering using positions and OpenShape features. Each cluster is moved into the empty room by using a network to predict their poses, while layouts inside the clusters are unchanged.

  2. Stage 2 (Object Placement) replaces the moved reference objects in the new room with target objects. Specifically, the moved reference objects and the floor of the empty room are combined into one point cloud and encoded via Concerto. Then, a second network predicts the pose of each target object given the scene-level Concerto features and a bunch of object-level features of the target object (OpenShape, category, size, etc). The aim is for each target object to take the place of a matching reference object in the new room.

  3. Stage 3 (Analogical Refinement) is training-free. It uses reweighted random walk graph matching followed by Hungarian algorithm to match replaced target objects and their corresponding reference objects. It then use gradient descent to optimize for relational and orientation consistency with the reference scene, as well as wall-clearance and opening-clearance (keep objects out of doorways). Finally, a projection step moves objects back inside the room and pushes overlapping objects apart.

Both networks in Stage 1 and Stage 2 are trained on 3D-FRONT rooms via layout recovery. Evaluation uses collision-free and in-boundary rates, LLM-judged positions and rotation scores, and PSA following LayoutVLM, as well as iRecall following InstructScene.

Critical strengths
  • The task is clearly defined. The use cases and applications discussed are concrete and real. For example, the real-to-sim application is quite sound, and I can see how the author's Concerto features can help.
  • The system works end-to-end and scores high on physical metrics.
  • The appendix is detailed and thorough, including lots of analysis such as what the layout decoder attention attends to. The authors clearly put lots of effort into this work.
Critical weaknesses

1. Major Weakness: The paper seems to overclaim its contributions

The paper calls Scene Retargeting "a flexible, scalable way to generate diverse 3D scene layouts" (Sec. 1). The authors claimed that the method transfers the reference scene's "spatial organization" and "draws its structure from a single exemplar rather than a learned generative prior". The paper also promised that "varying the floor plan and object instances yields diverse scenes that serve a similar function as the reference".

Compared to the claims, I find that the three stages shows much narrower contributions. However, I am willing to update my scores and comments if the authors are able to provide strong justifications or narrower their claims.

Here's a detailed breakdown:

  1. Room-level arrangement does not transfer (Inter-cluster relations are generative priors): The author claims the method transfers the reference scene's "spatial organization" and "draws its structure from a single exemplar rather than learned generative prior". However, cluster placements does not seem utilize the reference arragement much. For each cluster, the cluster placement network receives three inputs: averaged categories of its objects, sizes, and current poses (Eq.2 and Appendix B). It also inputs the boundary of the empty room. However, it never receives inter-cluster information from the reference scene. The only cluster relation information the target scene receive are the cluster centroids used as initial position of denoising, which I believe is too weak to be meaningful. Specifically, Appendix C states that inference adds positional noise with 0.5 scale in normalized room coordinates, while training perturbation use a 0.15 scale. So from my understanding the reference centroids are perturbed about 3x more than anything the network was trained to correct (or if 0.15 means variance, than 1.3x, which is still larger). Stage 3 doesn't receive said information either. They concern objects within a single cluster, and the wall-clearance term only matches walls next to where Stage 1 already placed each cluster (Appendix D.1). Therefore, it seems to me that room-level organization is not transferred. The inter-cluster relations are largely decided by the network's learned generative prior, which appears to be the opposite of the author's claims.

  2. The method is not diverse (Intra-cluster relations are static): The author claims that the model generates diverse layouts. However, objects inside each clusters are basically direct replicas of the reference scene. Stage 1 moves each cluster as a rigid unit. Stage 2 introduces some variation in object-level placement through denoising, but this variation seems to be immediately removed in stage 3, where the relational and orientation terms push the objects to replicate the offsets and headings relative to the cluster anchor as their reference counterparts. From my understanding, whatever the layout decoder predicts inside a cluster is pulled back towards a direct copy of the reference by the relational and orientation terms. The wall term or clearance term only makes small, local changes that doesn't seem to be meaningful enough to be called "flexible" or "diverse". Therefore, the intra-cluster relations seems static and I don't see how it is diverse. The qualitative results provided by the author further substantiates this, where most of the intra-cluster arrangements are exactly the same as their references (Fig 4, and especially Fig 12). While Section 1 says "varying the floor plan and object instances yields diverse scenes", this variation is supplied from outside the method; the method itself adds no variation inside the cluster. Cluster placement adds variety, but as mentioned in the last point, it comes from learned prior rather than scene reference. So it contradicts with the abstract's claim of "flexibly...adapts to distinctive floor plans." Essentially, the paper's framework sits on two extremes:

    • you either have strong transfer but weak variety (a static replica of the reference scene) as seen within each cluster, or
    • you have strong variety but weak transfer (pure generative priors) as seen between clusters.
  3. The method is not flexible (Clustering seems like an overly strong prior): Clustering basically assumes that a room naturally splits into compact groups that can be placed independently (Stage 1 did not transfer inter-cluster relations). I think this is roughly true for living rooms and bedrooms where you have strong localities. But for many other scene types such as classrooms, restaurants, or gyms, the layout is more about repeated units and locality is secondary. Kitchen and bathroom fixtures run along the walls. Libraries, supermarkets, and gyms are laid out around aisles and clearance spaces. I would appreciate if the authors can provide some insights that proves otherwise, or if the authors acknowledge that their inductive bias limits certain scene types.

Metrics such as iRecall does not reveal any of the above. I hypothesize the reason is that it extracts relation with short-range rules (e.g., adjacency needs gap below 0.22, facing and alignment needs distance below 0.6 etc.). Pairs this close are usually in the same cluster, so iRecall counts the very relation that the framework replicates (see the 2nd point above). Thus, iRecall mostly measures how closely the transfer copies the reference. It tends to ignore inter-cluster relations which tends to be further apart. It would also penalize a sensible change such as moving chairs around a table to a two rows along the long edge, etc.

Medium Weakness: If the final layout is a replica that satisfies constraint, why neural nets?

As mentioned earlier, two of the main issues are:

  1. Intra-cluster layout is static: Stage 3 seems to overwrite any diversity the intra-cluster decoder may have introduced. So the intra-cluster layout decoder's output barely survives into the final layout.
  2. Inter-cluster arrangement in Stage 1 comes from generative prior

For intra-cluster layout, since the matching in stage 3 comes from graph matching not the decoder, I don't understand the necessity of a neural net. The motivation is unclear to me. Also, the decoder also had no way to learn adaption anyways: in layout recovery (diffusion training), the objects in its scene context token are exactly the same objects it will place, so the model never sees a mismatch in size, count, or conflict with the wall. So I find it hard to imagine how the model can adapt to novel scene, or "diversity" and "flexibility" the paper promised in any way.

For inter-cluster layout, while generative prior offers some diversity, it is unclear how much better it is to some procedural algorithms (e.g., DFS, MCMC etc). I was unable to tell from the author's provided qualitative results whether the model truly learned scene statistics, or is simply finding arbitrary clearance that fits. For example, in Figure 4, I can easily write a ~30 line algorithm that achieves similar cluster arrangement without using your cluster placement network.

Collectively, these issues makes the motivation of the pipeline's complexity unjustified. If cluster placements are basically direct replica (intra-cluster) plus some generative prior not conditioned on inter-cluster arrangements from the reference scene, why not use a simpler approach (e.g., a procedural framework)? For example, one can extract each cluster's wall contacts, offsets between clusters, and simply solve a Constraint Satisfaction Problem (given you already fixed intra-cluster layout and that the cases in the paper are rather simple, I presume the computational complexity is bounded).

An ablation that keeps only graph matching, refinement, and projection may help, but I don't think it can settle the question. Simply arguing that directly pasting the reference coordinates would leave furniture outside boundaries or inside doorways would be even less convincing. So instead, I would prefer if the authors could explain, ideally from first principles, what the networks provide that these terms do not. An ablation cannot settle the question because you simply can't rule out the posibility that Stage 3 is simply not tuned for such task. Constraint problems like this can be solved in many ways: SA with Metropolis-Hasting acceptance criterion in Infinigen (Raistrick et al., CVPR 2024), Cluster-reparameterized gradient in R3L (Gu et al., ICML 2026), or the simplest, grid-based DFS in HSM (Pun et al., 3DV 2026).

Again, I don't expect the authors to actually reproduce and tune one of such algorithms for the sake of this question, because I believe this is a fundamental concern that should be addressed through first principles analysis rather than more experiments.

Minor Weakness: Potentially inaccurate explanations.

The paper attributed Concerto's benefit to its features being richer than category lables or texts. Correct me if I am wrong, but I believe a different property matters more here.

The layout decoder is trained by layout recovery, so during training the reference objects in the Concerto input are exactly those objects being placed. But at test time, they are different objects. Thus, for the decoder to still find the right counterpart, it must not learn to match objects by exact identity, but rather by invariance. My theory is that two design choices helped here (neither is about information richness):

  1. Concerto features likely stay similar for similar but different objects.
  2. Feature misalignment bewteen Concerto and OpenShape prevents shortcut learning.

Table 2(e) cannot tell apart these explanations. I think a more direct test might be to replace Concerto input with OpenShape token per transferred reference object, plus their positions, and compare layout recovery (training) against retargeting (inference). Both explanations expect retargeting to get worse, since per-object tokens lose the scene-level context the authors find important (Sec. 4.4); so what tells them apart is the gap between the two settings. If information richness is truly the reason, performance should degrade on both layout recovery and retargeting; if my explanation is right, layout recovery performance should stay as good or even improve, while retargeting performance degrades. If neither layout recovery nor retargeting performance degrades, Concerto may not be necessary at all.

But as suggested by the title, this is just a minor weakness. I am pointing this out simply to give some opinions that may help authors improve their paper. This weakness does not affect my review score, and the author can leave this question if time doesn't permit.

Questions for authors
  1. The method generates layout at two extremes. The inter-cluster relations come from a generative prior unconditional to inter-cluster arrangement in the reference, and the intra-cluster layout is a direct replica of the reference. Given this, how does the method transfer the reference’s “spatial organization” or “flexibly handle mismatched object instances”? If it cannot, could the authors narrow these claims, including which scene types the clustering prior suits? (see major weakness above).

  2. The paper argued against language-based methods, given that their verbalization "discards precise spatial metrics, retaining only coarse relational concepts" (Sec 2.2). However, formulating the author's scene retargeting problem as a constraint satisfaction problem directly from the reference scene loses none of these metrics. It might even be possible to use the same anchor offsets, orietnations, and wall clearances that stage 3 already uses. Stage 3 is where those metrtics entered the final layout in the existing framework anyways. Plus, as mentioned in the weaknesses, the layout decoder's intra-cluster output is pulled back towards these offsets, so the necessity of the object placement network is unclear. The cluster placement network does contribute varied cluster placement, but comes from learned generative prior and not from reference. So my question is: What information, if any, does the two neural nets contribute that a regular constraint solver formulation with the same metric constraints used in Stage 3 cannot recover? (see medium weakness for more details).

  3. Appendix C says that the layout decoder "begins from target inventorty poses", but Sec 3.1 defines the target inventory as unarranged. Am I missing something here? What do these poses refer to? Otherwise, this could be an evaluation concern.

  4. The paper's attribution of Concerto feature's benefits isn't very convincing. I think there exists a stronger hypothesis (See minor weakness above).

Reviewer_m46YOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Scene Retargeting: transferring the layout of a single furnished reference room to an empty target room with a different floor plan and object inventory. The method places functional clusters on the target floor plan, predicts object poses within them using Concerto scene features, and refines the result with graph matching and constraint-based optimization. On 3D-FRONT living rooms and bedrooms, it outperforms unconditional and text-conditioned layout-generation baselines on physical and semantic metrics. Ablations show that each stage and conditioning signal contributes. The paper also shows applications such as trajectory transfer and real-to-sim.

Critical strengths
  1. A well-motivated task. The paper studies transferring spatial organization from a single concrete exemplar to rooms with different floor plans and inventories, rather than sampling from a learned prior or a text description. This setting is relevant to embodied-AI simulation and scene construction.
  2. A clear and reasonable method design. Functional clusters serve as the unit of transfer, which separates global placement, within-cluster arrangement and explicit constraint handling (walls, openings, collisions). Conditioning on 3D foundation-model features avoids the information loss of text or scene-graph abstractions.
  3. Informative analysis. The paper includes module ablations, a cluster-size study, attention visualizations, failure cases and a detailed appendix. These help explain the method’s behavior and limitations.
Critical weaknesses
  1. Unequal reference information between the method and the baselines. The method uses reference 3D geometry, while text baselines receive a verbalized instruction that drops distance information. iRecall and PSA are built from reference relations that are closely aligned with the Stage 3 objectives. Without equal-information baselines, the gains cannot be attributed to the algorithm rather than the input.
  2. Unequal post-processing and unclear baseline reporting. CF/IB benefit from projection, collision handling and pruning that most baselines do not have. Baselines are evaluated on pruned and unpruned inventories and the higher overall score is reported, so it is unclear whether each metric comes from the same setting. The effect of pruning on the metrics is not reported.
  3. The contribution of the learned object decoder is not isolated. The parameter-free Stage 3 accounts for large gains (removing refinement lowers iRecall from 64.3 to 57.0, and removing physical constraints lowers CF from 97.0 to 88.4). The learned networks are trained on same-room perturbation recovery but used for cross-room transfer. There is no ablation without the learned object decoder, and no analysis of how matching or cluster-assignment errors affect the final layout.
Questions for authors

I would like the authors to address the three weaknesses above. I would be willing to raise my score if these concerns are adequately addressed.

Reviewer_mzZ4OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

In this paper, authors trying to transfer the spatial organization of a furnished reference scene to a target room with a different floor plan and furniture inventory, while preserving functional relationships and satisfying geometric constraints. The proposed framework has three stages. First, reference objects are grouped into functional clusters with spatial coordinates and OpenShape features, and model predicts their placement in the target room. Second, an object-level decoder uses cross-attention over Concerto features extracted from the transferred reference layout to arrange the target furniture. Third, the method establishes object correspondences and refines relative positions, orientations, wall clearances, and opening constraints. The two networks are trained through layout perturbation and recovery, avoiding the need for paired retargeting annotations. And this method gets good results on 75 living-room and bedroom pairs from 3D-FRONT.The paper also demonstrates trajectory transfer, real-to-sim transfer, and composition from multiple reference scenes.

Critical strengths
  1. The problem is well motivated and practically relevant.

Transferring an existing arrangement to a new space is useful for interior design, simulation, and reuse of interactive content. The formulation explicitly accounts for changes in room geometry and furniture inventory, making it a meaningful problem to study.

  1. The hierarchical design fits the task well.

Functional clusters provide a useful intermediate representation between individual objects and the full room. They allow related furniture, such as a sofa and coffee table, to move together before object-level differences and local constraints are resolved. The examples in Figure 4 illustrate how this design helps preserve recognizable furniture groupings.

3.The ablations provide useful evidence for the main components.

Removing cluster placement, Concerto features, or analogical refinement reduces iRecall by 13.2, 9.7, and 7.3 points, respectively. These results support the contribution of each stage. The appendix also examines cluster granularity and includes an identity-swap experiment showing that cross-attention responds to changes in object identity. These analyses help explain how the method works.

4.The system addresses both relational fidelity and geometric feasibility.

Under the reported evaluation protocol, the method performs well on both aspects. The multi-reference composition and same-inventory relocation examples also suggest that the cluster representation is reusable across related tasks. The reported runtime of approximately 6.9 seconds per scene is encouraging for practical use.

Critical weaknesses
  1. The “moving house” setting is interesting, but the training and test scenes are fairly simple, with basic furniture arrangements and no small objects. Larger-scale experiments on more complex, cluttered scenes would strengthen the paper.
  2. Evaluation is limited to 75 room pairs with similar inventories, with separate models for each room type. Broader tests covering larger inventory and room-size differences, along with multiple random seeds, would help assess robustness and generalization.
  3. The Concerto ablation shows that its features are useful, but their specific contribution remains unclear. Comparisons with category labels and coordinates, simple geometric features, and alternative encoders would help clarify this.
Questions for authors
  1. Where do the initial target object poses come from? Appendix C states that the object decoder starts from perturbed “target inventory poses,” whereas the task input is an unarranged object inventory. Are these poses sampled randomly, obtained through matching, or taken from the original target scene? If the original target layout is used, would this introduce information beyond the stated task inputs? How does the method perform without access to those poses?

  2. How much of the gain remains when reference information and post-processing are controlled? Could the authors include a baseline combining reference-scene alignment, object matching, and the same analogical refinement? Reporting performance before and after refinement would also help separate the contributions of learned placement, feature conditioning, and geometric optimization.

  3. What exactly is removed during pruning, and at what cost? How is the matching-confidence threshold selected? What fraction of the original inventory does the average removal of 1.6 objects represent? Are the removed objects mostly redundant furniture, or do they sometimes serve essential functions? Results with pruning disabled would be useful.

4.Although the paper includes LLM/VLM-based baselines, I wonder whether a strong general-purpose model could achieve similar results given comparable reference information and access to geometric tools. A direct comparison would help establish the benefit of the proposed specialized pipeline.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.