acceptodds
Withdrawn from ICLR 2027

ScenePilot: Plug-and-Play 3D Scene Repair across Generators

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-driven 3D scene generators have made substantial progress, yet their outputs often retain collisions, boundary violations, orientation errors, and structural inconsistencies. Existing correction mechanisms are typically coupled to generator-specific scene representations and execution pipelines, making repair difficult to reuse across heterogeneous systems. We introduce ScenePilot, a plug-and-play 3D scene repair policy designed to make corrective behavior transferable across generators. ScenePilot decouples repair reasoning from generator-specific execution through a canonical scene-editing interface, allowing one shared multimodal policy to operate across heterogeneous generators without modifying or retraining their underlying generation models. At inference time, ScenePilot performs conservative closed-loop repair: each round proposes at most one atomic edit, executes it tentatively, and retains the change only when a shared verifier confirms improvement. We further construct SceneReverse, which transforms clean 3D scenes into executable recovery trajectories and provides generator-independent supervision for learning atomic repair decisions. Combined with supervised learning and reward-guided policy optimization based on realized improvements, SceneReverse enables repair learning without relying on failure traces from any target generator. On SceneEval, the same frozen ScenePilot policy consistently improves scenes from five heterogeneous generators unseen during policy training, without backend-specific policy fine-tuning, using only lightweight backend adapters. These results support transferable 3D scene repair as a reusable capability that can be attached to existing generators through a canonical interface.

Primary area:
reinforcement learning
Keywords:
3D scene generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_UXNFOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

ScenePilot uses a shared Qwen3-VL-4B policy to repair indoor scenes through a canonical object-centric representation and backend adapters. It proposes atomic edits, retaining them only when a verifier’s combined physical, spatial and visual-semantic score improves. The policy is trained with SFT and GRPO on SceneReverse recovery trajectories constructed from perturbed 3D-FRONT rooms. The authors report 4–33% Overall score gains across five generators on 100 SceneEval prompts using one frozen policy.

Critical strengths
  1. The problem framing and interface design are clear. The canonical state, capability descriptor and accept-or-rollback verification are sensible abstractions.
  2. SceneReverse is documented with room-disjoint splits and action statistics.
  3. The evaluation uses paired comparisons across five generators and reports individual SceneEval metrics, including decreases in support and accessibility on ReSpace and Reason-3D.
Critical weaknesses
  1. Eq. 7 leaves Q_PHY, Q_SPA, normalization, ε and inference weights unspecified. Table 2 shows that training and verification help, but lacks a random-edit or collision-resolution proposer under the same verifier and budget. This control is particularly important if verification reuses SceneEval’s evaluation components.

  2. No repair comparisons are made against SceneReVis [1], LayoutVLM’s optimization stage, DiffuScene [2], PhyScene [3] or DeBaRA [4] on shared starting scenes. GPT-5.4 outperforms ScenePilot by 6.54 Overall points, but no quality-cost comparison is provided. The inference judge, call counts and per-scene time/cost are unspecified. Using GPT-5.4 for both reward and evaluation also raises possible judge-specific bias despite the Gemini check.

  3. T_max, repair rounds, acceptance/rollback rates and edit magnitudes are unreported. If Table 10 covers complete runs, only 2 to 12 of 100 terminate through STOP, suggesting budget-driven termination. Support and accessibility decline on ReSpace and Reason-3D, so Overall gains do not establish conservative repair.

  4. The policy’s prompt and history inputs differ across Fig. 2, Eq. 3, Algorithm 1 and G.2; instruction pairing in SceneReverse is also unexplained. Sec. 4.1 names five unseen generators, while A.4 lists four and clarifies tuning exclusion only for LayoutGPT. Policy/verifier inputs, each generator’s tuning status and the 100 evaluation prompts need clarification.

  5. The perceptual results are inconsistent across tables, and the human–VLM agreement statistics do not match the underlying ratings. The evaluation cohorts and denominators need clarification. Agreement is measured only across condition averages, while the human study’s scene count and statistical uncertainty are unreported.

Minor

  1. Table 4 calls populated hyperparameters “placeholders” despite describing a final run.
  2. The GRPO judge prompt mixes “Be generous” and “7 to 10” with a full 0 to 10 rubric; reward distributions and within-group variance would clarify the resulting signal.

References

[1] Zhao, Yang, et al. "SceneReVis: A Self-Reflective Vision-Grounded Framework for 3D Indoor Scene Synthesis via Multi-turn RL." arXiv preprint arXiv:2602.09432 (2026).

[2] Tang, Jiapeng, et al. "Diffuscene: Denoising diffusion models for generative indoor scene synthesis." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[3] Yang, Yandan, et al. "Physcene: Physically interactable 3d scene synthesis for embodied ai." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Maillard, Léopold, et al. "Debara: Denoising-based 3d room arrangement generation." Advances in Neural Information Processing Systems 37 (2024): 109202-109232.

Questions for authors

Refer to weaknesses.

Reviewer_QxWMOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents ScenePilot, a transferable policy for repairing 3D indoor scenes generated by a wide range of backends. Lightweight adapters translate each generator's native scenes and editing operations into a canonical interface. This lets a single, frozen multimodal policy suggest atomic edits across different systems while respecting the specific capabilities of each backend. Crucially, every proposal is only tentatively applied: it is kept if a shared verifier sees an improvement, and rolled back otherwise. Rather than relying on failure traces from individual generators, the authors build SceneReverse by perturbing clean scenes and reversing those perturbations into executable recovery paths. The policy is then trained with supervised fine-tuning followed by reward-guided optimization (GRPO).

Critical strengths

1.I find the canonical interface design practical. It separates what to repair from how a backend executes it, so a single frozen policy can work across different generators through lightweight adapters. This matters because it directly tackles the fragmentation issue in existing pipelines and suggests a viable way to reuse repair policies without retraining for each new backend.

2.Instead of collecting failure traces from target generators, the authors build SceneReverse by perturbing clean scenes and reversing those perturbations into recovery trajectories. I think this is a reasonable way to get process supervision, especially since gathering failure traces for every backend would be expensive and would undercut the whole point of transfer. The SFT and GRPO stages afterwards also help, mainly because they reward actual scene improvements rather than forcing the policy to imitate one specific recovery path.

Critical weaknesses

1.The paper's main selling point is that one frozen policy works across generators. But in practice, every new generator still needs a custom, hand-written adapter to handle its specific formats and coordinate systems. So the real burden hasn't disappeared, it's just shifted from retraining the policy to writing the adapter.

2.If the verifier is wrong, the closed loop will happily keep bad edits or reject good ones. The paper does some validation of this verifier, but it's still largely a black box built on top of another VLM judge. I would have liked to see a more rigorous analysis of where the verifier fails and how sensitive the system is to those failures. Without that, the safety guarantee at the heart of the method feels somewhat fragile.

Questions for authors

1.For the adapters, how much actual engineering effort does a new generator require? Is it just a few hundred lines of mapping code, or does someone need to understand the generator's internals in depth? And how much of an adapter can be reused for generators with similar representations? This directly affects whether "plug-and-play" is a fair description of the actual workflow.

2.For the verifier, how sensitive is repair performance to its mistakes? If the verifier wrongly accepts or rolls back an edit, does the loop tend to recover in later rounds, or does the error propagate? A short analysis of verifier failures, or an ablation that perturbs the verifier, would help clarify how much the safety guarantee depends on the verifier's accuracy.

Reviewer_nPaQOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents ScenePilot, a framework for repairing 3D indoor scenes produced by heterogeneous generators using a shared multimodal policy. Backend adapters convert native scenes into a canonical object-centric representation and translate the policy’s atomic editing actions into executable operations. A verifier accepts an edit only when it improves a weighted scene-quality score. The policy is trained through supervised fine-tuning on SceneReverse, a dataset constructed by reversing synthetic scene corruptions, followed by step-level GRPO.

Critical strengths

1. A useful formulation of transferable scene repair.

  • Separating repair decisions from generator-specific execution addresses a practical limitation of existing scene-generation pipelines. Reusing a shared policy through a canonical interface could reduce the need to develop and train a separate repair mechanism for each generator.

2. Encouraging evidence across multiple generators.

  • Table 1 reports improvements in Overall for all five evaluated generators using the same policy checkpoint. The within-generator paired comparisons are appropriate for heterogeneous scene outputs. The qualitative examples also illustrate potentially useful local corrections to furniture placement and arrangement.
Critical weaknesses

1. Limited methodological novelty and insufficiently demonstrated added value.

  • The framework primarily combines familiar components: a canonical scene-editing interface, synthetic corruption-and-recovery supervision, SFT followed by GRPO, and verifier-guided iterative editing. Several central elements, including trajectory-based supervision and multimodal iterative editing with SFT and GRPO, are already present in closely related work such as SceneReVis. The main contribution therefore appears to lie in integrating these components for cross-generator repair. The paper does not sufficiently establish what nontrivial methodological advance or generalizable insight emerges from this integration. In particular, the current comparisons leave open whether a simpler repair mechanism using the same canonical interface and verifier would achieve comparable improvements. This limits the evidence that the proposed combination constitutes a substantial research contribution.

2. Insufficient validation of conservative repair.

  • The acceptance rule ensures improvement in a weighted quality score, which does not necessarily imply preservation of already satisfied constraints. Table 1 reports regressions in individual metrics, including support and accessibility for ReSpace and accessibility for Reason-3D. While the verification ablation in Table 2 demonstrates its benefit to aggregate performance, it does not directly establish that the method avoids harmful or unnecessary edits, as posed in RQ4. Scene-level degradation rates, newly introduced violations, and the extent of modifications are needed to substantiate this central claim.
Questions for authors
  1. How does ScenePilot compare with a simpler repair mechanism using the same canonical interface, verifier, and editing budget?

  2. What evidence demonstrates that ScenePilot preserves already valid scene properties, beyond improvements in average aggregate scores? Could you provide experimental results showing how often previously satisfied constraints become violated after repair?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.