acceptodds
Under review as a conference paper at ICLR 2027

RAPP3R: Reinforcing Object Relationship for Physically Plausible 3D Reconstruction

Abstract

We present RAPP3R, a unified framework that combines architectural enhancements with a reinforcement learning strategy for physically plausible multi-object 3D scene reconstruction from a single image. Building on a pre-trained joint shape-pose generator that processes each object independently, RAPP3R equips the model to reason about interactions across the objects. Gated Inter-Object Attention introduces an inter-object branch while preserving the pre-trained model's behavior at initialization. Volume-RoPE further encodes each object's position and spatial extent, providing explicit spatial context for attention across objects. Building on these architectural changes, we train the model with GDPO to learn physical relationships in a scene, such as collisions and floating. We overcome the limitations of scene-level reinforcement learning through object-wise advantage computation. As a result, our experiments show that RAPP3R outperforms prior methods in both pose accuracy and physical plausibility.

Primary area:
generative models
Keywords:
multi-object, 3D, reconstruction, generation, reinforcement learning

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7o9oOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RAPP3R for physically plausible multi-object 3D reconstruction, combining inter-object attention, spatial encoding, and reinforcement learning. While the method improves reconstruction quality, its contributions are largely incremental combinations of existing techniques, and the experimental evaluation remains insufficient to fully support its claims of physical plausibility.

Critical strengths

The paper addresses a meaningful problem and demonstrates improvements in reconstruction accuracy and physical plausibility through its proposed framework.

Critical weaknesses
  1. Limited methodological novelty. The proposed framework largely combines existing techniques. Gated Inter-Object Attention integrates the inter-object attention mechanism from MIDI [1] with standard zero-initialized gating. Similarly, the RL formulation primarily adapts existing Flow-GRPO/GDPO techniques, with limited fundamental innovation in the learning algorithm.

  2. Insufficient justification of Volume-RoPE. The improvements attributed to Volume-RoPE may partly arise from extent-dependent attention scaling rather than more effective spatial relationship modeling. The current experiments do not sufficiently isolate these effects.

  3. Lack of competitive architectural baselines. The evaluation lacks simple yet competitive alternatives to Volume-RoPE, such as Center-RoPE with bounding-box size embeddings, Center-RoPE with learned extent-dependent attention bias, and 3D relative position bias. These comparisons are necessary to determine whether the proposed sinc-based formulation offers meaningful advantages over simpler spatial encoding strategies.

  4. Limited evaluation of physical plausibility. The reported Collision, Floating, and Ground metrics primarily measure geometric consistency rather than actual physical stability. Following simulation-based evaluations in REST3D [2], the authors should consider reporting stability under gravity, post-simulation object displacement, and the proportion of physically stable reconstructions.

References

[1] Huang et al., MIDI: Multi-Instance Diffusion for Single Image to 3D Scene Generation. CVPR, 2025.

[2] Ma et al., REST3D: Reconstructing Physically Stable 3D Scenes from a Single Image. 2026.

Questions for authors
  1. Have the authors tested simpler alternatives to Volume-RoPE, such as Center-RoPE with bounding-box size embeddings or learned extent bias?

  2. Can the physical stability of the reconstructed scenes be evaluated through simulation, using metrics similar to those in REST3D?

  3. How does the method compare with physics-based optimization approaches in terms of reconstruction quality and inference time?

5 comments

We appreciate the reviewer’s valuable feedback. Here are the responses to your specific concerns that are not addressed in Common Response.

1. Novelty

Please refer to Common Response 1.

2. Isolation of Volume-RoPE and Extent-Dependent Attention Scaling

If what the reviewer calls “extent-dependent attention scaling” refers to the sinc factor in Volume-RoPE, we would mention that the scaling itself is a core part of Volume-RoPE and cannot be separated from “spatial relationship modeling.” As shown in Figure 2 and Equation 1, the scaling term is mathematically equivalent to integrating (continuously averaging) the RoPE between a pair of objects. Using sinc is not an arbitrary or heuristic scaling - it is derived mathematically by considering the spatial and volumetric relationship between two objects. Therefore, the scaling is our “spatial relationship modeling,” and the improvements come from the scaling because it is the core of the proposed idea.

3. Alternate Approaches for Volume-RoPE

ADD-S↓[email protected]↑Collision↓Floating↓Ground↓
Center-RoPE + bbox size embedding0.05400.966112.200.9010.29
3D relative position bias0.06450.95957.761.678.39
Volume-RoPE (Ours)0.06850.95565.011.5011.20

Thank you for suggesting promising alternative approaches. We have tested alternative options and reported the results in the table. We are afraid that we excluded Center-RoPE with learned attention bias because it is not suitable for FlashAttention (FlashAttention computes QK multiplication and softmax at the same time for efficiency, but attention bias should be added between operations), takes a long time in RL sampling, and due to limited computational resources and the rebuttal period.

As shown in the table, the bbox-size embedding exhibits a deterioration in collision score compared to the SAM3D baseline. 3D relative position bias has more collisions than ours and is has even worse floating score compared to SAM3D.

Please note that attention biases are not suitable for FlashAttention. Therefore, RoPE-based approaches can benefit from FlashAttention more than the suggested attention-bias approaches. In addition, for bbox size embedding, extra MLPs and trainable embeddings are needed, increasing training and inference cost.

4. Physics Stability

Please refer to Common Response 4.

5. Quality and Time Compared to Optimization

ADD-S↓[email protected]↑Collision↓Floating↓Ground↓Time↓
SAM3D0.08180.745518.331.7417.4460.22
REST3D0.16170.50910.003.3427.241046.83
Ours0.06610.89097.781.9711.3163.22

(inference time was measured on RTX Pro 5000)

Here, we report quality and inference time comparison to physics-based optimization method, REST3D. We note that other methods (e.g. SPARCS, ϕ\phi-scene, etc.) mentioned in the paper did not release their source code so that we could not measure their inference time as the implementation details are unavailable. We used one sample (first frame) for each 12 scenes in YCB-V due to the limited rebuttal period.

As can be seen in the inference time, our approach requires similar time to the pre-trained model. On the other hand, REST3D required about 17 times longer time, with 977.45 seconds (among 1046.83 seconds) consumed at post-optimization. Moreover, despite longer inference time, optimization often produces bad results especially when there are multiple local minima (e.g. separated two collided objects in a wrong direction). Therefore, our RAPP3R outperformed REST3D in all the metrics except for collision which can be perfectly solved through optimization.

We additionally inform that PICASSO spent 2551.35 seconds per scene in the same 12 samples with (full 900 samples) quality comparison in the paper.

Thank you for the detailed responses. While I appreciate the practical improvements and the efficiency advantage over optimization-based methods, my main concerns remain insufficiently addressed. The mathematical derivation of volume-RoPE explains the origin of the sinc factor but does not establish whether the improvements arise from meaningful spatial relationship modeling or simply from extent-dependent attention modulation. The additional results show that Center-RoPE with bounding-box embeddings outperforms Volume-RoPE on four of five metrics, while standard Center-RoPE achieves fewer collisions. These results demonstrate the stronger competitiveness of simpler approaches. Compatibility with FlashAttention is a practical benefit but does not establish its methodological superiority. The comparison with REST3D demonstrates a compelling efficiency advantage. However, geometric consistency metrics alone cannot establish physical stability, which is central to the paper's claims. Overall, while I acknowledge the practical value of the proposed framework, the additional evidence remains insufficient to establish its core methodological contributions. Therefore, I am inclined to maintain my original score.

Reviewer_tSSXOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper addresses physically plausible multi-object 3D scene reconstruction from a single image, where independently generated objects often collide, float, or penetrate the ground. Prior work either adds inter-object attention to the generator or corrects physical violations at inference time through rejection sampling or scene-specific optimization. The former does not enforce physics, and the latter requires extra computation per scene and does not improve the generator itself. The authors fine-tune SAM3D with gated inter-object attention initialized to the pretrained behavior, a Volume-RoPE positional encoding that captures object position and extent, and reinforcement learning with collision, floating, free-space, and ground-truth pose rewards. The key difference is Object-wise GDPO, which assigns each object its own normalized advantage instead of a shared scene-level one, and the method is validated on YCB-V with ablations over attention, positional encoding, and advantage granularity.

Critical strengths
  1. The object-wise advantage formulation addresses a real flaw in scene-level RL for multi-object generation, where correctly placed objects can be penalized and wrongly placed ones rewarded. The ablation in Table 4 supports this: object-wise GDPO halves the collision ratio while the scene-level variant increases it relative to the pretrained model. This is the main reason I view the contribution as more than a reward-engineering exercise.

  2. The design preserves the pretrained model at initialization through zero-initialized gating, and the ablation shows this matters: naive inter-object attention collapses performance, while gated attention with Volume-RoPE recovers and improves it. This makes the approach practical for adapting existing joint shape-and-pose generators with little data, which is consequential given the method is trained on only 500 scenes.

Critical weaknesses

Weaknesses:

  1. All quantitative results come from a single benchmark of 12 tabletop scenes, so the generality of the method, especially its synthetic-to-real transfer claim, rests on narrow evidence; two of the three baselines (MIDI, RecGen) were designed and evaluated on indoor or differently structured scenes, which makes their weaker numbers hard to attribute to the proposed method rather than domain mismatch.

  2. The contribution of reinforcement learning itself is not isolated: the training objective includes a GT pose reward and an SFT loss, and the shape latent is fixed near ground truth, yet no SFT-only baseline is reported; the closest competing method, Scene-GRPO from PhyMix, is only approximated through the authors’ own scene-level variant rather than compared directly. Several results contradict the “outperforms on most metrics” framing and are not discussed: ground displacement is worse than SAM3D in Table 1, scene-level GDPO beats object-wise GDPO on ADD-S and floating in Table 4, and Center-RoPE beats Volume-RoPE on collision and ground in Table 3.

  3. The technical novelty is incremental: gated inter-object attention combines MIDI’s attention with zero-init gating from LLaMA-Adapter, object-wise GDPO adds an object index to GDPO’s normalization, and the reward terms largely follow PICASSO and SPARCS; Volume-RoPE is the most original piece but is a small positional-encoding variant.

Minor: no variance or multiple runs are reported although RL fine-tuning on 500 scenes is likely noisy and several gaps are small (ADD-S 0.0685 vs 0.0697); the norm-preservation violation of Volume-RoPE and the 5% collision threshold are asserted without any supporting experiment.

Questions for authors

Clarifications:

  1. Were MIDI and RecGen run with input normalization equivalent to the object-wise setting used for SAM3D in Appendix A.5? That setting alone improves SAM3D more than your method does, so it determines whether Table 1 is a fair comparison.

  2. What does an SFT-only variant without physical rewards achieve on the same metrics?

  3. At test time, is the shape latent generated or taken from ground truth, given that training fixes it near ground truth?

  4. Why does ground displacement get worse than SAM3D despite a dedicated ground reward?

Suggestion:

The floating distance of RAPP3R rises sharply after PICASSO refinement in Table 2. An analysis of why the learned prior conflicts with inference-time correction would strengthen the paper.

3 comments

We appreciate the reviewer’s valuable feedback. Here are the responses to the specific concerns that are not addressed in the Common Response.

1. Evaluation Dataset

ADD-S↓[email protected]↑Collision↓Ground↓
MIDI0.07270.8712‾\underline{0.8712}2.840.0236
RecGen0.0600‾\underline{0.0600}0.88363.53‾\underline{3.53}0.0335‾\underline{0.0335}
SAM3D0.06440.84598.230.0547
Ours0.05880.87043.880.0440
CD-S↓F-Score-S↑CD-O↓F-Score-O↑IoU-B↑
MIDI0.00950.91990.0216‾\underline{0.0216}0.90020.5637
RecGen0.01920.88880.01930.88730.5163‾\underline{0.5163}
SAM3D0.02240.88230.02190.87020.4512
Ours0.0146‾\underline{0.0146}0.8971‾\underline{0.8971}0.01520.8948‾\underline{0.8948}0.4983

We evaluated our method and baselines on 3D-Front, where MIDI was trained and evaluated. Scene-level ICP was used after all methods, following the MIDI evaluation pipeline.

As shown in the table, our method outperforms SAM3D on the 3D-Front test dataset (1000 scenes) using the same metrics as in our paper, except for floating. (We excluded floating scores because GT 3D-Front scenes include many floating objects, such as a framed picture on the wall, since it was not designed for physics simulation.) Given SAM3D's weak performance on 3D-Front, our RAPP3R achieved scores better than or near MIDI and RecGen, even though indoor scenes were not used in our training.

We also report metrics used in the MIDI paper. Across these metrics, our approach achieved the best or second-best score in 4/5 metrics, even though it was not trained with indoor samples (please note that MIDI was trained directly using 3D-Front). These results demonstrate the generalizability of our training pipeline.

2. Contribution of Physics RL

Please refer to Common Response 2.

3. Direct Comparison with PhyMix

First, we clarify that the PhyMix paper largely changed after their revision, which was unavailable at the time of our research and submission. Please refer to their v1 paper on arXiv.

We could not directly use PhyMix as a baseline because the code or model checkpoints were not released, and reproducing their results would require a massive effort in hyperparameter tuning for reinforcement learning and physical simulation.

The purpose of Table 4 is not to compare our results with an approximation of PhyMix. Its purpose is to emphasize why scene-level rewards can be a problem and to show the effectiveness of Object-Wise GDPO. Therefore, we should match all settings except how finely we apply GDPO. We mentioned PhyMix as an example of a coarse (scene-level) application of reinforcement learning rather than a method to be compared.

4. Trade-Off in Metrics

Please refer to Common Response 3.

5. Novelty

Please refer to Common Response 1.

6. Input Normalization

When (original) SAM3D receives a point map as input, it first normalizes the coordinates and then converts them to an embedding. The default SAM3D repository uses the scale of the entire scene for normalization. However, we found that using each object's scale works better in our setup, as explained in Section A.5. We emphasize that this input normalization is not a new feature or functionality but was originally part of SAM3D. We also clarify that the coordinate normalization in Section A.5 is a concept irrelevant to MIDI, RecGen, or general 3D reconstruction, but rather a detail change of SAM3D. We just reported this to note that we are not using the same setting as the original SAM3D.

7. Shape Latent

The shape latents are generated at test time, unlike fixing the ground truth during training.

8. Noise in Fine-Tuning and ADD-S

We agree that RL fine-tuning can possibly be noisy. However, looking at the collision and floating metrics, which we focus on, our approach reduces those errors by more than half compared to SAM3D. The gap in [email protected] (about 6%) also demonstrates the consistency of our method. Furthermore, consistent and superior performance in 3D-Front and physics stability over SAM3D shows that our performance gain is more than noise.

9. Norm-Preservation Violation

We would like to clarify that the norm-preservation violation is not an assertion. It simply follows from a mathematical property of the sinc function: it is not 1 almost everywhere. Therefore, we believe that a supporting experiment, which the reviewer is asking for, is irrelevant at this point.

10. Collision Threshold

Between two objects in contact, there can be a small intersection in meshes according to the representation, even in 3D scenes in practice because of the discreteness of computers, which is also not a problem in simulation. Therefore, if collisions between meshes are tested without modification, GT scenes would also be evaluated to show collisions. Furthermore, scenes with floating objects will get a better score than perfectly generated scenes. To address this issue in the metric, we shrink every object by 5% to allow contact between objects without severe penetration. The choice of the proportion “5” is just an arbitrary small number for thresholding.

We kindly remind the reviewer that the end of the author-reviewer discussion period is November 18th, AoE.

We first encourage the reviewer to check our rebuttal. If there remains any concerns that are not addressed, please let us know. Otherwise, we hope that the reviewer would possibly update the score.

Reviewer_se6wOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents RAPP3R, a framework for improving the physical plausibility of multi-object 3D reconstruction from a single view, using an image, depth map, and object masks as inputs. Building on SAM3D, the method introduces gated inter-object attention to exchange information across objects while preserving the pretrained model’s behavior at initialization. It also introduces Volume-RoPE, which incorporates object positions and spatial extents into attention through a bounding-volume-based positional encoding.

The pose-estimation component is adapted using Object-wise GDPO, which assigns separate advantages to individual objects. Training combines physical and geometric rewards, a ground-truth pose reward, and a supervised fine-tuning loss. Experiments on YCB-V, following adaptation on synthetic FoundationPose scenes, show improvements over the evaluated generative baselines on most reported metrics, with particularly substantial reductions in collisions and floating. Architectural and training ablations provide evidence for the proposed components, while also revealing trade-offs among alignment and physical metrics.

Critical strengths
  1. Meaningful improvements in the physical plausibility of reconstructed scenes. The method addresses a practical limitation of independently generated objects: plausible individual reconstructions can still produce inconsistent scene arrangements. Compared with the stronger, object-normalized SAM3D baseline, RAPP3R reduces the collision ratio from 10.24% to 5.01% and floating distance from 3.35 mm to 1.50 mm, while improving [email protected] from 0.8962 to 0.9556 (Table 1). These are substantial improvements on important aspects of scene reconstruction, and the use of synthetic adaptation data followed by real-world evaluation adds practical value.

  2. Coherent technical design supported by informative component comparisons. The combination of gated interaction, extent-aware positional encoding, and object-specific advantages is well motivated. Table 3 demonstrates the importance of spatial encoding for the tested inter-object attention mechanism, with Volume-RoPE providing the best alignment results among the architectural variants. Table 4 provides controlled evidence that object-wise advantages substantially reduce collisions compared with scene-level GDPO. Although the improvements involve trade-offs, these comparisons help explain the behavior of the proposed framework.

Critical weaknesses
  1. The methodological contribution is incremental. The broader motivation of incorporating physics into reconstruction is established in prior work such as PhyRecon, and inter-object attention has already been explored in methods such as MIDI. RAPP3R makes meaningful adaptations through gated attention, Volume-RoPE, and object-wise GDPO, but the overall contribution primarily lies in extending and combining established ideas within SAM3D. I therefore view the work as a useful incremental advance with limited conceptual novelty.

  2. The additional benefit of physics-guided reinforcement learning is not fully isolated. The final objective includes a supervised fine-tuning loss (Eq. 14; Appendix A.3), and the reward function includes direct ground-truth pose supervision (Eq. 5). However, the experiments do not include a matched SFT-only baseline using the same architecture, training data, trainable parameters, and comparable training budget. Consequently, it remains unclear how much of the improvement requires physics-guided RL beyond supervised adaptation. The existing ablations support architectural choices and object-wise advantages within the combined training framework, but they do not resolve this attribution question.

Questions for authors
  1. Additional benefit over supervised fine-tuning. Could the authors report a matched SFT-only baseline using the proposed architecture, the same training data and trainable parameters, and a comparable training budget? In particular, how does the trajectory-based supervised objective in Appendix A.3 perform without the reward-based policy update?

  2. Contribution of the physical rewards. Could the authors separate the contribution of the physical reward terms from that of the ground-truth pose reward? A comparison retaining the supervised loss and pose reward while removing the inter-object, ground, and depth rewards would help establish which improvements are specifically attributable to the physical objectives.

4 comments

We appreciate the reviewer’s valuable feedback. We believe that all concerns raised by reviewer se6w are addressed in the Common Response.

1. Novelty

Please refer to Common Response 1.

2. Contribution of Physics RL

Please refer to Common Response 2.

Thank you for the response and additional experiments. The new ablations clarify the contribution of the physical rewards. SFT-only already achieves similar reconstruction accuracy, while the full method provides clearer benefits in reducing collisions and improving stability in simulation. These results strengthen the evidence for the method’s practical value, although the gains in reconstruction accuracy are modest and I continue to view the methodological contribution as incremental. I therefore maintain my rating of 3 (weak acceptance).

We sincerely thank the reviewer for checking out our rebuttal.

Regarding the improvement of ADD-S, where the reviewer is further raising a concern, we first emphasize that our primary focus is on physical plausibility. Nevertheless, even though our method improves physical plausibility and considering the metrics trade-off (Common Response 3), our approach further consistently improves ADD-S, as demonstrated by 6% increment in accuracy. We additionally provide a comparison to SAM3D on full (1000) samples of 3D-Front test set that also shows enhancement in ADD-S, which was originally conducted for reviewer tSSX.

ADD-S↓[email protected]↑Collision↓Ground↓
SAM3D0.06440.84598.230.0547
Ours0.05880.87043.880.0440

(Floating is omitted since 3D-Front includes floating GT objects, such as a frame of picture on the wall, because the dataset was not originally designed considering physics.)

We also appreciate the reviewer for recognizing our approach’s strength. The reviewer highlighted our strengths as:

  • meaningful improvement in physics plausibility.
  • transfer from synthetic to real-world data.
  • coherent technical design with proper ablations.

As the reviewer noted, we believe our method brings improvement to the physically plausible 3D scene reconstruction through coherent and well-motivated technical components.

Comments on the paper

4 comments
AuthorsOpenReview ↗

Common Response [1/4]

We thank the reviewers for their constructive and insightful reviews. The reviewers noted that our proposed RAPP3R addresses a meaningful problem and offers improvements and contributions, including object-wise GDPO and Volume-RoPE, which are reasonably designed.

Here are the responses to common concerns: (1) novelty of our method (2) contribution of physics RL is not demonstrated through an ablation study.

Furthermore, to help explain the second concern, we leave our responses to the reviews of reviewer tSSX and 7o9o here: (3) trade-off between metrics (4) physics stability, respectively.

1. Novelty (se6w, tSSX, 7o9o)

We organize our contributions as follows:

  • Volume-RoPE. While Volume-RoPE may seem like simple, arbitrary scaling at first glance, we highlight that its new derivation requires mathematical insight. To the best of our knowledge, mathematically integrating rotary embeddings has not been addressed before, and we carefully designed it with continuity in 3D spaces in mind. Moreover, its simple form is an advantage of Volume-RoPE, as multiplying a diagonal matrix is computationally cheap. We also claim that Volume-RoPE is not just a small variation but an essential factor when applying inter-object attention. As shown in Table 3 and Common Response 4, Center-RoPE (direct application of RoPE) shows worse scores in pose accuracy (ADD-S), floating, and collision even compared with SAM3D.

  • Object-Wise GDPO. Using RL for multi-object reconstruction has been limitedly addressed by researchers yet, leaving many obstacles. For LLMs, image generation, or single object generation, conventional RL could directly be applied since they incorporate a single trajectory for a generation. On the other hand, in multi-object reconstruction, fine details of each object’s role is important, as explained in Section 3.2.2. However, multi-object reconstruction researchers have missed the granularity of objects in a scene, directly applying a coarse reward to a scene, including a more recent paper Scene-Designer [1]. Therefore, their reinforcement signal on each object may be weaker than object-wise reinforcing. We believe that our core contribution is in the insight on fine-grained reward decomposition rather than an indexing in the equation.

  • Physical-Plausibility Rewards. We acknowledge that our physical-plausibility rewards are largely influenced by previous optimization-based works (Yu et al., 2026; Huang et al., 2026b). However, the rewards are not just an application of physical losses in optimization. In optimization, due to the sum rule of differentiation, the gradient for each loss flows through backpropagation separately even though the losses are simply added. However, in reinforcement learning, it is difficult to adopt multiple objectives as the sum rule may not hold for rewards. Therefore, effective and detailed reward and regularizer design is needed, which we propose in the paper.

  • Gated Inter-Object Attention. We acknowledge that the gating mechanism is influenced by prior works (Zhang et al., 2024; Blattmann et al., 2023) as stated in line 168. However, we would like to emphasize the specificity of multi-object reconstruction. As it is much easier to train a generative model in a single-object setup, extending the single-object model to multi-object is important. Nevertheless, most of the prior works independently generate each object (SAM3D) or train multi-object components from scratch (SceneGen), missing the possibility of a gating mechanism. We also note that our other contribution, Volume-RoPE, is enabled by gated inter-object attention.

AuthorsOpenReview ↗

Common Response [2/4]

2. Contribution of Physics RL (se6w, tSSX)

ADD-S↓[email protected]↑Collision↓Floating↓Ground↓
SFT0.0690‾\underline{0.0690}0.927810.691.0811.86
GT-reward0.07040.9423‾\underline{0.9423}8.63‾\underline{8.63}6.639.05
Ours0.06850.95565.011.50‾\underline{1.50}11.20‾\underline{11.20}

Following feedback from reviewers se6w and tSSX, we compared our method with SFT-only (using LSFTL_{SFT} from Appendix A.3) and GT-reward-only (i.e., excluding physical-plausibility rewards) versions. Given their lower computational cost per training step, we trained them for 750 steps, which required similar training time, while our method was trained for only 500 steps. As a result, our method achieved the best scores on ADD-S, [email protected], and collision, and the second-best scores on the other metrics. While our approach is second-best on some metrics, the SFT-only version degrades on collision, and the GT-reward-only version degrades on ADD-S and floating compared to the SAM3D baseline in Table 1. Considering the trade-off between the metrics (in Common Response 3) and the first or second superiority across all metrics, we assert that our physics RL contributes to reconstruction quality and physical plausibility. In addition, superior physics stability can be observed in Common Response 4.

AuthorsOpenReview ↗

Common Response [3/4]

3. Trade-Off in Metrics (tSSX)

Our method does not reach first place in the ablation on some metrics because there are trade-offs between metrics. For instance, the output will exhibit less floating when an object penetrates another object below. On the other hand, if an object floats away, the collision score may be improved. REST3D [2] (Figure 9 in their paper) also reports a trade-off between geometric accuracy and physical plausibility. Due to the trade-off, it is possible but difficult to improve all the metrics, especially when the object geometries are imprecise. Therefore, it is important to consider multiple evaluation metrics improvement over the baseline (pre-trained model) collectively.

In Table 3, our Volume-RoPE uniquely improves ADD-S, with collision and floating scores improved over the pre-trained model. In contrast, Center-RoPE shows worse ADD-S and floating scores, which implies that the better collision score is attributable to floating objects.

For ground score, we acknowledge that our method is slightly worse than SAM3D. However, compared to other baselines such as “+RL+IOA” or Scene-Level GDPO in Table 4, the deterioration is small. As other scores for our method largely improve over SAM3D, we interpret the ground score as a trade-off metric.

We kindly request the reviewers check the additional physical stability scores, as they have less of a trade-off relationship with other metrics (the details follow in Common Response 4).

AuthorsOpenReview ↗

Common Response [4/4]

4. Physics Stability (7o9o)

Stable (%)↑Drift (mm)↓Lin Vel (m/s)↓Ang Vel (rad/s)↓
MIDI + ICP47.3553.330.40338.7947
RecGen81.9924.960.21517.1263
SAM3D89.6021.570.19974.8212
Ours91.0119.040.16373.8592

(An object is evaluated as stable when its drift is less than 5cm and 0.3rad.)

Although we did not explicitly enforce physical stability in our pipeline, we observed that our RAPP3R outperforms previous methods on simulation-based metrics on YCB-V designed similarly to REST3D, as reviewer 7o9o asked. We attribute these advances to improved physical plausibility, such as collision and floating, as violations of these properties may also degrade physical stability.

Stable (%)↑Drift (mm)↓Lin Vel (m/s)↓Ang Vel (rad/s)↓
SAM3D89.6021.570.19974.8212
+RL89.9223.030.17294.2923
+IOA28.0072.670.678610.3605
Center-RoPE89.5819.050.27954.9019
Volume-RoPE91.0119.040.16373.8592
Stable (%)↑Drift (mm)↓Lin Vel (m/s)↓Ang Vel (rad/s)↓
Scene-Level85.3328.340.20604.4171
Object-Wise91.0119.040.16373.8592
Stable (%)↑Drift (mm)↓Lin Vel (m/s)↓Ang Vel (rad/s)↓
SFT41.3846.440.36517.6000
GT-reward82.5725.000.32845.3486
Ours91.0119.040.16373.8592

In addition, we conducted a simulation-based ablation study. As a result, our proposed Volume-RoPE, Object-Wise GDPO, and Physics RL consistently performed best in all metrics. These results strengthen the claim of the physical plausibility of our method beyond the limitations of trade-offs in other methods.

References

[1] Liu et al., Scene-Designer: Learning Design Intelligence for Agentic 3D Scene Generation, arXiv 2610.23121, 2026.

[2] Ma et al., REST3D: Reconstructing physically stable 3d scenes from a single image. SIGGRAPH Asia, 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.