acceptodds
Under review as a conference paper at ICLR 2027

PAIR3D: Learning Physics-Aligned and Visually Consistent 3D Scene Layouts from Verifiable Rewards

Abstract

Modern 3D reconstruction can produce scenes that look right but behave wrong. Small errors in object layout or geometry may remain inconspicuous in the input view, yet become explicit under rigid-body dynamics through falling, toppling, interpenetration, or unstable contact responses. Recent methods use physical simulation to improve such reconstructions, but their scene-layout decisions are still largely worked out anew for each input through test-time search or prescribed diagnostic procedures. We ask whether this alignment behavior can instead be learned and reused. Our key observation is that the right action can be ambiguous even when its outcome is directly verifiable. We therefore introduce PAVER3D, which formulates physical scene alignment as sequential decision making with verifiable rewards. A multimodal relational policy reasons over current and target visual evidence, object states, and pairwise physical relations, and predicts objectlayout updates. Simulator outcomes train the policy with PPO, amortizing part of the per-scene optimization into reusable behavior while retaining a closed-loop simulator at deployment. In our current comparison with REST3D, PAVER3D improves physical stability more strongly, changes mIoU by +1.0% rather than −1.6398% relative to the initial reconstruction, and reduces measured alignment time from 42.23 to 4.50 seconds per scene. These results suggest that simulation can serve not only as a scene-specific optimizer, but also as a source of experience for learning how to align 3D scenes with physics.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Reinforcement Learning, Physically Compliant, Scene Reconstruction, Physics-Aligned

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_wroxOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Pair3D, a reinforcement-learning framework that learns a physics-aligned and visually consistent refinement of 3D scene layouts from verifiable rewards. Input: SAM3D generated scene Output: scene with plausible physics

Critical strengths
  1. Good research question with novelty.
  2. The reward design is reasonable.
  3. Demonstrated efficiency of the design.
Critical weaknesses
  1. The citation has the wrong title: "Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang style, et al. 3d-front: 3d houses ranked with objects. In IEEE/CVF International Conference on Computer Vision, 2021a." should be "3d-front: 3d furnished rooms with layouts and semantics"
  2. The citation has the wrong author and title: "Huan Fu, Rongfei Jia, Lin Gao, Mingli Song, Chao-Hui Shen style, et al. 3d-future: 3d furniture shaped with texture. International Journal of Computer Vision, 2021b" should be "Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, Dacheng Tao" and "3d-future: 3d furniture shape with texture".
  3. The evaluation metric coincides with the training reward, which favors PAIR3D over other baselines, and the evaluation metric is less restricted compared to the reward equation; why?
  4. Did not demonstrate the necessity of reinforcement learning; the author should compare with another simple strategy without learning, such as: REST3D with a simple floor-contact correction, put everything on the floor and resolve penetration between each object, then align with mask IoU.
  5. No in-the-wild comparison.
  6. Question about the necessity of the visual branch and relational branch, since in Table 2 ablation "Without vision branch" and "without relational branch" does not look a lot different than the full PAIR3D.
  7. The number of PAIR3D is different between Table 1 and 2; why?
Questions for authors

See weakness.

5 comments
AuthorsOpenReview ↗

Response to Reviewer #wrox (1/3)

Thank you for your careful review and for identifying the reference errors and questions about the evaluation. We have corrected the references, added comparisons and ablations, and clarified the experimental setup in the revised manuscript. We address your comments below.


W1/W2 — Corrections to the references

Thank you for catching these errors; we apologize for them. The errors were introduced when we converted our reference list into BibTeX format with a tool backed with AI function, and we did not check the converted entries against the published papers. The references themselves were selected by us, and both works are the datasets we use (Section 4.1); only the bibliographic metadata was wrong.

In the revision, we have corrected both entries, checked all references by hand against the publisher page or DBLP and found no other errors.

W3 — Training reward and evaluation criteria

The reward combines reductions in continuous physical, visual, and pose-prior costs with a step penalty and a success bonus. Training success requires physical acceptance F and a mask IoU at least as high as the initialization:

S(L)=F(L)∧[I(L)≥I0]S(\mathcal{L}) = F(\mathcal{L}) \land \left[ I(\mathcal{L}) \geq I_{0} \right]

Phys.-five reports the five physical conditions jointly, and mIoU reports visual agreement separately.

We did not explain the distinction between training acceptance and evaluation thresholds clearly enough in the original manuscript. We used looser evaluation thresholds because several baselines have very low success rates under the strict five-criterion conjunction; reporting only joint success can obscure the partial improvements they achieve. The looser thresholds help show these differences. We apply the same thresholds to all methods, without changing their output layouts.

The main evaluation uses 5 mm for Sup./Pen./RT and 5° for Tilt/RR. We also re-evaluated the same test layouts and residuals using the training thresholds: 0.5 mm for Sup./Pen., 2° for Tilt, 1.5 mm for RT, and 0.2° for RR.

Table 14 of the revised manuscript. Test-set pass rates (%) under the training acceptance thresholds.

\hline \text{Method} & \text{Sup.} & \text{Pen.} & \text{Tilt} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{SceneGen} & 0.0 & 90.0 & 1.0 & 0.0 & 0.0 & 0.0 \\\\ \text{SAM3D} & 1.0 & 87.0 & 29.0 & 9.0 & 11.0 & 1.0 \\\\ \text{REST3D} & 0.0 & 100.0 & 99.0 & 72.0 & 70.0 & 0.0 \\\\ \text{GPT-6} & 59.0 & 93.0 & 75.0 & 52.0 & 50.0 & 37.0 \\\\ \text{PAIR3D} & 100.0 & 100.0 & 99.0 & 83.0 & 77.0 & 75.0 \\\\ \hline \end{array}$$ PAIR3D reaches 75% Phys.-five under these tighter criteria, compared with 37% for GPT-6 Astra. Support and penetration pass in every PAIR3D scene. We also report the continuous errors below, which show the size of the residual errors rather than only whether they cross a threshold. Table 2 of the revised manuscript (selected rows). Mean per-scene maximum physical errors; lower is better. $$\begin{array}{lccccc} \hline \text{Method} & E_{\mathrm{sup}}\\;(\mathrm{mm}) & E_{\mathrm{pen}}\\;(\mathrm{mm}) & E_{\mathrm{tilt}}\\;({}^\circ) & E_{\mathrm{RT}}\\;(\mathrm{mm}) & E_{\mathrm{RR}}\\;({}^\circ) \\\\ \hline \text{SAM3D} & 39.43 & 6.40 & 5.29 & 13.92 & 3.63 \\\\ \text{REST3D} & 5.58 & 0 & 2.02 & 1.37 & 0.23 \\\\ \text{PAIR3D} & 0.03 & 0 & 0.78 & 1.29 & 0.27 \\\\ \hline \end{array}$$ PAIR3D has lower support, tilt, and translation errors, while REST3D has the lower mean rotation error. All methods are evaluated against the same physical criteria. The search comparisons also use PAIR3D's repair operators and verifier.
AuthorsOpenReview ↗

Response to Reviewer #wrox (2/3)

W4 — Comparison with non-learning methods

We added REST3D with fixed post-processing. It raises the support pass rate from 50% to 100% and Phys.-five from 48% to 96%. PAIR3D reaches 97% and improves mIoU by 1.62 points, with inference time reduced from 46.84 s to 4.57 s:

Table 13 of the revised manuscript (selected rows and columns). Effect of fixed post-processing.

\hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Sup.}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{REST3D-default} & 45.48 & 50.0 & 48.0 & 43.16 & 63{,}264.72 \\\\ \text{REST3D + fixed post-process} & 45.63 & 100.0 & 96.0 & 46.84 & 63{,}282.80 \\\\ \text{PAIR3D} & 47.25 & 100.0 & 97.0 & 4.57 & 24.90 \\\\ \hline \end{array}$$ We also compared Greedy and CEM using the same repair operators and physical/visual verifier as PAIR3D. Each search method uses 24 candidate rollouts plus initial/final checks: Table 3 of the revised manuscript (selected rows). Search with the same repair operators and verifier. $$\begin{array}{lcccc} \hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{Greedy (24 candidates)} & 46.46 & 19 & 11.68 & 26.0 \\\\ \text{CEM (3 × 8 candidates)} & 46.53 & 5 & 34.78 & 26.0 \\\\ \text{PAIR3D} & 47.25 & 97 & 4.57 & 24.9 \\\\ \hline \end{array}$$ With a comparable rollout budget, PAIR3D achieves substantially higher physical success and retains better image agreement. IT is seconds per scene and SC counts rollouts per scene. ### **W5 — Evaluation on real indoor scenes** We added comparisons between SAM3D and PAIR3D on 20 ScanNet reconstructions and 20 Replica reconstructions. The policy trained on 3D-F2Scene is transferred directly, without fine-tuning. ScanNet uses captured RGB images; Replica uses photorealistic renderings of scanned indoor environments. Table 5 of the revised manuscript (selected columns). Cross-dataset evaluation; all metrics are percentages. $$\begin{array}{llcccc} \hline \text{Dataset} & \text{Method} & \text{mIoU} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{ScanNet} & \text{SAM3D} & 51.96 & 0.0 & 0.0 & 0.0 \\\\ \text{ScanNet} & \text{PAIR3D} & 56.84 & 95.0 & 90.0 & 90.0 \\\\ \text{Replica} & \text{SAM3D} & 50.04 & 0.0 & 0.0 & 0.0 \\\\ \text{Replica} & \text{PAIR3D} & 53.01 & 80.0 & 75.0 & 70.0 \\\\ \hline \end{array}$$ Refinement raises mIoU on both datasets and brings Sup., Pen., and Tilt pass rates to 100%. Figures 4, 10, and 11 of the revised manuscript provide input-view and auxiliary-view comparisons of the seating, table, and cabinet arrangements.
AuthorsOpenReview ↗

Response to Reviewer #wrox (3/3)

W6 — Visual and relational ablations

We repeated the ablation with three training seeds. The original ablation Table 2 is now Table 4; all metrics and configurations are reproduced below:

Table 4 of the revised manuscript. Ablation results on the 3D-F2Scene validation dataset. Values are mean ± standard deviation over three training seeds. Bold denotes the best mean for each metric.

\text{Metric} & \text{PAIR3D} & \begin{array}{c}\text{w/o Scale}\\\\\text{Adjustment}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Reward}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Branch}\end{array} & \begin{array}{c}\text{w/o Relational}\\\\\text{Branch}\end{array}\\\\ \hline \text{mIoU}\\;(\\%)\uparrow & \mathbf{44.49}\pm0.07 & 43.57\pm0.24 & 42.40\pm0.10 & 43.35\pm0.31 & 43.44\pm0.34\\\\ \text{CD (mm)}\downarrow & \mathbf{316.79}\pm0.42 & 317.30\pm0.47 & 319.05\pm0.12 & 317.11\pm0.65 & 316.93\pm0.52\\\\ \text{F-Score}\\;(\\%)\uparrow & 12.05\pm0.04 & 12.02\pm0.08 & 11.97\pm0.07 & 12.06\pm0.10 & \mathbf{12.09}\pm0.05\\\\ \hline \text{Sup.}\\;(\\%)\uparrow & \mathbf{99.33}\pm0.58 & \mathbf{99.33}\pm0.58 & 98.00\pm2.00 & 99.00\pm1.00 & 98.33\pm0.58\\\\ \text{Pen.}\\;(\\%)\uparrow & 99.00\pm1.00 & \mathbf{99.67}\pm0.58 & 98.00\pm1.00 & 98.67\pm0.58 & 98.00\pm1.00\\\\ \text{Tilt}\\;(\\%)\uparrow & \mathbf{100.00}\pm0.00 & 99.67\pm0.58 & 98.67\pm2.31 & 99.67\pm0.58 & \mathbf{100.00}\pm0.00\\\\ \text{RT}\\;(\\%)\uparrow & 98.33\pm1.53 & 98.33\pm1.53 & 95.67\pm3.51 & \mathbf{98.67}\pm0.58 & 97.67\pm0.58\\\\ \text{RR}\\;(\\%)\uparrow & 99.33\pm0.58 & \mathbf{99.67}\pm0.58 & 99.00\pm1.00 & 99.00\pm0.00 & 99.00\pm0.00\\\\ \text{Phys.-five}\\;(\\%)\uparrow & \mathbf{98.33}\pm1.53 & 98.00\pm2.00 & 93.00\pm6.24 & \mathbf{98.33}\pm0.58 & 97.00\pm1.00\\\\ \end{array}$$ The visual branch improves mIoU by 1.14 points and reduces CD from 317.11 to 316.79 mm, while mean Phys.-five stays at 98.33%. Its main role is matching the target masks and depths. The node state already supplies physical diagnostics, so support, penetration, tilt, and motion information remains available without the visual branch. The relational branch adds relative geometry and contact information. It improves mIoU by 1.05 points and Phys.-five by 1.33 points, with a 1.00-point increase in both support and penetration pass rates. RT and RR increase by 0.66 and 0.33 points, respectively, while Tilt is unchanged. Figure 7 of the revised manuscript illustrates the interacting object constraints represented by this branch. F-Score is 12.05% for PAIR3D, 12.06% without the visual branch, and 12.09% without the relational branch. The full model achieves the highest mean mIoU, the lowest mean CD, and ties for the highest mean Phys.-five. ### **W7 — Test and validation results** Thank you for pointing this out. The difference in PAIR3D's performance between Table 1 and the original Table 2 (now Table 4 in the revised manuscript) is due to the use of different evaluation splits. Table 1 reports results on the test set (100 scenes) for comparison with existing methods, whereas the ablation table reports results on the validation set (100 separate scenes). We conducted ablation studies on the validation set to analyze the contributions of individual components during model development, while keeping the test set independent of model selection. The revised Table 4 also reports the mean and standard deviation over three training seeds. We have clarified the evaluation splits in Section 4.1 and in the captions of Tables 1 and 4.
Reviewer_uQaAOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper tackles the task of physically plausible single image 3D reconstruction, which has great potentials in sim-to-real and robotics applications. Instead of baseline methods that search in a large simulation space for physic correctness, which is extremely expensive, PAIR3D proposed by the paper learns a lightweight policy, that operates on the scene graph and can transfer to new scenes with a small amount of simulation call in the inference time. The extensive experiment results show the improvement from the proposed method and its components, and also its effectivenss comparing to the baselines.

Critical strengths

The motivation of the paper and the proposed system is inspiring. Turning per-scene heavy search into lightweight learnable policy is a scalable and valid way. A resuable physical policy runs with simulation in-the-loop sounds like the correct way for single image physically plausible 3D reconstruction, and replacing heavy search with learnt policy is good. Also, extensive experiments show strong improvement among other baselines, including both the accuracy, correctness, and the time consumption.

Critical weaknesses

The main weakness of the paper from my perspective is that, the evaluation is kind of unclear to explain the gains. The paper didn't clearly explain and list the action space of the learnt policy, i.e. the "actions per object and per pair". And also the ablation results show little performance changes when removing the vision and relational branches. It's also valid if the authors can show the REST3D baseline with the similar budget of simulation calls, e.g. 25 times, to more clearly show the gain from decreasing the simulation calls.

Another weakness is the soundness and presentation of the paper. The method section is very generic, and it'd be very helpful if the authors can provide more qualitative results on step-by-step refinement on each scene with clear examples. Also the experiment results should also show at least some qualitative results on in-the-wild images where SAM3D can work, and also there are no comparison to SimuScene baseline method.

Questions for authors
  1. What exactly are the actions? The definitions and formulations, and implementations of those actions are needed.
  2. Are there learning curves of the model training, and how does performance scale with the number of training scenes?
  3. How is a “simulation call” defined for each method? Some implementation details are needed.
  4. More details and results for GPT-6-Astra baseline is needed.
  5. Could the authors show the comparison to the SimuScene baseline, and if not, for what specific reason.
  6. Could the authors provide some qualitative results on in-the-wild scenes?

I'd like to raise my ratings if most of my questions can be solved or answered.

5 comments
AuthorsOpenReview ↗

Response to Reviewer #uQaA (1/3)

Thank you for your thoughtful comments on the method and evaluation. We have carefully considered your suggestions, clarified the action definitions, and added baseline comparisons and qualitative results in the revised manuscript. We address each of your comments below.


W1/Q1 — Object and pair actions

Each object has 18 actions: six translations along the signed world axes, six rotations about world-aligned axes through its local origin, and six increases/decreases of the three local scale components. The updates are:

t′=t+ε,d,ea\mathbf{t}' = \mathbf{t} + \varepsilon\\, d\\, \mathbf{e}_{a}

R′=exp⁡(εθ,[ea]×)R\mathbf{R}' = \exp\left(\varepsilon\theta\\,[\mathbf{e}_{a}]_{\times}\right)\mathbf{R}

sb′=q,sb,sc′=sc(c≠b)s_{b}' = q\\, s_{b}, \qquad s_{c}' = s_{c} \quad (c \neq b)

Here a and b select x, y, or z; ε is +1 or −1; d and θ are positive magnitudes; and q is a positive scale factor. The axis vector is a unit vector, the bracket with a cross subscript is its skew-symmetric cross-product matrix, and exp is the matrix exponential. Rotation and scaling preserve the world position of the local origin.

Each unordered object pair has eight horizontal translation actions: select either endpoint and then +x, -x, +y, or -y. Only the selected object moves. Its displacement depends on the continuous policy output, pairwise penetration depth, and contact direction.

\\#\text{actions} = 18N + 8|E| + 1

For N objects and unordered pair set E, the extra action is STOP. A unified categorical distribution selects one action per step. Shared linear heads predict the logits and Gaussian means, invalid or padded actions are masked, and rendering and collision geometry are synchronized after each update. Section 3.4 and Appendices A.3–A.4 now include these definitions.

W2 — Visual and relational branches

Both branches improve image alignment. The visual branch raises mIoU by 1.14 points, and the relational branch raises it by 1.05 points. We repeated the ablation with three training seeds:

Table 4 of the revised manuscript. Ablation results on the 3D-F2Scene validation dataset. Values are mean ± standard deviation over three training seeds. Bold denotes the best mean for each metric.

\text{Metric} & \text{PAIR3D} & \begin{array}{c}\text{w/o Scale}\\\\\text{Adjustment}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Reward}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Branch}\end{array} & \begin{array}{c}\text{w/o Relational}\\\\\text{Branch}\end{array}\\\\ \hline \text{mIoU}\\;(\\%)\uparrow & \mathbf{44.49}\pm0.07 & 43.57\pm0.24 & 42.40\pm0.10 & 43.35\pm0.31 & 43.44\pm0.34\\\\ \text{CD (mm)}\downarrow & \mathbf{316.79}\pm0.42 & 317.30\pm0.47 & 319.05\pm0.12 & 317.11\pm0.65 & 316.93\pm0.52\\\\ \text{F-Score}\\;(\\%)\uparrow & 12.05\pm0.04 & 12.02\pm0.08 & 11.97\pm0.07 & 12.06\pm0.10 & \mathbf{12.09}\pm0.05\\\\ \hline \text{Sup.}\\;(\\%)\uparrow & \mathbf{99.33}\pm0.58 & \mathbf{99.33}\pm0.58 & 98.00\pm2.00 & 99.00\pm1.00 & 98.33\pm0.58\\\\ \text{Pen.}\\;(\\%)\uparrow & 99.00\pm1.00 & \mathbf{99.67}\pm0.58 & 98.00\pm1.00 & 98.67\pm0.58 & 98.00\pm1.00\\\\ \text{Tilt}\\;(\\%)\uparrow & \mathbf{100.00}\pm0.00 & 99.67\pm0.58 & 98.67\pm2.31 & 99.67\pm0.58 & \mathbf{100.00}\pm0.00\\\\ \text{RT}\\;(\\%)\uparrow & 98.33\pm1.53 & 98.33\pm1.53 & 95.67\pm3.51 & \mathbf{98.67}\pm0.58 & 97.67\pm0.58\\\\ \text{RR}\\;(\\%)\uparrow & 99.33\pm0.58 & \mathbf{99.67}\pm0.58 & 99.00\pm1.00 & 99.00\pm0.00 & 99.00\pm0.00\\\\ \text{Phys.-five}\\;(\\%)\uparrow & \mathbf{98.33}\pm1.53 & 98.00\pm2.00 & 93.00\pm6.24 & \mathbf{98.33}\pm0.58 & 97.00\pm1.00\\\\ \end{array}$$ The node state already provides support, penetration, tilt, and motion measurements. Physical diagnostics therefore remain available when the visual branch is removed, and mean Phys.-five stays at 98.33%. The visual branch contributes target-versus-rendered mask and depth information for observation matching. The relational branch adds relative geometry and contact information. Removing it lowers both support and penetration pass rates by 1.00 point and Phys.-five by 1.33 points. Figure 7 of the revised manuscript gives an example of the coupled object corrections represented by these features.
AuthorsOpenReview ↗

Response to Reviewer #uQaA (2/3)

W3 — REST3D under a restricted simulation budget

We added REST3D-B24 with 24 online trajectories: eight for initialization and 16 for candidate evaluation. IT measures inference time per scene and SC counts physical rollouts per scene.

Table 13 of the revised manuscript (selected rows and columns). REST3D under default and restricted budgets.

\hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{REST3D-default} & 45.48 & 48.0 & 43.16 & 63{,}264.72 \\\\ \text{REST3D-B24} & 41.76 & 31.0 & 37.23 & 24.00 \\\\ \text{PAIR3D} & 47.25 & 97.0 & 4.57 & 24.90 \\\\ \hline \end{array}$$ At a comparable rollout budget, PAIR3D improves mIoU from 41.76% to 47.25% and Phys.-five from 31% to 97%, while reducing inference time from 37.23 s to 4.57 s. Every candidate in a parallel batch contributes one rollout to SC. ### **W4 — Intermediate refinement steps** We added the refinement sequence in Figure 7 of the revised manuscript, with a fixed main camera and local overhead views. At step 3, the policy corrects a tilted, floating cabinet. It then moves the sofa to reduce penetration at step 5 and lowers the table at step 6. Further descent causes floor penetration at step 7, which is corrected by step 11. The policy returns to the sofa at step 15 and removes the remaining overlap by step 16. Changing the table height affects both ground support and its contact with the sofa. The sequence shows the policy switching targets and revisiting objects as these constraints change. ### **W5/Q6 — Results on real indoor scenes** We added input-view and auxiliary-view comparisons on ScanNet and Replica in Figures 4, 10, and 11 of the revised manuscript. The examples show corrections to tilted seating, tables, and cabinets while preserving the overall arrangement. The quantitative evaluation uses 20 reconstructions per dataset, with the 3D-F2Scene policy applied without fine-tuning. ScanNet uses captured RGB images; Replica uses photorealistic renderings of scanned interiors. Table 5 of the revised manuscript (selected columns). Cross-dataset evaluation; all metrics are percentages. $$\begin{array}{llcccc} \hline \text{Dataset} & \text{Method} & \text{mIoU} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{ScanNet} & \text{SAM3D} & 51.96 & 0.0 & 0.0 & 0.0 \\\\ \text{ScanNet} & \text{PAIR3D} & 56.84 & 95.0 & 90.0 & 90.0 \\\\ \text{Replica} & \text{SAM3D} & 50.04 & 0.0 & 0.0 & 0.0 \\\\ \text{Replica} & \text{PAIR3D} & 53.01 & 80.0 & 75.0 & 70.0 \\\\ \hline \end{array}$$ Support, penetration, and tilt pass rates reach 100% on both datasets after refinement. The remaining failures arise from residual translation or rotation. ### **W6/Q5 — Comparison with SimuScene** At our ICLR 2027 submission in mid-September 2026, SimuScene was an arXiv preprint. It has since been listed as accepted at NeurIPS 2026, but its [GitHub repository](https://github.com/snuvclab/SimuScene) has not yet released the code. The method involves constructing a new dataset and fine-tuning SAM3D. During the rebuttal period, we contacted the authors by email to request the code and assistance with reproduction. We are awaiting their response and will add the comparison as soon as the required materials are available. ### **Q2 — Learning curves and training-set size** Figure 6 of the revised manuscript shows the learning curves over 20,000 environment steps, using 1,000-step bins and the mean and standard deviation across three seeds. Support improves early in training; motion stability improves more gradually. Phys.-five rises fastest during the first half and reaches its highest values later. We also varied the number of training scenes. Table 16 of the revised manuscript. Validation results (%) by number of training scenes; mean ± SD over three seeds. $$\begin{array}{lccccc} \hline \text{Metric} & 600 & 800 & 1{,}200 & 1{,}400 & 1{,}600 \\\\ \hline \text{mIoU} & 43.94 \pm 0.35 & 43.98 \pm 0.22 & 44.20 \pm 0.08 & 44.03 \pm 0.37 & 44.43 \pm 0.21 \\\\ \text{Phys.-five} & 72.67 \pm 6.11 & 76.00 \pm 4.58 & 79.00 \pm 4.58 & 80.67 \pm 3.21 & 79.67 \pm 2.31 \\\\ \hline \end{array}$$ Compared with 600 scenes, the larger training sets improve physical success and reduce variation across seeds. The highest mean Phys.-five is 80.67% with 1,400 scenes. The 1,600-scene setting gives the highest mIoU, 44.43%, and the lowest Phys.-five standard deviation, 2.31 points.
AuthorsOpenReview ↗

Response to Reviewer #uQaA (3/3)

Q3 — Counting simulation calls

One simulation call is one complete physical rollout of one scene configuration in one environment. Each candidate in a parallel batch counts once. SC is the total rollout count divided by the number of evaluated scenes.

  • REST3D-default includes Isaac Gym initialization and search trajectories, plus initial/final MuJoCo checks.
  • REST3D-B24 reports eight initialization trajectories and 16 candidate evaluations.
  • GPT-6 Astra counts every tool-executed physical rollout, including multiple calls within one decision.
  • PAIR3D includes initial verification, verification after each decision, and final verification: T+2 rollouts for T decisions.
  • Greedy and CEM each use 24 candidate rollouts plus initial/final checks, for 26 in total.

We report inference time alongside SC to capture the effects of batching and the different simulation pipelines.

Q4 — GPT-6 Astra baseline

GPT-6 Astra receives the RGB image, instance masks, and SAM3D scene, together with tools for object inspection/editing, rendering, and MuJoCo simulation. It chooses its repair sequence, verification calls, revisions, and stopping point. It can edit positions, rotations, and scales; object identities, meshes, masks, cameras, and physical parameters are fixed. Appendix C.1 gives the full prompt.

IT is seconds per scene and SC is rollouts per scene.

Table 1 of the revised manuscript (selected rows and columns). GPT-6 denotes GPT-6 Astra.

\hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{SAM3D} & 46.56 & 5.0 & \text{--} & \text{--} \\\\ \text{GPT-6} & 44.61 & 62.0 & 160.75 & 4.14 \\\\ \text{PAIR3D} & 47.25 & 97.0 & 4.57 & 24.90 \\\\ \hline \end{array}$$ In the three examples in Figure 5 of the revised manuscript, PAIR3D improves mIoU over GPT-6 Astra by 6.11, 4.46, and 2.00 points. GPT-6 Astra uses approximately 37,000–40,000 tokens per scene on average.
Reviewer_4qowOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PAIR3D, a reinforcement learning framework for refining object layouts reconstructed from single images. Instead of running per-scene optimization at inference time, the authors train a reusable policy that adjusts object translation, rotation, and scale using feedback from rendering and physics simulation.

The policy uses a multimodal relational scene graph containing object states, visual discrepancies, and pairwise physical relations. It is trained with PPO using rewards based on penetration, support, stability, and visual alignment, without requiring ground-truth corrected layouts.

On synthetic scenes initialized with SAM3D, PAIR3D achieves a 97% scene-level physical success rate and improves inference efficiency over REST3D, requiring 4.57 seconds and 24.9 simulator evaluations per scene.

Critical strengths

1. Learning a reusable refinement policy from physics feedback.

I like the formulation of scene refinement as a sequential decision-making problem. Instead of optimizing each scene independently, the method learns how to select and apply layout corrections across different scenes.

The use of verifiable rewards is particularly useful here, since there may be multiple physically valid layouts for a single image. The method does not require a unique ground-truth corrected pose and can instead learn from the measured consequences of its actions.

2. The evaluation considers both physical stability and visual consistency.

The paper does not optimize stability alone. It also penalizes changes that reduce agreement with the input image, preventing the policy from obtaining stable but visually incorrect configurations.

I also like that the authors report inference time and simulator evaluations. PAIR3D achieves 97% Physical-five success, compared with 48% for REST3D, while reducing average refinement time from 43.16 s to 4.57 s. These results support the benefit of amortizing scene refinement into a learned policy.

Critical weaknesses

1. Generalization is only evaluated on synthetic scenes from the same data generation pipeline.

The policy is trained on 1,600 scenes and evaluated on 100 test scenes generated using 3D-FUTURE assets and 3D-FRONT layouts, with SAM3D used for reconstruction initialization.

Although the scene-level split prevents direct overlap between training and test scenes, all scenes follow a similar generation process and come from the same underlying asset and layout distributions.

It is unclear how well the learned policy generalizes to real photographs, unfamiliar objects, different reconstruction models, or substantially different scene configurations.

This is particularly important because the main advantage of PAIR3D over per-scene optimization is supposed to be the transfer of learned refinement behavior to unseen scenes.

I would like to see evaluation on real-image benchmarks or at least an out-of-distribution test using different asset categories, scene layouts, or reconstruction backbones.

Recent methods such as REST3D (Ma et al., 2026), SimuScene (Lee et al., 2026), and Picasso (Yu et al., 2026) provide relevant settings for evaluating physics-aware scene reconstruction beyond a single synthetic distribution.

2. The paper does not fully isolate the benefit of reinforcement learning from the verifier and action space.

PAIR3D combines a structured action space, a multimodal relational representation, and a physics-based reward with PPO training. However, the main comparisons are against reconstruction or per-scene refinement systems that use different search spaces and optimization procedures.

It is therefore difficult to determine how much of the improvement comes from the learned policy rather than the carefully designed action space and verifier.

I would like to see a non-learning baseline using exactly the same action operators, physical and visual objectives, and simulation budget. For example, a greedy coordinate search or CEM-based optimizer operating on the same parameterized actions would provide a more direct comparison.

This would help establish whether the learned policy actually provides better update selection or mainly reduces computation by approximating an existing optimization procedure.

3. The physical evaluation mainly measures static stability rather than interaction readiness.

The Physical-five metric checks support, penetration, tilt, and residual translation and rotation under simulation. These criteria establish whether a reconstructed scene remains approximately stable under gravity.

However, a stable layout is not necessarily physically accurate during interaction. In particular, the method keeps object geometry fixed and does not evaluate whether the reconstructed scene reproduces contact behavior under external forces or manipulation.

The support-gap metric is also defined relative to the ground plane. It is unclear how the current formulation handles objects supported by other objects, such as items placed on tables, shelves, or stacked furniture.

I would like to see more complex support configurations and downstream interaction tests. Recent work on simulation-ready reconstruction, including REST3D and SimuScene (2026), makes this distinction between physical settling and usable scene reconstruction particularly relevant.

4. The evaluation would benefit from more analysis of the learned policy.

Table 2 shows that removing the visual reward reduces both visual alignment and physical success. However, removing the vision or relational branch leads to relatively small differences in Physical-five, which is already close to saturation.

It is unclear whether the learned policy uses relational information to resolve complex contact configurations or primarily learns common local corrections, such as moving objects onto the floor.

I would like to see performance broken down by failure type, number of objects, and scene complexity. Reporting success across multiple object counts and analyzing action sequences would help explain what the policy has actually learned.

Questions for authors
  1. How well does the learned policy generalize outside the synthetic training distribution? Could the authors evaluate on real images, unseen asset categories, or scenes reconstructed using a different backbone?

  2. How does PAIR3D compare with a non-learning optimizer using the same action space and verifier? A greedy search or CEM-based baseline with matched simulation budgets would help isolate the contribution of PPO training.

  3. How are non-ground support relationships handled? The support metric appears to measure object-to-floor gaps. Can the method handle objects resting on tables, shelves, or other reconstructed objects?

  4. What types of errors does the learned policy correct? Could the authors report results separately for floating objects, penetration, tilt, and multi-object support failures, as well as performance versus the number of objects?

  5. How does PAIR3D perform under interactive physics? Does the improvement in static stability translate to more reliable object manipulation or external-force interactions?

  6. What is the training cost compared with the inference savings? The paper reports fast test-time refinement but does not provide a complete accounting of training simulator calls and wall-clock cost. How many test scenes would be needed to offset this initial training cost compared with per-scene optimization?

4 comments
AuthorsOpenReview ↗

Response to Reviewer #4qow (1/2)

Thank you for your careful review and constructive suggestions. We have considered each point and added experiments on generalization, non-learning baselines, and physical interaction, along with further policy analysis. We have clarified the presentation in the revised manuscript and respond to your comments below.


W1/Q1 — Generalization beyond synthetic scenes

We added evaluations on 20 ScanNet reconstructions and 20 Replica reconstructions. The policy trained on 3D-F2Scene is applied directly to the SAM3D outputs, without fine-tuning. ScanNet provides captured RGB images, and Replica provides photorealistic renderings of scanned indoor environments.

Phys.-five requires the support, penetration, tilt, rollout translation, and rollout rotation criteria to be satisfied in the same scene.

Table 5 of the revised manuscript (selected columns). Cross-dataset evaluation; all metrics are percentages.

\hline \text{Dataset} & \text{Method} & \text{mIoU} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{ScanNet} & \text{SAM3D} & 51.96 & 0.0 & 0.0 & 0.0 \\\\ \text{ScanNet} & \text{PAIR3D} & 56.84 & 95.0 & 90.0 & 90.0 \\\\ \text{Replica} & \text{SAM3D} & 50.04 & 0.0 & 0.0 & 0.0 \\\\ \text{Replica} & \text{PAIR3D} & 53.01 & 80.0 & 75.0 & 70.0 \\\\ \hline \end{array}$$ PAIR3D improves mIoU by 4.88 points on ScanNet and 2.97 points on Replica. Sup., Pen., and Tilt reach 100% on both datasets; the remaining failures involve residual motion. Figures 4, 10, and 11 of the revised manuscript show the corresponding visual comparisons. These experiments evaluate transfer across data sources with SAM3D as the initializer. Transfer across reconstruction backbones and held-out asset categories requires separate evaluation. ### **W2/Q2 — Learned action selection** We added Greedy and CEM with the same repair operators and physical/visual verifier as PAIR3D. Greedy evaluates 24 candidates; CEM evaluates three rounds of eight candidates. With initial and final verification, each method uses 26 rollouts per scene. IT is inference time per scene and SC is the number of physical rollouts per scene. Table 3 of the revised manuscript (selected rows). Search and fixed post-processing on the test split. $$\begin{array}{lcccc} \hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{SAM3D} & 46.56 & 5 & \text{--} & \text{--} \\\\ \text{Greedy (24 candidates)} & 46.46 & 19 & 11.68 & 26.0 \\\\ \text{CEM (3 × 8 candidates)} & 46.53 & 5 & 34.78 & 26.0 \\\\ \text{REST3D + fixed post-process} & 45.63 & 96 & 46.84 & 63{,}282.8 \\\\ \text{PAIR3D} & 47.25 & 97 & 4.57 & 24.9 \\\\ \hline \end{array}$$ At about 25 rollouts, PAIR3D reaches 97% Phys.-five, compared with 19% for Greedy and 5% for CEM. REST3D with fixed post-processing reaches 96%; PAIR3D also improves mIoU from 45.63% to 47.25% and reduces inference time from 46.84 s to 4.57 s. Learning the update sequence therefore improves visual agreement and reduces the simulation effort needed for scene refinement. ### **W3/Q3/Q5 — Support relations and contact interaction** The current support criterion measures object-to-floor gaps for rigid furniture on a common floor. Tabletop, shelf, and stacked-object support would require support relations between objects, measurements against the corresponding contact surfaces, and training scenes containing these arrangements. We have clarified this scope in the revision. We added a contact-pushing experiment. After completing scene refinement with PAIR3D, we apply the same pushing protocol to the SAM3D and PAIR3D-refined layouts. In both cases, the same spherical pusher moves the cabinet toward a target 0.25 m along +x during 0–2 s. We then remove contact and observe settling until 4 s. Cabinet tilt (degrees) during the pushing experiment in Figure 9 of the revised manuscript. $$\begin{array}{lcccc} \hline \text{Method} & \text{0 s} & \text{1 s} & \text{2 s} & \text{4 s} \\\\ \hline \text{SAM3D} & 0.75 & 15.56 & 88.27 & 89.95 \\\\ \text{PAIR3D} & 0.45 & 1.23 & 4.13 & 0.10 \\\\ \hline \end{array}$$ The initial floor gap is 10.27 mm for SAM3D and 0.00 mm for PAIR3D. The SAM3D cabinet topples during pushing. At 4 s, the refined cabinet remains upright near the target, with 0.10° tilt and a 0.00 mm floor gap. Figure 9 shows the full sequence.
AuthorsOpenReview ↗

Response to Reviewer #4qow (2/2)

W4/Q4 — Failure modes and policy behavior

We added per-criterion results, continuous errors, an object-count analysis, and a step-by-step refinement example. On the 100-scene test split, all three PAIR3D failures violate only rollout translation.

Table 1 of the revised manuscript (selected rows and columns). Physical pass rates (%) on the test split.

\hline \text{Method} & \text{Sup.} & \text{Pen.} & \text{Tilt} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{SAM3D} & 9.0 & 87.0 & 81.0 & 30.0 & 88.0 & 5.0 \\\\ \text{PAIR3D} & 100.0 & 100.0 & 100.0 & 97.0 & 100.0 & 97.0 \\\\ \hline \end{array}$$ The continuous errors show the magnitude of the remaining violations. Each error is maximized within a scene and then averaged over scenes. Table 2 of the revised manuscript (selected rows). Mean per-scene maximum physical errors; lower is better. $$\begin{array}{lccccc} \hline \text{Method} & E_{\mathrm{sup}}\\;(\mathrm{mm}) & E_{\mathrm{pen}}\\;(\mathrm{mm}) & E_{\mathrm{tilt}}\\;({}^\circ) & E_{\mathrm{RT}}\\;(\mathrm{mm}) & E_{\mathrm{RR}}\\;({}^\circ) \\\\ \hline \text{SAM3D} & 39.43 & 6.40 & 5.29 & 13.92 & 3.63 \\\\ \text{REST3D} & 5.58 & 0 & 2.02 & 1.37 & 0.23 \\\\ \text{PAIR3D} & 0.03 & 0 & 0.78 & 1.29 & 0.27 \\\\ \hline \end{array}$$ Table 15 of the revised manuscript (selected columns). Results by object count N; all metrics are percentages. $$\begin{array}{llccccc} \hline N & \text{Method} & \text{mIoU} & \text{Pen.} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline 1 & \text{SAM3D} & 58.92 & 100.00 & 45.45 & 45.45 & 27.27 \\\\ 1 & \text{PAIR3D} & 60.84 & 100.00 & 90.91 & 90.91 & 90.91 \\\\ 2 & \text{SAM3D} & 46.38 & 80.95 & 0.00 & 4.76 & 0.00 \\\\ 2 & \text{PAIR3D} & 49.02 & 100.00 & 80.95 & 76.19 & 76.19 \\\\ 3 & \text{SAM3D} & 40.08 & 57.14 & 0.00 & 0.00 & 0.00 \\\\ 3 & \text{PAIR3D} & 43.30 & 100.00 & 90.48 & 76.19 & 76.19 \\\\ 4 & \text{SAM3D} & 45.15 & 66.67 & 4.76 & 4.76 & 0.00 \\\\ 4 & \text{PAIR3D} & 47.12 & 100.00 & 90.48 & 85.71 & 80.95 \\\\ 5 & \text{SAM3D} & 42.51 & 50.00 & 0.00 & 0.00 & 0.00 \\\\ 5 & \text{PAIR3D} & 45.85 & 95.00 & 60.00 & 50.00 & 50.00 \\\\ 6 & \text{SAM3D} & 23.51 & 50.00 & 0.00 & 0.00 & 0.00 \\\\ 6 & \text{PAIR3D} & 27.02 & 100.00 & 50.00 & 50.00 & 50.00 \\\\ 7 & \text{SAM3D} & 38.41 & 25.00 & 0.00 & 0.00 & 0.00 \\\\ 7 & \text{PAIR3D} & 41.95 & 75.00 & 100.00 & 100.00 & 75.00 \\\\ \hline \end{array}$$ mIoU improves in every group by 1.92–3.54 points. Residual motion limits the five- and six-object groups, while penetration limits the seven-object group. The five-object group also retains support and tilt failures, with 90% pass rates for both. Figure 7 of the revised manuscript follows the policy as it corrects a cabinet, separates a sofa, lowers a table, restores ground contact, and returns to the sofa to remove remaining overlap. The sequence shows how edits to one object change the constraints on another. In the three-seed validation ablation, removing the relational branch reduces mIoU from 44.49% to 43.44% and Phys.-five from 98.33% to 97.00%. ### **Q6 — Training cost and amortization** Training, initial scene-cache generation, and validation take 55,088.14 s (15.30 h) and 186,163 physical rollouts in total. PPO training uses 20,000 environment transitions. Default REST3D takes 43.16 s and 63,264.72 rollouts per scene; PAIR3D takes 4.57 s and 24.90 rollouts. Dividing the upfront cost by the per-scene savings gives: $$\text{Time:}\quad \left\lceil \frac{55088.14}{43.16 - 4.57} \right\rceil = 1428$$ $$\text{Rollouts:}\quad \left\lceil \frac{186163}{63264.72 - 24.90} \right\rceil = 3$$ The upfront cost is recovered after 1,428 scenes in wall-clock time or three scenes in rollout count. These estimates use the measured mean inference costs. A parallel batch contributes one rollout per candidate, so rollout savings and wall-clock savings are reported separately.

Thank you for the responses and additional experiments. My concerns have been addressed, and I will maintain my rating of weak accept.

AuthorsOpenReview ↗

Official Comment by Authors

Thank you for following up and for your helpful suggestions during the review.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.