- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces Pair3D, a reinforcement-learning framework that learns a physics-aligned and visually consistent refinement of 3D scene layouts from verifiable rewards. Input: SAM3D generated scene Output: scene with plausible physics
Critical strengths
- Good research question with novelty.
- The reward design is reasonable.
- Demonstrated efficiency of the design.
Critical weaknesses
- The citation has the wrong title: "Huan Fu, Bowen Cai, Lin Gao, Ling-Xiao Zhang, Jiaming Wang style, et al. 3d-front: 3d houses ranked with objects. In IEEE/CVF International Conference on Computer Vision, 2021a." should be "3d-front: 3d furnished rooms with layouts and semantics"
- The citation has the wrong author and title: "Huan Fu, Rongfei Jia, Lin Gao, Mingli Song, Chao-Hui Shen style, et al. 3d-future: 3d furniture shaped with texture. International Journal of Computer Vision, 2021b" should be "Huan Fu, Rongfei Jia, Lin Gao, Mingming Gong, Binqiang Zhao, Steve Maybank, Dacheng Tao" and "3d-future: 3d furniture shape with texture".
- The evaluation metric coincides with the training reward, which favors PAIR3D over other baselines, and the evaluation metric is less restricted compared to the reward equation; why?
- Did not demonstrate the necessity of reinforcement learning; the author should compare with another simple strategy without learning, such as: REST3D with a simple floor-contact correction, put everything on the floor and resolve penetration between each object, then align with mask IoU.
- No in-the-wild comparison.
- Question about the necessity of the visual branch and relational branch, since in Table 2 ablation "Without vision branch" and "without relational branch" does not look a lot different than the full PAIR3D.
- The number of PAIR3D is different between Table 1 and 2; why?
Questions for authors
See weakness.
5 comments
Response to Reviewer #wrox (2/3)
W4 — Comparison with non-learning methods
We added REST3D with fixed post-processing. It raises the support pass rate from 50% to 100% and Phys.-five from 48% to 96%. PAIR3D reaches 97% and improves mIoU by 1.62 points, with inference time reduced from 46.84 s to 4.57 s:
Table 13 of the revised manuscript (selected rows and columns). Effect of fixed post-processing.
\hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Sup.}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{REST3D-default} & 45.48 & 50.0 & 48.0 & 43.16 & 63{,}264.72 \\\\ \text{REST3D + fixed post-process} & 45.63 & 100.0 & 96.0 & 46.84 & 63{,}282.80 \\\\ \text{PAIR3D} & 47.25 & 100.0 & 97.0 & 4.57 & 24.90 \\\\ \hline \end{array}$$ We also compared Greedy and CEM using the same repair operators and physical/visual verifier as PAIR3D. Each search method uses 24 candidate rollouts plus initial/final checks: Table 3 of the revised manuscript (selected rows). Search with the same repair operators and verifier. $$\begin{array}{lcccc} \hline \text{Method} & \text{mIoU}\\;(\\%) & \text{Phys.-five}\\;(\\%) & \text{IT (s)} & \text{SC} \\\\ \hline \text{Greedy (24 candidates)} & 46.46 & 19 & 11.68 & 26.0 \\\\ \text{CEM (3 × 8 candidates)} & 46.53 & 5 & 34.78 & 26.0 \\\\ \text{PAIR3D} & 47.25 & 97 & 4.57 & 24.9 \\\\ \hline \end{array}$$ With a comparable rollout budget, PAIR3D achieves substantially higher physical success and retains better image agreement. IT is seconds per scene and SC counts rollouts per scene. ### **W5 — Evaluation on real indoor scenes** We added comparisons between SAM3D and PAIR3D on 20 ScanNet reconstructions and 20 Replica reconstructions. The policy trained on 3D-F2Scene is transferred directly, without fine-tuning. ScanNet uses captured RGB images; Replica uses photorealistic renderings of scanned indoor environments. Table 5 of the revised manuscript (selected columns). Cross-dataset evaluation; all metrics are percentages. $$\begin{array}{llcccc} \hline \text{Dataset} & \text{Method} & \text{mIoU} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{ScanNet} & \text{SAM3D} & 51.96 & 0.0 & 0.0 & 0.0 \\\\ \text{ScanNet} & \text{PAIR3D} & 56.84 & 95.0 & 90.0 & 90.0 \\\\ \text{Replica} & \text{SAM3D} & 50.04 & 0.0 & 0.0 & 0.0 \\\\ \text{Replica} & \text{PAIR3D} & 53.01 & 80.0 & 75.0 & 70.0 \\\\ \hline \end{array}$$ Refinement raises mIoU on both datasets and brings Sup., Pen., and Tilt pass rates to 100%. Figures 4, 10, and 11 of the revised manuscript provide input-view and auxiliary-view comparisons of the seating, table, and cabinet arrangements.Response to Reviewer #wrox (3/3)
W6 — Visual and relational ablations
We repeated the ablation with three training seeds. The original ablation Table 2 is now Table 4; all metrics and configurations are reproduced below:
Table 4 of the revised manuscript. Ablation results on the 3D-F2Scene validation dataset. Values are mean ± standard deviation over three training seeds. Bold denotes the best mean for each metric.
\text{Metric} & \text{PAIR3D} & \begin{array}{c}\text{w/o Scale}\\\\\text{Adjustment}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Reward}\end{array} & \begin{array}{c}\text{w/o Visual}\\\\\text{Branch}\end{array} & \begin{array}{c}\text{w/o Relational}\\\\\text{Branch}\end{array}\\\\ \hline \text{mIoU}\\;(\\%)\uparrow & \mathbf{44.49}\pm0.07 & 43.57\pm0.24 & 42.40\pm0.10 & 43.35\pm0.31 & 43.44\pm0.34\\\\ \text{CD (mm)}\downarrow & \mathbf{316.79}\pm0.42 & 317.30\pm0.47 & 319.05\pm0.12 & 317.11\pm0.65 & 316.93\pm0.52\\\\ \text{F-Score}\\;(\\%)\uparrow & 12.05\pm0.04 & 12.02\pm0.08 & 11.97\pm0.07 & 12.06\pm0.10 & \mathbf{12.09}\pm0.05\\\\ \hline \text{Sup.}\\;(\\%)\uparrow & \mathbf{99.33}\pm0.58 & \mathbf{99.33}\pm0.58 & 98.00\pm2.00 & 99.00\pm1.00 & 98.33\pm0.58\\\\ \text{Pen.}\\;(\\%)\uparrow & 99.00\pm1.00 & \mathbf{99.67}\pm0.58 & 98.00\pm1.00 & 98.67\pm0.58 & 98.00\pm1.00\\\\ \text{Tilt}\\;(\\%)\uparrow & \mathbf{100.00}\pm0.00 & 99.67\pm0.58 & 98.67\pm2.31 & 99.67\pm0.58 & \mathbf{100.00}\pm0.00\\\\ \text{RT}\\;(\\%)\uparrow & 98.33\pm1.53 & 98.33\pm1.53 & 95.67\pm3.51 & \mathbf{98.67}\pm0.58 & 97.67\pm0.58\\\\ \text{RR}\\;(\\%)\uparrow & 99.33\pm0.58 & \mathbf{99.67}\pm0.58 & 99.00\pm1.00 & 99.00\pm0.00 & 99.00\pm0.00\\\\ \text{Phys.-five}\\;(\\%)\uparrow & \mathbf{98.33}\pm1.53 & 98.00\pm2.00 & 93.00\pm6.24 & \mathbf{98.33}\pm0.58 & 97.00\pm1.00\\\\ \end{array}$$ The visual branch improves mIoU by 1.14 points and reduces CD from 317.11 to 316.79 mm, while mean Phys.-five stays at 98.33%. Its main role is matching the target masks and depths. The node state already supplies physical diagnostics, so support, penetration, tilt, and motion information remains available without the visual branch. The relational branch adds relative geometry and contact information. It improves mIoU by 1.05 points and Phys.-five by 1.33 points, with a 1.00-point increase in both support and penetration pass rates. RT and RR increase by 0.66 and 0.33 points, respectively, while Tilt is unchanged. Figure 7 of the revised manuscript illustrates the interacting object constraints represented by this branch. F-Score is 12.05% for PAIR3D, 12.06% without the visual branch, and 12.09% without the relational branch. The full model achieves the highest mean mIoU, the lowest mean CD, and ties for the highest mean Phys.-five. ### **W7 — Test and validation results** Thank you for pointing this out. The difference in PAIR3D's performance between Table 1 and the original Table 2 (now Table 4 in the revised manuscript) is due to the use of different evaluation splits. Table 1 reports results on the test set (100 scenes) for comparison with existing methods, whereas the ablation table reports results on the validation set (100 separate scenes). We conducted ablation studies on the validation set to analyze the contributions of individual components during model development, while keeping the test set independent of model selection. The revised Table 4 also reports the mean and standard deviation over three training seeds. We have clarified the evaluation splits in Section 4.1 and in the captions of Tables 1 and 4.
Response to Reviewer #wrox (1/3)
Thank you for your careful review and for identifying the reference errors and questions about the evaluation. We have corrected the references, added comparisons and ablations, and clarified the experimental setup in the revised manuscript. We address your comments below.
W1/W2 — Corrections to the references
Thank you for catching these errors; we apologize for them. The errors were introduced when we converted our reference list into BibTeX format with a tool backed with AI function, and we did not check the converted entries against the published papers. The references themselves were selected by us, and both works are the datasets we use (Section 4.1); only the bibliographic metadata was wrong.
In the revision, we have corrected both entries, checked all references by hand against the publisher page or DBLP and found no other errors.
W3 — Training reward and evaluation criteria
The reward combines reductions in continuous physical, visual, and pose-prior costs with a step penalty and a success bonus. Training success requires physical acceptance F and a mask IoU at least as high as the initialization:
S(L)=F(L)∧[I(L)≥I0]
Phys.-five reports the five physical conditions jointly, and mIoU reports visual agreement separately.
We did not explain the distinction between training acceptance and evaluation thresholds clearly enough in the original manuscript. We used looser evaluation thresholds because several baselines have very low success rates under the strict five-criterion conjunction; reporting only joint success can obscure the partial improvements they achieve. The looser thresholds help show these differences. We apply the same thresholds to all methods, without changing their output layouts.
The main evaluation uses 5 mm for Sup./Pen./RT and 5° for Tilt/RR. We also re-evaluated the same test layouts and residuals using the training thresholds: 0.5 mm for Sup./Pen., 2° for Tilt, 1.5 mm for RT, and 0.2° for RR.
Table 14 of the revised manuscript. Test-set pass rates (%) under the training acceptance thresholds.
\hline \text{Method} & \text{Sup.} & \text{Pen.} & \text{Tilt} & \text{RT} & \text{RR} & \text{Phys.-five} \\\\ \hline \text{SceneGen} & 0.0 & 90.0 & 1.0 & 0.0 & 0.0 & 0.0 \\\\ \text{SAM3D} & 1.0 & 87.0 & 29.0 & 9.0 & 11.0 & 1.0 \\\\ \text{REST3D} & 0.0 & 100.0 & 99.0 & 72.0 & 70.0 & 0.0 \\\\ \text{GPT-6} & 59.0 & 93.0 & 75.0 & 52.0 & 50.0 & 37.0 \\\\ \text{PAIR3D} & 100.0 & 100.0 & 99.0 & 83.0 & 77.0 & 75.0 \\\\ \hline \end{array}$$ PAIR3D reaches 75% Phys.-five under these tighter criteria, compared with 37% for GPT-6 Astra. Support and penetration pass in every PAIR3D scene. We also report the continuous errors below, which show the size of the residual errors rather than only whether they cross a threshold. Table 2 of the revised manuscript (selected rows). Mean per-scene maximum physical errors; lower is better. $$\begin{array}{lccccc} \hline \text{Method} & E_{\mathrm{sup}}\\;(\mathrm{mm}) & E_{\mathrm{pen}}\\;(\mathrm{mm}) & E_{\mathrm{tilt}}\\;({}^\circ) & E_{\mathrm{RT}}\\;(\mathrm{mm}) & E_{\mathrm{RR}}\\;({}^\circ) \\\\ \hline \text{SAM3D} & 39.43 & 6.40 & 5.29 & 13.92 & 3.63 \\\\ \text{REST3D} & 5.58 & 0 & 2.02 & 1.37 & 0.23 \\\\ \text{PAIR3D} & 0.03 & 0 & 0.78 & 1.29 & 0.27 \\\\ \hline \end{array}$$ PAIR3D has lower support, tilt, and translation errors, while REST3D has the lower mean rotation error. All methods are evaluated against the same physical criteria. The search comparisons also use PAIR3D's repair operators and verifier.