- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents InnerScene, a dataset that connects densely packed storage scenes with the sequential hand-object placement actions that create them. It contains 10,821 completed scenes, 23,814 reconstructed assets corresponding to real physical objects, and roughly 1.9 million images captured from eight synchronized viewpoints at the completion of each placement step. To support data collection, the authors develop an open capture setup that preserves the spatial constraints of containers while allowing clear observation of the placement process. The annotation pipeline integrates metric depth alignment, asset registration, object tracking, hand reconstruction, contact refinement, and VLM-based trajectory filtering. The dataset is applied to tasks such as conditional layout generation, single-view instance-level reconstruction, and selected cases of human-to-robot placement transfer. Experimental results show improvements on downstream tasks and include ablation studies with synthetic annotations; however, the authors also explicitly note that these findings do not demonstrate the overall accuracy of real trajectory annotations across the entire dataset.
Critical strengths
-
A valuable combination of static and temporal supervision. Linking instance geometry, metric arrangements, intermediate states, and placement trajectories addresses an important gap between observing a packed storage scene and understanding how it was formed. Progressive acquisition is a sensible response to occlusion, and the reported scale could make the resource useful for several research tasks (Sections 3.1-3.5).
-
Promising evidence of practical utility. Fine-tuning PartCrafter improves scene CD from 0.286 to 0.123 and object F-score from 0.453 to 0.862 under the reported protocol (Table 3). The layout experiments, including unseen assets and containers in Appendix F.3, offer complementary evidence that the data contain useful supervision. The manuscript also appropriately distinguishes manually initialized robot trajectory transfer from policy learning.
Critical weaknesses
-
The quality of the central real 4D annotations remains insufficiently established. Table 4(a) evaluates only four rendered scenes with 50 objects; the dynamic hand benchmark is also synthetic. The real-data retention rate of 80.31% measures passage through a VLM overlay screen, not independent object-pose, hand-joint, or contact accuracy. The paper acknowledges this limitation, but a representative real-data audit is still needed because downstream users would rely on these labels as supervision. Absolute retained-trajectory counts and coverage across occlusion or packing difficulty are not reported. The open rig also leaves uncertainty about how faithfully recorded paths respect the complete geometry of an enclosed cabinet.
-
Important evaluation definitions and internal consistency checks are incomplete. Table 3's caption refers to Appendix G for CD normalization, F-score thresholds, and object matching, but Appendix G explicitly states that normalization and thresholds have not been supplied and does not specify a complete matching protocol. Appendix C.3 likewise leaves the calibration constraint abstract. Moreover, Section 3.4 fixes the object during hand-only contact refinement, yet Table 4(b) changes RS from 0.19 to 0.11 even though Section 5 defines RS using reconstructed-object scale. This requires clarification. The layout stability result also includes physical relaxation and uses a coarse rotation criterion, so it does not independently validate the learned annotations or collision-free geometry.
Questions for authors
-
Can the authors provide an independently annotated, representative real-capture subset with object translation/rotation error, hand-joint error, and contact error, stratified by occlusion or packing difficulty? Please also report the absolute number of retained placement trajectories and the proportion of scenes with complete retained placement histories.
-
In Table 4(b), why does RS change after contact optimization if the object geometry and pose remain fixed? Is the implemented metric different from the object-scale definition in Section 5, or does the optimization change additional variables?
-
What are the exact Table 3 evaluation split, normalization, point sampling, F-score threshold, object-assignment procedure, and penalties for missing or extra objects? Please also specify the board-anchor implementation and correspondence construction for Equation (3).
-
Is there a controlled comparison that isolates the contribution of InnerScene supervision under matched training budgets and post-processing? In particular, do layout baselines receive the same physics relaxation, and how often do retained human trajectories violate the geometry of the intended closed enclosure?
4 comments
W2: Evaluation Protocol
Thank you for these comments. As these concerns are closely related to Q2–Q4, we address them together below, following the order of the questions.
W2–Q2: RS Changes After Contact Refinement
Thank you for this question. Our final refinement includes a separate reprojection-based object-scale optimization step, in addition to hand-contact refinement. We fix the object’s rotation and translation but allow its scale to be optimized.
Our pipeline combines a calibration-board metric reference, dynamic Depth Anything estimates, and static multi-view VGGT estimates. Residual scale mismatches can remain between the hand and object branches after alignment. The additional scale refinement addresses these mismatches, reducing the need for the hand pose alone to compensate for scale errors. RS remains the object-scale error defined in Section 5. The row labelled “+ contact opt.” in Table 4(b) includes both scale and contact refinement, so the RS reduction from 0.19 to 0.11 reflects the additional object-scale optimization. We have revised the manuscript accordingly to clarify the optimization procedure and the ablation label.
W2–Q3: Evaluation Protocol and Calibration Details
Thank you for requesting these details. We clarify the evaluation protocol, metric anchoring, and correspondence construction below.
Table 3 Evaluation Protocol.
-
Evaluation split. For the reconstruction task in Table 3, we use a 9:1 scene-level training/test split, evaluating 137 test scenes with four images per scene (548 images). All views and intermediate states from the same scene remain in the same split.
-
Normalization and CD. Predictions and ground truth are compared in a shared normalized scene frame, with the reference scene’s longest bounding-box side scaled to 2. Scene-level evaluation uses global alignment; object-level evaluation independently normalizes and aligns each matched pair to assess shape quality. We report bidirectional, non-squared Chamfer distance, summing the two directional means in normalized coordinate units.
-
Point sampling. We uniformly sample 10,000 surface points per object mesh from both predictions and ground truth. These points are concatenated for scene-level evaluation and evaluated separately for each matched object pair at the object level.
-
F-score threshold. We compute F-score using a distance threshold of 0.1 in the corresponding normalized coordinate space, reporting scores on a 0–1 scale.
-
Object assignment. Our progressive placement records assign an object ID to each placed instance and link it to its corresponding reference mesh. For instance-conditioned methods, we preserve this association between the input instance and its predicted mesh, allowing direct matching to the reference mesh through the recorded object ID. For PartCrafter, we use Hungarian matching based on rendered-mask IoU.
-
Missing or extra objects. Progressive placement records provide a complete object list, including occluded objects, and the corresponding object count is supplied during evaluation. All evaluated instance-level outputs contain the prescribed number of valid meshes. We therefore evaluate every predicted instance after assignment without an additional object-count penalty. Incorrect geometry remains reflected in the object- and scene-level metrics.
Board Anchor and Correspondence Construction for Equation (3).
-
Board-based metric anchoring. All eight cameras observe the same AprilTag 36h11 board, whose measured geometry defines the metric world frame. Camera poses are estimated from detected tag corners and held fixed. For depth outputs observing the board, we constrain their scales using
where is the camera-frame depth of board corner , computed from the known board geometry and calibrated camera pose, and weights each corner observation. This term makes the board-derived metric anchor explicit; we have included it in Equation (3), replacing the abstract notation .
-
Correspondence construction. The set pairs VGGT and Depth Anything 3 depth predictions at identical pixels in the same image. To construct , we back-project source pixels using board-initialized depth and project them into the target camera, retaining correspondences that pass image-boundary, visibility, and depth-consistency checks. Cross-time correspondences are restricted to already-settled objects. Correspondences remain fixed during each scale-fitting solve, which optimizes only multiplicative depth scales, with camera parameters and additive depth offsets unchanged.
We have also clarified these implementation details in the revised manuscript.
W2–Q4: Layout Evaluation and Enclosure Compatibility
Thank you for these questions. We clarify the comparison protocol, the evidence for InnerScene’s contribution, and trajectory compatibility with enclosure boundaries below.
-
Controlled Comparison under a Shared Evaluation Protocol. All layouts generated by our method and all baselines undergo the same physics-relaxation procedure to allow unsupported objects to settle. All methods also use the same object sets and container dimensions and are evaluated from identical rendering viewpoints under a common protocol. This shared post-processing ensures a fair comparison without giving our method an exclusive advantage from physical relaxation.
-
Evidence for InnerScene’s Contribution. Our ablation in Appendix F.4 and Table 12 compares the original Qwen planner with its InnerScene-tuned counterpart. Fine-tuning improves the support score from 1.13 to 2.75 and the placement score from 1.13 to 2.50, showing that InnerScene supervision helps the planner learn support relationships and object placement.
Our layout-generation results (Section 4.1, Table 2) achieve the highest scores across realism, density, support, placement, and semantic plausibility, together with 100% physical stability under the reported protocol. The human study (Appendix F.2) reports an average preference share of 71.8% among 50 participants. Furthermore, the model generalizes zero-shot to 100 external Objaverse objects and 10 unseen containers (Section 4.1 and Appendix F.3). Together, these results support InnerScene’s value in helping models learn complex stacking, support relationships, and dense, semantically plausible arrangements.
-
Quantitative Enclosure Compatibility. As detailed in W1 & Q1, we construct finite-thickness wall meshes from the furniture dimensions and center coordinates, preserving the designated opening. We check hand–wall intersections using bounding-box filtering followed by SAT-based triangle–box intersection tests. A trajectory is boundary-compliant only if no intersection is detected in any evaluated frame. 99.6% of retained hand trajectories satisfy this criterion, providing direct quantitative evidence of their geometric feasibility within the modeled cuboidal enclosure boundaries.
We thank the reviewer for recognizing the value of our dataset, the design of our capture and annotation pipeline, and the promising results of the downstream applications. We are currently preparing the dataset for public release and expect to make it available soon. Below, we address each concern and question and provide additional clarification and experimental evidence.
W1 & Q1: Annotation Accuracy, Trajectory Retention, and Enclosure Compatibility
Thank you for this suggestion. Our manuscript provides complementary evidence for the quality and practical utility of InnerScene’s annotations: synthetic benchmarks quantify annotation errors against known ground truth, VLM screening checks real trajectories for visible reprojection inconsistencies, and robot demonstrations establish the practical utility of selected annotations. Following your suggestion, we further strengthen this evidence with an independent evaluation on representative real captures, described below.
Independent Real-Capture Annotation Validation.
To further assess annotation accuracy on real data, we constructed four real-world scenes using objects and layouts from InnerScene and acquired independent reference measurements for each scene. The scenes contain 15, 20, 25, and 30 objects, respectively, covering different packing densities and occlusion conditions. Following the marker-based motion-capture approach of HOT3D (Banerjee et al., CVPR 2025), we collected independent motion references at an external professional motion-capture facility equipped with an OptiTrack system. Object poses were recovered from tracked rigid marker configurations, while reference hand joint positions were recovered from the tracked hand markers. Each object was separately scanned using an RGB-D camera, with multi-view depth observations fused into an independent reference mesh. Reference and estimated trajectories were temporally synchronized and transformed into a common metric coordinate frame before evaluation.
We report mean object translation and rotation errors, root-relative hand MPJPE following MagicHOI (Wang et al., ICCV 2025), object geometry Chamfer distance (CD), hand–object penetration volume (SIV), and the trajectory retention rate after quality screening.
As shown in the above table, across the four scenes, object translation error ranges from 1.9 to 2.7 cm, object rotation error from 3.21 to 4.13°, root-relative hand MPJPE from 4.29 to 4.93 mm, and object CD from 0.79 to 0.91 cm. These independent measurements provide quantitative validation of object pose, local hand pose, and object geometry on real captures, further supporting the accuracy and reliability of our annotation pipeline and the quality of the resulting dataset.
Retained Trajectories and Complete Placement Histories. We recorded 183,502 placement trajectories, of which 147,370 (80.31%) were retained after quality screening. Among the 10,821 scenes, 7,677 (70.95%) retain complete placement histories, meaning that all recorded placement trajectories within each scene are retained after screening.
Quantitative Enclosure-Compatibility Evaluation. Although capture is performed without physical walls, our open setup reproduces the spatial constraints of each storage type through boundary markings and height references defining the footprint, designated entry side, and height limit. To quantitatively assess whether recorded hand trajectories respect these constraints, we construct finite-thickness wall meshes using the associated furniture dimensions and center coordinates while preserving the designated opening. We check hand–wall intersections using bounding-box filtering followed by SAT-based triangle–box intersection tests. A trajectory is boundary-compliant only if no intersection is detected in any evaluated frame. Across the retained trajectories, 99.6% satisfy this criterion, providing quantitative evidence of their geometric feasibility within the modeled cuboidal enclosures.