acceptodds
Desk rejected at ICLR 2027

InnerScene: A 4D Hand-object Interaction Dataset for Densely Packed Inner Storage Scenes

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

3D scene data increasingly support perception, generation, and manipulation, yet densely stocked inner-container scenes remain difficult to capture with the geometry and action history these tasks jointly require. A stocked cabinet is the result of many placements, but its final state hides the insertion paths through shrinking free space that produced it; walls and clutter conceal even the objects inside. Directly synthesizing this linked 4D record is difficult because each placement changes the geometry and demands plausible hand–object contact. We introduce InnerScene, to our knowledge the first large-scale, real-captured 4D dataset of densely packed inner-container scenes linking reconstructed object assets and metric layouts to synchronized multi-view observations and estimated hand–object placement trajectories. We record placements progressively in an open capture setup that preserves container spatial constraints, then register objects and motion in a shared metric frame. The resulting data support three practical uses: conditional layout generation populates virtual cabinets with specified objects for retrieval and rearrangement research; single-view instance-level reconstruction turns a cabinet-opening image into a 3D inventory for reasoning about access; and human-to-robot trajectory transfer turns selected recorded insertions into executed robot placements in crowded cabinets. InnerScene thus ties what is stored, where it is stored, and how it was placed into one resource for building, understanding, and acting in inner-container scenes.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Scene Generation, 4D Scene Datasets, Inner-container Scenes, Hand–object Interaction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_bNU3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents InnerScene, a dataset that connects densely packed storage scenes with the sequential hand-object placement actions that create them. It contains 10,821 completed scenes, 23,814 reconstructed assets corresponding to real physical objects, and roughly 1.9 million images captured from eight synchronized viewpoints at the completion of each placement step. To support data collection, the authors develop an open capture setup that preserves the spatial constraints of containers while allowing clear observation of the placement process. The annotation pipeline integrates metric depth alignment, asset registration, object tracking, hand reconstruction, contact refinement, and VLM-based trajectory filtering. The dataset is applied to tasks such as conditional layout generation, single-view instance-level reconstruction, and selected cases of human-to-robot placement transfer. Experimental results show improvements on downstream tasks and include ablation studies with synthetic annotations; however, the authors also explicitly note that these findings do not demonstrate the overall accuracy of real trajectory annotations across the entire dataset.

Critical strengths
  1. A valuable combination of static and temporal supervision. Linking instance geometry, metric arrangements, intermediate states, and placement trajectories addresses an important gap between observing a packed storage scene and understanding how it was formed. Progressive acquisition is a sensible response to occlusion, and the reported scale could make the resource useful for several research tasks (Sections 3.1-3.5).

  2. Promising evidence of practical utility. Fine-tuning PartCrafter improves scene CD from 0.286 to 0.123 and object F-score from 0.453 to 0.862 under the reported protocol (Table 3). The layout experiments, including unseen assets and containers in Appendix F.3, offer complementary evidence that the data contain useful supervision. The manuscript also appropriately distinguishes manually initialized robot trajectory transfer from policy learning.

Critical weaknesses
  1. The quality of the central real 4D annotations remains insufficiently established. Table 4(a) evaluates only four rendered scenes with 50 objects; the dynamic hand benchmark is also synthetic. The real-data retention rate of 80.31% measures passage through a VLM overlay screen, not independent object-pose, hand-joint, or contact accuracy. The paper acknowledges this limitation, but a representative real-data audit is still needed because downstream users would rely on these labels as supervision. Absolute retained-trajectory counts and coverage across occlusion or packing difficulty are not reported. The open rig also leaves uncertainty about how faithfully recorded paths respect the complete geometry of an enclosed cabinet.

  2. Important evaluation definitions and internal consistency checks are incomplete. Table 3's caption refers to Appendix G for CD normalization, F-score thresholds, and object matching, but Appendix G explicitly states that normalization and thresholds have not been supplied and does not specify a complete matching protocol. Appendix C.3 likewise leaves the calibration constraint abstract. Moreover, Section 3.4 fixes the object during hand-only contact refinement, yet Table 4(b) changes RS from 0.19 to 0.11 even though Section 5 defines RS using reconstructed-object scale. This requires clarification. The layout stability result also includes physical relaxation and uses a coarse rotation criterion, so it does not independently validate the learned annotations or collision-free geometry.

Questions for authors
  1. Can the authors provide an independently annotated, representative real-capture subset with object translation/rotation error, hand-joint error, and contact error, stratified by occlusion or packing difficulty? Please also report the absolute number of retained placement trajectories and the proportion of scenes with complete retained placement histories.

  2. In Table 4(b), why does RS change after contact optimization if the object geometry and pose remain fixed? Is the implemented metric different from the object-scale definition in Section 5, or does the optimization change additional variables?

  3. What are the exact Table 3 evaluation split, normalization, point sampling, F-score threshold, object-assignment procedure, and penalties for missing or extra objects? Please also specify the board-anchor implementation and correspondence construction for Equation (3).

  4. Is there a controlled comparison that isolates the contribution of InnerScene supervision under matched training budgets and post-processing? In particular, do layout baselines receive the same physics relaxation, and how often do retained human trajectories violate the geometry of the intended closed enclosure?

4 comments

We thank the reviewer for recognizing the value of our dataset, the design of our capture and annotation pipeline, and the promising results of the downstream applications. We are currently preparing the dataset for public release and expect to make it available soon. Below, we address each concern and question and provide additional clarification and experimental evidence.

W1 & Q1: Annotation Accuracy, Trajectory Retention, and Enclosure Compatibility

Thank you for this suggestion. Our manuscript provides complementary evidence for the quality and practical utility of InnerScene’s annotations: synthetic benchmarks quantify annotation errors against known ground truth, VLM screening checks real trajectories for visible reprojection inconsistencies, and robot demonstrations establish the practical utility of selected annotations. Following your suggestion, we further strengthen this evidence with an independent evaluation on representative real captures, described below.

  • Independent Real-Capture Annotation Validation.

    To further assess annotation accuracy on real data, we constructed four real-world scenes using objects and layouts from InnerScene and acquired independent reference measurements for each scene. The scenes contain 15, 20, 25, and 30 objects, respectively, covering different packing densities and occlusion conditions. Following the marker-based motion-capture approach of HOT3D (Banerjee et al., CVPR 2025), we collected independent motion references at an external professional motion-capture facility equipped with an OptiTrack system. Object poses were recovered from tracked rigid marker configurations, while reference hand joint positions were recovered from the tracked hand markers. Each object was separately scanned using an RGB-D camera, with multi-view depth observations fused into an independent reference mesh. Reference and estimated trajectories were temporally synchronized and transformed into a common metric coordinate frame before evaluation.

    We report mean object translation and rotation errors, root-relative hand MPJPE following MagicHOI (Wang et al., ICCV 2025), object geometry Chamfer distance (CD), hand–object penetration volume (SIV), and the trajectory retention rate after quality screening.

    Scene# objsTrans. Error ↓ (cm)Rot. Error ↓ (°)Root-relative Hand MPJPE ↓ (mm)Object CD ↓ (cm)SIV ↓ (cm³)Retention ↑ (%)
    Scene 1152.43.2114.5740.912.23100%
    Scene 2202.13.6544.2910.893.18100%
    Scene 3251.94.1284.6320.812.7596%
    Scene 4302.73.9864.9320.793.22100%

    As shown in the above table, across the four scenes, object translation error ranges from 1.9 to 2.7 cm, object rotation error from 3.21 to 4.13°, root-relative hand MPJPE from 4.29 to 4.93 mm, and object CD from 0.79 to 0.91 cm. These independent measurements provide quantitative validation of object pose, local hand pose, and object geometry on real captures, further supporting the accuracy and reliability of our annotation pipeline and the quality of the resulting dataset.

  • Retained Trajectories and Complete Placement Histories. We recorded 183,502 placement trajectories, of which 147,370 (80.31%) were retained after quality screening. Among the 10,821 scenes, 7,677 (70.95%) retain complete placement histories, meaning that all recorded placement trajectories within each scene are retained after screening.

    Retained Placement TrajectoriesScenes with Complete Retained Placement Histories
    147,370 / 183,502 (80.31%)7,677 / 10,821 (70.95%)
  • Quantitative Enclosure-Compatibility Evaluation. Although capture is performed without physical walls, our open setup reproduces the spatial constraints of each storage type through boundary markings and height references defining the footprint, designated entry side, and height limit. To quantitatively assess whether recorded hand trajectories respect these constraints, we construct finite-thickness wall meshes using the associated furniture dimensions and center coordinates while preserving the designated opening. We check hand–wall intersections using bounding-box filtering followed by SAT-based triangle–box intersection tests. A trajectory is boundary-compliant only if no intersection is detected in any evaluated frame. Across the retained trajectories, 99.6% satisfy this criterion, providing quantitative evidence of their geometric feasibility within the modeled cuboidal enclosures.

W2: Evaluation Protocol

Thank you for these comments. As these concerns are closely related to Q2–Q4, we address them together below, following the order of the questions.

W2–Q2: RS Changes After Contact Refinement

Thank you for this question. Our final refinement includes a separate reprojection-based object-scale optimization step, in addition to hand-contact refinement. We fix the object’s rotation and translation but allow its scale to be optimized.

Our pipeline combines a calibration-board metric reference, dynamic Depth Anything estimates, and static multi-view VGGT estimates. Residual scale mismatches can remain between the hand and object branches after alignment. The additional scale refinement addresses these mismatches, reducing the need for the hand pose alone to compensate for scale errors. RS remains the object-scale error defined in Section 5. The row labelled “+ contact opt.” in Table 4(b) includes both scale and contact refinement, so the RS reduction from 0.19 to 0.11 reflects the additional object-scale optimization. We have revised the manuscript accordingly to clarify the optimization procedure and the ablation label.

W2–Q3: Evaluation Protocol and Calibration Details

Thank you for requesting these details. We clarify the evaluation protocol, metric anchoring, and correspondence construction below.

Table 3 Evaluation Protocol.

  • Evaluation split. For the reconstruction task in Table 3, we use a 9:1 scene-level training/test split, evaluating 137 test scenes with four images per scene (548 images). All views and intermediate states from the same scene remain in the same split.

  • Normalization and CD. Predictions and ground truth are compared in a shared normalized scene frame, with the reference scene’s longest bounding-box side scaled to 2. Scene-level evaluation uses global alignment; object-level evaluation independently normalizes and aligns each matched pair to assess shape quality. We report bidirectional, non-squared Chamfer distance, summing the two directional means in normalized coordinate units.

  • Point sampling. We uniformly sample 10,000 surface points per object mesh from both predictions and ground truth. These points are concatenated for scene-level evaluation and evaluated separately for each matched object pair at the object level.

  • F-score threshold. We compute F-score using a distance threshold of 0.1 in the corresponding normalized coordinate space, reporting scores on a 0–1 scale.

  • Object assignment. Our progressive placement records assign an object ID to each placed instance and link it to its corresponding reference mesh. For instance-conditioned methods, we preserve this association between the input instance and its predicted mesh, allowing direct matching to the reference mesh through the recorded object ID. For PartCrafter, we use Hungarian matching based on rendered-mask IoU.

  • Missing or extra objects. Progressive placement records provide a complete object list, including occluded objects, and the corresponding object count is supplied during evaluation. All evaluated instance-level outputs contain the prescribed number of valid meshes. We therefore evaluate every predicted instance after assignment without an additional object-count penalty. Incorrect geometry remains reflected in the object- and scene-level metrics.

Board Anchor and Correspondence Construction for Equation (3).

  • Board-based metric anchoring. All eight cameras observe the same AprilTag 36h11 board, whose measured geometry defines the metric world frame. Camera poses are estimated from detected tag corners and held fixed. For depth outputs observing the board, we constrain their scales using

    Eboard=λb∑q,jwqj(sqDq(uqj)−zqjboard)2,E_{\mathrm{board}} = \lambda_b \sum_{q,j} w_{qj} \left(s_q D_q(u_{qj}) - z_{qj}^{\mathrm{board}}\right)^2,

    where zqjboardz_{qj}^{\mathrm{board}} is the camera-frame depth of board corner jj, computed from the known board geometry and calibrated camera pose, and wqjw_{qj} weights each corner observation. This term makes the board-derived metric anchor explicit; we have included it in Equation (3), replacing the abstract notation Scal\mathcal{S}_{\mathrm{cal}}.

  • Correspondence construction. The set P\mathcal{P} pairs VGGT and Depth Anything 3 depth predictions at identical pixels in the same image. To construct C\mathcal{C}, we back-project source pixels using board-initialized depth and project them into the target camera, retaining correspondences that pass image-boundary, visibility, and depth-consistency checks. Cross-time correspondences are restricted to already-settled objects. Correspondences remain fixed during each scale-fitting solve, which optimizes only multiplicative depth scales, with camera parameters and additive depth offsets unchanged.

We have also clarified these implementation details in the revised manuscript.

W2–Q4: Layout Evaluation and Enclosure Compatibility

Thank you for these questions. We clarify the comparison protocol, the evidence for InnerScene’s contribution, and trajectory compatibility with enclosure boundaries below.

  • Controlled Comparison under a Shared Evaluation Protocol. All layouts generated by our method and all baselines undergo the same physics-relaxation procedure to allow unsupported objects to settle. All methods also use the same object sets and container dimensions and are evaluated from identical rendering viewpoints under a common protocol. This shared post-processing ensures a fair comparison without giving our method an exclusive advantage from physical relaxation.

  • Evidence for InnerScene’s Contribution. Our ablation in Appendix F.4 and Table 12 compares the original Qwen planner with its InnerScene-tuned counterpart. Fine-tuning improves the support score from 1.13 to 2.75 and the placement score from 1.13 to 2.50, showing that InnerScene supervision helps the planner learn support relationships and object placement.

    Our layout-generation results (Section 4.1, Table 2) achieve the highest scores across realism, density, support, placement, and semantic plausibility, together with 100% physical stability under the reported protocol. The human study (Appendix F.2) reports an average preference share of 71.8% among 50 participants. Furthermore, the model generalizes zero-shot to 100 external Objaverse objects and 10 unseen containers (Section 4.1 and Appendix F.3). Together, these results support InnerScene’s value in helping models learn complex stacking, support relationships, and dense, semantically plausible arrangements.

  • Quantitative Enclosure Compatibility. As detailed in W1 & Q1, we construct finite-thickness wall meshes from the furniture dimensions and center coordinates, preserving the designated opening. We check hand–wall intersections using bounding-box filtering followed by SAT-based triangle–box intersection tests. A trajectory is boundary-compliant only if no intersection is detected in any evaluated frame. 99.6% of retained hand trajectories satisfy this criterion, providing direct quantitative evidence of their geometric feasibility within the modeled cuboidal enclosure boundaries.

Reviewer_7zcyOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces InnerScene, a real-captured dataset of densely populated storage scenes. Its distinguishing feature is the joint annotation of reconstructed object assets, metric scene layouts, synchronized multi-view observations, and estimated hand–object trajectories for the placements that produced each scene. The authors report 10,821 complete scenes, 23,814 reconstructed object assets spanning eight categories, and approximately 1.9 million stable-state images. The paper evaluates the dataset through three applications: conditional layout generation, single-view instance-level reconstruction, and human-to-robot placement trajectory transfer. It also includes synthetic evaluations of the annotation pipeline and ablations of the layout model.

Critical strengths
  1. The dataset addresses an important gap between static scene data and manipulation data. A stocked cabinet contains both a final arrangement and a history of insertions through gradually changing free space, but existing resources rarely preserve these two aspects together. By linking instance assets, intermediate layouts, multi-view observations, and hand–object motion, InnerScene could support research on layout generation, reconstruction under occlusion, and manipulation in confined spaces. I consider the underlying task highly valuable; if released with appropriate quality control and documentation, the dataset could become a useful and influential resource for several research communities.
  2. The progressive acquisition strategy is well suited to the occlusion structure of the problem. Observing each object after placement, before later objects further occlude it, is a sensible way to obtain persistent instance identities and more reliable geometric registration. The design also retains the intermediate states and action history that are difficult to recover from a finished scene alone. I further appreciate the explicit definitions of scenes, images, objects, and trajectories, which avoid presenting synchronized views or intermediate states as independent scenes.
Critical weaknesses
  1. The real-world accuracy of the central 4D annotations requires further validation. The dataset’s main contribution is not merely the number of scenes, but the availability of metric object and hand–object trajectories. However, the quantitative annotation evaluation in Table 4 is conducted on rendered data: four artist-constructed static scenes with 50 objects, together with a separate synthetic dynamic benchmark. The 80.31% retention rate measures acceptance by a VLM inspecting reprojection overlays, rather than trajectory accuracy. Plausible reprojection can still be consistent with errors in depth, object pose, hand pose, or contact, and the selected robot demonstrations do not provide a quantitative substitute. These issues do not negate the value of the collected data, but they currently limit our ability to judge the reliability of the annotations. Even a limited independent evaluation on representative real captures would substantially strengthen the paper and clarify the downstream settings in which the labels are sufficiently accurate.
  2. The open capture rig may introduce a domain gap relative to enclosed storage scenes. The open rig is a reasonable practical compromise for multi-view acquisition, but removing the container walls also removes wall, ceiling, shelf, and door interactions that may affect insertion paths and hand motion in real cabinets. The current draft therefore does not fully establish how well the recorded trajectories and observations represent the enclosed settings emphasized in the motivation. Additional evidence from real enclosed cabinets, or a more detailed analysis of this gap, would help clarify the appropriate scope of the dataset’s claims.
Questions for authors
  1. Could the authors provide quantitative annotation errors on even a small but representative subset of real captures, using reference measurements independent of the proposed pipeline? Useful measurements would include object translation and rotation error, hand joint error, relative hand–object error, and failure rates under different levels of occlusion.
  2. Could the authors provide more detailed dataset statistics, including the number of containers and capture rigs, object instances per category, human participants, total recorded and retained trajectories, and the distribution of objects per scene? How frequently are the same physical objects and containers reused across scenes?
  3. How closely does the open rig reproduce the motion constraints of an enclosed cabinet? Are wall, top, shelf, or door contacts present in the recorded trajectories? Is there any quantitative evaluation on real enclosed cabinets, for either the trajectory annotations or the single-view reconstruction task?
3 comments

We thank the reviewer for recognizing the value of our dataset and the design of our progressive acquisition strategy. We are currently preparing the dataset for public release and expect to make it available soon. Below, we address each concern and question and provide additional experiments and results.

W1 & Q1: Validation of Real-World Annotation Accuracy

Thank you for this suggestion. Our manuscript provides complementary evidence for the quality and practical utility of InnerScene’s annotations: synthetic benchmarks quantify annotation errors against known ground truth, VLM screening checks real trajectories for visible reprojection inconsistencies, and robot demonstrations establish the practical utility of selected annotations. Following your suggestion, we further strengthen this evidence with an independent evaluation on representative real captures, described below.

Independent Real-Capture Annotation Validation:

To further assess annotation accuracy on real data, we constructed four real-world scenes using objects and layouts from InnerScene and acquired independent reference measurements for each scene. The scenes contain 15, 20, 25, and 30 objects, respectively, covering different packing densities and occlusion conditions. Following the marker-based motion-capture approach of HOT3D (Banerjee et al., CVPR 2025), we collected independent motion references at an external professional motion-capture facility equipped with an OptiTrack system. Object poses were recovered from tracked rigid marker configurations, while reference hand joint positions were recovered from the tracked hand markers. Each object was separately scanned using an RGB-D camera, with multi-view depth observations fused into an independent reference mesh. Reference and estimated trajectories were temporally synchronized and transformed into a common metric coordinate frame before evaluation.

We report mean object translation and rotation errors, root-relative hand MPJPE following MagicHOI (Wang et al., ICCV 2025), object geometry Chamfer distance (CD), hand–object penetration volume (SIV), and the trajectory retention rate after quality screening.

Scene# ObjectsTrans. Error ↓ (cm)Rot. Error ↓ (°)Root-relative Hand MPJPE ↓ (mm)Object CD ↓ (cm)SIV ↓ (cm³)Retention ↑ (%)
Scene 1152.43.2114.5740.912.23100%
Scene 2202.13.6544.2910.893.18100%
Scene 3251.94.1284.6320.812.7596%
Scene 4302.73.9864.9320.793.22100%

As shown in the above table, across the four scenes, object translation error ranges from 1.9 to 2.7 cm, object rotation error from 3.21 to 4.13°, root-relative hand MPJPE from 4.29 to 4.93 mm, and object CD from 0.79 to 0.91 cm. These independent measurements provide quantitative validation of object pose, local hand pose, and object geometry on real captures, further supporting the accuracy and reliability of our annotation pipeline and the quality of the resulting dataset.

W2 & Q3: Open Capture Design and Enclosure Compatibility

Thank you for raising this concern. Below, we explain our open capture design, how it reproduces furniture constraints, and our quantitative evaluation of trajectory compatibility with the enclosure geometry.

  • Open Capture for Reliable Observation. Physical cabinet walls, together with densely packed objects, introduce severe occlusion, making it difficult to continuously observe hands and objects during placement. We therefore use an open setup to enable synchronized multi-view acquisition while explicitly representing the real furniture’s spatial constraints.

  • Reproducing Furniture Spatial Constraints. To reproduce the spatial conditions and geometric constraints of real furniture, we configure each capture setup as follows:

    1. Footprint and bottom support. Boundary markings define the furniture’s footprint, while the capture surface provides physical bottom support.

    2. Entry side. Objects are placed through the designated entry side.

    3. Height limit. Height references represent the furniture’s upper boundary.

    4. Participant instructions. Participants are specifically trained to execute placements through the designated opening and prevent both the hand and the manipulated object from crossing the prescribed enclosure boundaries.

    Because InnerScene’s storage spaces are cuboidal, their main geometric constraints can be fully represented by the footprint, entry side, and height limit defined above. Together with participant training, these constraints support the capture of hand–object placement motions that respect the boundaries of real furniture.

  • Quantitative Evaluation Against Target Cabinet Geometry. Following your suggestion, to further assess enclosure compatibility, we construct wall meshes for all enclosure faces except the designated entry face to simulate the geometry of the corresponding real cabinet, using the furniture dimensions, center coordinates, and a specified wall thickness. In a shared metric coordinate frame, we check hand–wall intersections using bounding-box filtering followed by SAT-based triangle–box intersection tests. A trajectory is boundary-compliant only if no intersection is detected in any evaluated frame. As shown in the table below, 99.6% of retained hand trajectories satisfy this criterion. Since the wall meshes reproduce the dimensions and enclosure boundaries of the corresponding real furniture, this result provides quantitative evidence that the captured hand motions respect the geometric constraints of real cabinets, supporting the effectiveness of our open capture design and participant instructions.

    Evaluated trajectoriesBoundary-compliant (%) ↑With detected collisions (%) ↓
    All retained hand trajectories99.60.4

    Together, the spatial constraints, participant training, and 99.6% boundary-compliance rate support our open setup’s effectiveness in capturing hand motions that respect the geometric constraints of real cuboidal cabinets. Our original manuscript also presents zero-shot layout generalization to external Objaverse assets and unseen container dimensions, alongside human-to-robot trajectory transfer in real cabinets, providing complementary evidence of the resulting data’s utility for tasks performed under container constraints.

Q2: Dataset Statistics

Thank you for requesting these details. We provide additional dataset statistics below.

StatisticValue
Complete storage scenes10,821
Object categories235
Recorded placement trajectories183,502
Retained placement trajectories147,370 (80.31%)
Objects per scene: range10–33
Objects per scene: mean / median17 / 17
  • Container Configurations, Capture Setups, and Participants. InnerScene covers eight storage types with 64 size configurations: refrigerators (5), kitchenware cabinets (5), cleaning-supply cabinets (13), food-storage cabinets (11), shoe cabinets (5), medicine cabinets (5), dressing cabinets (7), and desk storage (13). Each configuration denotes a distinct storage-type-and-size combination. We use eight category-specific open capture-rig setups, adjusting each setup to accommodate the corresponding size range. 12 participants performed the placement sequences.

  • Reuse Across Captures. Based on our capture records, each object is reused in approximately 177 capture sequences on average, while each recorded container-size configuration is reused in approximately 169 sequences.

Due to space constraints, the objects in each category and their corresponding instance counts will be provided with the dataset release.

Reviewer_4AoAOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces a new dataset of hand-object interactions of densely packed scenes inside shelfs. They capture people filling a storage container with one object at a time. A multi view capture system is used. The dataset contains 3D reconstructed object meshes, object and hand trajectories. Since tracking objects inside shelfs is difficult due to the occlusion they record the placing in an open, restricted area. The container is added virtually after the reconstruction is done. The reconstruction pipeline is evaluated on synthetic blender scenes with hand object trajectories from GraspXL. The dataset is used for 3 downstream task: object packing, single view reconstruction and robot policy learning.

Critical strengths
  • I think a high quality dataset of hand object interactions in tight, difficult to look into spaces would be very valuable for the community.
Critical weaknesses
  • The reconstruction pipeline is evaluated on synthetic Blender scenes with hand trajectories from GraspXL. But it's not clear how close these synthetic scenes are to the real ones. The real scenes are densely packed in a tight space. How was ensured that the synthetic replication are similarly dense without introucing intersections between the generated hand and the objects.
  • The main distinguishing points of this dataset, the hands and object trajectories are not used in the applications or are not evaluated. The hand trajectories are only used for the robot policy learning but the quantitative evaluation is missing. In addition, neither the layout generation nor the single view reconstruction utilize the hand or object trajectories. An experiment that shows methods trained on Innerscene generalize zero-shot to other datasets would be very valuable, especially if compared to other similar datasets like MessyKitchens.
Questions for authors
  • What is the number of distinct objects in the dataset? According to the paper it is 23814, but it is hard to believe that over 20k physical objects were used.
3 comments

We thank the reviewer for recognizing the value of our high-quality dataset of hand–object interactions in densely packed storage spaces. We are currently preparing the dataset for public release and expect to make it available soon. Below, we address each concern and question and present additional experimental results.

W1: Evaluation on Synthetic Scenes

Thank you for raising this concern. Static reconstruction and dynamic hand–object annotation are evaluated separately, with the dynamic benchmark constructed by adding hand–object motions to the synthetic scenes. Below, we describe synthetic scene construction, real–synthetic consistency evaluation, and hand-motion generation and collision checking.

  • Reproducing Real-Scene Layouts in Synthetic Scenes. We selected real InnerScene captures as references and constructed four Blender scenes containing 20, 25, 25, and 30 objects, respectively, yielding 100 object instances drawn from 50 distinct object assets. These assets were selected from Objaverse based on their semantic and geometric similarity to the captured InnerScene objects. A professional 3D artist reproduced the reference scenes’ object counts and spatial arrangements, with manual adjustments to reflect their dense configurations. The resulting synthetic scenes preserve the dense layouts of the real references while providing known object geometry and poses for evaluation.

  • Evaluating Real–Synthetic Visual Consistency. To further quantify how closely these synthetic scenes resemble their real references, we conducted a paired visual consistency evaluation using Gemini 3.8 Flash and GPT-6.1 Sol. We evaluated paired real and synthetic images from corresponding viewpoints. Both models scored object composition, packing density, spatial arrangement, and occlusion complexity on a 1–5 consistency scale, with higher scores indicating greater consistency. Each image pair was evaluated three times in both real-first and synthetic-first orders. For each evaluator, we report the mean scores across image pairs and repeated evaluations below. Overall is the equal-weight average of the four criteria.

    EvaluatorObject CompositionDensityArrangementOcclusionOverall
    Gemini 3.8 Flash4.314.464.083.944.20
    GPT-6.1 Sol4.194.353.964.074.14

    As shown in the above table, both evaluators report high overall consistency scores (4.20/5 and 4.14/5), with the highest scores for packing density (4.46/5 and 4.35/5). These results provide quantitative visual evidence that the synthetic validation scenes closely reproduce the dense layouts of their real references.

  • Generating Hand Trajectories with Scene-Level Collision Checking. We follow a four-step procedure to ensure valid, penetration-free trajectories:

    1. Grasp specification. We specify the object grasping order, approximate grasp region, and approach direction for each object.

    2. Candidate generation. GraspXL generates multiple candidate grasping trajectories per object, conditioned on its mesh and the specified grasp conditions.

    3. Scene-level collision filtering. We instantiate each candidate in the synthetic scene and check for mesh interpenetration throughout the motion between (i) the hand and the grasped object, (ii) the hand and surrounding objects or the container, and (iii) the transported object and surrounding objects or the container. Any candidate with detected interpenetration is discarded.

    4. Motion selection. We manually select the most natural motion from the remaining collision-free candidates.

We additionally validated our annotation pipeline on four real captures using independent reference measurements, obtaining object translation errors of 1.9–2.7 cm and root-relative hand MPJPE of 4.29–4.93 mm. The experimental setup and full results are provided in our response to Reviewer 7zcy, W1 & Q1.

W2: Hand–Object Trajectory Applications and Cross-Dataset Generalization

Thank you for raising this concern. Below, we clarify the original robot experiment and present additional quantitative policy-learning and cross-dataset generalization results.

  • Human-to-Robot Trajectory Transfer. Our original robot application demonstrates human-to-robot trajectory transfer rather than policy learning. We retarget the recorded hand and object trajectories, together with the registered scene geometry, to a robot gripper and execute selected placements in real cabinets. The robot follows the transferred insertion path through surrounding clutter. These demonstrations provide practical evidence that InnerScene’s annotations contain actionable motion and spatial information for robot placement in densely packed storage spaces, connecting captured human interactions to executable robot manipulation.

  • Quantitative Evaluation of Robot Policy Learning. Our original robot application transfers individual trajectories. To exploit InnerScene’s large-scale hand–object annotations, we initiated a follow-up policy-learning study immediately after submitting the manuscript to ICLR. We report preliminary results below.

    Using the same retargeting procedure, we convert recorded demonstrations into robot gripper trajectories in physics simulation. After execution and filtering out failed grasps, we obtain 1,232 robot trajectories for training. Unlike existing simulation datasets such as RoboTwin (Mu et al., CVPR 2025), which primarily feature loosely arranged tabletop objects, our trajectories capture object placement inside densely packed containers. We fine-tune π0.5\pi_{0.5} (Black et al., CoRL 2025) on these trajectories and evaluate it on 20 manually constructed simulation tasks, each requiring the robot to remove objects from a densely packed container and then place them back inside. Evaluation tasks are separate from the training demonstrations, and success requires completing both the removal and replacement of all objects. We report the success rates below.

    PolicyPretrained success rate ↑Success rate after InnerScene fine-tuning ↑
    π0.5\pi_{0.5}0%25%

    The preliminary results above indicate that InnerScene provides actionable supervision for robot policy learning in densely packed storage spaces, which existing datasets cannot offer.

  • Zero-Shot Layout Generalization. Our manuscript already evaluates zero-shot generalization to external Objaverse assets and unseen containers in Section 4.1 and Appendix F.3. The model trained on InnerScene is applied without additional training. Figure 4 shows that it produces dense, well-supported layouts and outperforms the evaluated baselines. These results show that training on InnerScene enables zero-shot layout generation with objects from another dataset.

  • Zero-Shot Cross-Dataset Reconstruction on MessyKitchens. Following your suggestion, we directly evaluate PartCrafter and SAM3D fine-tuned on InnerScene on MessyKitchens, without additional training on MessyKitchens. We randomly select 20 real scenes and evaluate four images per scene. Each pretrained model and its fine-tuned counterpart use identical test images, object counts, and evaluation settings. Following the MessyKitchens evaluation protocol, we globally align each reconstructed scene to its ground truth using Sim(3) ICP, then compute scene-level and object-level Chamfer distance (CD) and IoU.

    MethodScene CD ↓Scene IoU ↑Object CD ↓Object IoU ↑
    PartCrafter0.2350.1010.5320.069
    PartCrafter fine-tuned on InnerScene0.2130.2010.4500.095
    SAM3D0.0730.4080.0680.438
    SAM3D fine-tuned on InnerScene0.0620.4220.0590.453

    As shown in the above table, both models improve across all four metrics, demonstrating the value of InnerScene as training supervision that improves zero-shot reconstruction on another real-world dataset.

Q1: Number of Distinct Objects

Thank you for this question. InnerScene contains 1035 distinct physical objects across 235 categories. Data collection and processing are organized into batches, each covering multiple scene-placement sequences. During data processing, each object is reconstructed once per batch, and its asset is reused across sequences within that batch; the same object may be reconstructed again in another batch. Thus, the reported figure of 23,814 refers to reconstructed asset outputs, including repeated reconstructions, rather than distinct physical objects. We have also clarified this distinction in the revised manuscript.

Comments on the paper

1 comment

We thank the reviewers for their constructive feedback.

We have uploaded a revised manuscript, with additions highlighted in blue. The updates are:

  1. Added independent real-capture annotation validation in Section 5, Table 5, and Appendix C.5. (Reviewers 7zcy, bNU3)
  2. Added real–synthetic visual consistency evaluation and collision-filtering details in Appendix C.5 and Table 7. (Reviewer 4AoA)
  3. Added robot policy-learning and zero-shot MessyKitchens reconstruction experiments in Appendices H.2 and G.2, Tables 14 and 13. (Reviewer 4AoA)
  4. Added detailed dataset statistics and clarified physical-object counts, retained trajectories, and complete placement histories in Section 3.5 and Appendix D. (Reviewers 4AoA, 7zcy, bNU3)
  5. Added capture-constraint details and quantitative enclosure-compatibility evaluation in Appendices C.1 and C.5. (Reviewers 7zcy, bNU3)
  6. Added evaluation-protocol, calibration, scale-refinement, and shared physics post-processing details in Sections 3.4 and 4.1 and Appendices C.3, C.4, and G.1. (Reviewer bNU3)

Thank you again for your time and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.