acceptodds
Under review as a conference paper at ICLR 2027

SituHOI: Scalable Capture of Scene-Situated Full-Body Human-Object Interaction

Abstract

Modeling human-object interaction (HOI) in real environments requires jointly representing whole-body motion, manipulated objects, and scene geometry, yet existing datasets trade off scalable capture against structured, scene-aligned supervision. We introduce , a large-scale RGB-D dataset containing 18,429 annotated clips across 16 indoor rooms and approximately 86 hours of quality-screened activity. It pairs recordings from a single moving RGB-D camera with static scene scans, object geometry, language descriptions, and estimated human and object trajectories. Using the scans as persistent references, our pipeline combines camera localization and depth-based metric alignment with human mesh recovery, object tracking, and temporal refinement to register all geometric annotations in a shared floor-aligned frame. These data support , a goal-conditioned joint motion prediction benchmark with variable observation and prediction durations. Given observed histories, scene and object geometry, and a terminal object pose, models predict future human and object motion. We provide as a reference predictor and evaluate world-space accuracy and human-object coordination. On a MoCap subset, our pipeline reduces human G-MPJPE from 195.4 to 68.3 mm and object translation error from 43.0 to 28.7 mm. SharedHOI leads four adapted baselines on all three aggregate FlexHOI-Bench metrics. Data will be released upon acceptance. An anonymized code repository is available at https://anonymous.4open.science/r/iclr2027-anonymous-code-1DFB.

Primary area:
datasets and benchmarks
Keywords:
3D Human–Object Interaction, human-object prediction

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_YJwqOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces SituHOI, a large-scale dataset for scene-situated full-body human-object interaction (HOI). To address the limitations of existing capture systems in scalability and structured scene-aligned supervision, the authors develop a capture and reconstruction pipeline based on a single moving RGB-D camera, together with prior scene and object scans. The resulting dataset contains approximately 86 hours of recordings and 18,429 annotated clips across 16 indoor rooms, providing human and object trajectories, scene geometry, and language descriptions in a shared coordinate frame.

Critical strengths
  1. A valuable and well-motivated dataset contribution. The paper addresses an important limitation of existing HOI datasets: the difficulty of jointly obtaining full-body motion, object trajectories, scene geometry, and semantic annotations at scale. SituHOI offers a compelling combination of these properties through a relatively flexible single-camera capture setup. The dataset is reasonably large, and its scene-aligned multimodal annotations make it a valuable resource for scene-aware motion modeling and embodied intelligence. I particularly appreciate the clear positioning against existing datasets and the practical motivation behind the capture protocol.

  2. Comprehensive validation and a useful benchmark. The paper goes beyond dataset construction by providing a systematic evaluation of reconstruction quality and downstream utility. The independent MoCap validation and component ablations support the reliability of the proposed pipeline. FlexHOI-Bench provides a meaningful and standardized setting for joint human-object motion prediction across different temporal horizons. More importantly, the cross-dataset transfer experiments demonstrate that SituHOI can benefit existing HOI models, providing convincing evidence of its broader research value. Overall, the dataset, benchmark, and experimental evaluation form a coherent and substantial contribution.

Critical weaknesses

The main limitations concern the extent of validation rather than the overall motivation or technical soundness. In particular, the diversity of participants and object categories remains somewhat limited, and the independent MoCap evaluation covers a relatively small subset of the collected data. These limitations do not substantially diminish the contribution, but further analysis would strengthen confidence in the dataset's general applicability.

Questions for authors
  1. Generalization across scenes and participants. The dataset covers 16 indoor rooms and 15 participants. Could the authors provide further discussion or evaluation of generalization to unseen rooms, participants, or object categories? In particular, a room-disjoint benchmark evaluation would help clarify the extent to which the proposed models generalize beyond the training environments.

  2. Annotation accuracy and interaction consistency. The independent MoCap validation provides useful evidence of reconstruction quality. However, precise human-object contact modeling can be sensitive to relatively small geometric errors. Could the authors further discuss the reliability of the recovered trajectories during close-contact interactions, potentially including additional contact-distance or penetration-related statistics?

  3. Dataset accessibility and future extensions. Could the authors clarify the planned release scope, including processed trajectories, object and scene meshes, and benchmark evaluation scripts? It would also be interesting to discuss how the proposed capture framework could be extended to articulated objects or more complex interaction scenarios.

Reviewer_4k7POpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes SituHOI, a large-scale scene-grounded HOI dataset captured using a single moving RGB-D camera, providing aligned 3D human motion, object trajectories, scene geometry, and language annotations. The paper further introduces FlexHOI-Bench and SharedHOI for goal-conditioned HOI motion prediction, validated through MoCap comparisons, benchmark evaluations, and cross-dataset transfer.

Critical strengths
  1. This paper introduces an important large-scale dataset for 3D human-object interaction understanding. Unlike existing HOI datasets that often focus on human-object interaction alone, SituHOI provides human motion, object motion, scene geometry, and language descriptions in a common 3D coordinate system. I think this is an important contribution for future HOI research.
  2. The proposed data collection pipeline is very interesting. Capturing full-body HOI using a single moving RGB-D camera instead of expensive multi-view MoCap systems could substantially improve the scalability of future HOI dataset collection. Figure 1 provides a good overview of the pipeline.
  3. One of the strongest advantages of this paper is that the dataset is not only introduced but also validated through a downstream benchmark. Table 4 is particularly convincing as it demonstrates that pretraining on SituHOI benefits existing HOI methods on BEHAVE, showing the usefulness of the dataset beyond the proposed SharedHOI framework.
Critical weaknesses
  1. While SituHOI contains a large amount of recorded data, it is difficult to understand how diverse the actual human-object interactions are. According to Appendix Table 5, the dataset contains only 13 dynamic object types and 15 participants. Since the paper emphasizes the importance of scalable and diverse HOI capture, it would be important to distinguish the amount of recorded interaction data from the diversity of the collected interactions.
  2. The main motivation of FlexHOI-Bench is to evaluate human-object coordination. However, the evaluation of human-object interaction uses wrist-object proximity with a 20 cm threshold rather than actual contact or physical interaction. Considering that the dataset provides 3D human and object geometry, I think the evaluation formulation does not fully capture the richness of the proposed dataset.
Questions for authors
  1. In Table 1, SituHOI is compared with many existing HOI datasets. Could you further explain what is the most important advantage of SituHOI compared with datasets such as ParaHome, HOI-M3, and HUMOTO? Is it primarily the data scale, capture accessibility, or scene-grounded interaction representation?
  2. In L063-L071, the paper emphasizes the scalability of the data capture pipeline. However, the method still requires prior static scene scans, dedicated object scans, and several reconstruction and optimization stages. How much does this reduce the actual data collection and annotation burden compared with existing approaches?
  3. Figure 2 shows that the object distribution is imbalanced across the dataset. How much does the diversity of object categories and interaction types affect the usefulness of SituHOI for general HOI modeling?
  4. According to Appendix Table 5, there are only 15 participants and 13 dynamic object types. While the dataset duration is large, could you explain how the dataset achieves sufficient diversity in human motion and object manipulation?
  5. In Table 2, MoCap validation is performed on approximately 30 minutes of data from two participants. How representative is this subset of the entire 86-hour dataset? Could reconstruction accuracy differ substantially across object types or environments?
  6. Why is the terminal object pose provided as an input to FlexHOI-Bench? While goal-conditioned prediction is a meaningful problem, the future object pose is generally unknown in human-object motion forecasting. It would be great to further explain the practical application of this benchmark setting.
  7. In L451-L455, proximity F1 is defined using wrist-object distance below 20 cm. Why is such a large threshold used to define interaction? Wouldn't this count the hand as interacting with an object even when there is no actual contact?
  8. The proposed dataset includes full-body human motion and scene geometry. However, the benchmark mainly evaluates root translation, object translation, and wrist-object proximity. How does FlexHOI-Bench evaluate whether the generated motion is physically plausible and consistent with the surrounding scene?
  9. In Table 3, SharedHOI outperforms existing methods, but the improvement in root translation error over DecHOI is relatively small. What is the main methodological contribution of SharedHOI compared with existing joint human-object motion prediction methods?
  10. In Figure 3, SharedHOI jointly models human and object motion. What is the benefit of the shared modeling framework compared with separately predicting human and object motion while conditioning on the same object goal?
  11. The benchmark provides scene geometry as input. How much does scene information actually contribute to the final prediction? What kind of interaction would be difficult to predict without scene geometry?
  12. The paper introduces language descriptions as an important part of SituHOI, but FlexHOI-Bench does not appear to directly use this information for motion prediction. What kind of downstream tasks would benefit most from the aligned language annotations?
  13. Could you clarify whether the train/test splits evaluate generalization to unseen participants, objects, and scenes? Appendix D.1 indicates recording-level separation, while the held-out test recordings can share room identities with training.
Reviewer_wB9TOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

SituHOI provides an 86-hour RGBD dataset of scene grounded full body interactions using a single moving camera. It also establishes the FlexHOI-Bench prediction task across flexible time horizons. The author also introcudes SharedHOI, a unified transformer reference model that jointly predicts human motion, object trajectories, and relative contact coordination conditioned on terminal object poses.

Critical strengths

The strongest part of this paper is the scale of the presented dataset. It can offer valuable size of data to the community to train larger model.

Critical weaknesses

The critical weaknesses are two folds:

  1. On the data side, the quality remain questionable. The hand motion doesn't seem good enough (Figure 3 right side, hand pose next to the box is not ideal). The dataset object skewed toward large, easy to track objects like boxes and chairs, while smaller ones are not well represented (due to the limitation in tracking algorithm)
  2. On the evaluation side, the benchmark relies almost exclusively on root ADE, object ADE, and coarse proximity F1. Crucial metrics for physical plausibility are missing.
Questions for authors
  1. How are hand poses being fitted?
  2. Does the algorithm scales better with more diverse scenarios? The user set 16 scenes for the dataset and requires some pre-scanning of the scene. Is this step a must? Any chance to make it easier for scaling?
  3. Are there any consideration of physical plausibility when constructing the dataset and benchmark? Any insight on whether to include / not include the related metrics will be useful for the paper.
3 comments
AuthorsOpenReview ↗

Response to Reviewer wB9T

We appreciate the reviewer’s recognition of the scale and potential value of SituHOI. Below, we address the concerns about hand reconstruction, object-scale coverage, scalability, and physical plausibility with quantitative evidence, while clarifying the scope of each evaluation.

W1(a): Reconstruction Quality

We evaluate the pipeline on an independently measured subset captured with synchronized RGB-D and MoCap. The full pipeline achieves a G-MPJPE of 68.3 mm, an RA-MPJPE of 47.6 mm, and an object-local wrist error of 51.6 mm. Under the same calibration and evaluation samples, the wrist error increases to 61.5 mm without human temporal regularization and to 55.2 mm without the depth-based surface loss. These controlled comparisons show that both components improve body and wrist-level human–object alignment.

To further investigate the functional effect of the remaining reconstruction error, we additionally conducted physics-based lifting simulations using the reconstructed hand poses. We provide the complete protocol and results in our response to W2. Without modifying the reconstructed hand geometry or initial hand–object configuration, the hands supported the objects throughout the simulated lifting motions.

The right panel of Fig. 3 visualizes SharedHOI predictions rather than the hand reconstruction pipeline. SharedHOI predicts root translation and 21 body rotations; finger articulation is not part of its output state. The rendered fingers therefore should not be interpreted as a prediction or quantitative evaluation of finger motion. We will revise the visualization and caption to remove this ambiguity. If space permits, we will separately provide source-RGB and fitted-SMPL-X overlays as qualitative reconstruction evidence.

W1(b): Small-Object Coverage

SituHOI includes objects spanning substantially different physical scales. For example, the cup and bottle meshes have approximate bounding-box dimensions of 13.216.818.3 cm and 10.214.218.9 cm and appear in 666 and 533 interaction clips, respectively. Together, they contribute 1,199 clips, representing 11.6% of the 10,335 dynamic-object clips. By comparison, box and chair—the two most frequent larger-object categories—account for 3,084 clips, or 29.8%. Thus, small-object interactions are represented by more than one thousand clips rather than isolated examples. We will add metric object dimensions to Fig. 2 to make both the physical-scale and frequency distributions explicit.

Q1: Hand Fitting

We do not use a separate hand-specific post-processing system. OSX initializes the SMPL-X pose and shape from RGB, while root translation is initialized separately by back-projecting depth in the detected person region. We then jointly refine the root translation/orientation, body pose, and both hand poses at the sequence level; body shape, jaw, and eye parameters remain fixed.

The fitting uses 63 matched RTMW body, foot, and hand landmarks, 40 of which correspond to the two hands. Detections below 0.70 confidence are discarded, and hand landmarks receive a weight of 0.65. The first stage combines robust 2D reprojection, body and hand priors, preservation of the image-based initialization, and temporal regularization over joints, vertices, rotations, root translation, and the body latent code. The second stage adds a bidirectional nearest-vertex loss between masked RGB-D person points and the complete SMPL-X surface.

Q2: Scalability and Prescanning

Our pipeline can be scaled to additional environments: it has no room-specific learned parameters, and the same protocol was applied across all 16 rooms. It requires a metric geometric reference, but not necessarily a complete room mesh. In SituHOI, we scan each room once because we aim to provide complete room geometry and globally scene-aligned trajectories; the scan is then reused across all recordings in that room.

If complete room geometry is unnecessary, a local map can instead be captured beforehand or reconstructed incrementally from the interaction RGB-D stream. This replaces the global localization front end while leaving downstream human and object fitting unchanged. Thus, prescanning supports the richer SituHOI data specification, while the underlying single-camera capture pipeline can operate with lighter local reconstruction.

AuthorsOpenReview ↗

Cont. Response to Reviewer wB9T

W2 and Q3: Physical Plausibility

To complement the existing benchmark metrics, we conducted two additional forms of physical evaluation:

  1. physics-based lifting simulations using reconstructed hand and object trajectories; and
  2. physical-consistency evaluations of the motions predicted by SharedHOI and the adapted baselines.

Physics-Based Lifting Simulation

We evaluated ten lifting motions from SituHOI using an effective friction coefficient of 0.5, a compliant contact stiffness of 300 N/m per contact, and a simulation timestep of 1/960 s. The object was modeled as a free rigid body under gravity. We retained the stored hand geometry, object mesh, and initial hand–object relative pose, and used the same contact setting across all experiments.

The objective was to test whether the reconstructed hand trajectories could support the objects throughout the corresponding lifting motions without post-hoc modification of the hand poses.

Across the ten experiments, we obtained:

MeasurementResult
Average object displacement98.79 mm
Average maximum hand–object relative drift1.67 mm
Average hand–object relative rotation change0.75°
Average hand-region penetration4.60 mm
Average total normal contact force61.47 N

The simulations also remained stable with timesteps of 1/240 s and 1/480 s and in a separate one-second stationary-hold test. In the corresponding no-hand control, the objects fell freely under gravity.

This experiment provides a functional diagnostic under a common compliant-contact model: it evaluates whether the reconstructed hand configurations can maintain the captured lifting behavior without adjusting the stored hand poses. It complements the MoCap experiment, which directly evaluates spatial trajectory accuracy.

Physical Metrics for Predicted Motions

We additionally evaluated three metrics on the frozen predictions, without applying physics-based post-processing:

  • body–floor penetration;
  • foot sliding; and
  • non-hand body–object penetration.

The evaluation uses a fixed, recording-balanced set of 135 non-overlapping windows from different rooms. The windows were selected independently of the prediction results. All methods use the same samples, three training seeds, and the representative long-forecast setting (H=0.5) s and (F=3.5) s.

MethodBody-floor depth (mm) ↓Foot sliding (cm/s) ↓Non-hand body–object penetration (%) ↓
CHOIS36.35 ± 3.5134.72 ± 2.161.51 ± 0.47
DecHOI38.82 ± 1.0137.72 ± 0.612.29 ± 0.05
InterDiff41.55 ± 2.8840.37 ± 2.341.89 ± 0.24
OMOMO48.02 ± 4.9148.80 ± 1.261.67 ± 0.23
SharedHOI33.96 ± 1.6132.82 ± 1.941.70 ± 0.26

Body–floor penetration is the frame-averaged maximum body-mesh depth below the shared scene floor. Foot sliding measures horizontal sole velocity on identical reference-defined support intervals for all methods. Non-hand body–object penetration is the percentage of area-uniform object-surface samples located more than 5 mm inside the non-hand body region; hands are excluded because intentional hand–object contact should not be counted as unwanted collision.

SharedHOI obtains the lowest mean body–floor penetration and foot sliding. CHOIS obtains the lowest non-hand body–object penetration, while SharedHOI remains comparable to the other methods on this metric.

The lifting simulations evaluate whether reconstructed hand configurations can functionally support object motion, whereas these three metrics evaluate the physical consistency of predicted full-body trajectories. Together, they complement the existing trajectory-accuracy and human–object coordination metrics.

We will revise the paper to clarify the hand-fitting and benchmark scope, report object-scale statistics, state the required room/object preparation explicitly, and add the complete protocols and quantitative results for the new physical-consistency diagnostics.

Thank you for your thorough rebuttal.

  1. I am still not convinced on the accuracy of the reconstructed hand given the state-of-the-art methods’ performance on vision-based human motion reconstruction.
  2. In your physical simulation experiment, while I appreciate the effort, the 61.47N average normal force suggest that objects can potentially be supported via interpenetration penalty forces rather than real grasps.
  3. On the small-object cases, if I understand the number correctly, the cup and bottle you used seems to be taller than a phone, which is quite not a “small object”.
  4. For several points you mentioned that you will revise, I would like to see the revised paper with the revision highlighted.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.