- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces SituHOI, a large-scale dataset for scene-situated full-body human-object interaction (HOI). To address the limitations of existing capture systems in scalability and structured scene-aligned supervision, the authors develop a capture and reconstruction pipeline based on a single moving RGB-D camera, together with prior scene and object scans. The resulting dataset contains approximately 86 hours of recordings and 18,429 annotated clips across 16 indoor rooms, providing human and object trajectories, scene geometry, and language descriptions in a shared coordinate frame.
Critical strengths
-
A valuable and well-motivated dataset contribution. The paper addresses an important limitation of existing HOI datasets: the difficulty of jointly obtaining full-body motion, object trajectories, scene geometry, and semantic annotations at scale. SituHOI offers a compelling combination of these properties through a relatively flexible single-camera capture setup. The dataset is reasonably large, and its scene-aligned multimodal annotations make it a valuable resource for scene-aware motion modeling and embodied intelligence. I particularly appreciate the clear positioning against existing datasets and the practical motivation behind the capture protocol.
-
Comprehensive validation and a useful benchmark. The paper goes beyond dataset construction by providing a systematic evaluation of reconstruction quality and downstream utility. The independent MoCap validation and component ablations support the reliability of the proposed pipeline. FlexHOI-Bench provides a meaningful and standardized setting for joint human-object motion prediction across different temporal horizons. More importantly, the cross-dataset transfer experiments demonstrate that SituHOI can benefit existing HOI models, providing convincing evidence of its broader research value. Overall, the dataset, benchmark, and experimental evaluation form a coherent and substantial contribution.
Critical weaknesses
The main limitations concern the extent of validation rather than the overall motivation or technical soundness. In particular, the diversity of participants and object categories remains somewhat limited, and the independent MoCap evaluation covers a relatively small subset of the collected data. These limitations do not substantially diminish the contribution, but further analysis would strengthen confidence in the dataset's general applicability.
Questions for authors
-
Generalization across scenes and participants. The dataset covers 16 indoor rooms and 15 participants. Could the authors provide further discussion or evaluation of generalization to unseen rooms, participants, or object categories? In particular, a room-disjoint benchmark evaluation would help clarify the extent to which the proposed models generalize beyond the training environments.
-
Annotation accuracy and interaction consistency. The independent MoCap validation provides useful evidence of reconstruction quality. However, precise human-object contact modeling can be sensitive to relatively small geometric errors. Could the authors further discuss the reliability of the recovered trajectories during close-contact interactions, potentially including additional contact-distance or penetration-related statistics?
-
Dataset accessibility and future extensions. Could the authors clarify the planned release scope, including processed trajectories, object and scene meshes, and benchmark evaluation scripts? It would also be interesting to discuss how the proposed capture framework could be extended to articulated objects or more complex interaction scenarios.
Response to Reviewer wB9T
We appreciate the reviewer’s recognition of the scale and potential value of SituHOI. Below, we address the concerns about hand reconstruction, object-scale coverage, scalability, and physical plausibility with quantitative evidence, while clarifying the scope of each evaluation.
W1(a): Reconstruction Quality
We evaluate the pipeline on an independently measured subset captured with synchronized RGB-D and MoCap. The full pipeline achieves a G-MPJPE of 68.3 mm, an RA-MPJPE of 47.6 mm, and an object-local wrist error of 51.6 mm. Under the same calibration and evaluation samples, the wrist error increases to 61.5 mm without human temporal regularization and to 55.2 mm without the depth-based surface loss. These controlled comparisons show that both components improve body and wrist-level human–object alignment.
To further investigate the functional effect of the remaining reconstruction error, we additionally conducted physics-based lifting simulations using the reconstructed hand poses. We provide the complete protocol and results in our response to W2. Without modifying the reconstructed hand geometry or initial hand–object configuration, the hands supported the objects throughout the simulated lifting motions.
The right panel of Fig. 3 visualizes SharedHOI predictions rather than the hand reconstruction pipeline. SharedHOI predicts root translation and 21 body rotations; finger articulation is not part of its output state. The rendered fingers therefore should not be interpreted as a prediction or quantitative evaluation of finger motion. We will revise the visualization and caption to remove this ambiguity. If space permits, we will separately provide source-RGB and fitted-SMPL-X overlays as qualitative reconstruction evidence.
W1(b): Small-Object Coverage
SituHOI includes objects spanning substantially different physical scales. For example, the cup and bottle meshes have approximate bounding-box dimensions of 13.216.818.3 cm and 10.214.218.9 cm and appear in 666 and 533 interaction clips, respectively. Together, they contribute 1,199 clips, representing 11.6% of the 10,335 dynamic-object clips. By comparison, box and chair—the two most frequent larger-object categories—account for 3,084 clips, or 29.8%. Thus, small-object interactions are represented by more than one thousand clips rather than isolated examples. We will add metric object dimensions to Fig. 2 to make both the physical-scale and frequency distributions explicit.
Q1: Hand Fitting
We do not use a separate hand-specific post-processing system. OSX initializes the SMPL-X pose and shape from RGB, while root translation is initialized separately by back-projecting depth in the detected person region. We then jointly refine the root translation/orientation, body pose, and both hand poses at the sequence level; body shape, jaw, and eye parameters remain fixed.
The fitting uses 63 matched RTMW body, foot, and hand landmarks, 40 of which correspond to the two hands. Detections below 0.70 confidence are discarded, and hand landmarks receive a weight of 0.65. The first stage combines robust 2D reprojection, body and hand priors, preservation of the image-based initialization, and temporal regularization over joints, vertices, rotations, root translation, and the body latent code. The second stage adds a bidirectional nearest-vertex loss between masked RGB-D person points and the complete SMPL-X surface.
Q2: Scalability and Prescanning
Our pipeline can be scaled to additional environments: it has no room-specific learned parameters, and the same protocol was applied across all 16 rooms. It requires a metric geometric reference, but not necessarily a complete room mesh. In SituHOI, we scan each room once because we aim to provide complete room geometry and globally scene-aligned trajectories; the scan is then reused across all recordings in that room.
If complete room geometry is unnecessary, a local map can instead be captured beforehand or reconstructed incrementally from the interaction RGB-D stream. This replaces the global localization front end while leaving downstream human and object fitting unchanged. Thus, prescanning supports the richer SituHOI data specification, while the underlying single-camera capture pipeline can operate with lighter local reconstruction.