acceptodds
Withdrawn from ICLR 2027

Collaborative Human-Object Interaction Reconstruction from a Monocular Video

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Collaborative human-object interaction (HOI) reconstruction requires recovering how several people manipulate a shared object over time. Existing video-based HOI reconstruction methods fall short in this scenario, as they neither align multiple human-object pairs in a shared 4D space nor reason about how contacts among multiple humans and the shared object evolve over time. To address this, we present COMBO, a 4D reconstruction method for reconstructing collaborative HOIs from a monocular video. Our key insight is that, in collaborative scenarios, the humans and the object are tightly coupled through contacts that evolve over time, and these time-varying contacts provide the constraint needed for joint reconstruction. We design a two-stage approach: human-anchored initialization first establishes a shared 4D space, using human motions to calibrate depth, and initializes the object. Subsequently, temporal contact-guided optimization builds a proximity-based temporal contact graph from 2D and 3D cues, without a learned contact predictor. Unlike static or single-pair contact formulations, our graph captures temporally varying contacts across participants, guiding the joint alignment of all entities in a shared 4D space. Experiments on the CORE4D benchmark show lower reconstruction errors than competing video-based methods. Our results on in-the-wild and generated videos demonstrate the generalization capability of COMBO. Code will be publicly available upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
4D reconstruction, Collaborative Human-Object Interaction, human-object interaction

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_mQGKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes COMBO, a training-free framework for 4D collaborative HOI reconstruction from monocular video using human-anchored initialization and a proximity-based temporal contact graph for joint human-object refinement. Experiments demonstrate improved reconstruction performance over existing methods on CORE4D, supported by ablation studies and qualitative evaluations on in-the-wild and generated videos.

Critical strengths
  1. This is a pioneering paper that tackles collaborative human-object reconstruction from monocular video in training-free manner.
  2. COMBO is not dependent on known 3D object templates or ground-truth camera parameters. Considering that these are common limitations of previous HOI reconstruction methods, this improves the applicability of the proposed framework.
  3. Temporal contact graph effectively couples multiple humans and an object in collaborative interaction through temporally coupled contact.
  4. Table 1 demonstrates substantial improvements in Chamfer Distance for both human and object reconstruction. Figure 3 also shows promising qualitative results for collaborative interaction reconstruction.
Critical weaknesses
  1. While the problem formulation is important, the proposed methodology sounds more of an integration of existing 3D reconstruction methods with proximity-based contact optimization rather than a fundamentally new reconstruction framework. In particular, the temporal contact graph is constructed using 2D/3D proximity and temporal filtering. It is difficult to understand how this graph performs temporal interaction reasoning beyond identifying frame-wise contact relationships. More explanation on the core methodological contribution would strengthen this paper. Also, as proximity-based temporal contact graph is major part of this method, extensive comparisons with off-the-shelf dense contact estimation models, such as DECO (Tripathi et al., ICCV 2023), InteractVLM (Dwivedi et al., CVPR 2025), HACO (Jung et al., NeurIPS 2025) are needed.
  2. The proposed method heavily relies on the initial human and object reconstruction to estimate contact relationships. However, contact is inferred from the proximity of these initial predictions, which may already contain substantial 3D errors. Since the estimated contacts are subsequently used to refine the human and object geometry, it is unclear how the method prevents incorrect initial contact estimates from propagating throughout the optimization. The evaluation on error rate (F1-score) of contact compared with GT is needed.
Questions for authors
  1. In L219-L223, the paper reconstructs object geometry from the first video frame using an image-to-3D model. What happens when the object is severely occluded or only a small part of the object is visible in the first frame? How much does the quality of initial object geometry influence the final reconstruction?
  2. In Equation 2, the temporal contact graph is represented as a sequence of frame-wise contact graphs. Could you further explain what makes the proposed graph a temporal reasoning mechanism rather than a sequence of independently estimated binary contact labels?
  3. In Equation 3 and Equation 4, the contact relationships are determined using 2D and 3D proximity. However, geometric proximity does not necessarily indicate physical contact. How does the method distinguish actual contact from a hand being close to an object without touching it?
  4. For proximity-based temporal contact graph, why not learn contact estimation directly or use off-the-shelf contact estimation models like DECO (Tripathi et al., ICCV 2023), InteractVLM (Dwivedi et al., CVPR 2025), or HACO (Jung et al., NeurIPS 2025)? My concern is that contact-based object optimization might not be accurate if the contact itself comes from erroneous initial 3D human and object reconstruction. Why such method is employed and could you evaluate if this approach is better than learned contact predictor?
  5. In Equation 6, the contact loss is computed using the closest hand vertices to the object. However, actual collaborative interaction often involves multiple contact regions with different spatial constraints. How does the proposed formulation preserve the correct contact locations rather than simply minimizing the distance between hands and the object?
  6. The temporal refinement assumes that contacting hands remain approximately static relative to the object. How does the method handle sliding contacts or interactions where the grasping hand moves along the object surface?
  7. In the joint human-object refinement, only human arm poses are optimized (as explained in L188) while human root translation and orientation remain fixed. Why does the method restrict the optimization to the arms? Collaborative interaction can also involve full-body coordination and need optimization on corresponding body parts.
  8. The improvements in Table 1 are very substantial, particularly for object reconstruction. How much of this improvement comes from the foundation models used for initialization, and how much comes from the proposed temporal contact graph and joint optimization? Table 2 provides component ablations, but it would be great to explain this distinction more clearly. Also, if most of the performance gap from Table 1 comes from foundation models, it is ambiguous if the proposed method truly proposed effective approach for collaborative human-object interaction reconstruction.
  9. In Appendix A.4, the paper reports approximately 40 minutes for reconstructing a 70-frame sequence. Considering that the paper emphasizes reconstructing collaborative HOIs from everyday videos, what is the expected practical application of this computationally expensive pipeline?
  10. In Figure 3 and Figure 4, the qualitative results are promising. However, the contact graph appears to focus on hand-object interactions. How would COMBO handle collaborative interactions involving other body parts, such as supporting a large object with the torso or shoulder?
Reviewer_ySh9OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The authors introduce COMBO, a training-free framework designed to reconstruct collaborative human-object interactions from monocular video. The method begins by initializing human motion, scene depth, and object geometry using pretrained foundation models. It employs reconstructed human meshes to calibrate metric depth and initializes an object mesh along with a 6-DoF trajectory.

In the second stage, a temporal graph of hand-object contacts is built using both 2D and 3D proximity data. This graph then guides an optimization process focused solely on the object, followed by a joint refinement of the object and the arm poses of the humans involved.

The authors evaluate their framework on a selected subset of the CORE4D-Real dataset and report improvements in both human and object reconstruction compared to adapted CARI4D and HOI-PAGE baselines. Additionally, they present qualitative results from captured and generated videos.

Critical strengths
  1. This paper presents a significant advancement in monocular human-object interaction (HOI) reconstruction. While most existing video-based HOI methods focus on a single human interacting with one object, collaborative interactions often involve multiple humans maintaining contact with the same moving object. Therefore, the need to model simultaneous contacts and transfers of contact is clearly justified.

  2. The proposed pipeline does not rely on pre-scanned object templates or ground-truth camera parameters. Instead, it integrates pretrained human reconstruction, depth estimation, image-to-3D reconstruction, and object tracking into a cohesive metric reconstruction pipeline. This approach is practically beneficial, as collecting object templates and calibrated multi-view recordings can be costly.

  3. The two-stage optimization process is straightforward. By first optimizing the object while keeping human motion fixed, the number of variables during the initial contact correction is minimized. This is followed by a joint refinement stage that allows the human arms and the object to adjust in unison. The ablation study presented in Table 2 demonstrates that the complete model yields improved Chamfer distances for humans, objects, and the combined results when compared to models lacking either stage of optimization.

  4. COMBO reports strong reconstruction results on its evaluated CORE4D sequences. In the template-free setting, Table 1 reports an object Chamfer distance of 28.86 cm compared with 134.28 cm for the adapted CARI4D baseline. The paper also provides detailed implementation settings and visual examples of collaborative carrying and handover.

Critical weaknesses

Major Weaknesses

1. The primary contribution of this work is the temporal contact graph; however, the evaluation of its contact predictions is not conducted independently from simpler proximity constraints.

The main innovation of COMBO is the proximity-based temporal contact graph, designed to identify when individuals grasp, release, or transfer a shared object. Nevertheless, the paper lacks a quantitative assessment of the predicted contact edges.

To construct the graph, the method thresholds the geometric mean of 2D and 3D proximity scores and applies a nine-frame median filter. These processes produce a binary decision regarding contact, but it's essential to note that physical contact does not equate to proximity. For instance, a hand may stay close to an object after releasing it or approach an object before actually grasping it. Furthermore, inaccurate depth estimates for humans or objects might lead to a genuinely contacting hand appearing to be separated in 3D space.

The paper does not provide metrics such as contact precision, recall, F1 score, or contact-transition accuracy. Additionally, it does not demonstrate whether the contact graph accurately identifies the moments when handovers begin and end. This aspect is crucial because the advantage claimed over single-person methods stems from modeling the changing nature of contact relationships.

The ablation studies presented do not isolate this contribution either. Table 2 removes human temporal refinement, object-only optimization, and joint refinement, but it does not replace the proposed temporal graph with simpler alternatives. For example, the authors could have employed frame-wise geometric proximity without temporal filtering, fixed contact assignments, or used contact estimates from only one person. Implementing these controls would help clarify whether the observed improvements arise from temporal contact reasoning or merely from incorporating hand-object distance constraints into the optimizer.

Without these experiments, the results indicate that the complete contact-guided optimization pipeline enhances reconstruction, but they do not confirm that the proposed temporal graph accurately captures collaborative contact dynamics or contributes beyond what standard proximity-based fitting provides.

2. The method estimates contact points from an initial reconstruction and keeps them fixed during optimization. However, it does not demonstrate that incorrect initial contacts can be corrected.

Section 3.3 constructs the contact graph using the initialized human meshes, object masks, and calibrated object depth. Then, Section 3.4 utilizes these estimated contact edges to optimize the positions of the objects and human arms.

This creates a dependence between the initial reconstruction and the constraints used to improve it.

If the object is initially misplaced, a hand that is genuinely in contact may appear too far away and receive no active contact edge. Conversely, a misaligned hand may be close enough to the reconstructed object to be assigned a false contact constraint. Once these binary decisions are made, the optimizer treats them as the contact relationships that should be enforced.

The paper does not address updating the contact graph after the poses of the object and humans change. As a result, an incorrect contact assignment may remain active throughout the optimization process, and a missed contact may never be included.

This is especially concerning for collaborative manipulation scenarios where the initial pose of an occluded object might be inaccurate, leading to changes in contact assignments during handovers.

The authors should compare the current fixed-contact strategy with an alternating approach that re-estimates contacts after refining object or joint poses. Additionally, it would be beneficial to perturb the initialized object pose and assess how often the correct contact relationships are recovered.

The current method may perform well when the foundation model initialization is nearly accurate. However, the paper does not demonstrate that temporal contact-guided optimization can recover from incorrect initial contact assignments, despite presenting this recovery as a key contribution.

3. The baseline comparison does not adequately isolate the benefits of collaborative reconstruction from the effects of adapting methods designed for different tasks.

The main comparison includes CARI4D, HOI-PAGE, and MMHOI-Net, which address related but distinct problems.

CARI4D reconstructs one human interacting with one object. COMBO adapts this approach by running it separately for each human and averaging the resulting object translations and rotations, without any joint refinement after this fusion. While this method provides a reasonable way to create a two-human baseline, it does not test whether CARI4D's outputs could be improved through simple shared-object optimization.

HOI-PAGE, initially a text-driven human-object interaction generation method, has had its video diffusion stage removed by COMBO, which supplies the input video as a reference for reconstruction. This alteration changes its original task and pipeline, making the resulting performance difficult to interpret as a representative reconstruction baseline.

MMHOI-Net directly models multiple humans and objects, but the numbers presented in Table 1 are taken from its original paper under a different evaluation protocol. Although this manuscript acknowledges the discrepancy, the comparison is still presented alongside other reconstruction results.

I recognize that directly comparable collaborative video reconstruction methods are limited. Nevertheless, the strongest evidence for COMBO should stem from controlled variants of its own pipeline.

For instance, the authors could maintain the same human estimates, depth calibration, canonical object mesh, and initialized object trajectory, and then compare independent human-object fitting, standard joint optimization without temporal contacts, and the proposed graph-guided optimization.

This approach would help determine whether COMBO's improvements arise from collaborative contact reasoning rather than stronger foundational model initialization or differences in how competing methods were adapted.

4. The quantitative evaluation uses a selected CORE4D subset, but the paper does not adequately establish performance on the challenging collaborative interactions it claims to address.

Section 4.1 states that the evaluation utilizes the full test set of the authors' benchmark. However, Appendix B.1 clarifies that the benchmark was constructed by extracting carrying segments from CORE4D-Real S1, selecting the camera view where both humans and the object are most visible, and removing sequences with thin objects, inadequate visibility, or inconsistent ground truth.

As a result, the evaluation consists of 150 sequences and 10,930 frames across five object categories.

While these decisions may be reasonable for constructing a reliable reconstruction benchmark, especially given some inaccurate annotations, they also exclude conditions central to the paper's motivation.

Collaborative manipulation often involves occluded hands, partially visible objects, complex object geometries, and variations in which participant supports the object. Selecting the clearest camera view and excluding challenging shapes can significantly reduce these complications.

The authors should report how many sequences were excluded under each criterion and provide performance metrics broken down by object category, visibility, and interaction type. Furthermore, I would expect an evaluation specifically targeting handovers, joining and leaving interactions, and simultaneous two-person contact.

Without such a breakdown, the results demonstrate strong performance on a curated subset of collaborative carrying sequences but do not fully substantiate the claims regarding general collaborative interactions and changing contact relationships.

5. The ablation study does not determine which components of the proposed optimization are responsible for the observed improvements, and several individual metrics deteriorate.

COMBO utilizes human temporal refinement, human-anchored depth calibration, object initialization, proximity-based contact estimation, object-only optimization, and joint human-object refinement.

However, Table 2 only examines three components: human temporal refinement, object-only optimization, and joint refinement.

It does not consider human-anchored depth calibration, the temporal median contact filter, the differences between 2D and 3D contact cues, the contact drift term, the penetration term, or the sequential optimization schedule against a matched single-stage alternative.

These omissions are particularly significant because the proposed method primarily relies on combining pretrained estimates through geometric constraints.

The numerical results indicate that not every component enhances every metric. For example, removing joint refinement improves PA-MPJPE from 6.17 to 6.08 cm and reduces object rotation error from 15.26 to 15.17 degrees. Similarly, removing object-only optimization results in a slightly lower rotation error.

The authors acknowledge that optimizing the joint configuration can trade individual pose accuracy for relative alignment. While this is a reasonable explanation, the paper does not directly measure the relative hand-object contact consistency that is allegedly improved enough to justify this trade-off.

A more thorough ablation study should isolate geometric and temporal losses, compare stage orderings, and report a contact-specific or relative-configuration metric alongside the existing Chamfer distances.

This would clarify which components are truly impactful and whether the claimed improvement in collaborative interaction consistency is distinct from an enhancement in overall mesh alignment.

6. The human-anchored depth and object initialization pipeline introduces significant sources of systematic error that are not evaluated separately.

COMBO estimates a single global depth scale and shift by aligning predicted human-region depth with depth rendered from reconstructed human meshes. It applies the same calibration to the object.

This process relies on the reconstructed human meshes providing a reliable metric reference. Any systematic errors in the predicted human dimensions or camera-space positioning can affect the estimated depth calibration, thereby impacting the object’s scale and translation.

Related work, such as HAC, has analyzed the use of reconstructed humans as metric references. Although COMBO uses a different scale-and-shift formulation, it does not discuss how its calibration performs when the human estimates are inaccurate.

Object reconstruction introduces another dependency. COMBO generates the canonical object mesh from the first frame using SAM 3D, aligns its orientation and scale, and then maintains the geometry fixed for the remainder of the video. If the first frame is heavily occluded or reveals only one side of the object, the generated geometry may be inaccurate. While contact refinement can correct object motion and human arm poses, it cannot rectify an incorrect canonical object shape.

The authors acknowledge first-frame occlusion as a limitation but do not measure its effects or differentiate the gains from depth calibration, object initialization, and contact optimization.

A controlled experiment employing ground-truth or perturbed object geometry, alternative initialization frames, and adjusted human scale would help identify which errors the method can effectively correct.

This is especially crucial for a template-free reconstruction method, as its accuracy relies heavily on the quality of the generated object mesh as well as the subsequent contact-guided optimization.

Minor Weaknesses

  1. The temporal contact graph captures the relationships of contact that can hold, break, and re-form. However, mathematically, it consists of a sequence of independently estimated binary hand-object edges followed by median filtering. There is no explicit model for grasp states, contact persistence, or transfer dynamics. The authors should clarify the distinction between the temporal representation and learned or structured temporal contact reasoning.

  2. The conventions for coordinate frames need clarification. The method utilizes per-frame depth and camera intrinsics, human translations, and object poses, but it does not clearly explain how camera motion is managed when transforming all entities into a shared world coordinate frame. This is particularly important for the claim that the method supports monocular videos without requiring ground-truth camera parameters.

  3. The evaluation of generalization in real-world scenarios is limited. The user study includes only nine videos, consisting of two actual recordings and seven generated videos. The prompt for generating videos specifically requests fixed camera settings, full visibility, minimal occlusion, and simple backgrounds. These conditions are favorable for the proposed pipeline. While the user and VLM scores provide useful perceptual evidence, they do not demonstrate broad generalization to unconstrained collaborative interactions.

  4. The method is computationally expensive, despite being training-free. Appendix A.4 reports that it takes approximately 40 minutes to reconstruct a 70-frame video on an RTX A6000. Although this timeframe may be reasonable for an optimization-based reconstruction system, the practical limitations should be discussed more explicitly, especially considering the paper's goal of producing scalable collaborative interaction data.

Summary

COMBO addresses a significant problem and offers a valuable reconstruction pipeline for two-person object manipulation. Its key strength lies in its ability to combine pretrained models of humans, scenes, and objects with joint optimization, allowing multiple human hands to effectively manipulate a single object.

However, there is limited evidence supporting the proposed temporal contact graph. The paper does not evaluate the accuracy of the contact predictions, nor does it separate these predictions from simpler proximity-based constraints. Furthermore, the graph is constructed based on initial geometry and remains fixed during optimization, which raises questions about the method's ability to recover from incorrect initial contact assignments.

The baseline comparisons are challenging to interpret because the main competing methods are adapted from single-person reconstruction or text-driven generation. Additionally, the evaluation utilizes a curated subset of CORE4D, excluding instances with difficult visibility conditions and thin objects.

While I find the results promising, the current experiments do not adequately demonstrate that the proposed temporal contact representation is responsible for the observed improvements or that the method can generalize to the more complex collaborative interactions outlined in the motivation.

Questions for authors

Questions

  1. Quantitative validation of the temporal contact graph. The authors are requested to report contact precision, recall, F1, and contact-transition accuracy, distinguishing simultaneous carrying, initial grasp, release, and handover. Please also compare: (i) 2D-only proximity, (ii) 3D-only proximity, (iii) combined 2D/3D proximity without temporal filtering, and (iv) the proposed temporally filtered graph. This would establish whether the contact representation actually captures the changing relationships that motivate the method. (Refer to major weakness 1.)

  2. Robustness to incorrect initial contact assignments. Please perturb the initial object translation and rotation and report how contact accuracy and reconstruction error change. Compare the current fixed-contact graph against an alternating approach that re-estimates contact edges after object or human refinement. This would show whether COMBO can recover from poor initialization or only correct small errors around an already reliable estimate. (Refer to major weakness 2.)

  3. Controlled collaborative optimization baseline. Please evaluate variants using identical human meshes, calibrated depth, canonical object geometry, and initial object poses: (i) independent human-object fitting, (ii) independent fitting followed by simple object-pose fusion, (iii) joint fitting without temporal contact edges, and (iv) full COMBO. This would more clearly isolate the benefit of jointly modeling multiple humans and temporal contacts. Please also clarify the interpretation of the adapted CARI4D and HOI-PAGE results and keep MMHOI-Net's different evaluation protocol clearly separated. (Refer to major weakness 3.)

  4. Evaluation population and difficult interactions. Please report the number of CORE4D S1 sequences removed for thin objects, visibility, and inconsistent ground truth. Provide results by object category, human-object occlusion, and interaction type, especially handover, joining, leaving, and simultaneous two-person carrying. These breakdowns would show whether the method handles actual contact changes rather than mainly well-observed carrying sequences. (Refer to major weakness 4.)

  5. Complete component and loss ablations. Please ablate human-anchored depth calibration, the 2D/3D contact threshold, temporal filtering window, contact drift loss, penetration loss, and the two-stage optimization order. Include a matched single-stage optimizer and report hand-object distance, penetration, and contact stability alongside reconstruction errors. These controls would establish why the proposed combination and schedule are necessary. (Refer to major weakness 5.)

  6. Error propagation from pretrained initialization. Please compare the normal pipeline against settings using ground-truth or deliberately perturbed human scale, object geometry, and camera parameters where available. Vary the object initialization frame, including less visible frames, and report how object Chamfer, translation, rotation, and contact accuracy change. The authors should also explain whether the single global depth scale and shift remain reliable under camera movement and inaccurate human-body predictions. (Refer to major weakness 6.)

  7. Practical generalization and coordinate conventions. Please explain how per-frame depth, human translations, and object poses are transformed into a shared world coordinate system when the camera moves. Report runtime broken down by initialization, object reconstruction, tracking, and optimization, and clarify the intended practical operating regime given the approximately 40-minute processing time for 70 frames. Finally, the nine-video user/VLM study should be presented as limited perceptual evidence rather than sufficient proof of broad generalization. (Refer to minor weaknesses 2–4.)

References

  1. Xie et al., CARI4D: Category Agnostic 4D Reconstruction of Human-Object Interaction, CVPR 2026.
  2. Li and Dai, HOI-PAGE: Zero-Shot Human-Object Interaction Generation with Part Affordance Guidance, ICML 2026.
  3. Kogashi et al., MMHOI: Modeling Complex 3D Multi-Human Multi-Object Interactions, WACV 2026.
  4. Liu et al., CORE4D: A 4D Human-Object-Human Interaction Dataset for Collaborative Object Rearrangement, CVPR 2025.
  5. Yang et al., Humans as Checkerboards: Calibrating Camera Motion Scale for World-Coordinate Human Mesh Recovery, ICCV 2025.
  6. Chen et al., SAM 3D: 3Dfy Anything in Images, CVPR 2026.
  7. Zhang et al., ProxyPose: 6-DoF Pose Tracking via Video-to-Video Translation, 2026.
Reviewer_NLnzOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The work proposes COMBO, a training-free method for reconstructing collaborative human-object interaction (HOI) from an RGB video. The method does not require an object template and GT camera information. COMBO works in two stages: first the humans and object are initialized using several foundation models (SAM3, SAM 3D, Depth Anything 3, Sapiens2, NLF, and ProxyPose), then the initial estimates are refined with the help of the proposed Temporal Contact Graph. The graph encodes the contacts between hands of each human and the object over time and is built from 2D and 3D proximity cues computed from stage 1 predictions. COMBO is evaluated on a subset of CORE4D-Real and compared with CARI4D (single person method, which is run twice) and HOI-PAGE (adapted to reconstruction). Evaluation is complemented with user study and VLM-based rating on 2 in-the-wild and 7 generated clips.

Critical strengths
  • Underexplored and well-motivated problem. Reconstruction of multi-person human-object interactions from monocular videos is a clear gap in capturing HOI with concrete applications for downstream tasks.
  • COMBO has significant generalization potential. The method is training-free, template-free and relies on a number of foundation models that are not limited by the narrow availability of interaction-specific data. The generalization is shown qualitatively on several in-the-wild and synthesized videos.
Critical weaknesses

The weaknesses of the work are concentrated in the evaluation:

  1. Weak baselines. Object mesh reconstruction plays significant role in the quality of results, so a fair comparison with baselines (e.g.,CARI4D) would share similar object mesh reconstruction and tracking pipeline (i.e. upper half of Table 1 unifies starting object mesh, but tracking in CARI4D still relies on a set of older models: FoundationPose+UniDepth vs. ProxyPose+DepthAnything3). The current setting does not make the source of the performance gain clear: using a newer stack of large models or the actual design of the method.
  2. Actual benefit of contact graph is not demonstrated. Current ablation only tests Stage 1 + either half of Stage 2 optimization. A fairer comparison would test Stage 1 only as well, then improvement of the full method over it would be the actual benefit of the contact graph (and optimization on top of it). Moreover ablation table is not comparable to Table 1, because it uses a narrower set of sequences.
  3. No contact-based metrics in evaluation. The work claims to improve "physical plausibility" (Line 084), yet the evaluation does not quantify it.
  4. Configurations with more than 2 people are not tested. The text uses "several" to refer to multi-person human-object interaction, yet the configuration with more than two people is not demonstrated in the work (e.g., qualitative evaluation on some multi-person segment of HOI-M3 or MMHOI).
  5. Small evaluation set. Section 4.1 claims "using the full test set of our benchmark" (Line 315), while Section B.1 describes that the test set of CORE4D-Real is thinned by extracting segments, excluding objects, etc. Moreover, only S1 subset of CORE4D is used for evaluation.

Minor:

  1. Confusing naming of stages. Introduction and Figure 2 use Stage 1 and Stage 2 to refer to initialization and optimization respectively. At the same time, Table 2 and Section 4.4 use Stage 1 and Stage 2 (as well as stage-1 and stage-2) to refer to parts of optimization pipeline. So in line with original notation these are stage 2.1 and stage 2.2.
Questions for authors

Could the authors please clarify the gaps in the evaluation mentioned above, namely:

  1. Have the variants of COMBO with older foundation models been tested (e.g., initialization from FoundationPose+UniDepth) or is it possible to evaluate CARI4D with a newer stack of foundation models (e.g., ProxyPose+Depth Anything 3)?
  2. What are the results of COMBO with Stage 1 only (i.e., without contact optimization)?
  3. Would it be possible to evaluate the quality of contacts to support the claim of improved physical plausibility?
  4. Has COMBO been tested with more than 2 people in a sequence?
  5. What is the primary reason for using only the S1 subset of CORE4D-Real? Will the evaluation sequence list with segment boundaries be released to ensure reproducibility?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.