Major Weaknesses
1. The primary contribution of this work is the temporal contact graph; however, the evaluation of its contact predictions is not conducted independently from simpler proximity constraints.
The main innovation of COMBO is the proximity-based temporal contact graph, designed to identify when individuals grasp, release, or transfer a shared object. Nevertheless, the paper lacks a quantitative assessment of the predicted contact edges.
To construct the graph, the method thresholds the geometric mean of 2D and 3D proximity scores and applies a nine-frame median filter. These processes produce a binary decision regarding contact, but it's essential to note that physical contact does not equate to proximity. For instance, a hand may stay close to an object after releasing it or approach an object before actually grasping it. Furthermore, inaccurate depth estimates for humans or objects might lead to a genuinely contacting hand appearing to be separated in 3D space.
The paper does not provide metrics such as contact precision, recall, F1 score, or contact-transition accuracy. Additionally, it does not demonstrate whether the contact graph accurately identifies the moments when handovers begin and end. This aspect is crucial because the advantage claimed over single-person methods stems from modeling the changing nature of contact relationships.
The ablation studies presented do not isolate this contribution either. Table 2 removes human temporal refinement, object-only optimization, and joint refinement, but it does not replace the proposed temporal graph with simpler alternatives. For example, the authors could have employed frame-wise geometric proximity without temporal filtering, fixed contact assignments, or used contact estimates from only one person. Implementing these controls would help clarify whether the observed improvements arise from temporal contact reasoning or merely from incorporating hand-object distance constraints into the optimizer.
Without these experiments, the results indicate that the complete contact-guided optimization pipeline enhances reconstruction, but they do not confirm that the proposed temporal graph accurately captures collaborative contact dynamics or contributes beyond what standard proximity-based fitting provides.
2. The method estimates contact points from an initial reconstruction and keeps them fixed during optimization. However, it does not demonstrate that incorrect initial contacts can be corrected.
Section 3.3 constructs the contact graph using the initialized human meshes, object masks, and calibrated object depth. Then, Section 3.4 utilizes these estimated contact edges to optimize the positions of the objects and human arms.
This creates a dependence between the initial reconstruction and the constraints used to improve it.
If the object is initially misplaced, a hand that is genuinely in contact may appear too far away and receive no active contact edge. Conversely, a misaligned hand may be close enough to the reconstructed object to be assigned a false contact constraint. Once these binary decisions are made, the optimizer treats them as the contact relationships that should be enforced.
The paper does not address updating the contact graph after the poses of the object and humans change. As a result, an incorrect contact assignment may remain active throughout the optimization process, and a missed contact may never be included.
This is especially concerning for collaborative manipulation scenarios where the initial pose of an occluded object might be inaccurate, leading to changes in contact assignments during handovers.
The authors should compare the current fixed-contact strategy with an alternating approach that re-estimates contacts after refining object or joint poses. Additionally, it would be beneficial to perturb the initialized object pose and assess how often the correct contact relationships are recovered.
The current method may perform well when the foundation model initialization is nearly accurate. However, the paper does not demonstrate that temporal contact-guided optimization can recover from incorrect initial contact assignments, despite presenting this recovery as a key contribution.
3. The baseline comparison does not adequately isolate the benefits of collaborative reconstruction from the effects of adapting methods designed for different tasks.
The main comparison includes CARI4D, HOI-PAGE, and MMHOI-Net, which address related but distinct problems.
CARI4D reconstructs one human interacting with one object. COMBO adapts this approach by running it separately for each human and averaging the resulting object translations and rotations, without any joint refinement after this fusion. While this method provides a reasonable way to create a two-human baseline, it does not test whether CARI4D's outputs could be improved through simple shared-object optimization.
HOI-PAGE, initially a text-driven human-object interaction generation method, has had its video diffusion stage removed by COMBO, which supplies the input video as a reference for reconstruction. This alteration changes its original task and pipeline, making the resulting performance difficult to interpret as a representative reconstruction baseline.
MMHOI-Net directly models multiple humans and objects, but the numbers presented in Table 1 are taken from its original paper under a different evaluation protocol. Although this manuscript acknowledges the discrepancy, the comparison is still presented alongside other reconstruction results.
I recognize that directly comparable collaborative video reconstruction methods are limited. Nevertheless, the strongest evidence for COMBO should stem from controlled variants of its own pipeline.
For instance, the authors could maintain the same human estimates, depth calibration, canonical object mesh, and initialized object trajectory, and then compare independent human-object fitting, standard joint optimization without temporal contacts, and the proposed graph-guided optimization.
This approach would help determine whether COMBO's improvements arise from collaborative contact reasoning rather than stronger foundational model initialization or differences in how competing methods were adapted.
4. The quantitative evaluation uses a selected CORE4D subset, but the paper does not adequately establish performance on the challenging collaborative interactions it claims to address.
Section 4.1 states that the evaluation utilizes the full test set of the authors' benchmark. However, Appendix B.1 clarifies that the benchmark was constructed by extracting carrying segments from CORE4D-Real S1, selecting the camera view where both humans and the object are most visible, and removing sequences with thin objects, inadequate visibility, or inconsistent ground truth.
As a result, the evaluation consists of 150 sequences and 10,930 frames across five object categories.
While these decisions may be reasonable for constructing a reliable reconstruction benchmark, especially given some inaccurate annotations, they also exclude conditions central to the paper's motivation.
Collaborative manipulation often involves occluded hands, partially visible objects, complex object geometries, and variations in which participant supports the object. Selecting the clearest camera view and excluding challenging shapes can significantly reduce these complications.
The authors should report how many sequences were excluded under each criterion and provide performance metrics broken down by object category, visibility, and interaction type. Furthermore, I would expect an evaluation specifically targeting handovers, joining and leaving interactions, and simultaneous two-person contact.
Without such a breakdown, the results demonstrate strong performance on a curated subset of collaborative carrying sequences but do not fully substantiate the claims regarding general collaborative interactions and changing contact relationships.
5. The ablation study does not determine which components of the proposed optimization are responsible for the observed improvements, and several individual metrics deteriorate.
COMBO utilizes human temporal refinement, human-anchored depth calibration, object initialization, proximity-based contact estimation, object-only optimization, and joint human-object refinement.
However, Table 2 only examines three components: human temporal refinement, object-only optimization, and joint refinement.
It does not consider human-anchored depth calibration, the temporal median contact filter, the differences between 2D and 3D contact cues, the contact drift term, the penetration term, or the sequential optimization schedule against a matched single-stage alternative.
These omissions are particularly significant because the proposed method primarily relies on combining pretrained estimates through geometric constraints.
The numerical results indicate that not every component enhances every metric. For example, removing joint refinement improves PA-MPJPE from 6.17 to 6.08 cm and reduces object rotation error from 15.26 to 15.17 degrees. Similarly, removing object-only optimization results in a slightly lower rotation error.
The authors acknowledge that optimizing the joint configuration can trade individual pose accuracy for relative alignment. While this is a reasonable explanation, the paper does not directly measure the relative hand-object contact consistency that is allegedly improved enough to justify this trade-off.
A more thorough ablation study should isolate geometric and temporal losses, compare stage orderings, and report a contact-specific or relative-configuration metric alongside the existing Chamfer distances.
This would clarify which components are truly impactful and whether the claimed improvement in collaborative interaction consistency is distinct from an enhancement in overall mesh alignment.
6. The human-anchored depth and object initialization pipeline introduces significant sources of systematic error that are not evaluated separately.
COMBO estimates a single global depth scale and shift by aligning predicted human-region depth with depth rendered from reconstructed human meshes. It applies the same calibration to the object.
This process relies on the reconstructed human meshes providing a reliable metric reference. Any systematic errors in the predicted human dimensions or camera-space positioning can affect the estimated depth calibration, thereby impacting the object’s scale and translation.
Related work, such as HAC, has analyzed the use of reconstructed humans as metric references. Although COMBO uses a different scale-and-shift formulation, it does not discuss how its calibration performs when the human estimates are inaccurate.
Object reconstruction introduces another dependency. COMBO generates the canonical object mesh from the first frame using SAM 3D, aligns its orientation and scale, and then maintains the geometry fixed for the remainder of the video. If the first frame is heavily occluded or reveals only one side of the object, the generated geometry may be inaccurate. While contact refinement can correct object motion and human arm poses, it cannot rectify an incorrect canonical object shape.
The authors acknowledge first-frame occlusion as a limitation but do not measure its effects or differentiate the gains from depth calibration, object initialization, and contact optimization.
A controlled experiment employing ground-truth or perturbed object geometry, alternative initialization frames, and adjusted human scale would help identify which errors the method can effectively correct.
This is especially crucial for a template-free reconstruction method, as its accuracy relies heavily on the quality of the generated object mesh as well as the subsequent contact-guided optimization.
Minor Weaknesses
-
The temporal contact graph captures the relationships of contact that can hold, break, and re-form. However, mathematically, it consists of a sequence of independently estimated binary hand-object edges followed by median filtering. There is no explicit model for grasp states, contact persistence, or transfer dynamics. The authors should clarify the distinction between the temporal representation and learned or structured temporal contact reasoning.
-
The conventions for coordinate frames need clarification. The method utilizes per-frame depth and camera intrinsics, human translations, and object poses, but it does not clearly explain how camera motion is managed when transforming all entities into a shared world coordinate frame. This is particularly important for the claim that the method supports monocular videos without requiring ground-truth camera parameters.
-
The evaluation of generalization in real-world scenarios is limited. The user study includes only nine videos, consisting of two actual recordings and seven generated videos. The prompt for generating videos specifically requests fixed camera settings, full visibility, minimal occlusion, and simple backgrounds. These conditions are favorable for the proposed pipeline. While the user and VLM scores provide useful perceptual evidence, they do not demonstrate broad generalization to unconstrained collaborative interactions.
-
The method is computationally expensive, despite being training-free. Appendix A.4 reports that it takes approximately 40 minutes to reconstruct a 70-frame video on an RTX A6000. Although this timeframe may be reasonable for an optimization-based reconstruction system, the practical limitations should be discussed more explicitly, especially considering the paper's goal of producing scalable collaborative interaction data.
Summary
COMBO addresses a significant problem and offers a valuable reconstruction pipeline for two-person object manipulation. Its key strength lies in its ability to combine pretrained models of humans, scenes, and objects with joint optimization, allowing multiple human hands to effectively manipulate a single object.
However, there is limited evidence supporting the proposed temporal contact graph. The paper does not evaluate the accuracy of the contact predictions, nor does it separate these predictions from simpler proximity-based constraints. Furthermore, the graph is constructed based on initial geometry and remains fixed during optimization, which raises questions about the method's ability to recover from incorrect initial contact assignments.
The baseline comparisons are challenging to interpret because the main competing methods are adapted from single-person reconstruction or text-driven generation. Additionally, the evaluation utilizes a curated subset of CORE4D, excluding instances with difficult visibility conditions and thin objects.
While I find the results promising, the current experiments do not adequately demonstrate that the proposed temporal contact representation is responsible for the observed improvements or that the method can generalize to the more complex collaborative interactions outlined in the motivation.