- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes TraceForcing-HOI, a framework that integrates the persistent identity, contact information, and interaction state of hands and objects for fine-grained Hand-Object Interaction Video understanding, and constructs it through Fast-Slow tracking to be fed into MLLM. It also induces the learng of temporal identity consistency and contact attribution through Contact-Aware Trace Forcing, achieving improved performance over existing methods on the HanDyVQA benchmark.
Critical strengths
- Defines an important problem well
- The method seems reasonable and sound, especially the separation of fast perception and slow propagation for video HOI understanding and its contact-aware trace forcing.
- Clear performance improvement over baselines.
- The method is well ablated.
Critical weaknesses
Overall: There are some methodological questions, which seem to remain unresolved in the paper.
- The motivation emphasizes preserving contact events that occur over short-lived contact through dense interaction perception. However, the final active object and contact representation are constructed only on a sparse timeline. Therefore, it is unclear whether the contact events occurring between the sampled slow frames are actually preserved.
- The proposed contact representation is derived not from explicit contact estimation, but from the morphological overlap of an enlarged hand and an object mask. This method provides a practical approximation, but it can produce false contact evidence in closed or spatially near cases even without physical interaction. Needs to be validated quantitatively regarding contact quality and should investigate how noisy contact intention labels affect the proposed alignment objectives.
- Persistent instance identity is a core feature of the proposed framework, but identity consistency, tracking association accuracy, or robustness in occlusion is not directly measured quantitatively or qualitatively.
Questions for authors
- Can Traceforcing-HOI be applied to long video qa? Seems like HanDyVQA is too short.
- Multiple components are handled in Traceforcing-HOI, and it seems that pseudo-labels are used extensively. It would be helpful to analyze which type of perception error leads to the final reasoning error.
3 comments
Rebuttal by Authors - Part 2/2
(W3) Quantitative Evaluation of Identity Consistency and Tracking Association
To directly quantify the quality of the automatically constructed traces, we add a cross-dataset evaluation on EPIC-KITCHENS VISOR [1]. This dataset provides manual pixel-level annotations with cross-frame instance correspondence, which allows us to test whether HOI objects are correctly localized and whether the same object is consistently associated over time. We compare against HOIST-Former [6] (CVPR 2024), a method for video instance segmentation and tracking of hand-held objects.
We apply our existing pipeline directly to VISOR, without any additional training or fine-tuning. The evaluation covers 679 video subsequences and 3,260 manually annotated frames from the validation set, with an average subsequence length of 12.76 s and a maximum of 93.13 s. We use the manually annotated timestamps as the Slow timeline, and the Fast branch and HOIST-Former use the same input frames. All object masks and tracks are constructed automatically from RGB, and only objects interacting with the hands are scored, on the manually annotated frames.
| Method | Video AP50 ↑ | Video AP50:95 ↑ | IDF1 ↑ |
|---|---|---|---|
| HOIST-Former | 22.12 | 7.79 | 39.55 |
| w/o Fast | 17.33 | 6.89 | 40.60 |
| Ours | 20.43 | 8.13 | 41.05 |
Without any additional training or fine-tuning, our method achieves better Video AP50:95 and IDF1. The former summarizes trace quality over multiple matching thresholds and the latter measures the accuracy of cross-frame identity correspondence, showing that our method is stronger in overall spatio-temporal matching and identity preservation and can be applied directly to an external dataset. HOIST-Former was trained specifically on VISOR and is stronger at recognizing small objects in this scenario, and therefore obtains a higher Video AP50. In addition, removing the Fast branch lowers all three metrics, further showing that dense interaction observations improve the quality of the automatically constructed traces. We will add the evaluation protocol and the analysis of these results to the revision.
(Q1) Applicability to Longer Videos
The VISOR experiment above (W3) already applies the representation construction pipeline zero-shot to video clips that average 12.76 s and reach 93.13 s, considerably longer than the roughly 5 s clips of HanDyVQA, and interacting-object localization and cross-frame identity association remain effective over these longer time spans. In addition, the Qwen2.5-VL backbone used by TraceForce-HOI natively supports dynamic frame sampling and long video inputs. Our framework can therefore accommodate longer video inputs. We also provide zero-shot out-of-domain inference cases on HOIGen-1M [7] in Appendix Section F.
(Q2) Perception Errors and Failure Case Analysis
We have added representative failure cases and their analysis in Appendix Section G and Figure 12. The typical failures fall into two types. The first is the omission of small interacting objects: when the pen and the paper share a merged identity, or when the identity color of the pencil is overwritten in the HOI Mask Video, these tools may become less distinguishable as separate manipulated objects. The second is the incorrect attribution of hand actions: in local views that lack auxiliary references such as the relative position of the two hands, a small-parameter model may focus more on action details such as finger motion and pay relatively less attention to left/right hand identity. In addition, Table 6 shows that missing identity cues mainly affects Process understanding, while missing contact cues mainly affects Objects recognition and Location-related questions.
Future work could explore agentic reasoning that uses the cross-frame identities and local interaction cues provided by our Persistent Interaction Traces to guide the model to actively revisit the relevant moments and regions, verifying and correcting its judgments on object recognition and action attribution.
References
[1] Darkhalil et al. EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. NeurIPS 2022.
[2] Su et al. Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing. ICLR 2026.
[3] Zhang et al. Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications. ECCV 2022.
[4] Yan et al. Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation. CVPR 2026.
[5] Prakash et al. How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions. CVPR 2025.
[6] Narasimhaswamy et al. HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild. CVPR 2024.
[7] Liu et al. HOIGen-1M: A Large-Scale Dataset for Human-Object Interaction Video Generation. CVPR 2025.
Dear Reviewer CEEC,
Thank you again for your constructive feedback. We have responded to all the concerns raised in your review, including clarifying how short-lived interactions are preserved in the Fast-Slow design, explaining the rationale and filtering mechanism of 2D contact cues, providing a zero-shot quantitative evaluation of tracking association on EPIC-KITCHENS VISOR, and adding discussions on longer video applicability and failure cases. As the discussion period is approaching its end, we would appreciate knowing whether these additions and clarifications have addressed your concerns.
Rebuttal by Authors - Part 1/2
We thank Reviewer CEEC for the thoughtful and constructive feedback on our work. We have added OOD and failure case analyses to the appendix (Sections F and G) and report additional quantitative results below. We will release our code upon acceptance. The following is our response to the comments and questions in the review.
We agree that the Fast–Slow design and the way information is passed between the two branches should be explained more clearly. In simplified form: given 5n Fast frames and n Slow frames, we first build candidate object tracklets from the Fast observations and take the earliest valid observation of each tracklet as its seed. The Slow branch then performs bidirectional propagation over "the n Slow frames + this seed frame". Finally, only the instance masks on the Slow frames are retained to generate the HOI Mask Video and the trace marks. The complete procedure is given in Appendix A and Algorithm 1.
Our design goal is to discover objects that have taken part in an interaction and to keep tracking and marking them after the contact ends, so that briefly manipulated objects are less likely to be missed under sparse sampling. To this end, the Fast branch discovers interacting objects from dense observations, while the Slow branch concentrates mask propagation on the sparse frames actually consumed by the MLLM. With the number of input frames unchanged, the model still receives the masks and cross-frame identities of the relevant objects discovered in the dense observations. The quantitative trace evaluation on VISOR [1] (W3) below confirms this.
The key point is that, for an object whose brief contact falls entirely between two Slow sampling times, it is precisely because the object took part in an interaction that it continues to be marked and tracked after the contact ends, forming a complete trace. These marks implicitly carry the cue that the object was involved in an interaction, helping the model connect its position and state before and after the contact and thereby reason about the interaction process and its outcome. We will further clarify this design principle and the information path in the revision, and explain the relation between persistent object tracking and per-frame interaction-state encoding.
Our contact evidence is built on interaction-aware hand–object parsing. InterFormer [2] (ICLR 2026) improves parsing consistency through contact-boundary guidance and hand–object co-occurrence constraints. The discovery of candidate objects therefore already incorporates interaction semantics, and the 2D contact regions subsequently built from the hand masks and the persistent object masks further provide local cues on where the interaction occurs.
For cases that are adjacent in 2D but separated in 3D, proximity alone does not make an object an interacting object: active objects are determined by the interaction parsing of InterFormer and then matched to the persistent tracks, while the 2D contact region is used only to indicate the local interaction location. During training, the positives of Contact Alignment are exactly these active objects; objects that merely overlap the contact region with ambiguous ownership are neither used as positives nor included in the negatives, so pseudo-contact caused by proximity does not enter the positive or negative sets.
It is worth noting that constructing 2D contact cues in image space is a common way of representing local interaction in HOI research. EgoHOS [3] (ECCV 2022) generates contact-boundary supervision from the intersection of dilated hand and object masks, and InterFormer [2] (ICLR 2026) adopts the same construction; SCAR [4] (CVPR 2026) estimates the contact region from the intersection of dilated contours and combines it with depth information to build its interaction representation. LatentAct [5] (CVPR 2025) also uses 2D hand–object overlap regions together with estimated hand meshes to build 3D contact annotations. These works show that a contact approximation in 2D image space can serve as an effective auxiliary cue for interaction modeling.
We adopt this representation also because it shares spatial coordinates with the RGB visual tokens and the instance masks, so that local interaction locations can be directly associated with the participating objects, without additional 3D object meshes or hand–object 3D registration. The ablations in Appendix C.2 (Table 6) and Appendix C.3 (Table 7) further support its effectiveness for the task.