acceptodds
Under review as a conference paper at ICLR 2027

TraceForcing: Learning from Persistent Interaction Traces for Fine-Grained Hand-Object Interaction Video Understanding

Abstract

Fine-grained hand-object interaction (HOI) understanding is not merely about recognizing hands, objects, and actions in individual frames. It requires preserving a persistent interaction trace: who interacts with what, where contact occurs, and how these relations evolve into manipulation processes and state changes over time. However, existing approaches typically model entities, trajectories, or semantic relations as isolated structured cues, lacking a unified mechanism to preserve localized contact evidence and persistent hand-object identities throughout videos. In this work, we propose TraceForce-HOI, a unified framework that models HOI videos through Persistent Interaction Traces, a structured representation capturing hand entities, object instances, contact evidence, and interaction state evolution. To construct reliable traces, we introduce Fast-Slow Trace Identification, which combines dense interaction-aware perception with sparse temporal propagation to capture short-lived contact dynamics while maintaining consistent instance identities over time. The resulting traces are encoded in both pixel and latent spaces through HOI mask videos and instance-aware trace marks, exposing entity identities, active-object states, and localized interaction evidence to the model. Furthermore, we introduce causal identity and contact-attribution constraints to regularize intermediate visual representations, ensuring that interaction evidence remains anchored to the correct entities throughout the video. Extensive experiments demonstrate that TraceForce-HOI achieves significant improvements in fine-grained HOI video understanding, particularly on tasks requiring persistent instance association and interaction evolution modeling, including procedural understanding, state transition recognition, and object part-level comprehension.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Understanding, Hand-Object Interaction, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_CEECOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes TraceForcing-HOI, a framework that integrates the persistent identity, contact information, and interaction state of hands and objects for fine-grained Hand-Object Interaction Video understanding, and constructs it through Fast-Slow tracking to be fed into MLLM. It also induces the learng of temporal identity consistency and contact attribution through Contact-Aware Trace Forcing, achieving improved performance over existing methods on the HanDyVQA benchmark.

Critical strengths
  1. Defines an important problem well
  2. The method seems reasonable and sound, especially the separation of fast perception and slow propagation for video HOI understanding and its contact-aware trace forcing.
  3. Clear performance improvement over baselines.
  4. The method is well ablated.
Critical weaknesses

Overall: There are some methodological questions, which seem to remain unresolved in the paper.

  1. The motivation emphasizes preserving contact events that occur over short-lived contact through dense interaction perception. However, the final active object and contact representation are constructed only on a sparse timeline. Therefore, it is unclear whether the contact events occurring between the sampled slow frames are actually preserved.
  2. The proposed contact representation is derived not from explicit contact estimation, but from the morphological overlap of an enlarged hand and an object mask. This method provides a practical approximation, but it can produce false contact evidence in closed or spatially near cases even without physical interaction. Needs to be validated quantitatively regarding contact quality and should investigate how noisy contact intention labels affect the proposed alignment objectives.
  3. Persistent instance identity is a core feature of the proposed framework, but identity consistency, tracking association accuracy, or robustness in occlusion is not directly measured quantitatively or qualitatively.
Questions for authors
  1. Can Traceforcing-HOI be applied to long video qa? Seems like HanDyVQA is too short.
  2. Multiple components are handled in Traceforcing-HOI, and it seems that pseudo-labels are used extensively. It would be helpful to analyze which type of perception error leads to the final reasoning error.
3 comments
AuthorsOpenReview ↗

Rebuttal by Authors - Part 1/2

We thank Reviewer CEEC for the thoughtful and constructive feedback on our work. We have added OOD and failure case analyses to the appendix (Sections F and G) and report additional quantitative results below. We will release our code upon acceptance. The following is our response to the comments and questions in the review.


(W1) Preservation of Short-Lived Interaction Cues and the Fast–Slow Design

We agree that the Fast–Slow design and the way information is passed between the two branches should be explained more clearly. In simplified form: given 5n5n Fast frames and nn Slow frames, we first build candidate object tracklets from the Fast observations and take the earliest valid observation of each tracklet as its seed. The Slow branch then performs bidirectional propagation over "the nn Slow frames + this seed frame". Finally, only the instance masks on the Slow frames are retained to generate the HOI Mask Video and the trace marks. The complete procedure is given in Appendix A and Algorithm 1.

Our design goal is to discover objects that have taken part in an interaction and to keep tracking and marking them after the contact ends, so that briefly manipulated objects are less likely to be missed under sparse sampling. To this end, the Fast branch discovers interacting objects from dense observations, while the Slow branch concentrates mask propagation on the sparse frames actually consumed by the MLLM. With the number of input frames unchanged, the model still receives the masks and cross-frame identities of the relevant objects discovered in the dense observations. The quantitative trace evaluation on VISOR [1] (W3) below confirms this.

The key point is that, for an object whose brief contact falls entirely between two Slow sampling times, it is precisely because the object took part in an interaction that it continues to be marked and tracked after the contact ends, forming a complete trace. These marks implicitly carry the cue that the object was involved in an interaction, helping the model connect its position and state before and after the contact and thereby reason about the interaction process and its outcome. We will further clarify this design principle and the information path in the revision, and explain the relation between persistent object tracking and per-frame interaction-state encoding.


(W2) Rationale and Effectiveness of the 2D Contact Evidence

Our contact evidence is built on interaction-aware hand–object parsing. InterFormer [2] (ICLR 2026) improves parsing consistency through contact-boundary guidance and hand–object co-occurrence constraints. The discovery of candidate objects therefore already incorporates interaction semantics, and the 2D contact regions subsequently built from the hand masks and the persistent object masks further provide local cues on where the interaction occurs.

For cases that are adjacent in 2D but separated in 3D, proximity alone does not make an object an interacting object: active objects are determined by the interaction parsing of InterFormer and then matched to the persistent tracks, while the 2D contact region is used only to indicate the local interaction location. During training, the positives of Contact Alignment are exactly these active objects; objects that merely overlap the contact region with ambiguous ownership are neither used as positives nor included in the negatives, so pseudo-contact caused by proximity does not enter the positive or negative sets.

It is worth noting that constructing 2D contact cues in image space is a common way of representing local interaction in HOI research. EgoHOS [3] (ECCV 2022) generates contact-boundary supervision from the intersection of dilated hand and object masks, and InterFormer [2] (ICLR 2026) adopts the same construction; SCAR [4] (CVPR 2026) estimates the contact region from the intersection of dilated contours and combines it with depth information to build its interaction representation. LatentAct [5] (CVPR 2025) also uses 2D hand–object overlap regions together with estimated hand meshes to build 3D contact annotations. These works show that a contact approximation in 2D image space can serve as an effective auxiliary cue for interaction modeling.

We adopt this representation also because it shares spatial coordinates with the RGB visual tokens and the instance masks, so that local interaction locations can be directly associated with the participating objects, without additional 3D object meshes or hand–object 3D registration. The ablations in Appendix C.2 (Table 6) and Appendix C.3 (Table 7) further support its effectiveness for the task.

AuthorsOpenReview ↗

Rebuttal by Authors - Part 2/2

(W3) Quantitative Evaluation of Identity Consistency and Tracking Association

To directly quantify the quality of the automatically constructed traces, we add a cross-dataset evaluation on EPIC-KITCHENS VISOR [1]. This dataset provides manual pixel-level annotations with cross-frame instance correspondence, which allows us to test whether HOI objects are correctly localized and whether the same object is consistently associated over time. We compare against HOIST-Former [6] (CVPR 2024), a method for video instance segmentation and tracking of hand-held objects.

We apply our existing pipeline directly to VISOR, without any additional training or fine-tuning. The evaluation covers 679 video subsequences and 3,260 manually annotated frames from the validation set, with an average subsequence length of 12.76 s and a maximum of 93.13 s. We use the manually annotated timestamps as the Slow timeline, and the Fast branch and HOIST-Former use the same input frames. All object masks and tracks are constructed automatically from RGB, and only objects interacting with the hands are scored, on the manually annotated frames.

MethodVideo AP50 ↑Video AP50:95 ↑IDF1 ↑
HOIST-Former22.127.7939.55
w/o Fast17.336.8940.60
Ours20.438.1341.05

Without any additional training or fine-tuning, our method achieves better Video AP50:95 and IDF1. The former summarizes trace quality over multiple matching thresholds and the latter measures the accuracy of cross-frame identity correspondence, showing that our method is stronger in overall spatio-temporal matching and identity preservation and can be applied directly to an external dataset. HOIST-Former was trained specifically on VISOR and is stronger at recognizing small objects in this scenario, and therefore obtains a higher Video AP50. In addition, removing the Fast branch lowers all three metrics, further showing that dense interaction observations improve the quality of the automatically constructed traces. We will add the evaluation protocol and the analysis of these results to the revision.


(Q1) Applicability to Longer Videos

The VISOR experiment above (W3) already applies the representation construction pipeline zero-shot to video clips that average 12.76 s and reach 93.13 s, considerably longer than the roughly 5 s clips of HanDyVQA, and interacting-object localization and cross-frame identity association remain effective over these longer time spans. In addition, the Qwen2.5-VL backbone used by TraceForce-HOI natively supports dynamic frame sampling and long video inputs. Our framework can therefore accommodate longer video inputs. We also provide zero-shot out-of-domain inference cases on HOIGen-1M [7] in Appendix Section F.


(Q2) Perception Errors and Failure Case Analysis

We have added representative failure cases and their analysis in Appendix Section G and Figure 12. The typical failures fall into two types. The first is the omission of small interacting objects: when the pen and the paper share a merged identity, or when the identity color of the pencil is overwritten in the HOI Mask Video, these tools may become less distinguishable as separate manipulated objects. The second is the incorrect attribution of hand actions: in local views that lack auxiliary references such as the relative position of the two hands, a small-parameter model may focus more on action details such as finger motion and pay relatively less attention to left/right hand identity. In addition, Table 6 shows that missing identity cues mainly affects Process understanding, while missing contact cues mainly affects Objects recognition and Location-related questions.

Future work could explore agentic reasoning that uses the cross-frame identities and local interaction cues provided by our Persistent Interaction Traces to guide the model to actively revisit the relevant moments and regions, verifying and correcting its judgments on object recognition and action attribution.


References

[1] Darkhalil et al. EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. NeurIPS 2022.

[2] Su et al. Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing. ICLR 2026.

[3] Zhang et al. Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications. ECCV 2022.

[4] Yan et al. Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation. CVPR 2026.

[5] Prakash et al. How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions. CVPR 2025.

[6] Narasimhaswamy et al. HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild. CVPR 2024.

[7] Liu et al. HOIGen-1M: A Large-Scale Dataset for Human-Object Interaction Video Generation. CVPR 2025.

Dear Reviewer CEEC,

Thank you again for your constructive feedback. We have responded to all the concerns raised in your review, including clarifying how short-lived interactions are preserved in the Fast-Slow design, explaining the rationale and filtering mechanism of 2D contact cues, providing a zero-shot quantitative evaluation of tracking association on EPIC-KITCHENS VISOR, and adding discussions on longer video applicability and failure cases. As the discussion period is approaching its end, we would appreciate knowing whether these additions and clarifications have addressed your concerns.

Reviewer_c9xEOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The authors propose a mid-level representation for modeling hand-object interactions based on masks for hands, interaction traces, and contact masks. They propose an elaborate pipeline for computing this structured mid-level representation using off-the-shelf HoI models like InterFormer. They then utilize their mid-level representation to fine-tune a VLM to align the intermediate features of the VLM's vision encoder with their mid-level structured representation. They fine-tune the VLM on HanDyVQA, a VQA dataset for HoI understanding, and show better performance than the baselines on most categories.

Critical strengths

The fast-slow sampling of video frames is an interesting approach for obtaining dense interaction-rich representations.

Critical weaknesses
  1. Writing: The description of the method is extremely difficult to parse despite careful reading, and the motivation for the elaborate loss functions is not well defined. Certain design decisions, like their approach to constructing contact masks, also look quite hand-crafted for the dataset that they are training on.
  2. Weak Baselines: Both proprietary and open models used for comparisons are quite old, raising the question whether the proposed method really adds any performance improvements over current models.
    1. Furthermore, the comparisons are not truly apples-to-apples. The authors provide HoI mask videos as additional inputs to their model. At a minimum, I would expect a comparison with a set-of-marks-style [1] video with hands and objects highlighted with pixel boundaries and (optionally) marked with numeric labels explained in the prompts and fed to the model.
  3. Generality of design decisions: Looking at the details for the mid-level representation construction, the design seems quite hand-crafted. This raises concerns about its generality to other HoI datasets or in-the-wild scenes. Evidence from either would help assess the robustness of the intermediate representation construction pipeline.
Questions for authors
  1. Some experiments showing that their mid-level representation pipeline can scale to datasets beyond HanDyQA would be helpful.
  2. For contact mask mining, how do you handle cases where the masks for the hand and object might be adjacent in 2D but not actually in contact in 3D space?
  3. Can the authors share results for the baselines where they are also provided the HoI masks as SoM prompts or in another reasonable manner to ascertain the necessity of their training pipeline?
6 comments
AuthorsOpenReview ↗

Rebuttal by Authors - Part 1/3

We thank Reviewer c9xE for the thoughtful and constructive feedback on our work. We have added OOD and failure case analyses to the appendix (Sections F and G) and report additional quantitative results below. We will release our code upon acceptance. The following is our response to the comments and questions in the review.


(W1) Method Pipeline and Motivation of the Training Objectives

We agree that the manuscript should explain the motivation and the pipeline of each component more clearly. The two auxiliary objectives correspond to two requirements of fine-grained interaction understanding: Track Alignment helps the model keep the identity of the same object consistent across time, so that observations before and after manipulation can be integrated. Contact Alignment associates contact evidence with the active objects, helping the model learn which objects take part in the interaction.

Appendix A and Algorithm 1 give the complete automatic trace construction procedure. For ease of understanding, a simplified version is as follows: 5n5n Fast frames are used to discover interacting objects, and the earliest valid observation of each candidate tracklet is taken as its seed; the Slow branch performs bidirectional propagation over the sequence formed by the nn Slow frames and this seed, after which only the masks on the Slow frames are retained to generate the HOI Mask Video and the trace marks.

In addition, the quantitative trace evaluation on VISOR [1] and the zero-shot open-ended QA on HOIGen-1M [2] (W3 & Q1) added below provide further evidence for the out-of-domain applicability of our method. We will further refine the writing in the revision so that the method description is clearer and easier to follow.


(W2) Comparison with the Latest Specialized Method and General-Purpose Model

We add comparisons with R4DSG [3] (ACM MM 2026), an egocentric video understanding method based on relative 4D scene graph memory, and with GPT-6 Astra, a state-of-the-art general-purpose model.

Objects APActionProcessLocationStatePartsAvg5Latency (s)
R4DSG54.277.884.271.170.468.574.476.37
GPT-6 Astra80.782.484.880.074.380.780.429.43
GPT-6 Astra Agentic83.687.784.882.875.983.082.8139.11
TraceForce-HOI-3B66.684.686.076.880.073.380.113.52

The results show that TraceForce-HOI outperforms R4DSG. GPT-6 Astra performs better on object recognition, part understanding, and spatial localization, reflecting its strong capability in spatial visual-semantic recognition and fine-detail understanding. In the agentic mode, tool use during reasoning further improves understanding, but also adds the overhead of tool execution and the subsequent processing of tool results. In contrast, TraceForce-HOI achieves the best results on both Process and State, indicating that its advantage lies in understanding manipulation processes and state changes. Notably, with only a 3B backbone, TraceForce-HOI reaches an overall performance close to that of GPT-6 Astra at less than half of its latency, and roughly one tenth of the latency of the agentic setting.


(W2 & Q3) SoM-Style Baseline with the Same HOI Mask Video

We add a SoM-style visual-prompting baseline: using the same Qwen2.5-VL-3B backbone, the same automatically constructed HOI Mask Video is fed together with the RGB video and the model is trained with SFT; each instance is marked with a fixed color, and the meaning of the marks is explained in the prompt. The masks are provided as a separate video stream that is temporally aligned with the RGB video, which is exactly the same input form as ours.

The results show that adding the HOI masks improves the model, indicating that these structured interaction cues are valuable in themselves. However, the full method still achieves higher results on all six task categories. This gap comes from two designs beyond mask prompting: the trace marks strengthen object identity and add active-state information, while the auxiliary alignment constrains cross-frame instance correspondence and the association between contact evidence and objects. The individual contribution of each design can be read from the existing ablations: with the same RGB + HOI Mask Video input, adding only Trace Forcing (Appendix Table 5, w/o trace marks) or only the trace marks (Table 2, third row) both clearly outperform this baseline.

Objects APActionProcessLocationStatePartsAvg5
Qwen2.5-VL-3B-SFT64.270.766.166.368.957.365.9
Qwen2.5-VL-3B + HOI Mask64.977.077.368.471.769.472.8
TraceForce-HOI-3B66.684.686.076.880.073.380.1
AuthorsOpenReview ↗

Rebuttal by Authors - Part 2/3

(W3 & Q1) Cross-Dataset Generalization

We add cross-dataset validation from two aspects, automatic representation construction and out-of-domain video understanding: the former tests whether the pipeline transfers directly and produces usable traces, and the latter tests whether the trained model can exploit these structures in external HOI videos.

Zero-Shot Trace Construction on VISOR

To directly quantify the quality of the automatically constructed traces, we add a cross-dataset evaluation on EPIC-KITCHENS VISOR [1]. This dataset provides manual pixel-level annotations with cross-frame instance correspondence, which allows us to test whether HOI objects are correctly localized and whether the same object is consistently associated over time. We compare against HOIST-Former [4] (CVPR 2024), a method for video instance segmentation and tracking of hand-held objects.

We apply our existing pipeline directly to VISOR, without any additional training or fine-tuning. The evaluation covers 679 video subsequences and 3,260 manually annotated frames from the validation set, with an average subsequence length of 12.76 s and a maximum of 93.13 s. We use the manually annotated timestamps as the Slow timeline, and the Fast branch and HOIST-Former use the same input frames. All object masks and tracks are constructed automatically from RGB, and only objects interacting with the hands are scored, on the manually annotated frames.

MethodVideo AP50 ↑Video AP50:95 ↑IDF1 ↑
HOIST-Former22.127.7939.55
w/o Fast17.336.8940.60
Ours20.438.1341.05

Without any additional training or fine-tuning, our method achieves better Video AP50:95 and IDF1. The former summarizes trace quality over multiple matching thresholds and the latter measures the accuracy of cross-frame identity correspondence, showing that our method is stronger in overall spatio-temporal matching and identity preservation and can be applied directly to an external dataset. HOIST-Former was trained specifically on VISOR and is stronger at recognizing small objects in this scenario, and therefore obtains a higher Video AP50. In addition, removing the Fast branch lowers all three metrics, further showing that dense interaction observations improve the quality of the automatically constructed traces. We will add the evaluation protocol and the analysis of these results to the revision.

Out-of-Domain Open-Ended QA on HOIGen-1M

To examine the applicability of the model in external scenarios, we select real-world interaction videos from HOIGen-1M [2] (CVPR 2025), a dataset that contains over one million video clips from diverse sources and covers more than 15,000 object categories and more than 7,000 interaction types. Without any additional fine-tuning on these videos, we compare with the baseline through open-ended question answering.

The case studies have been added to Appendix Section F. In the videos of painting a pot and hammering a piece of wood, both models recognize the main action, while TraceForce-HOI further describes the division of labor in which one hand stabilizes the target object and the other operates the tool. This support-and-manipulation relation also appears in the drink-pouring video, where the model clearly distinguishes the hand holding the receiving container from the hand performing the pour. For a clip containing consecutive operations, the answer also covers the stage transition from stirring with a spoon to adding material from a small container. These cases provide qualitative support for the cross-scenario applicability of our method: across different tools and activities, the model still links actions to the specific participating objects and their roles, and describes how the interaction evolves over time.

AuthorsOpenReview ↗

Rebuttal by Authors - Part 3/3

(Q2) Rationale and Effectiveness of the 2D Contact Evidence

Our contact evidence is built on interaction-aware hand–object parsing. InterFormer [5] (ICLR 2026) improves parsing consistency through contact-boundary guidance and hand–object co-occurrence constraints. The discovery of candidate objects therefore already incorporates interaction semantics, and the 2D contact regions subsequently built from the hand masks and the persistent object masks further provide local cues on where the interaction occurs.

For cases that are adjacent in 2D but separated in 3D, proximity alone does not make an object an interacting object: active objects are determined by the interaction parsing of InterFormer and then matched to the persistent tracks, while the 2D contact region is used only to indicate the local interaction location. During training, the positives of Contact Alignment are exactly these active objects; objects that merely overlap the contact region with ambiguous ownership are neither used as positives nor included in the negatives, so pseudo-contact caused by proximity does not enter the positive or negative sets.

It is worth noting that constructing 2D contact cues in image space is a common way of representing local interaction in HOI research. EgoHOS [6] (ECCV 2022) generates contact-boundary supervision from the intersection of dilated hand and object masks, and InterFormer [5] (ICLR 2026) adopts the same construction; SCAR [7] (CVPR 2026) estimates the contact region from the intersection of dilated contours and combines it with depth information to build its interaction representation. LatentAct [8] (CVPR 2025) also uses 2D hand–object overlap regions together with estimated hand meshes to build 3D contact annotations. These works show that a contact approximation in 2D image space can serve as an effective auxiliary cue for interaction modeling.

We adopt this representation also because it shares spatial coordinates with the RGB visual tokens and the instance masks, so that local interaction locations can be directly associated with the participating objects, without additional 3D object meshes or hand–object 3D registration. The ablations in Appendix C.2 (Table 6) and Appendix C.3 (Table 7) further support its effectiveness for the task.


References

[1] Darkhalil et al. EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. NeurIPS 2022.

[2] Liu et al. HOIGen-1M: A Large-Scale Dataset for Human-Object Interaction Video Generation. CVPR 2025.

[3] Ma et al. R4DSG: Relative 4D Scene Graph Memory for Object-Centric Question Answering in Long Egocentric Video. ACM MM 2026.

[4] Narasimhaswamy et al. HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild. CVPR 2024.

[5] Su et al. Interaction-aware Representation Modeling with Co-occurrence Consistency for Egocentric Hand-Object Parsing. ICLR 2026.

[6] Zhang et al. Fine-Grained Egocentric Hand-Object Segmentation: Dataset, Model, and Applications. ECCV 2022.

[7] Yan et al. Open-world Hand-Object Interaction Video Generation Based on Structure and Contact-aware Representation. CVPR 2026.

[8] Prakash et al. How Do I Do That? Synthesizing 3D Hand Motion and Contacts for Everyday Interactions. CVPR 2025.

Reviewer_gfzaOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates the fine-grained understanding of hand-object interactions in first-person videos, with a focus on the relationship between transient interaction evidence and object identity across frames. The authors propose TraceForce-HOI, which organizes information about the left and right hands, object instances, active objects, and local contact information through continuous interaction traces. The method constructs trajectories using dense interaction parsing and sparse mask propagation, encodes them as color-masked videos and learnable trajectory labels, and constrains the visual representations in the intermediate layers of the language model via trajectory alignment and contact alignment losses. Additionally, ablation experiments on components, input representations, interaction clues, and training objectives are provided in the main manuscript and appendices.

Critical strengths
  1. The research motivation is clear, and the experimental results align with the goal of fine-grained interaction understanding. Linking evidence from brief contacts to the sustained identity of objects is a meaningful problem in first-person interactive video understanding. This paper achieves significant improvements over the baseline of fine-tuning a backbone RGB model and further outperforms structured baselines that use the same perceptual results.

  2. The input representations and training constraints form a coherent framework, supported by systematic ablation results. The method provides interaction structures through masked videos and learnable labels, and then facilitates the model’s utilization of these structures by aligning trajectories with interactions. The main text and appendices examine input formats, interaction cues, and auxiliary objectives, respectively. In particular, the results demonstrate that incorporating full Trace Forcing while retaining interactive inputs yields significant improvements.

Critical weaknesses
  1. The path by which information about brief contacts is retained from dense perception to the final representation remains unclear. The Fast branch uses dense sampling to detect interaction objects, but the final visual input retains only frames from the Slow timeline, and contact masks are also reconstructed independently on these frames. For brief contacts that occur entirely between two Slow sampling timepoints, the paper does not clearly explain how their contact relationships and temporal information are incorporated into the final representation. Dense detection can help preserve interacting objects, but this differs from preserving specific contact events.

  2. The reliability of automatically generated supervision and its error propagation have not been fully evaluated. Both alignment objectives rely on external perception models and contact evidence constructed based on two-dimensional proximity relationships. The authors have adopted rules such as region filtering, trajectory consistency checks, and ambiguity resolution, but they have not quantified the remaining identity association and contact attribution errors, nor have they analyzed the impact of these errors on training.

Questions for authors
  1. How does contact alignment handle contact attribution when multiple objects are involved in an interaction simultaneously? Section 3.4.2 and Equation (2) pool the contact regions into a frame-level representation qtq_t and treat all active objects as the set of positive samples. When both hands are manipulating different objects simultaneously, how does this approach distinguish which object each local contact belongs to? Does the implementation use the per-object contact masks CtkC_t^k defined in Appendix A.3 to construct alignments separately? If only a single aggregated representation is used, how can we prevent the loss from being primarily determined by one of the active objects that is easier to match?

  2. For brief contacts that occur only in Fast-sampled frames and do not appear in any Slow-sampled frames, what information is included in the final interaction trajectory? Detail the specific propagation path.

  3. When automatically constructed interaction trajectories contain errors, is Trace Forcing still effective? Instance associations and active object attributions are generated through segmentation, propagation, and spatial matching, and are simultaneously used for input representation and loss supervision. Aside from filtering out tokens with ambiguous spatial attributions, how does the model handle identity switches or incorrect object attributions caused by occlusions?

5 comments
AuthorsOpenReview ↗

Rebuttal by Authors - Part 1/2

We thank Reviewer gfza for the thoughtful and constructive feedback on our work. We have added OOD and failure case analyses to the appendix (Sections F and G) and report additional quantitative results below. We will release our code upon acceptance. The following is our response to the comments and questions in the review.


(W1 & Q2) Preservation of Short-Lived Interaction Cues and the Fast–Slow Design

We agree that the Fast–Slow design and the way information is passed between the two branches should be explained more clearly. In simplified form: given 5n5n Fast frames and nn Slow frames, we first build candidate object tracklets from the Fast observations and take the earliest valid observation of each tracklet as its seed. The Slow branch then performs bidirectional propagation over "the nn Slow frames + this seed frame". Finally, only the instance masks on the Slow frames are retained to generate the HOI Mask Video and the trace marks. The complete procedure is given in Appendix A and Algorithm 1.

Our design goal is to discover objects that have taken part in an interaction and to keep tracking and marking them after the contact ends, so that briefly manipulated objects are less likely to be missed under sparse sampling. To this end, the Fast branch discovers interacting objects from dense observations, while the Slow branch concentrates mask propagation on the sparse frames actually consumed by the MLLM. With the number of input frames unchanged, the model still receives the masks and cross-frame identities of the relevant objects discovered in the dense observations. The quantitative trace evaluation on VISOR [1] (W2) below confirms this.

The key point is that, for an object whose brief contact falls entirely between two Slow sampling times, it is precisely because the object took part in an interaction that it continues to be marked and tracked after the contact ends, forming a complete trace. These marks implicitly carry the cue that the object was involved in an interaction, helping the model connect its position and state before and after the contact and thereby reason about the interaction process and its outcome. We will further clarify this design principle and the information path in the revision, and explain the relation between persistent object tracking and per-frame interaction-state encoding.


(W2) Quantitative Evaluation of Trace Quality and Identity Association

To directly quantify the quality of the automatically constructed traces, we add a cross-dataset evaluation on EPIC-KITCHENS VISOR [1]. This dataset provides manual pixel-level annotations with cross-frame instance correspondence, which allows us to test whether HOI objects are correctly localized and whether the same object is consistently associated over time. We compare against HOIST-Former [2] (CVPR 2024), a method for video instance segmentation and tracking of hand-held objects.

We apply our existing pipeline directly to VISOR, without any additional training or fine-tuning. The evaluation covers 679 video subsequences and 3,260 manually annotated frames from the validation set, with an average subsequence length of 12.76 s and a maximum of 93.13 s. We use the manually annotated timestamps as the Slow timeline, and the Fast branch and HOIST-Former use the same input frames. All object masks and tracks are constructed automatically from RGB, and only objects interacting with the hands are scored, on the manually annotated frames.

MethodVideo AP50 ↑Video AP50:95 ↑IDF1 ↑
HOIST-Former22.127.7939.55
w/o Fast17.336.8940.60
Ours20.438.1341.05

Without any additional training or fine-tuning, our method achieves better Video AP50:95 and IDF1. The former summarizes trace quality over multiple matching thresholds and the latter measures the accuracy of cross-frame identity correspondence, showing that our method is stronger in overall spatio-temporal matching and identity preservation and can be applied directly to an external dataset. HOIST-Former was trained specifically on VISOR and is stronger at recognizing small objects in this scenario, and therefore obtains a higher Video AP50. In addition, removing the Fast branch lowers all three metrics, further showing that dense interaction observations improve the quality of the automatically constructed traces. We will add the evaluation protocol and the analysis of these results to the revision.

AuthorsOpenReview ↗

Rebuttal by Authors - Part 2/2

(W2 & Q3) Effectiveness of Trace Forcing When the Traces Contain Errors

Identity switches caused by occlusion are first handled during trace construction: fragmented tracklets produced before and after an occlusion are iteratively merged and re-propagated as a single instance, and the two hands are jointly propagated as two fixed identities (Appendix A.2). On top of this, Trace Forcing uses only supervision that passes validity filtering, which is applied at two stages: feature aggregation and the alignment constraints. Specifically, region-level aggregation keeps only tokens whose coverage, purity, and spatial ownership satisfy the criteria, reducing interference from non-target regions on the instance features. Track Alignment uses the most recent valid historical observation of the same ID as the target and stops gradients through the historical targets, so that the current alignment loss does not back-propagate into historical representations. Therefore, even if an occasional identity switch remains, it affects only the single adjacent pair at the switch point; since the loss is bounded by the margin, the error does not accumulate over time. Contact Alignment is computed only when a valid contact region and an active object are both present, and objects that heavily overlap the contact region but have ambiguous ownership are excluded from the negative set, reducing erroneous constraints caused by ambiguity. Both alignment objectives serve as auxiliary terms with small weights and are optimized jointly with the answer supervision.

The ablation in Table 7 (Appendix C.3) provides a direct comparison of the actual training effect. With all input representations unchanged and the same automatically constructed traces, removing either alignment objective lowers the overall QA performance, and process understanding and state judgment decrease accordingly. This shows that Trace Forcing is effective on automatically constructed traces. Moreover, even with coarser traces obtained by per-frame extraction and IoU-based association, the interaction-trace representation still clearly outperforms the RGB baseline (Table 2, second row).


(Q1) Contact Attribution and Alignment with Multiple Objects

Contact Alignment applies a constraint to each active object separately, so an easily matched object cannot stand in for the others. At the representation level, which object each local contact belongs to is given by spatial correspondence: the contact mark and the trace mark of the contacted object are superimposed on the same local visual tokens. The per-object contact regions CtkC_t^k retain the contact attribution of each object and are used for track ranking. For alignment, we merge them into CtC_t and pool it into a joint contact representation qtq_t, while each object keeps its own representation ht,kh_{t,k}. Simultaneously active objects are treated as equally important interaction targets, with no need to designate a primary object.

Specifically, for each valid active object, we take it as the positive and compute a contrastive loss between qtq_t and ht,kh_{t,k}. All terms share the same negative set, other active objects are not used as negatives, and the losses are averaged over all active objects. Therefore, even if qtq_t already matches one object well, the loss terms of the remaining objects are not reduced: the lower the similarity between qtq_t and an object, the larger the loss of that object. We will further clarify the form of Eq. (2) in the revision.


References

[1] Darkhalil et al. EPIC-KITCHENS VISOR Benchmark: VIdeo Segmentations and Object Relations. NeurIPS 2022.

[2] Narasimhaswamy et al. HOIST-Former: Hand-held Objects Identification, Segmentation, and Tracking in the Wild. CVPR 2024.

Dear Reviewer gfza,

Thank you again for your thoughtful and constructive feedback. We have responded to all the concerns raised in your review, including clarifying the Fast-Slow information propagation path for short-lived contacts, providing a zero-shot quantitative evaluation of trace quality on EPIC-KITCHENS VISOR, analyzing the robustness of Trace Forcing under noisy trajectories, and detailing multi-object contact attribution. As the discussion period is approaching its end, we would appreciate knowing whether these additions and clarifications have addressed your concerns.

Comments on the paper

1 comment

We sincerely thank the AC and all reviewers for their time and constructive feedback. In our rebuttal, we clarified the key design choices of our method. We added comparisons with the recent specialized method R4DSG, the frontier model GPT-6 Astra, and a SoM-style baseline that uses the same HOI masks. We further provided a zero-shot evaluation of trace quality on VISOR, zero-shot out-of-domain cases on HOIGen-1M, and failure-case analysis (Appendix F and G). Detailed responses are provided in our individual replies under each review. As the discussion period is approaching its end, we would greatly appreciate knowing whether our responses have addressed your concerns, and we would be glad to provide any further clarification.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.