acceptodds
Under review as a conference paper at ICLR 2027

Who Does What to Whom? Role Binding for Interaction-Aware Video Segmentation

Abstract

Interaction-aware referring video object segmentation (InterRVOS) asks not only which objects are referred to, but who does what to whom: the model must segment the Actor and Target of a described interaction according to their asymmetric semantic roles. Unlike conventional referring segmentation, these roles are not intrinsic object identities but are defined by the interaction between participants. Existing methods use separate Actor and Target tokens while leaving this interaction implicit, which can lead to ambiguous or relationally inconsistent role predictions. We instead formulate InterRVOS as relation-conditioned role binding, where the interaction serves as an explicit intermediate state that connects multimodal understanding to role-specific segmentation. Based on this view, we introduce R3Refer, which first learns an input-dependent, visually grounded interaction representation. It then performs directed role reasoning, where the interaction conditions Actor prediction and the resulting Actor representation further guides Target reasoning. Finally, triadic refinement jointly coordinates the Interaction, Actor, and Target representations before mask decoding. Experiments on InterRVOS-127K show consistent improvements in both Actor and Target segmentation, while evaluations on conventional RVOS benchmarks demonstrate the broader effectiveness of relation-aware role modeling.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Referring Video Object Segmentation

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_EKmZOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

Brief Summary: The paper tackles the task of Referring Video Object Segementation; here the model is provided with a video sequence + a natural language query and the model is required to provide bounding box / instance segmentation mask for the referred objects in the video. This doesn't require temporal grounding, so no timestamps for the objects (aka not spatio-temporal).

The main idea is to use Qwen3-VL with query-vectors (interaction/actor/target), and interaction query are used with video features to have auxillary mask loss; separately the three query-vectors are combined to get text-outputs which are then passed to SAM3 to get the final segmentation output.

Experiments on InterRVOS-127K show significant improvement over previous baselines.

Critical strengths

Pros:

  1. Paper studies an important problem in referring expression.

  2. Authors show cross-dataset generalization (appendix A). Qualitative Visualizations are good.

Critical weaknesses

Cons:

  1. The main idea is incremental, it uses specific query vectors for interaction/actor/target but this is not generalizable outside of this specific use-case, and even here seems to be working on a specific dataset not other datasets (see point4).

  2. The baselines are not strong. Obvious baselines like simply prompting Qwen3-VL to give points/bboxes and the passed to SAM3, or even directly giving the prompt to SAM3.

  3. Only Qwen3-VL baseline is used, authors should also try InternVL for fair comparison with ReVIOSa. Given the backbone change it is unclear where the gains are coming from.

  4. It seems like it doesn't generalize to other datasets? Such as being worse than VIRST on Ref-DAVIS and Ref-Youtube-VOS? Authors haven't expanded on this observation either.

  5. (Minor) It is tough to understand why we need dice loss just for auxillary use case when the final segmentation is via SAM3 directly?

Questions for authors

see weakness

Reviewer_TmeJOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper tackles the problem of interaction-aware referring video object segmentation (InterRVOS), where a model must split an action description into the actor and the target. Its first localizes the interaction explicitly, then binds roles through asymmetric actor-to-target propagation and joint three-way alignment, and finally decodes masks with SAM3's concept-prompt segmentation and tracking. Experiments are conducted on the InterRVOS-127K, Ref-DAVIS, Ref-YouTube-VOS, MeViS and ReVOS datasets.

Critical strengths
  1. The paper's motivation is clear. it identifies that existing InterRVOS methods treat Actor and Target as parallel, independent predictions while leaving the defining interaction implicit, and it argues that the asymmetric Actor–Target dependency should be resolved through an explicit, visually grounded interaction state.
  2. The experimental evaluation is also comprehensive, covering the InterRVOS-127K benchmark across Actor, Target, and standard RVOS settings, cross-dataset generalization on Ref-DAVIS, Ref-YouTube-VOS, MeViS, and ReVOS, and ablations that demonstrate the effectiveness of each component.
Critical weaknesses
  1. The core modules are mostly combinations of existing techniques, e.g., cross-attention cascades, query alignment, and query-based decoding, so the method feels more like a engineering integration than a genuinely new mechanism.
  2. The experimental comparison seems unfair. The best results use an Qwen3-VL-8B backbone while most competing methods use InterVL2.5-1B-4B models, so the reported gains cannot be cleanly attributed to the method itself rather than to the stronger backbone.
  3. Some formulas are not rigorous: Eq. 7 uses inconsistent symbols (e.g., Q_T^cal vs. Q_AT^cal, and cos(·,·) vs. sim(·,·)).
  4. In Section 3.5, the interaction branch is supervised using the Actor–Target union tracks, but a union of two role masks is not a physically meaningful "interaction region", so it is unclear whether the model truly learns the interaction or merely a coarse bounding region that contains both participants.
  5. On the more general Ref-DAVIS, Ref-YouTube-VOS, and ReVOS benchmarks, R³ Refer is clearly beaten by VIRST (e.g., nearly 3 points on ReVOS).
Questions for authors

Please refer to the weaknesses.

Reviewer_FJKnOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper formulates InterRVOS as relation-conditioned role binding, with an explicit interaction representation connecting multimodal understanding to role-specific segmentation. R^3Refer introduces interaction queries grounded through Dynamic Interaction Reasoning (DIR), followed by Directed Role Propagation (DRP) and Triadic Cross-Role Alignment (TCRA) to refine the Interaction, Actor, and Target representations. SAM3 PCS then decodes the calibrated role queries into temporal masks. Experiments show strong performance on InterRVOS-127K, while component ablations support the effectiveness of the proposed modules.

Critical strengths

1. Clear motivation and architectural design.

Explicitly grounding interaction queries with Actor-Target union masks and using them to refine role representations is a clear and well motivated design. The strong InterRVOS-127K results, particularly for Target segmentation, demonstrate the effectiveness of the overall framework.

2. Supporting component ablations.

The ablations provide evidence that DIR, DRP, and TCRA each contribute to the final segmentation performance.

Critical weaknesses

1. Insufficient methodological and experimental detail.

Several important details are missing, including the Actor-Target pair fusion operation, the annotations used for language modeling supervision, and the parameter count of the added MHA modules. Eq. (7) is also inconsistent with its accompanying definitions. For Table 3, it is unclear whether DRP/TCRA are retained in the "w/o DIR" setting and how the "action classification" and "region feature" variants are implemented. Similarly, Table 5 does not explain how R^3Refer's calibrated role queries are incorporated into the PVS variants or whether DIR/IARC and text conditioning are held fixed. These omissions hinder reproducibility and make the source of the PCS gains difficult to assess.

2. The contribution of interaction reasoning is not sufficiently isolated.

SAM3 PCS receives role specific text phrases that may already provide strong appearance cues for identifying the participants. Without separating text conditioning from latent role query conditioning, it remains unclear how much of the improvement comes from the proposed interaction representations rather than text conditioned object localization.

Questions for authors
  1. Clarify the methodological details and ablation configurations identified above.

  2. Compare the full model with query-only and phrase-only PCS to isolate the contributions of calibrated role queries and role-specific text phrases.

  3. Compare DRP/TCRA with parameter-matched standard self-attention under otherwise identical settings.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.