acceptodds
Under review as a conference paper at ICLR 2027

CaReVOS: Causal Reasoning Across Missing Events for Video Object Segmentation

Abstract

Reasoning video object segmentation requires models to identify and segment objects from implicit queries that depend on understanding events and object relations. Existing settings, however, largely assume that the interactions defining these roles are observable. To study causal reasoning across missing events, we formulate a partial-observation setting in which models infer and ground the relevant participants from observations before and after an unobserved interaction. Based on this formulation, we introduce CaReVOS, a benchmark comprising 1,432 videos, 6,386 reasoning expressions, and 373,828 object mask annotations. It includes consequence, source, multi-solution, and counterfactual questions, with human-verified answers and masks. Because the interaction is hidden, answering these questions requires relating the observed setup to its outcome before selecting the target. To improve both this inference and the resulting segmentation, we develop Event-OPD. Supervised fine-tuning teaches target grounding, while privileged on-policy distillation lets a full-event teacher guide a student that sees only the opening and closing segments. Extensive evaluations show that existing referring and reasoning segmentation models struggle under this setting, while adaptation to CaReVOS yields substantial gains. Our distilled model achieves 58.33 , 61.98 , and 60.16 &, exceeding the strongest trained baseline by 2.21 points and its supervised initialization by 1.62 points.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
video object segmentation, multimodal reasoning, causal reasoning, counterfactual reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_AZdjOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CaReVOS, a benchmark for identifying event participants from the opening and closing segments of a video while the intervening interaction is hidden from the reasoner. The selected objects are then segmented throughout the full video. The authors propose Event-OPD, which combines supervised answer and mask training with privileged on-policy distillation from a frozen copy of the SFT model that observes the complete event. Distillation aligns both next-token distributions and segmentation-token representations. The method achieves 60.16 J&F, improving over its SFT initialization by 1.62 points.

Critical strengths

The combination of missing-event inference and privileged on-policy distillation is well motivated. A teacher with access to the complete event can provide additional evidence when guiding a student that only observes the setup and outcome. Extending this guidance to the segmentation-token representation also connects answer-level distillation to spatial grounding. I consider this combination the main methodological highlight.

CaReVOS provides a concrete benchmark for studying this setting, with human-verified questions and object masks across multiple video sources. Requiring pixel-level outputs makes target selection errors observable and adds a useful grounding component to missing-event reasoning.

Critical weaknesses
  1. Ambiguity under partial observation is insufficiently quantified. The same opening and closing observations may support different hidden interactions and, potentially, different target identities. Although the authors describe manual verification and ambiguity filtering, the paper does not quantify how reliably independent annotators can identify the intended targets without seeing the hidden event. The multi-solution category addresses multiple jointly valid targets, but does not by itself resolve uncertainty between alternative explanations. This matters because evaluation against one reference target set could penalize another answer that is also defensible from the available evidence.

  2. Evidence for improved reasoning remains limited. The reported gains are measured primarily through segmentation quality, which combines target selection and mask accuracy. It is unclear whether the model generates explicit reasoning traces, whether thinking mode is enabled, and which reasoning content is actually distilled. The improvement over SFT is 1.62 J&F points, while giving the teacher the complete event adds 0.86 points over the restricted-view teacher. These gains are encouraging, but their stability and reasoning-specific contribution remain uncertain without repeated runs, a training-budget-matched continued-SFT control, and a separate evaluation of target selection or event inference.

Questions for authors
  1. How do you determine that the target identity is sufficiently constrained by the visible segments and question? Do you have an independent annotation study in which annotators never see the hidden event? How are alternative but defensible target sets handled during evaluation?

  2. Is Qwen3.5's thinking mode enabled during SFT, on-policy rollouts, and evaluation? Do the generated responses contain explicit event-inference rationales, or mainly target descriptions? Please clarify which tokens receive distillation supervision and provide representative outputs. A controlled comparison with and without explicit reasoning would help assess its contribution.

  3. How stable is the 1.62-point improvement across runs, and how does Event-OPD compare with continuing SFT for the same additional training budget? Can you separately evaluate target selection to distinguish reasoning gains from improvements in mask prediction?

As a possible extension, have you explored varying the hidden interval or constructing multiple validated queries and alternative explanations per clip? Such augmentation could help test robustness to different observation patterns, provided that each example is checked for answerability.

Reviewer_HCdJOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The work focuses on reasoning video object segmentation specifically across partial observations missing central events. It introduces a benchmark CaReVOS, of 1432 videos collected from 7 datasets/benchmarks (STAR, OOPS, EPIC-KITCHENS, VOST, NExT-QA, GroundMoRe and Minimal Video Pairs), 6386 reasoning and 373828 object mask annotations. The questions span four defined categories, consequence, source ,multi solution and counterfactual.

It proposes Event-OPD, a two stage framework of SFT then privileged OPD that aims to connect a VLM to video segmenter via learned object representation and also distills from a teacher model with access to complete events and optimizes the student using token level KL divergence and segmentation representation alignment. Their proposed method attains 60.16 J&F, outperforming the trained external baseline and its SFT initialization.

Critical strengths
  • The proposed setting is interesting because identifying objects through an unseen interaction is different from recognizing an explicitly observed action. It requires combining information from a prior state and an effect or outcome, making it potentially relevant to abductive reasoning and causal event understanding/reasoning.

  • The work provides a reasonably large benchmark drawn from several sources with a well thought out and detailed annotation and a human verification pipeline.

  • The work evaluates numerous existing methods and provides useful ablations on training strategies, loss components, observed segments, and privileged teacher information.

Critical weaknesses
  • My primary concern is the extent to which CaReVOS genuinely evaluates causal reasoning. Several question types appear potentially solvable through visual state comparison or event-participant identification without understanding the underlying causal relationship. For example, consequence questions may be answered by identifying objects whose states changed, while multi-solution questions may involve detecting objects that changed locations. This is particularly pronounced in the counterfactual questions category. In Figure 7D, the question asks, "If the bottle were empty, which performer would have stayed dry?", with the woman as the target. The hypothetical intervention changes the outcome experienced by the woman, but does not change the identity of the entity to be segmented. A model could therefore identify the participant affected by the factual event and produce the correct mask without reasoning about the counterfactual outcome. A similar issue arises in Figure 5 with the question about an empty drawer.
    More generally, different factual, causal, and counterfactual questions can yield identical segmentation targets even though they require different reasoning processes. Since the benchmark evaluates the resulting masks, it cannot determine whether a model has correctly inferred the causal relationship or simply identified the appropriate event participant.
    Based on the provided question generation prompt and verification details, the annotation pipeline includes safeguards against shortcuts, these do not establish or necessarily constrain that causal inference is necessary to solve the benchmark. Furthermore, although the annotation procedure is detailed, the paper does not report inter-annotator agreement or demonstrate that humans can reliably identify the correct answers using only the visible segments.

  • The results do not isolate improvements in causal reasoning. The main metrics reported in the paper, J&F do not distinguish correct reasoning with poor segmentation vs incorrect reasoning with good segmentation. i.e improvements are entangled cannot be attributed to better causal reasoning rather than improved localization or tracking.
    While the authors themselves acknowledge this in appendix D, mask accuracy alone is not equivalent to correctly inferring or reasoning about the missing event. Some analysis or ablations on whether Event-OPD improves the model's ability to infer missing events, or if it mainly improves the representations and mask predictions associated with the final target, would have been better in supporting the paper's claims of causal reasoning.

Questions for authors
  • The teacher model is initialized from the SFT student and frozen, rather than independently trained on full-event input. There could be some potential problems here as the teacher itself has not been optimized to use the hidden events. Comparison with a full-observation SFT teacher or further details on this design choice would be helpful.
  • As there are Teacher-advantage conditions, what proportion of training examples activates the token KL and segmentation alignment terms, respectively? Does the privileged teacher provide advantages primarily for specific question categories or event types?
  • Could more details be provided on the human verification stage of the dataset? How many reviewers? What were instructions to reviewer? What percentages of samples were reviewed? Is there any annotator agreement? As a large part of the pipeline utilizes models to generate the samples, these information on the verification stage could help solidify the reliability of the dataset.
Reviewer_qTB7OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces causal reasoning across missing events, a new task for video object segmentation in which the key interaction is unobserved: the model must infer the object responsible for a change from only the before- and after-event segments, then segment and track it throughout the full video. The authors build CaReVOS, a human-verified dataset spanning consequence, source, multi-solution, and counterfactual questions. They also propose Event-OPD, a training framework that augments supervised fine-tuning with privileged on-policy distillation, in which a teacher with access to the full video guides the student on its own sampled generations via token-level KL and segmentation-representation alignment. Experiments show the method reaches J&F = 60.16 on the test set, outperforming the strongest baseline by 2.21 points, with ablations confirming the contribution of each component.

Critical strengths
  • Novel, substantive task formulation. "Causal reasoning across missing events" targets a real blind spot, existing reasoning segmentation assumes the key interaction is visible. Here the model must reason abductively from before/after segments and ground the result at pixel level. The strongest zero-shot model reaches only 47.58 J&F, confirming a genuine challenge. A clearly defined, discriminating new task is a lasting contribution.

  • Method matched to the task, with ablation support. Event-OPD's privileged distillation exploits the task's information asymmetry (teacher sees the full video, student only the visible segments). Restricting the teacher to the same segments drops J&F from 60.16 to 59.30, showing the gain comes from observing the hidden event rather than generic distillation.

Critical weaknesses
  • The trainable SAM2 decoder may leak the hidden segment. Section 3.1 forbids frames outside the observed set from altering the inferred target, but the constraint applies only to the reasoner's input. The mask decoder and projector are trainable (Sec. 5.1) and the mask loss in Eq. 4 covers every frame including the hidden interval, so hidden-segment gradients reach the decoder and shape the mapping from the segmentation-token state to the segmenter prompt. A degenerate solution is thus reachable: the reasoner emits an under-determined target representation and the full-video decoder latches onto whatever object changes. The protocol is unfalsifiable as stated, and partial-observation reasoning cannot be disentangled from full-video segmentation.

  • The paper's own SFT model already beats every baseline before the method is applied. SFT only reaches 58.54 (Table 3), above the strongest trained baseline (FeVOS-R1 SFT, 57.95). Table 2 omits it, so the visible comparison is Sa2VA (SFT) 53.82 to 60.16, inviting attribution of about 6 points to the framework when only 1.62 is attributable. The 2.21-point headline also conflates backbone (Qwen3.5-4B vs. InternVL2.5-4B and others), architecture, and recipe.

Questions for authors
  • SFT runs 2 epochs and Event-OPD adds 2 more, so 58.54 to 60.16 is confounded with longer training. Please report SFT at 4 epochs.

  • The teacher is a frozen SFT copy that only sees more frames, yet its test score is never reported. it bounds the distillation signal and measures what observing the interaction is worth.

  • Please report retention, rejection, and manual-correction rates for each data construction stage. Without them, the two-segment need and single-segment gate of Section 3.2 cannot be assessed, which matters given that the opening segment alone reaches 54.71, or 91 percent of the 60.16 from both segments.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.