acceptodds
Under review as a conference paper at ICLR 2027

Think Visually Across Space and Time: Grounded Reasoning for Video Segmentation

Abstract

Video Reasoning Segmentation (VRS) requires a model to identify an implicitly described target and predict its pixel-level mask trajectory, while resolving the spatial relations and temporal events that determine the answer. Text-only Chain-of-Thought (CoT) can explain a target choice without making the underlying visual references explicit, leaving a *video reference gap* between linguistic inference and the evidence distributed across frames. We introduce **Temporal Chain of Video Primitives (TCoVP)**, a spatiotemporally grounded form of Video CoT that organizes query-relevant observations into a chronological reasoning chain. Each primitive associates an entity reference with temporal support and timestamped spatial evidence, and the chain interleaves these references with language to guide reasoning about which entity is relevant, when the queried event holds, and where its evidence appears. The selected target primitives then provide box prompts and temporal support for video mask prediction. We construct TCoVP-SFT-15k and TCoVP-RL-13k to learn this representation through supervised fine-tuning and chain-aligned reinforcement learning. Extensive experiments across five VRS benchmarks and three model backbones demonstrate the strong performance of TCoVP, highlighting the effectiveness of organizing spatial and temporal evidence within the reasoning process. All models and code will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video Reasoning Segmentation, Temporal Chain of Video Primitives, Chain-aligned Reinforcement Learning, Multimodal Large Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_gp5oOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Chain of Video Primitives for Video Reasoning Segmentation, aiming to address the "video reference gap" between text-based reasoning and visual evidence. TCoVP interleaves natural-language reasoning with chronologically ordered video primitives, each containing an entity identifier, a temporal span, and a timestamped bounding box. The selected target primitives are then used as box prompts for SAM3 to generate segmentation masks. The method is trained using SFT on TCoVP-SFT-15k, followed by reinforcement learning on TCoVP-RL-13k with rewards for structural validity, temporal alignment, spatiotemporal grounding, and target accuracy. Experiments across five benchmarks and three MLLM backbones show strong performance,

Critical strengths
  1. Motivation is clear: The paper identifies a specific limitation of text-only CoT in VRS: the "video reference gap" between reasoning and visual evidence. This provides a clear motivation for the proposed method.
  2. The video primitives make intermediate reasoning steps explicit and verifiable, allowing rewards to be applied to intermediate steps rather than only the final answer.
  3. The oracle experiment on GroundMoRe is very interesting, which shows that replacing predicted temporal intervals with ground-truth intervals improves J&F from 53.7 to 85.5.
Critical weaknesses
  1. Limited novelty. Open-o3 Video already uses timestamped spatial evidence for video reasoning. The paper could better clarify how TCoVP differs from this closely related work.

  2. Missing weight ablation. The paper does not provide an ablation study on the temporal and spatial reward weights w_t and w_s, making it unclear how sensitive the method is to their choices.

Questions for authors

see weakness

3 comments
AuthorsOpenReview ↗

Response to Reviewer gp5o

We thank the reviewer for the helpful feedback. We clarify the distinction from Open-o3 Video below and have added the reward-weight ablation and analysis to the revised manuscript, with the added text highlighted in red.

W1: Novelty relative to Open-o3 Video

TCoVP addresses the video reference gap in video reasoning segmentation, which requires distinguishing the target from contextual entities and, for motion-grounded queries, identifying when the queried action holds. To meet these demands, we introduce a temporal primitive chain that unifies contextual evidence, target references, and temporal supports, together with chain-aligned evidence supervision and primitive-to-mask execution. These components connect grounded reasoning to the identity, location, and temporal extent of the segmentation output. The table below clarifies these contributions relative to Open-o3 Video:

AspectOpen-o3 VideoTCoVP (ours)
Problem addressedGrounded reasoning for video question answeringResolve implicit references for video reasoning segmentation: which entity is the target and when the queried action holds
CoT formulationObject names and boxes linked to observation timestampsChronologically ordered, entity-linked primitives with explicit temporal support intervals; selected target primitives provide mask prompts and action-valid output intervals
Data and evidence coverage5.9k newly annotated spatiotemporal CoT examples for video QA; the remaining 24.1k SFT examples are drawn from existing CoT datasets and are predominantly text-only CoT. During RL, 70% of training samples do not include intermediate spatiotemporal evidence.A unified primitive representation across SFT-15k and RL-13k, covering contextual evidence, target references, and temporal support intervals, with corresponding intermediate evidence available for RL supervision.
RL SupervisionTimestamp proximity, temporally gated spatial matching, and task-specific answer rewardsTemporal evidence precision-recall, intermediate box-and-support matching, and joint spatiotemporal target-set matching

As summarized above, TCoVP organizes evidence into chronologically ordered, entity-linked primitives with explicit temporal support intervals. This representation links observations of the entity across frames, distinguishes contextual entities from segmentation targets, and separates observation timestamps from action-valid intervals. Interleaved with language, the primitives support reasoning about relations and event sequences; selected target primitives then directly provide mask prompts and constrain output intervals for motion-grounded queries.

The data and rewards make this reasoning-to-segmentation connection learnable. While Open-o3 Video combines spatiotemporal QA annotations with predominantly text-only CoT data, TCoVP-SFT-15k and TCoVP-RL-13k share a unified representation of contextual evidence, target references, and temporal supports. Our chain-aligned rewards evaluate temporal evidence precision and recall, intermediate box-and-support alignment, and the final spatiotemporal target set. Together, these components provide a unified approach to resolving implicit video references through explicitly supervised evidence and executable target primitives.

W2: Sensitivity to temporal and spatial reward weights.

We thank the reviewer for highlighting the need for a weight ablation. We add an experiment comparing three configurations of wtw_t and wsw_s. These weights balance temporal overlap and spatial box overlap in the target primitive accuracy reward, Ri,j=wtTi,j+wsSi,jR_{i,j}=w_tT_{i,j}+w_sS_{i,j}. The table below reports J&F scores, with the first two columns specifying the reward weight ratios ws:wtw_s:w_t for GroundMoRe and other training sources, respectively.

GroundMoRe ws:wtw_s:w_tOther sources ws:wtw_s:w_tGroundMoReReVOSLong-RVOSReasonVOS
1:11:151.164.464.169.8
2:12:151.664.864.571.0
1:22:153.765.365.671.6

Our default configuration achieves the best performance across all four benchmarks. Across the three configurations, J&F varies by 0.9, 1.5, and 1.8 points on ReVOS, Long-RVOS, and ReasonVOS, respectively, and by 2.6 points on GroundMoRe.

With the weights for other sources fixed at ws:wt=2:1w_s:w_t=2:1, switching GroundMoRe from 2:12:1 to 1:21:2 increases its J&F from 51.6 to 53.7 (+2.1). This benefit from greater temporal emphasis is consistent with GroundMoRe's requirement to localize the query-relevant action interval. For the other sources, the default configuration emphasizes spatial precision to encourage accurate box prompts for mask propagation.

Sorry for my late reply. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score. Best of luck with your paper!

AuthorsOpenReview ↗

Comment by authors

We sincerely thank you for taking the time to carefully review our paper and rebuttal, and for your thoughtful feedback throughout the rebuttal process. Your comments have helped us clarify our contributions and strengthen the experimental analysis.

Reviewer_yqkFOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Chain of Video Primitives (TCoVP) for Video Reasoning Segmentation (VRS), which organizes visual evidence into chronological reasoning chains with explicit spatial and temporal grounding. The authors construct two training datasets and employ supervised fine-tuning followed by chain-aligned reinforcement learning. The predicted target primitives guide SAM3 for mask propagation. Experiments across five VRS benchmarks and three MLLM backbones demonstrate substantial improvements over existing methods, particularly on motion-grounded tasks.

Critical strengths
  1. Promising reasoning paradigm. As illustrated in Figure 1 and the qualitative results, TCoVP produces more concise, focused, and visually grounded reasoning traces, leading to more accurate target identification. This structured reasoning approach offers a promising direction for training multimodal models to reason more effectively with visual evidence.
  2. Well-motivated problem formulation. The paper identifies an important limitation of existing VRS methods: the lack of explicit spatiotemporal correspondence between language-based reasoning and visual evidence. The proposed temporal primitive chain provides an intuitive and meaningful approach to bridging this gap.
  3. Comprehensive experiments and clear presentation. The paper provides extensive evaluations across multiple benchmarks and model backbones, supported by ablation studies and informative visualizations. The substantial performance improvements and well-organized presentation strengthen the credibility of the proposed approach.
Critical weaknesses
  1. Fairness of Comparisons and Additional Supervision. Can Veason-R1 and VideoSEG-O3 also be trained on TCoVP-SFT-15k and TCoVP-RL-13k? Does TCoVP benefit from additional annotations or supervision unavailable to these baselines? If applicable, data-matched comparisons would help clarify whether the gains come from the proposed method or richer supervision.
  2. Insufficient Ablation of Temporal Chain Design. The contributions of chronological ordering and entity ID reuse remain unclear. Could the authors evaluate these components separately while keeping spatial and temporal supervision unchanged?
  3. Limited Entity Consistency Supervision. The proposed rewards mainly rely on spatial and temporal matching, potentially overlooking cross-frame identity and relational consistency. Could the authors clarify this limitation and provide quantitative evaluations of entity consistency or relational grounding?
Questions for authors

How sensitive is TCoVP's performance to the quality of annotations generated by Kimi-K2.5 and verified by Qwen2.5-VL? Could using newer or larger models further improve annotation quality and downstream performance? Since data construction is a one-time cost and the resulting datasets can be reused across models, understanding this relationship would be valuable for assessing the approach's potential and scalability.

5 comments
AuthorsOpenReview ↗

Response to Reviewer yqkF (Part 1)

We thank the reviewer for the thoughtful feedback on comparison fairness, temporal chain design, and annotation and reasoning quality. All additional experiments reported in our responses are included in the revised manuscript, with the added text highlighted in red.

W1: Fairness of Comparisons and Additional Supervision

1. Can the baselines be trained on our datasets? Neither method directly supports our datasets in their current form.

  • Veason-R1: Its 8-frame sampling can omit evidence frames in our 16-frame annotations. Reuse requires adapting frame sampling and aligning timestamps, boxes, reasoning traces, and answer targets.
  • VideoSEG-O3: Its multi-turn agentic training requires action-observation trajectories, whereas our datasets provide single-turn primitive reasoning. Reuse requires constructing compatible interaction trajectories and adapting the supervision and reward interfaces.

2. Training data and supervision across methods. Supervision differs across methods in both scale and form, as summarized below.

MethodSeparate foundational alignment stageCoT supervised fine-tuningReinforcement learning
Veason-R1None5.8k CoT examples from ReVOS10k examples from ReVOS
VideoSEG-O3100k video-QA data + 100k temporal-grounding data + 270.2k image-segmentation data + 45k video-segmentation dataApproximately 6k VTS-CoT examplesReVOS (1.7k) + MeViS (0.6k), reported as source dataset sizes
TCoVP (ours)None15k examples spanning ReVOS, MeViS, Long-RVOS, and GroundMoReApproximately 13k examples from the same four sources

As the table shows, VideoSEG-O3 includes an additional alignment stage with training data on a substantially larger scale than our task-specific corpus, spanning video QA, temporal grounding, and image/video segmentation. Veason-R1 also uses more ReVOS examples during RL than TCoVP. These differences show that the baselines themselves draw on substantial training resources, and that data exposure varies across stages and sources.

To further distinguish the effects of our method from differences in training data and supervision, we present controlled comparisons below with matched training examples and additional ablations that preserve intermediate grounding supervision.

3. Data-matched and supervision-controlled comparisons. Table 3 in the main paper provides a controlled comparison between TCoVP and Text-CoT with answer-VP, using the same Qwen2.5-VL-3B backbone, training examples, data volume, and training configuration.

SettingReVOSReasonVOSLong-RVOSGroundMoReMeViSMacro avg.
Text-CoT with answer-VP62.065.260.546.354.857.8
TCoVP-3B65.371.665.653.757.362.7

To further distinguish the structural contribution from richer supervision, we conduct two additional SFT+RL ablations.

  • w/o Chronological Ordering: For each entity, we randomly permute its complete primitive records among the positions occupied by that entity in the reasoning trace. Each record retains its entity ID, bounding box, observation timestamp, and temporal support interval.

  • w/o Entity ID Reuse: We append a distinct numerical suffix to each object identifier occurrence, so repeated references use different identifiers (e.g., person_1, person_2). This preserves the original object-name stem, primitive order, and all spatial and temporal values while removing exact identifier reuse.

Both variants retain the surrounding natural-language reasoning and the same intermediate spatial and temporal annotations. We train each variant through SFT and RL with the same backbone, training examples, schedule, and hyperparameters as TCoVP-3B. All entries below are J&F scores.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.765.365.671.6
w/o Chronological Ordering45.663.262.267.1
w/o Entity ID Reuse43.463.561.366.3

Removing chronological ordering reduces J&F by 2.1-8.1 points, and removing exact entity ID reuse reduces it by 1.8-10.3 points across the four benchmarks.

Together, these results show that TCoVP improves performance under matched training examples and data volume, and that chronological ordering and entity ID reuse contribute beyond the intermediate grounding supervision shared by the ablation variants. The gains therefore cannot be attributed solely to more training examples or richer supervision.

AuthorsOpenReview ↗

Response to Reviewer yqkF (Part 2)

W2: Ablations of Temporal Chain Design

We thank the reviewer for this suggestion. To address this concern, we provide controlled SFT+RL ablations that evaluate each component separately while retaining the same spatial and temporal annotations and grounding rewards. We also provide complementary inference-time interventions on TCoVP-3B.

1. Controlled ablations with unchanged spatial and temporal supervision. We independently modify the SFT reasoning traces as follows:

  • w/o Chronological Ordering: For each entity, we randomly permute its complete primitive records among the positions occupied by that entity in the reasoning trace. Each record retains its entity ID, bounding box, observation timestamp, and temporal support interval. This disrupts chronological ordering within an entity while preserving the grounding evidence and ID reuse.
  • w/o Entity ID Reuse: We append a distinct numerical suffix to each object identifier occurrence, so repeated references use different identifiers (e.g., person_1, person_2). This preserves the original object-name stem, primitive order, and all spatial and temporal values while removing exact identifier reuse.

Both variants retain the surrounding natural-language reasoning and all intermediate spatial and temporal annotations. We train each variant through SFT and RL with the same backbone, training examples, schedule, and hyperparameters as TCoVP-3B. During RL, the structural validity reward does not enforce chronological ordering or identifier reuse; the temporal-alignment, spatiotemporal-grounding, and target-accuracy rewards remain unchanged. All entries below are J&F scores.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.765.365.671.6
w/o Chronological Ordering45.663.262.267.1
w/o Entity ID Reuse43.463.561.366.3

Disrupting chronological ordering reduces J&F by 2.1-8.1 points, and removing exact entity ID reuse reduces it by 1.8-10.3 points across the four benchmarks. The largest drops occur on GroundMoRe (8.1 and 10.3 points, respectively), where the task requires identifying both the target and its query-relevant action interval. These results support the contribution of both chronological ordering and exact entity ID reuse beyond the spatial and temporal grounding supervision shared by all variants.

2. Complementary inference-time interventions. Without retraining, we independently apply the same two modifications to the primitives in the <think> block generated by TCoVP-3B. We retain the original video, query, prompt, and surrounding natural-language reasoning. We provide the modified block within the assistant message, followed by <answer>, and let the model generate a new answer without access to the original answer. The new answer primitives enter the same SAM3 segmentation pipeline. The table reports segmentation J&F / final-answer bounding-box IoU.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.7 / 59.465.3 / 65.465.6 / 64.371.6 / 65.1
w/o Chronological Ordering47.7 / 50.360.4 / 59.959.7 / 56.566.4 / 60.9
w/o Entity ID Reuse45.3 / 49.161.8 / 61.162.1 / 59.665.8 / 59.2

Both interventions reduce both metrics on every benchmark. Removing chronological ordering reduces J&F by 4.9-6.0 points, while removing exact entity ID reuse reduces it by 3.5-8.4 points. The accompanying declines in final-answer box IoU indicate that these effects extend to answer generation before mask propagation. These interventions demonstrate the model's sensitivity to both aspects of the temporal chain and complement the controlled training ablations.

AuthorsOpenReview ↗

Response to Reviewer yqkF (Part 3)

W3: Entity Consistency and Relational Grounding

We agree that spatial and temporal matching alone does not explicitly enforce cross-frame identity or relational consistency. In our training pipeline, SFT provides supervision through reasoning chains with consistent entity identifiers, allowing the model to learn to associate observations of the same entity across time. RL primarily reinforces the spatial and temporal accuracy of these chains, but does not explicitly supervise cross-frame identity or relational consistency. This limitation is acknowledged in our limitations discussion and illustrated by the relational-role failure case.

To quantitatively assess these aspects, we conduct a human evaluation of 200 randomly sampled model-generated reasoning chains. For entity consistency, annotators examine pairs of references to the same predicted entity ID at consecutive appearances and determine whether they refer to the same real-world object. The score is the number of consistent pairs divided by the number of evaluable pairs. For relational grounding accuracy, annotators assess whether the actions and states attributed to participating entities in the reasoning chain are supported by the corresponding video segment; the score is the number of correct relations divided by the number of evaluable relations.

Human evaluation metricResult
Entity consistency99%
Relational grounding accuracy92%

These results suggest that the generated reasoning chains generally maintain entity consistency and ground entity-related actions and states in the corresponding video evidence. The observed error rates are 1% for entity consistency and 8% for relational grounding across evaluable reference pairs and relations, respectively. While entity consistency is high in the audited samples, relational grounding still leaves room for improvement. Explicit identity and relational consistency rewards therefore remain a direction for future work.

Q: Annotation Quality and Scalability

We conduct a human evaluation of 500 randomly sampled dataset examples, assessing entity consistency, relational grounding, and spatial and temporal agreement with human annotations. As shown below, the results indicate reliable entity associations and temporal alignment in the audited samples. Contextual box imprecision and the remaining 3% relational error rate identify concrete opportunities for improving annotation quality.

Human evaluation metricResult
Entity consistency100%
Relational grounding accuracy97%
Mean contextual box IoU0.79
Mean temporal-span IoU0.93

Regarding sensitivity to annotation quality, these annotations provide both SFT supervision and reference evidence for chain-aligned RL, so annotation errors can affect both learning stages. Our pipeline uses target boxes derived from ground-truth masks and applies consistency checks and semantic filtering to improve supervision reliability. With the current annotations, TCoVP consistently outperforms its text-only CoT counterpart across all five benchmarks, increasing the macro-average J&F from 57.8 to 62.7 under the same backbone, data volume, and training configuration (Table 3). Its strong performance across three backbone families further supports the utility and reusability of the constructed supervision. Together, these results demonstrate that the current annotation pipeline supports effective learning of spatiotemporally grounded reasoning for segmentation.

Regarding newer or larger annotation models, stronger generators or verifiers potentially improve contextual localization and relational grounding, providing more accurate training signals. Their downstream benefit remains an empirical question that requires controlled comparisons with fixed data size, backbone, and training budget. We agree that dataset reuse makes this investment valuable: the same TCoVP-SFT-15k and TCoVP-RL-13k datasets support Qwen2.5-VL-3B, Qwen3-VL-4B, and Molmo2-4B in our experiments. Thus, the offline construction cost is amortized across models, supporting scalability and motivating further improvements in annotation quality.

Reviewer_dVhMOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes Temporal Chain of Video Primitives (TCoVP), a framework for video reasoning segmentation that explicitly grounds intermediate reasoning in spatial and temporal evidence. Each primitive contains an entity reference, a timestamped bounding box, and a temporal span. These primitives are interleaved with natural-language reasoning, and the selected target primitives provide prompts for video mask propagation. The authors construct two training datasets and combine supervised fine-tuning with chain-aligned reinforcement learning. Experiments across five benchmarks and three model backbones demonstrate improvements over existing methods.

Critical strengths
  1. Well-motivated representation. The paper addresses an important gap between linguistic reasoning and visual grounding.

  2. Strong empirical results. TCoVP achieves competitive performance across multiple benchmarks and backbones.

  3. Useful diagnostic experiments. The reward ablations, intermediate grounding evaluations, and SAM2/SAM3 comparisons strengthen the empirical analysis. The oracle experiments are especially informative in identifying temporal localization as a major remaining bottleneck on GroundMoRe.

Critical weaknesses
  1. The contribution of individual design choices is not fully isolated. The main representation comparison introduces both structured primitives and additional intermediate grounding supervision. It therefore does not establish how much improvement specifically comes from chronological ordering or persistent entity identifiers.

  2. Annotation reliability needs stronger validation. The training pipeline uses model-generated reasoning, contextual boxes, and temporal annotations. Although consistency checks and semantic filtering are described, quantitative human evaluation would better establish the reliability of the resulting supervision.

  3. Grounded traces do not necessarily establish faithful reasoning. Improved intermediate box and temporal accuracy supports better localization, but does not demonstrate that the model uses the stated contextual evidence to reach its final decision. The relational-role failure case further illustrates this distinction.

  4. Efficiency and robustness are underexplored. The paper does not report inference latency or generated reasoning length. Moreover, the fixed sampling of 16 frames may limit performance on brief events and long videos, but the sensitivity to sampling density is not quantified.

Questions for authors
  1. Could you provide ablations that remove chronological ordering or persistent entity identifiers while retaining comparable grounding supervision and training budgets?

  2. What are the human-verified error rates for contextual boxes, entity associations, and temporal spans? How much does each filtering stage improve annotation quality?

  3. Does perturbing or removing relevant contextual primitives change the final prediction in the expected way? Such experiments could help assess whether the generated evidence meaningfully supports target selection.

  4. What is the inference cost relative to text-only CoT and existing VRS methods? How does accuracy vary with the number of sampled frames, video duration, and event length?

6 comments
AuthorsOpenReview ↗

Response to Reviewer dVhM (Part 1)

We thank the reviewer for the constructive suggestions on structural ablations, annotation reliability, primitive interventions, and efficiency. All additional experiments reported in our responses have been incorporated into the revised manuscript, with the added text highlighted in red.

W1 and Q1: Isolating chronological ordering and persistent entity identifiers

We thank the reviewer for this suggestion. We agree that the main representation comparison does not separately isolate chronological ordering and persistent entity identifiers, as it also introduces intermediate grounding supervision. To examine the contributions of chronological ordering and persistent entity identifiers, we provide two complementary experiments with TCoVP-3B: controlled SFT+RL ablations with comparable grounding supervision and training budgets, and inference-time interventions on the existing model.

1. Controlled SFT+RL ablations. We independently modify the SFT reasoning traces in two ways:

  • w/o Chronological Ordering: For each entity, we randomly permute its complete primitive records among the positions occupied by that entity in the reasoning trace. Each record retains its identifier, bounding box, observation timestamp, and temporal support interval. This disrupts chronological ordering within an entity while preserving its grounding evidence and identifier reuse.
  • w/o Entity ID Reuse: We append a distinct numerical suffix to each object identifier occurrence, so that repeated references use different identifiers (e.g., person_1, person_2). This preserves the original object-name stem, primitive order, and all spatial and temporal values while removing exact identifier reuse.

Both variants retain the surrounding natural-language reasoning and the same intermediate spatial and temporal annotations. We train each variant through SFT and RL using the same backbone, training examples, training schedule, and hyperparameters as TCoVP-3B. During RL, the structural validity reward does not enforce chronological ordering or identifier reuse. The temporal-alignment, spatiotemporal-grounding, and target-accuracy rewards remain in place, preserving comparable grounding supervision throughout training. The table below reports performance on the four benchmarks, with all scores measured by J&F.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.765.365.671.6
w/o Chronological Ordering45.663.262.267.1
w/o Entity ID Reuse43.463.561.366.3

Removing chronological ordering reduces J&F by 2.1-8.1 points, while replacing persistent entity identifiers with occurrence-specific identifiers reduces it by 1.8-10.3 points across the four benchmarks. The largest drops occur on GroundMoRe, where the query requires identifying the target together with its relevant action interval. These results support the value of both design choices even when intermediate grounding supervision and the training budget remain comparable.

2. Inference-time interventions. We apply the same two modifications independently to the reasoning traces generated by TCoVP-3B, without retraining. For each intervention, we retain the original video, query, and prompt, modify only the primitives in the generated <think> block, and provide the modified block within the assistant message, followed by <answer>, as in the original format. The model then continues generating a new answer without access to the original answer. The surrounding natural-language reasoning remains unchanged, and the new answer primitives enter the same SAM3 segmentation pipeline. The table below reports segmentation J&F / final-answer bounding-box IoU.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.7 / 59.465.3 / 65.465.6 / 64.371.6 / 65.1
w/o Chronological Ordering47.7 / 50.360.4 / 59.959.7 / 56.566.4 / 60.9
w/o Entity ID Reuse45.3 / 49.161.8 / 61.162.1 / 59.665.8 / 59.2

Both interventions reduce both metrics on every benchmark. Removing chronological ordering reduces J&F by 4.9-6.0 points, while replacing persistent entity identifiers with occurrence-specific identifiers reduces it by 3.5-8.4 points. The accompanying declines in final-answer box IoU show that the effects extend to answer generation before mask propagation. These results demonstrate sensitivity to primitive ordering and identifier reuse at inference time.

AuthorsOpenReview ↗

Response to Reviewer dVhM (Part 2)

W2 and Q2: Annotation Reliability

We conduct a human evaluation of 500 randomly sampled dataset examples, covering entity consistency, relational grounding, contextual box alignment, and temporal-span alignment.

Human evaluation metricResult
Entity consistency100%
Relational grounding accuracy97%
Mean contextual box IoU0.79
Mean temporal-span IoU0.93

For entity consistency, annotators examine pairs of references to the same entity ID at consecutive appearances and determine whether they refer to the same real-world object. The score is the number of consistent pairs divided by the number of evaluable pairs. For relational grounding accuracy, annotators assess whether the actions and states attributed to participating entities in the reasoning chain are supported by the corresponding video segment; the score is the number of correct relations divided by the number of evaluable relations. The observed error rates are therefore 0% for entity consistency and 3% for relational grounding on the audited samples. These results suggest that the dataset's reasoning chains generally maintain entity consistency and ground entity-related actions and states in the corresponding video evidence. For spatial and temporal correctness, we compare the dataset annotations with human annotations using box IoU and temporal IoU, respectively. The spatial and temporal IoU mean scores of 0.79 and 0.93 provide further support for the reliability of the dataset annotations.

We also report sample retention rates, each computed relative to the input to the corresponding filtering stage, to characterize their selectivity:

Filtering stageSample retention rate
Consistency checking85.2%
Primitive filtering78.6%

W3 and Q3: Reasoning faithfulness and primitive interventions

We thank the reviewer for this suggestion. To examine whether the generated primitives affect the final prediction, we add three inference-time interventions using TCoVP-3B without retraining. For each intervention, we retain the original video, query, and prompt, modify only the primitives in the generated <think> block, and provide the modified block within the assistant message, followed by <answer>, as in the original format. The model then continues generating a new answer without access to the original answer. The surrounding natural-language reasoning remains unchanged, and the new answer primitives enter the same SAM3 segmentation pipeline. We consider three independent modifications:

(1) Name-only. We retain object names but remove the primitives' spatial and temporal values and special tags.

(2) Box perturbation. We randomly translate each box while preserving its width and height. The horizontal and vertical displacements are Δx=0.2wcos⁡θ\Delta x=0.2w\cos\theta and Δy=0.2hsin⁡θ\Delta y=0.2h\sin\theta, where w,hw,h are the box width and height and θ∼U(0,2π)\theta\sim\mathcal{U}(0,2\pi) is sampled independently for each box. We round these displacements to integer pixels and limit them to keep the box inside the image. All other primitive fields remain unchanged.

(3) Temporal-span perturbation. For each interval [s,e][s,e] of length L=e−sL=e-s, we independently add offsets ϵs,ϵe∼U(−0.5L,0.5L)\epsilon_s,\epsilon_e\sim\mathcal{U}(-0.5L,0.5L) to its endpoints. We clip the endpoints to the video duration and round to one decimal place. Repeated occurrences of the same entity and interval share the same perturbation. Only <time> changes; boxes and observation timestamps <t> remain unchanged.

The table reports segmentation J&F / final-answer bounding-box IoU on a 0-100 scale. The TCoVP-3B results serve as the reference.

SettingGroundMoReReVOSLong-RVOSReasonVOS
TCoVP-3B53.7 / 59.465.3 / 65.465.6 / 64.371.6 / 65.1
Name-only38.9 / 38.455.3 / 52.358.7 / 53.455.4 / 47.7
Box perturbation46.9 / 42.853.4 / 50.255.1 / 48.960.6 / 49.4
Temporal-span perturbation45.1 / 56.063.5 / 63.463.3 / 60.968.0 / 61.9

All three interventions reduce both metrics across all four benchmarks. Name-only reduces J&F by 6.9-16.2 points, and box perturbation reduces it by 6.8-11.9 points, indicating that object names and the retained language alone do not preserve the performance. Temporal-span perturbation also consistently degrades performance, with the largest J&F drop on GroundMoRe (8.6 points), where segmentation depends on the query-relevant action interval. The accompanying decreases in final-answer box IoU show that the effects extend to answer generation before mask propagation. These results support the functional relevance of the explicit spatiotemporal primitives to final prediction.

AuthorsOpenReview ↗

Response to Reviewer dVhM (Part 3)

W4 and Q4: Inference Efficiency, Generation Length, and Sensitivity to Temporal Sampling

We thank the reviewer for highlighting efficiency and temporal sampling. We provide additional measurements of inference and segmentation latency, input/output token counts, and performance with 8, 16, and 32 sampled frames. These results show that TCoVP maintains comparable inference cost to the text-only CoT method Veason-R1-3B, and achieves lower total latency than the other evaluated VRS methods on ReVOS. The sampling study supports 16 frames as a practical default, while quantifying the benefits and limitations of denser sampling for long videos and brief actions.

For the direct comparison with Veason-R1-3B, we use the same NVIDIA L20 GPU and the same vLLM inference and SAM2 segmentation framework. The following table reports average seconds per video sample, with MLLM inference and video segmentation measured separately.

BenchmarkVeason-R1-3B: inference (s)Veason-R1-3B: segmentation (s)TCoVP-3B: inference (s)TCoVP-3B: segmentation (s)
GroundMoRe5.888.116.275.93
ReVOS6.148.095.497.61
Long-RVOS6.4527.526.3823.18
ReasonVOS6.5514.197.4314.70

TCoVP's MLLM inference time remains close to that of Veason-R1-3B, with differences ranging from -0.65 to +0.88 seconds per sample. Its segmentation time is lower on GroundMoRe, ReVOS, and Long-RVOS, and slightly higher on ReasonVOS.

We additionally compare the average total inference latency per video sample with other VRS methods on ReVOS, with values reported in whole seconds.

MethodTotal latency on ReVOS (s/video)
VISA-13B45
VRS-HQ-13B39
VideoSEG-O3-4B29
Veason-R1-3B14
TCoVP-3B13

TCoVP has the lowest latency among these methods: approximately 3.5x, 3.0x, and 2.2x faster than VISA-13B, VRS-HQ-13B, and VideoSEG-O3-4B, respectively, while remaining close to Veason-R1-3B. These are comparisons of the evaluated systems, whose model sizes and execution procedures differ. TCoVP generates grounded reasoning chain in a single autoregressive response and then executes the selected target primitives for mask prediction, without iterative tool interaction during reasoning.

To characterize sequence length, we report average input and generated output token counts per sample.

BenchmarkVeason-R1-3B: inputVeason-R1-3B: outputTCoVP-3B: inputTCoVP-3B: output
GroundMoRe33842272562475
ReVOS33782072354477
Long-RVOS33932152377410
ReasonVOS33862142378404

Our method uses 24.3%-30.3% fewer input tokens than Veason-R1-3B while generating longer outputs (404-477 vs. 207-227 tokens) that include explicit spatiotemporal evidence. Together with the latency results above, these measurements show that our method incorporates explicit spatiotemporal evidence into reasoning while maintaining a comparable inference cost.

Finally, we vary the number of uniformly sampled frames at inference for TCoVP-3B. All entries below are J&F scores.

Sampled framesGroundMoReReVOSLong-RVOSReasonVOS
850.264.663.067.0
16 (default)53.765.365.671.6
3254.465.065.869.7

Increasing the frame count from 8 to 16 improves all four benchmarks by 3.5, 0.7, 2.6, and 4.6 points, respectively, demonstrating the value of greater temporal coverage. Increasing it further to 32 yields small gains on GroundMoRe (+0.7) and Long-RVOS (+0.2), but decreases ReVOS (-0.3) and ReasonVOS (-1.9). Thus, denser sampling does not produce uniformly better accuracy, and 16 frames offers a practical balance between temporal coverage and the number of input observations.

Regarding video duration, Long-RVOS specifically evaluates long-video scenarios: performance increases from 63.0 at 8 frames to 65.6 at 16 frames and 65.8 at 32 frames. Regarding event length, GroundMoRe emphasizes motion-grounded queries that require localizing the relevant action interval, making it particularly relevant to brief events; its scores increase from 50.2 to 53.7 and 54.4. These results provide benchmark-level evidence for both settings and suggest that denser sampling remains useful for temporally demanding queries.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.