- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Temporal Chain of Video Primitives for Video Reasoning Segmentation, aiming to address the "video reference gap" between text-based reasoning and visual evidence. TCoVP interleaves natural-language reasoning with chronologically ordered video primitives, each containing an entity identifier, a temporal span, and a timestamped bounding box. The selected target primitives are then used as box prompts for SAM3 to generate segmentation masks. The method is trained using SFT on TCoVP-SFT-15k, followed by reinforcement learning on TCoVP-RL-13k with rewards for structural validity, temporal alignment, spatiotemporal grounding, and target accuracy. Experiments across five benchmarks and three MLLM backbones show strong performance,
Critical strengths
- Motivation is clear: The paper identifies a specific limitation of text-only CoT in VRS: the "video reference gap" between reasoning and visual evidence. This provides a clear motivation for the proposed method.
- The video primitives make intermediate reasoning steps explicit and verifiable, allowing rewards to be applied to intermediate steps rather than only the final answer.
- The oracle experiment on GroundMoRe is very interesting, which shows that replacing predicted temporal intervals with ground-truth intervals improves J&F from 53.7 to 85.5.
Critical weaknesses
-
Limited novelty. Open-o3 Video already uses timestamped spatial evidence for video reasoning. The paper could better clarify how TCoVP differs from this closely related work.
-
Missing weight ablation. The paper does not provide an ablation study on the temporal and spatial reward weights w_t and w_s, making it unclear how sensitive the method is to their choices.
Questions for authors
see weakness
3 comments
Sorry for my late reply. After carefully reading all the reviews and the authors’ rebuttal, I have decided to maintain my original score. Best of luck with your paper!
Comment by authors
We sincerely thank you for taking the time to carefully review our paper and rebuttal, and for your thoughtful feedback throughout the rebuttal process. Your comments have helped us clarify our contributions and strengthen the experimental analysis.
Response to Reviewer gp5o
We thank the reviewer for the helpful feedback. We clarify the distinction from Open-o3 Video below and have added the reward-weight ablation and analysis to the revised manuscript, with the added text highlighted in red.
W1: Novelty relative to Open-o3 Video
TCoVP addresses the video reference gap in video reasoning segmentation, which requires distinguishing the target from contextual entities and, for motion-grounded queries, identifying when the queried action holds. To meet these demands, we introduce a temporal primitive chain that unifies contextual evidence, target references, and temporal supports, together with chain-aligned evidence supervision and primitive-to-mask execution. These components connect grounded reasoning to the identity, location, and temporal extent of the segmentation output. The table below clarifies these contributions relative to Open-o3 Video:
As summarized above, TCoVP organizes evidence into chronologically ordered, entity-linked primitives with explicit temporal support intervals. This representation links observations of the entity across frames, distinguishes contextual entities from segmentation targets, and separates observation timestamps from action-valid intervals. Interleaved with language, the primitives support reasoning about relations and event sequences; selected target primitives then directly provide mask prompts and constrain output intervals for motion-grounded queries.
The data and rewards make this reasoning-to-segmentation connection learnable. While Open-o3 Video combines spatiotemporal QA annotations with predominantly text-only CoT data, TCoVP-SFT-15k and TCoVP-RL-13k share a unified representation of contextual evidence, target references, and temporal supports. Our chain-aligned rewards evaluate temporal evidence precision and recall, intermediate box-and-support alignment, and the final spatiotemporal target set. Together, these components provide a unified approach to resolving implicit video references through explicitly supervised evidence and executable target primitives.
W2: Sensitivity to temporal and spatial reward weights.
We thank the reviewer for highlighting the need for a weight ablation. We add an experiment comparing three configurations of wt and ws. These weights balance temporal overlap and spatial box overlap in the target primitive accuracy reward, Ri,j=wtTi,j+wsSi,j. The table below reports J&F scores, with the first two columns specifying the reward weight ratios ws:wt for GroundMoRe and other training sources, respectively.
Our default configuration achieves the best performance across all four benchmarks. Across the three configurations, J&F varies by 0.9, 1.5, and 1.8 points on ReVOS, Long-RVOS, and ReasonVOS, respectively, and by 2.6 points on GroundMoRe.
With the weights for other sources fixed at ws:wt=2:1, switching GroundMoRe from 2:1 to 1:2 increases its J&F from 51.6 to 53.7 (+2.1). This benefit from greater temporal emphasis is consistent with GroundMoRe's requirement to localize the query-relevant action interval. For the other sources, the default configuration emphasizes spatial precision to encourage accurate box prompts for mask propagation.