- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes VideoMend, a framework that improves video reasoning by explicitly modeling changes between sampled frames through Ref-Delta Chain-of-Thought (CoT) and reinforcing visually grounded reasoning with Delta-Mask GRPO. Experiments show consistent improvements across six video benchmarks, with particularly strong gains on temporal reasoning tasks.
Critical strengths
-
Intuitive approach. Ref-Delta CoT explicitly separates static scene information from inter-frame changes, providing a clear and interpretable way to address temporal discontinuity and single-frame bias in video reasoning.
-
Strong empirical validation. The paper evaluates six benchmarks and provides comprehensive ablation studies, demonstrating the effectiveness of its key components and improvements in temporal reasoning and visual grounding.
Critical weaknesses
-
Insufficient evidence supporting the central claim. While the authors claim that Ref-Delta CoT can mitigate single-frame bias by explicitly modeling inter-frame changes, it remains unclear whether the model actually relies on the generated Deltas to make predictions, rather than simply producing plausible temporal descriptions. Although Fig. 6 shows improved Delta visual correctness, this does not necessarily establish their causal contribution to the final predictions. More direct evidence, such as evaluating how removing or perturbing Delta descriptions affects the model's answers, would help substantiate the central claim.
-
Limited contribution of the proposed components. Tab. 2 suggests that most improvements come from the RL stage, while Ref-Delta CoT and data augmentation provide relatively modest improvements. Since the RL stage is built on GRPO, this raises the concern that the overall improvements may be primarily driven by reward design rather than the proposed reasoning structure or data augmentation. More comparisons are needed to disentangle the contributions of RL optimization, reward design, training data, and Ref-Delta CoT.
Questions for authors
- It is unclear how many questions truly require temporal information rather than static visual cues. I suggest testing with single-frame inputs and shuffled frames to verify whether the improvements actually come from Ref-Delta CoT. Further concern from Tab. 7 shows that ReverseFilm does not improve with VideoMend.
- Insufficient validation of the Ref-Delta CoT design. The ablation without Ref-Delta markers preserves the original reasoning content, making it difficult to isolate the benefit of the proposed reasoning structure. Could the authors compare Ref-Delta CoT against standard CoT and simpler frame-transition descriptions under the same training setup?
- Quality of the paper. Figs. 2 and 3 are low-resolution, and the style is quite different from the other figures, which lowers the quality of the paper.
- Some equations need clearer definitions, especially Eq. (5), where the timestamp reward is defined as a sum of three terms but is claimed to lie within [0, 1], while missing the details of , , and .
- The Delta-Mask reward encourages robustness when non-critical intervals are masked but does not explicitly enforce sensitivity when critical temporal evidence is removed. Could the authors explain how this design helps the model identify and rely on essential temporal information?
- Sec 3.3 states that the Ref-Delta annotations are generated using the video, question, and ground-truth answer. Although answer-leakage filtering is applied, could this conditioning encourage post-hoc reasoning that supports the known answer rather than independently identifying relevant temporal evidence? An ablation using answer-independent annotations would help clarify this issue.
2 comments
Q1.2: Why does ReverseFilm not improve? The current training objectives do not directly constrain forward/reverse playback discrimination, and gains in understanding actions and changes have not translated into higher ReverseFilm accuracy. Specifically, the antonymous-action augmentation in §3.2 distinguishes action semantics, Ref-Delta describes state changes, and DM-GRPO strengthens reliance on critical evidence. ReverseFilm instead asks which video is played backward. For example, a door changing from open to closed can depict either a real closing action or reversed footage of an opening action; correctly describing the state change alone does not establish whether the playback direction corresponds to a real process.
Q3: Figure quality. We will replace Figures 2 and 3 with vector graphics and standardize font sizes, line styles, and layout.
Q4: The range and components of Equation (5). We retain the sum of three components in the original Equation (5) and clarify that the total score ranges from zero to three. The components assess temporal order, interval overlap, and temporal coverage, each with a score between zero and one. The timestamp reward is zero if a format, parsing, interval-validity, or duration check fails. Specifically,
s_{\mathrm{cov},i}=\operatorname{clip}\left(\frac{C_i-C_{\min}}{C_{\max}-C_{\min}},0,1\right),\qquad C_i=\frac{|\bigcup_d[s_d,e_d]|}{T}.$$ The original **§3.4, lines 295–304** already specifies that monotonicity checks interval start times with a one-second tolerance, overlap checks whether adjacent intervals avoid overlaps longer than three seconds, and coverage uses the merged Delta-interval duration divided by the video duration. $C_{\min},C_{\max}$ are coverage thresholds; we will include their actual values in the revision. The total is weighted by $\lambda_{\mathrm{ts}}$ in the original **Equation (6)**. **Q5: Is sensitivity to critical evidence constrained?** **We have directly tested sensitivity to critical temporal evidence through matched masking and temporal-reversal controls.** Table A4 uses the same questions and matches the number of masked frames. VideoMend drops by **1.6 / 12.4 points** under non-critical / critical masking, versus **2.2 / 6.0 points** for the base model. Reverse consistency in Table A3 also increases from **14.2%** for the base model to **41.3%**. Together, these results support stability to non-essential content and use of critical temporal information when answering. The original **Equation (3), lines 288–289** already distinguishes the two masking cases: correctness under non-critical masking provides a bonus to correct original-video responses, discouraging reliance on removable content. Critical masking sets only $r_{dm}=0$, retaining the base and timestamp rewards. A performance drop after removing critical evidence is expected, so this component remains reward-neutral. **Table A4: Interval masking.** The same 500 temporal questions are evaluated with 6/32 frames masked in each condition, preserving temporal positions. The evidential role of the intervals is checked manually. Cells report correct / total (accuracy). This subset is reported separately from the full benchmarks; critical masking is not assumed to make every question entirely unanswerable. | Model | Original input | Non-critical masking | Critical masking | |---|---:|---:|---:| | Qwen2.5-VL-7B | 350/500 (70.0%) | 339/500 (67.8%) | 320/500 (64.0%) | | SFT-only | 370/500 (74.0%) | 359/500 (71.8%) | 326/500 (65.2%) | | VideoMend | 392/500 (78.4%) | 384/500 (76.8%) | 330/500 (66.0%) | **Q6: Does answer-guided annotation encourage post-hoc reasoning?** **Answer-independent annotation retains most of the gains: 57.2 versus 57.5 with answer-guided annotation (Table A5).** This control fixes the teacher, 11,906 video/QA examples, and SFT setup, changing only answer access during annotation. It complements the original rule-based and judge-based leakage filters (**Appendix B.3**), audit of References, Deltas, timestamps, and summaries (**986/1,000†** correct trajectories; **Appendix C.3**), and prompts without answers during RL and evaluation (**Appendix B.2.2**). **Table A5: Annotation controls with matched sample counts, single runs.** Avg. denotes mean accuracy across the six benchmarks after SFT. The two matched groups use exactly the same video/QA examples. The original full-data row is a reference only; † denotes submitted results. | Annotation setting | Samples | Avg. | | --- | ---: | ---: | | Answer-guided, original full-data setting | 13,248 | 57.6† | | Answer-guided, matched subset | 11,906 | 57.5 | | Without answers, matched subset | 11,906 | 57.2 |
We appreciate your careful review and positive assessment of the Ref-Delta design and evaluation. We address each point using the original manuscript and additional controls.
W1: Does the model use its generated Deltas? Yes. Delta interventions support the model's use of their semantic content when answering. The original §4.4, Figure 6, and Appendix C.4 already evaluate visual grounding without access to answers, showing an increase in correctness from 63.4% to 92.2%†. To test answer dependence, we hold the video, question, and References fixed, edit only the Deltas, and regenerate the summary and answer (Table A1). Reversing direction/order reduces TempCompass accuracy by 13.2 points, whereas paraphrasing with unchanged meaning reduces it by only 0.4 points.
Table A1: Delta interventions, answer-change rate / accuracy (%). Change rates are measured against the unedited answer to the same question. All conditions use the same continuation and decoding settings; the Reference control matches the amount of edited text. † Submitted results.
W2 / Q2: Are the gains mainly from RL, and is the structure better than simpler descriptions? Both SFT and RL contribute, and Ref-Delta still outperforms free-form CoT under the same vanilla GRPO training. The original §4.3, Table 2 reports the progression 54.4 → 57.6 → 61.0, with gains of 3.2 and 3.4 points from SFT and RL, respectively. The marker ablation deliberately preserves the reasoning content to isolate explicit state/change labels; retaining most of the gains is consistent with retaining the descriptions of changes. Table A2 further compares complete representations with the teacher, video/QA data, and training setup held fixed: under vanilla GRPO, Ref-Delta achieves 58.7 versus 57.4 for free-form CoT, a gain of 1.3 points.
Table A2: Trajectory-format controls, 32 frames. Avg. denotes mean accuracy across the six benchmarks. † Submitted results and their six-benchmark means. DM-GRPO reports the mean ± sample standard deviation across three seeds; other added results are from single runs. The original marker and augmentation ablations report TempCompass / six-benchmark means of 74.8 / 59.6† and 74.2 / 60.0†, respectively.
Q1.1: Do single-frame and shuffled-frame controls support temporal dependence? Shuffling and semantic-reversal controls jointly support reliance on temporal relations; single-frame performance provides a static-input reference. The original TVBench Table 8 already reports gains on Moving Direction, 30.6 → 41.8†, and Action Antonym, 72.1 → 78.2†. In the added controls, VideoMend achieves 41.3% reverse consistency, versus 14.2% for the base model (Table A3). Reversal uses a shared subset of semantically reversible questions, excludes ambiguous answer mappings, and does not select examples based on whether the original prediction was correct. Consistency is defined as
=\frac{1}{|S_{\mathrm{rev}}|} \sum_{i\in S_{\mathrm{rev}}}\mathbf1[\hat y_i^{\mathrm{rev}}=m(y_i)].$$ Here, $m(y_i)$ is the ground-truth answer after reversal. All models use the same frame sets, shuffled permutations, and treatment of temporal positions. This measures correct semantic responses, not arbitrary answer changes. We interpret the shuffling drops together with reversal and interval interventions. **Table A3: Temporal input controls.** Shuffled reports the accuracy drop; single frame reports accuracy using the middle frame. Both are listed as TempCompass / MVBench. | Model | Reverse consistency | Shuffled drop | Single-frame accuracy | |---|---:|---:|---:| | Qwen2.5-VL-7B | 14.2% | 1.9 / 1.2 | 61.8 / 47.5 | | SFT-only | 27.5% | 3.6 / 2.0 | 62.3 / 48.6 | | VideoMend | 41.3% | 5.8 / 3.1 | 63.0 / 49.8 |