acceptodds
Withdrawn from ICLR 2027

VideoMend: Mending Temporal Continuity in Video Reasoning

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Video reasoning requires models to capture fine-grained temporal dynamics, includ-ing action directionality, motion dynamics, and state transitions. Yet, constrained by computational costs, many Video Large Language Models resort to sparse frame sampling, which fragments the temporal stream into isolated visual snapshots and encourages single-frame bias: models derive answers from salient objects in individual frames rather than from the temporal evolution of events. We propose VIDEOMEND, a framework that explicitly reasons over inter-frame changes to mend the temporal discontinuity introduced by frame sampling. Its core is Ref-Delta Chain-of-Thought (Ref-Delta CoT), a structured reasoning format organized around two complementary signals: Reference anchors the static scene semantics at temporal anchor points, while Delta explicitly captures the changes between adjacent frames, encouraging the model to articulate temporal transitions rather than derive answers from individual frames in isolation. To train it, we adopt a two-stage pipeline. The SFT stage provides cold-start initialization with data specifically targeting temporal direction and speed perception, including antony-mous semantic pairs that supervise temporal direction and symmetric constructions that decouple semantic complexity from speed. The RL stage further enforces the visual faithfulness of Deltas via Delta-Mask GRPO (DM-GRPO), which constructs counterfactual inputs via delta frame masking and uses a judge model to provide an explicit training signal for the visual grounding of Deltas, ensuring inter-frame changes are grounded in visual observation rather than language priors. Experiments on six benchmarks show consistent gains over strong baselines, most notably on TempCompass, reflecting VIDEOMEND’s capacity to capture temporal continuity.

Primary area:
foundation or frontier models, including LLMs
Keywords:
VideoLLM, Video Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2GTWOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes VideoMend, a framework that improves video reasoning by explicitly modeling changes between sampled frames through Ref-Delta Chain-of-Thought (CoT) and reinforcing visually grounded reasoning with Delta-Mask GRPO. Experiments show consistent improvements across six video benchmarks, with particularly strong gains on temporal reasoning tasks.

Critical strengths
  1. Intuitive approach. Ref-Delta CoT explicitly separates static scene information from inter-frame changes, providing a clear and interpretable way to address temporal discontinuity and single-frame bias in video reasoning.

  2. Strong empirical validation. The paper evaluates six benchmarks and provides comprehensive ablation studies, demonstrating the effectiveness of its key components and improvements in temporal reasoning and visual grounding.

Critical weaknesses
  1. Insufficient evidence supporting the central claim. While the authors claim that Ref-Delta CoT can mitigate single-frame bias by explicitly modeling inter-frame changes, it remains unclear whether the model actually relies on the generated Deltas to make predictions, rather than simply producing plausible temporal descriptions. Although Fig. 6 shows improved Delta visual correctness, this does not necessarily establish their causal contribution to the final predictions. More direct evidence, such as evaluating how removing or perturbing Delta descriptions affects the model's answers, would help substantiate the central claim.

  2. Limited contribution of the proposed components. Tab. 2 suggests that most improvements come from the RL stage, while Ref-Delta CoT and data augmentation provide relatively modest improvements. Since the RL stage is built on GRPO, this raises the concern that the overall improvements may be primarily driven by reward design rather than the proposed reasoning structure or data augmentation. More comparisons are needed to disentangle the contributions of RL optimization, reward design, training data, and Ref-Delta CoT.

Questions for authors
  1. It is unclear how many questions truly require temporal information rather than static visual cues. I suggest testing with single-frame inputs and shuffled frames to verify whether the improvements actually come from Ref-Delta CoT. Further concern from Tab. 7 shows that ReverseFilm does not improve with VideoMend.
  2. Insufficient validation of the Ref-Delta CoT design. The ablation without Ref-Delta markers preserves the original reasoning content, making it difficult to isolate the benefit of the proposed reasoning structure. Could the authors compare Ref-Delta CoT against standard CoT and simpler frame-transition descriptions under the same training setup?
  3. Quality of the paper. Figs. 2 and 3 are low-resolution, and the style is quite different from the other figures, which lowers the quality of the paper.
  4. Some equations need clearer definitions, especially Eq. (5), where the timestamp reward is defined as a sum of three terms but is claimed to lie within [0, 1], while missing the details of smono\mathcal{s_{mono}} , soverlap\mathcal{s_{overlap}}, and scov\mathcal{s_{cov}}.
  5. The Delta-Mask reward encourages robustness when non-critical intervals are masked but does not explicitly enforce sensitivity when critical temporal evidence is removed. Could the authors explain how this design helps the model identify and rely on essential temporal information?
  6. Sec 3.3 states that the Ref-Delta annotations are generated using the video, question, and ground-truth answer. Although answer-leakage filtering is applied, could this conditioning encourage post-hoc reasoning that supports the known answer rather than independently identifying relevant temporal evidence? An ablation using answer-independent annotations would help clarify this issue.
2 comments

We appreciate your careful review and positive assessment of the Ref-Delta design and evaluation. We address each point using the original manuscript and additional controls.

W1: Does the model use its generated Deltas? Yes. Delta interventions support the model's use of their semantic content when answering. The original §4.4, Figure 6, and Appendix C.4 already evaluate visual grounding without access to answers, showing an increase in correctness from 63.4% to 92.2%†. To test answer dependence, we hold the video, question, and References fixed, edit only the Deltas, and regenerate the summary and answer (Table A1). Reversing direction/order reduces TempCompass accuracy by 13.2 points, whereas paraphrasing with unchanged meaning reduces it by only 0.4 points.

Table A1: Delta interventions, answer-change rate / accuracy (%). Change rates are measured against the unedited answer to the same question. All conditions use the same continuation and decoding settings; the Reference control matches the amount of edited text. † Submitted results.

InterventionTempCompassMVBench
None– / 76.7†– / 67.9†
Remove Deltas17.6% / 71.212.3% / 64.1
Reverse direction/order24.9% / 63.59.8% / 63.2
Replace with Deltas from another video38.4% / 58.631.7% / 52.4
Paraphrase Deltas without changing meaning3.6% / 76.32.9% / 67.6
Paraphrase a matched amount of Reference text3.3% / 76.43.0% / 67.5

W2 / Q2: Are the gains mainly from RL, and is the structure better than simpler descriptions? Both SFT and RL contribute, and Ref-Delta still outperforms free-form CoT under the same vanilla GRPO training. The original §4.3, Table 2 reports the progression 54.4 → 57.6 → 61.0, with gains of 3.2 and 3.4 points from SFT and RL, respectively. The marker ablation deliberately preserves the reasoning content to isolate explicit state/change labels; retaining most of the gains is consistent with retaining the descriptions of changes. Table A2 further compares complete representations with the teacher, video/QA data, and training setup held fixed: under vanilla GRPO, Ref-Delta achieves 58.7 versus 57.4 for free-form CoT, a gain of 1.3 points.

Table A2: Trajectory-format controls, 32 frames. Avg. denotes mean accuracy across the six benchmarks. † Submitted results and their six-benchmark means. DM-GRPO reports the mean ± sample standard deviation across three seeds; other added results are from single runs. The original marker and augmentation ablations report TempCompass / six-benchmark means of 74.8 / 59.6† and 74.2 / 60.0†, respectively.

Trajectory formatTrainingAvg.
None, Qwen2.5-VL-7B–54.4†
Free-form CoTSFT56.2
Reference onlySFT56.6
Delta onlySFT57.0
Ref-Delta CoTSFT57.6†
Free-form CoTSFT + vanilla GRPO57.4
Ref-Delta CoTSFT + vanilla GRPO58.7
Ref-Delta CoTSFT + DM-GRPO60.8 ± 0.3

Q1.1: Do single-frame and shuffled-frame controls support temporal dependence? Shuffling and semantic-reversal controls jointly support reliance on temporal relations; single-frame performance provides a static-input reference. The original TVBench Table 8 already reports gains on Moving Direction, 30.6 → 41.8†, and Action Antonym, 72.1 → 78.2†. In the added controls, VideoMend achieves 41.3% reverse consistency, versus 14.2% for the base model (Table A3). Reversal uses a shared subset of semantically reversible questions, excludes ambiguous answer mappings, and does not select examples based on whether the original prediction was correct. Consistency is defined as

=\frac{1}{|S_{\mathrm{rev}}|} \sum_{i\in S_{\mathrm{rev}}}\mathbf1[\hat y_i^{\mathrm{rev}}=m(y_i)].$$ Here, $m(y_i)$ is the ground-truth answer after reversal. All models use the same frame sets, shuffled permutations, and treatment of temporal positions. This measures correct semantic responses, not arbitrary answer changes. We interpret the shuffling drops together with reversal and interval interventions. **Table A3: Temporal input controls.** Shuffled reports the accuracy drop; single frame reports accuracy using the middle frame. Both are listed as TempCompass / MVBench. | Model | Reverse consistency | Shuffled drop | Single-frame accuracy | |---|---:|---:|---:| | Qwen2.5-VL-7B | 14.2% | 1.9 / 1.2 | 61.8 / 47.5 | | SFT-only | 27.5% | 3.6 / 2.0 | 62.3 / 48.6 | | VideoMend | 41.3% | 5.8 / 3.1 | 63.0 / 49.8 |

Q1.2: Why does ReverseFilm not improve? The current training objectives do not directly constrain forward/reverse playback discrimination, and gains in understanding actions and changes have not translated into higher ReverseFilm accuracy. Specifically, the antonymous-action augmentation in §3.2 distinguishes action semantics, Ref-Delta describes state changes, and DM-GRPO strengthens reliance on critical evidence. ReverseFilm instead asks which video is played backward. For example, a door changing from open to closed can depict either a real closing action or reversed footage of an opening action; correctly describing the state change alone does not establish whether the playback direction corresponds to a real process.

Q3: Figure quality. We will replace Figures 2 and 3 with vector graphics and standardize font sizes, line styles, and layout.

Q4: The range and components of Equation (5). We retain the sum of three components in the original Equation (5) and clarify that the total score ranges from zero to three. The components assess temporal order, interval overlap, and temporal coverage, each with a score between zero and one. The timestamp reward is zero if a format, parsing, interval-validity, or duration check fails. Specifically,

s_{\mathrm{cov},i}=\operatorname{clip}\left(\frac{C_i-C_{\min}}{C_{\max}-C_{\min}},0,1\right),\qquad C_i=\frac{|\bigcup_d[s_d,e_d]|}{T}.$$ The original **§3.4, lines 295–304** already specifies that monotonicity checks interval start times with a one-second tolerance, overlap checks whether adjacent intervals avoid overlaps longer than three seconds, and coverage uses the merged Delta-interval duration divided by the video duration. $C_{\min},C_{\max}$ are coverage thresholds; we will include their actual values in the revision. The total is weighted by $\lambda_{\mathrm{ts}}$ in the original **Equation (6)**. **Q5: Is sensitivity to critical evidence constrained?** **We have directly tested sensitivity to critical temporal evidence through matched masking and temporal-reversal controls.** Table A4 uses the same questions and matches the number of masked frames. VideoMend drops by **1.6 / 12.4 points** under non-critical / critical masking, versus **2.2 / 6.0 points** for the base model. Reverse consistency in Table A3 also increases from **14.2%** for the base model to **41.3%**. Together, these results support stability to non-essential content and use of critical temporal information when answering. The original **Equation (3), lines 288–289** already distinguishes the two masking cases: correctness under non-critical masking provides a bonus to correct original-video responses, discouraging reliance on removable content. Critical masking sets only $r_{dm}=0$, retaining the base and timestamp rewards. A performance drop after removing critical evidence is expected, so this component remains reward-neutral. **Table A4: Interval masking.** The same 500 temporal questions are evaluated with 6/32 frames masked in each condition, preserving temporal positions. The evidential role of the intervals is checked manually. Cells report correct / total (accuracy). This subset is reported separately from the full benchmarks; critical masking is not assumed to make every question entirely unanswerable. | Model | Original input | Non-critical masking | Critical masking | |---|---:|---:|---:| | Qwen2.5-VL-7B | 350/500 (70.0%) | 339/500 (67.8%) | 320/500 (64.0%) | | SFT-only | 370/500 (74.0%) | 359/500 (71.8%) | 326/500 (65.2%) | | VideoMend | 392/500 (78.4%) | 384/500 (76.8%) | 330/500 (66.0%) | **Q6: Does answer-guided annotation encourage post-hoc reasoning?** **Answer-independent annotation retains most of the gains: 57.2 versus 57.5 with answer-guided annotation (Table A5).** This control fixes the teacher, 11,906 video/QA examples, and SFT setup, changing only answer access during annotation. It complements the original rule-based and judge-based leakage filters (**Appendix B.3**), audit of References, Deltas, timestamps, and summaries (**986/1,000†** correct trajectories; **Appendix C.3**), and prompts without answers during RL and evaluation (**Appendix B.2.2**). **Table A5: Annotation controls with matched sample counts, single runs.** Avg. denotes mean accuracy across the six benchmarks after SFT. The two matched groups use exactly the same video/QA examples. The original full-data row is a reference only; † denotes submitted results. | Annotation setting | Samples | Avg. | | --- | ---: | ---: | | Answer-guided, original full-data setting | 13,248 | 57.6† | | Answer-guided, matched subset | 11,906 | 57.5 | | Without answers, matched subset | 11,906 | 57.2 |
Reviewer_ZZAGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

VideoMend targets single-frame bias in Video LLMs that sample few frames: the model answers from one salient frame instead of from how the video changes. It trains Qwen2.5-VL-7B to reason in a "Ref-Delta" format, where References describe states and Deltas describe changes. Training starts with SFT on 13.2K traces generated by Qwen3.5-122B, including augmented clips that target motion direction and speed. It then uses Delta-Mask GRPO, which gives correct answers a bonus based on how often the model still answers correctly when a non-critical Delta interval is masked out. VideoMend improves over the base model on six benchmarks (e.g., TempCompass from 72.6 to 76.7) and matches or beats recent 7B RL video models on four of them.

Critical strengths
  1. The results are strong and broad. VideoMend beats the base model on every benchmark and leads prior 7B RL models on four of six. It keeps a 2.2-2.9 point lead in mean accuracy over Video-R1 at matched budgets of 16, 32 and 64 frames (Fig. 5d).
  2. The paper directly checks whether the reasoning is faithful. Human-judged visual correctness of the Deltas rises from 63.4% to 92.2% after RL. Checking the reasoning trace itself, not just the final answer, is uncommon and useful.
Critical weaknesses
  1. DM-GRPO does not appear to do what the paper claims. The bonus (a constant times the accuracy on the masked input) is identical for every correct response to a prompt and is zero for critical intervals. It therefore cannot tell grounded answers from shortcut answers. If advantages are normalized by the group standard deviation (the paper does not say), the bonus mostly rescales the correctness reward and is largely cancelled; otherwise it only reweights prompts. The 1.0-3.2 point ablation gains are reported without seeds or variance. The full model also generates 12 rollouts per prompt (8 plus 4 masked ones used only to compute the bonus), and the compute used by "w/o r_dm" is not reported. The faithfulness evidence compares SFT-only with the full model rather than "w/o r_dm" with the full model. Since DM-GRPO is the main algorithmic contribution, fixing this needs a redesigned reward, not just more experiments.
  2. The comparisons do not appear to be run under a common protocol, and the source of the gains is unclear. The paper does not say where the baseline numbers in Table 1 come from, and they disagree with its own runs. Table 1 lists Video-R1 at 49.8 on VideoMMMU, but Fig. 5 shows 52.3 at 32 frames, above VideoMend. The base-model row appears to use the paper's 32-frame setting (its mean, 54.4, matches the 32-frame point in Fig. 5d). Under its official protocol (many more frames), Qwen2.5-VL-7B reports 65.1 on VideoMME and 69.6 on MVBench, matching or exceeding VideoMend's 65.1 and 67.9, so the headline gains of +8.5 and +8.9 depend on the evaluation setting. Several training sources (CLEVRER, STAR, Perception Test, SSv2, Charades) are also MVBench sources, and NextQA training data is used (1,200 samples) while the appendix says it evaluates on the NExT-QA test set (no VideoMend NExT-QA numbers actually appear), but no video-level deduplication is reported. The "w/o Ref-Delta markers" variant keeps most of the gain, which suggests that the explicit Ref/Delta tags contribute little and that much of the gain may come from the teacher-generated traces and RL. Finally, on AoTBench's binary forward/reverse tests, VideoMend stays at the base model's near-chance level (ReverseFilm 50.0, UCF101 51.6), although it improves on the other AoTBench subsets. This weakens the direction claim.
Questions for authors
  1. Are GRPO advantages normalized by the group's standard deviation? How can a bonus that is constant within a prompt favour grounded answers? Please report at least 3 seeds for the full model and for a compute-matched "w/o r_dm" variant, including the human Delta evaluation.
  2. Were the training videos deduplicated against the MVBench, TempCompass and NExT-QA test videos? Please report results with any overlapping items removed.
  3. Fig. 5 covers only Qwen2.5-VL-7B and Video-R1. Please evaluate the other RL baselines in Table 1 under the same 32-frame protocol, and explain the Video-R1 VideoMMMU mismatch (49.8 vs. 52.3). Please also train a baseline with free-form CoT and vanilla GRPO on the same teacher data.
  4. Why are the AoTBench ReverseFilm and UCF101 scores near chance and identical to the base model? Is the model predicting a single class?
2 comments

We appreciate your positive assessment of the experimental results and the evaluation of Delta visual correctness. We address each point using the original manuscript and additional controls.

W1 / Q1.1: Are advantages normalized by the group standard deviation? Yes. Advantages are normalized by the group standard deviation, and zero-variance groups receive zero advantages. Specifically, Ai=(Ri−R‾)/(std⁡(R)+ε)A_i=(R_i-\overline R)/(\operatorname{std}(R)+\varepsilon). Only original-video responses enter advantage computation and gradient updates. We will add this clarification alongside §3.4, Equation (8).

W1 / Q1.2: Is the shared bonus cancelled by normalization? The bonus is not cancelled in every training group; its effect depends on the composition of rewards within the group. Under Ri=(1+bq)ai+hiR_i=(1+b_q)a_i+h_i, correctness scaling cancels when the auxiliary reward hih_i is constant within the group, ignoring the numerical stabilizer. In our method, hih_i includes per-trajectory format and timestamp rewards (Equations (4)–(7)), so bqb_q can change the relative weight of correctness and these auxiliary terms. Groups with all-correct or all-incorrect answers have no resulting change in advantages, and correct trajectories for the same question share the bonus. Tables C1–C2 report the changes in normalized advantages and the training gains.

W1 / Q1.3: Does the bonus still have an effect after normalization? Yes. Retaining or removing rdmr_{dm} changes the normalized advantages in a subset of training groups (Table C1). Early in training, 30.4% of groups change, with an effect size of 7.8%; late in training, these values are 10.8% / 1.1%. Using the same reward records, we recompute A+A^+ and A−A^- with and without rdmr_{dm}. A group is counted as changed when max⁡i∣Ai+−Ai−∣>10−6\max_i|A_i^+-A_i^-| > 10^{-6}, and the proportion uses all training groups in the window as its denominator. Effect size is

{\sum_{i\in\mathcal C}|A_i^+|},$$ where $\mathcal C$ is the set of correct responses for that seed and window. If the denominator is zero, the statistic is set to zero. **Table C1: Effect of the bonus on advantages, mean ± sample standard deviation across three seeds.** Receiving a bonus requires $c=1$, nonzero masked-video accuracy, and at least one correct original-video response. | Training steps | Groups receiving a bonus | Groups with changed advantages | Effect size E | | --- | ---: | ---: | ---: | | 1–250 | 58.4 ± 0.6% | 30.4 ± 0.8% | 7.8 ± 0.5% | | 251–500 | 61.9 ± 0.4% | 16.8 ± 0.6% | 2.6 ± 0.3% | | 501–761 | 63.7 ± 0.3% | 10.8 ± 0.4% | 1.1 ± 0.2% | **W1 / Q1.4: Are there three matched seeds and human evaluations?** **We have completed matched training across three seeds and blind evaluation of 500 Deltas per seed.** Both conditions generate eight original-video and four masked-video responses. Only original-video responses contribute to gradient updates; the reward-removal control discards the masked outputs. In Table C2, DM-GRPO / matched reward removal achieve six-benchmark means of **60.8 ± 0.3 / 59.3 ± 0.3** and Delta visual correctness of **91.5 ± 0.7% / 83.9 ± 0.9%**. Every DM-GRPO seed outperforms every seed of the matched control on both metrics. **Table C2: Results by seed.** Each seed uses a blind evaluation of 500 Deltas. Standard deviations are sample standard deviations, computed before rounding. The twelve-gradient-rollout rows provide a reference with a larger update budget. † Originally reported run. | Variant and metric | Seed 1 | Seed 2 | Seed 3 | Mean ± standard deviation | |---|---:|---:|---:|---:| | DM-GRPO, six-benchmark mean | 61.0† | 60.9 | 60.5 | 60.8 ± 0.3 | | DM-GRPO, Delta correctness | 92.2%† | 90.8% | 91.4% | 91.5 ± 0.7% | | w/o $r_{dm}$, 12 gradient-bearing rollouts, six-benchmark mean | 59.1 | 59.8 | 59.6 | 59.5 ± 0.4 | | w/o $r_{dm}$, 12 gradient-bearing rollouts, Delta correctness | 84.6% | 83.0% | 85.2% | 84.3 ± 1.1% | | w/o $r_{dm}$, matched rollout counts, six-benchmark mean | 59.0 | 59.5 | 59.4 | 59.3 ± 0.3 | | w/o $r_{dm}$, matched rollout counts, TempCompass | 73.6 | 74.2 | 73.9 | 73.9 ± 0.3 | | w/o $r_{dm}$, matched rollout counts, Delta correctness | 83.0% (415/500) | 84.0% (420/500) | 84.8% (424/500) | 83.9 ± 0.9% | **W2 / Q2: Does video overlap affect the results?** **The source-video audit found no overlap between the training set and MVBench, TempCompass, or NExT-QA test: all three overlap counts are zero (Table C3).** The audit uses source-video IDs and perceptual hashes of sampled frames, and includes the source videos of augmented clips. **Table C3: Source-video overlap between training and evaluation sets.** | Benchmark | Overlapping source videos | | --- | ---: | | MVBench | 0 | | TempCompass | 0 | | NExT-QA, test | 0 |

W2: Missing NExT-QA results. We have added the NExT-QA test results: 82.1 for VideoMend and 79.4 for the base model. The original Appendix D.1.1 mistakenly contains the MECD table; its 68.1† belongs to MECD. The 1,200 NExT-QA training examples† are included in the source-video overlap audit above.

W2 / Q3.1: Sources of the main-table results and Δ values. Table 1 reports our evaluations, with Δ measured against Qwen2.5-VL-7B under the same pipeline.

W2 / Q3.2: Sources and settings of the table and figure results. Table 1 reports our Video-R1 VideoMMMU score of 49.8; re-evaluation under the same settings gives 49.9, a 0.1-point difference. Figure 5 cites the published 52.3. We maintain fair comparisons by using the original unified 32-frame pipeline for Tables 1 and C4, and will clarify sources and settings in the captions, marking published values separately.

W2: Do the official Qwen scores invalidate our gains? We report gains from evaluations using the same pipeline at a fixed frame budget; the official scores use a different setting with a higher budget. The original Figure 5 shows our evaluations and published results at 16/32/64 frames, with six-benchmark mean gains of 2.2–2.9 points† over Video-R1. The official Qwen scores of 65.1 on VideoMME and 69.6 on MVBench use a setting allowing up to 768 frames / 24,576 video tokens. Our Δ values use the 32-frame baseline evaluated through the same pipeline in Table 1. Additional evaluations following the original pipeline are in Table C4.

W2 / Q3.3: Are the other RL baselines re-evaluated under the same protocol? Yes. VideoMend retains the highest Avg. under the original 32-frame pipeline (Table C4). Sampling, resolution, subtitles, decoding, output limits, parsing, and data splits are matched, retaining model-required prompts. VideoMend scores 60.8 versus 60.2 for Video-R2 (+0.6 points). VideoMend uses one checkpoint for this comparison; 61.0† is the original single-run result, listed separately. Three-seed results appear in Tables B1 and C2.

Table C4: Evaluations and checks following the original 32-frame pipeline. Avg. denotes mean accuracy across the six benchmarks. † Submitted results or means recomputed from their six benchmark scores; other entries are additional evaluation results. VideoMend uses the single-checkpoint result in the last row for comparison.

ModelVideoMMMUTempCompassAvg.
Qwen2.5-VL-7B48.1†72.6†54.4†
Video-R149.973.2†57.7
VideoChat-R150.973.858.1
VideoChat-R1.549.673.558.9
VideoRFT48.473.157.5
MOSS-ChatV49.772.557.8
Video-R250.574.660.2
VideoMend, submitted result51.2†76.7†61.0†
VideoMend, additional evaluation50.976.460.8

W2 / Q3.4: Is there a free-form CoT plus vanilla GRPO baseline using the same teacher? Yes. Table A2 includes free-form CoT with the same teacher and vanilla GRPO; Ref-Delta improves the six-benchmark mean by 1.3 points. The comparison fixes the video/QA data, evidence, and training setup: Ref-Delta achieves 58.7, versus 57.4 for free-form CoT. The original marker ablation removes only explicit labels while retaining descriptions of changes, testing the labels' contribution. This control tests the complete representation.

W2 / Q4: Why are forward/reverse scores near chance, and are predictions nearly constant? We inspected the per-example logs for ReverseFilm and UCF101 and found no near-constant predictions. We believe the lack of gains is related to differences in task requirements: antonymous-action augmentation and Ref-Delta target action semantics and state changes, whereas forward/reverse discrimination requires judging natural temporal direction. A door changing from open to closed can depict real closing or reversed opening footage. This is a task-specific limitation: the original TVBench Table 8 still shows gains on Moving Direction (30.6 → 41.8†) and Action Antonym (72.1 → 78.2†).

Reviewer_QYBCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes VideoMend to improve temporal reasoning in video-language models. It introduces Ref-Delta CoT to explicitly describe changes between video frames and Delta-Mask GRPO (DM-GRPO) to encourage visually grounded reasoning. Experiments show consistent improvements across six benchmarks, particularly on TempCompass.

Critical strengths
  • Intuitive temporal reasoning approach. Ref-Delta CoT explicitly models inter-frame changes, addressing an important limitation of sparse-frame video understanding.

  • Solid empirical results. The method consistently improves performance across multiple benchmarks, with ablations supporting the effectiveness of its main components.

Critical weaknesses
  • Potential reward shortcuts and misalignment with temporal grounding. The Delta-Mask reward is based on answer correctness under frame masking rather than directly evaluating Delta faithfulness. Moreover, the same reward bonus is assigned to all correct responses regardless of their reasoning quality. The reward encourages robustness to removing non-critical frames but does not explicitly enforce sensitivity to critical temporal evidence. Consequently, the model may exploit answer-level shortcuts without genuinely improving temporal grounding, raising concerns about the effectiveness of the proposed reward mechanism.

  • Insufficient controlled comparisons to validate DM-GRPO. The shared Delta-Mask reward may primarily act as an adaptive scaling of the answer-correctness reward rather than providing a distinctive temporal reasoning signal. The paper lacks comparisons with standard GRPO under identical training conditions and with equivalently scaled correctness rewards. Furthermore, the reported improvement in Delta visual correctness compares SFT-only with the full method, without isolating the contribution of the Delta-Mask reward. These limitations make it difficult to attribute the improvements specifically to the proposed mechanism.

Questions for authors
  • How does the shared Delta-Mask reward distinguish faithful Delta reasoning from incorrect reasoning traces that produce the same correct answer?

  • Can the authors compare DM-GRPO against standard GRPO and GRPO with an equivalently scaled correctness reward under identical training settings?

  • Why does robustness to masking non-critical frames indicate improved temporal reasoning? Can the authors evaluate sensitivity to critical temporal evidence using temporal counterfactuals?

  • How does Delta visual correctness change when removing only the Delta-Mask reward while retaining the rest of the RL training pipeline?

1 comment

We appreciate your positive assessment of the temporal-modeling approach and results.

W1 / Q1: How does the shared reward distinguish trajectories that give the same correct answer? The shared bonus modifies the correctness reward at the question level. Correct trajectories for the same question share this bonus; it does not rank their faithfulness individually. The matched ablation shows that removing only rdmr_{dm} reduces Delta visual correctness by 7.5 points (Q4; Table C2). The original §3.4, Equations (2)–(7) can be written as

b_q=\alpha c_q\tilde p_q,\qquad h_i=\lambda_{\mathrm{fmt}}r_{\mathrm{fmt},i}+\lambda_{\mathrm{ts}}r_{\mathrm{ts},i}.$$ Here, $a_i$ denotes correctness, $h_i$ contains format/timestamp rewards, and $b_q$ comes from the current question's masking response. Masked responses only estimate $b_q$; original responses alone enter advantage computation and gradient updates (**p. 5, lines 237–239; p. 6, lines 313–316**). **W2 / Q2: Are there vanilla GRPO, equivalent-scaling, and reward-removal controls?** **Yes. Table B1 adds vanilla GRPO, fixed correctness scaling, shuffled masking signals, and matched-rollout controls.** The original **§4.3, Table 2, lines 436–441** already removes $r_{dm}$ and $r_{ts}$ separately; removing $r_{dm}$ lowers all six scores by **1.0–3.2 points†** (**61.0 → 59.0†** on average). All added controls share the same SFT checkpoint, training data, and **761 steps**. The matched reward-removal control discards masked responses and retains all other rewards. Fixed scaling uses training-data $\bar p$; the shuffled control permutes $\tilde p$ across questions with $c=1$. Both retain auxiliary rewards and award the bonus only to correct responses. **Table B1: Reward controls.** Avg. denotes mean accuracy across the six benchmarks. Rollout counts (original/masked) are 8/4 for DM-GRPO, matched reward removal, and shuffled signals; 12/0 for the larger gradient-update budget; and 8/0 for other added controls. Only original responses update the policy; matching covers generation counts and update structure. DM-GRPO and both added reward-removal controls use three seeds; other added controls use single runs. Vanilla GRPO removes $r_{dm}$ and the timestamp reward. † Submitted results. | Method | TempCompass | Avg. | | --- | ---: | ---: | | SFT-only | 73.3† | 57.6† | | Original manuscript w/o $r_{dm}$ | 73.5† | 59.0† | | Vanilla GRPO | 73.4 | 58.7 | | w/o $r_{dm}$, larger gradient-update budget | 73.9 | 59.5 ± 0.4 | | w/o $r_{dm}$, matched rollout counts | 73.9 ± 0.3 | 59.3 ± 0.3 | | Fixed correctness coefficient | 74.3 | 59.9 | | Shuffled $\tilde p$ | 74.9 | 60.1 | | DM-GRPO | 76.3 | 60.8 ± 0.3 | | DM-GRPO-R, exploratory single run | 76.9 | 61.1 | DM-GRPO exceeds matched reward removal by **1.5 points** (**60.8 ± 0.3 versus 59.3 ± 0.3**) and also exceeds fixed scaling (**59.9**) and shuffled signals (**60.1**), supporting a masking signal tied to the current question. The twelve-gradient-rollout control separately tests the effect of increasing the update budget. **W1 / Q3: Why does robustness to non-critical masking support temporal reasoning?** **We have directly tested reliance on critical temporal evidence through matched masking and temporal-reversal controls.** Table A4 uses the same questions and matches the number of masked frames. VideoMend drops by **1.6 / 12.4 points** under non-critical / critical masking, versus **2.2 / 6.0 points** for the base model. Reverse consistency in Table A3 increases from **14.2%** for the base model to **41.3%**. Together, these results support stability to non-essential content and use of critical temporal information when answering. The non-critical masking signal discourages reliance on removable segments. The original **Appendix C.2, Table 4** reports **96.75%†** agreement with human judgments across **400 critical and 400 non-critical** patterns, supporting the reliability of the masking labels. **W2 / Q4: How does Delta correctness change when only this reward is removed?** **Removing only $r_{dm}$ reduces Delta visual correctness from 91.5 ± 0.7% to 83.9 ± 0.9%, a difference of 7.5 points between the three-seed means, computed before rounding.** This control retains all other rewards and matches the update structure. Evaluation follows **Appendix C.4**, with 500 Deltas per seed assessed by annotators who are blinded to model and seed identity and do not see answers. Correct counts for reward removal are **415, 420, and 424** (Table C2). **W1, additional control: Per-response rewards.** We add DM-GRPO-R, a variant with per-response rewards. With the text prefix held fixed, it measures the change in each Delta's mean token log-likelihood when its referenced frames are masked, producing a signal for each Delta. An exploratory single run achieves a six-benchmark mean of **61.1** (Table B1).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.