acceptodds
Withdrawn from ICLR 2027

Video Reversal Curse: Diagnosing and Aligning Temporal Reasoning in Diffusion VLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Large Language Models can exhibit the Reversal Curse: after learning that event A precedes event B, they may fail to infer that event B follows event A. We show that video-language models exhibit the analogous Video Reversal Curse, combining directional bias with weak reliance on temporal order: they answer cause-to-effect questions more reliably than paired effect-to-cause questions and may ignore frame reversals or shuffles. Although dVLMs provide bidirectional attention and iterative denoising, these capabilities alone do not ensure symmetric temporal reasoning. We propose ReVID, which combines Temporal-Symmetric GRPO, training on paired forward/backward questions with rewards for balanced accuracy and temporal reliance, and Temporal Visual Contrastive Decoding, which contrasts original and shuffled-video logits at inference. We introduce RevVideo, a 3,334-question paired benchmark, and the Temporal Symmetry Score. ReVID improves RevVideo accuracy by 8.37 points over its backbone, reduces the forward/backward gap to 1.78 points, and achieves the best TSS of 40.56, while improving dVLM performance on TempCompass and TVBench.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion-based MLLMs, Video Understanding, Video Reversal Curse

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_9br3OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies the Video Reversal Curse, where VLMs struggle with backward temporal reasoning and often ignore frame order. It proposes ReVID, which combines temporally symmetric reinforcement learning and contrastive decoding for diffusion VLMs, along with a new benchmark called RevVideo.

Critical strengths
  1. The paper demonstrates the effect of the Video Reversal Curse and proposes ReVID, which combines Temporal-Symmetric GRPO and Temporal Visual Contrastive Decoding to alleviate this effect.
  2. The paper introduces RevVideo benchmark, which provides paired forward/backward questions and a Temporal Symmetry Score (TSS) to evaluate bidirectional temporal reasoning.
Critical weaknesses
  1. Insufficient evidence for the advantage of diffusion VLMs. The proposed SFT, temporal rewards, and contrastive decoding can also be applied to autoregressive VLMs. Although Appendix G evaluates TVCD on an AR model, a fair comparison using the complete ReVID pipeline is missing. It remains unclear whether the improvements mainly come from the diffusion architecture or the proposed training and decoding strategies.
  2. Missing strong baselines and fair comparisons. Relevant temporal reasoning methods, such as Video-R1, Time-R1, and SEASON, are discussed but not directly compared in the experiments. Including these baselines under comparable settings (e.g., trained with the same data) would better demonstrate the effectiveness of ReVID.
Questions for authors
  1. Questionable reward design (Eqs. 3–5). The Temporal Reliance Reward can be positive even when ordered and shuffled videos achieve the same accuracy ((\mu=0.8)). Moreover, the group-level symmetry reward may cancel out under standard GRPO advantage normalization. It is unclear how these rewards effectively encourage temporal reasoning.
  2. Insufficient analysis of the Video Reversal Curse. The discussion is mostly high-level and lacks quantitative evidence to support the proposed explanations. In particular, it remains unclear how severely this problem affects dVLMs and whether their architectural properties actually provide an advantage over AR-VLMs.
  3. Insufficient details on benchmark construction. The RevVideo construction pipeline lacks important implementation details, particularly regarding how temporal reversibility filtering is performed.
  4. SFT + TS-GRPO without TVCD performs worse than the original backbone (47.65% vs. 50.57% in Table 8), raising concerns about the effectiveness of the proposed training strategy independently of TVCD.
  5. Unclear temporal symmetry in qualitative examples. Some forward/backward QA pairs appear not to correspond to exactly symmetric temporal transitions or supporting frames. The authors should clarify how temporal symmetry is defined and verified in these examples.
Reviewer_myfHOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper investigates the video reversal curse problem for dVLM. Specifically, the model suffers from Static Semantic Inertia (i.e.,. model reliance on static object cues) and Unidirectional Cognitive Bias (i.e., answer cause-to-effect questions more reliably than paired effect-to-cause questions). The paper assumes auto-regressive LMs are the root and diffusion-based VLMs (dVLM) might be a principled cure.

The authors introduces a targeted benchmark and training method to improve the performance of dVLMs on this aspect. The proposed ReVID employs an GRPO-variant to trains on paired questions with symmetry rewards, combining inference time modification TVCD, which contrasts original and shuffled-video logits during decoding under an adaptive plausibility constraint.

Dataset-side, the constructed RevVideo are sourced from public temporal reasoning datasets, with an LLM assisted pipeline to curate paired forward/backward questions to propose the reversal curse challenge.

Experimental results show the proposed GRPO method improves dVLM's performance on the RevVideo Bench, as well as general temporal reasoning benchmarks such as TVBench and TempCompass.

Critical strengths
  • The temporal reversal curse is an interesting problem to explore, not specific to dVLMs.
  • The curated dataset might be valuable for the community.
Critical weaknesses
  • The method part yields limited novelty and the improvements might due to the similar task format. The author is encouraged to conduct a in-depth analysis of the performance again for decomposing the gain from the distribution and the symmertry rewards design.
  • The assumption of the dVLM might be more principled cure for the curse problem is not well consolidated. As recent models have shown decent performance on those temporal reasoning tasks such as Qwen-3.5. Comparing dVLMs and AR-VLMs with a more controlled settings are nice to have for drawing a solid conclusion.
  • The evaluated models are rather out-of-dated, such as Qwen-VL-Chat, leading the results less convincing as recent ARM might not yield similar issues anymore. Besides, including more benchmarks such as TOMATO [1] and T3 [2] (text-only) can provide more diagnostic signals.
  • The paper has signficant format issues such as wrong citation format (citep instead of cite), empty lines after formulas, rendering poor readability.

Refs

[1] TOMATO: Assessing Visual Temporal Reasoning Capabilities in Multimodal Foundation Models, ICLR 2025

[2] Temporal Reasoning Transfer from Text to Video, ICLR 2025

Questions for authors
  • What's the model ReVID (Ours) represent in Table 3 and Table 4?
  • How many human annotators were employed and are they well compensated?
Reviewer_vUbROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies directional asymmetry in video question answering and introduces RevVideo, a benchmark of paired forward and backward temporal questions. It proposes ReVID, combining supervised fine-tuning, temporal-symmetric reinforcement learning, and temporal visual contrastive decoding for diffusion vision-language models. Experiments report improved accuracy and joint correctness on RevVideo across two diffusion backbones, alongside gains on temporal subsets of TempCompass and TVBench.

Critical strengths
  1. RevVideo pairs forward and backward questions about the same video, enabling evaluation of joint correctness and revealing directional failures that aggregate question-answering accuracy can obscure.

  2. The study evaluates two diffusion backbones and additional temporal benchmark subsets, with full component-combination ablations that provide useful evidence about the interaction between training and inference-time correction.

Critical weaknesses
  1. The learning signal from the proposed rewards is unclear. Section 4.2 defines the symmetry reward using group-level success rates, but does not specify advantage normalization. If this reward is shared across rollouts and advantages are centered within each group, it cancels exactly. Moreover, Eq. (4) rewards equal positive success rates on ordered and shuffled videos, despite claiming to require temporal dependence.

  2. The evaluation statistics contain unresolved inconsistencies. Weighting the full ReVID category accuracies in Table 2 by the counts in Table 1 yields 57.46%, whereas 58.94% is reported. Table 12 also contains inconsistent decompositions: the SFT + TS-GRPO row implies 47.90% overall accuracy but reports 47.65%. These discrepancies require clarified aggregation rules and corrected statistics to establish the reported gains reliably.

  3. The architectural explanation is not adequately isolated. Table 15 compares AR + TVCD against ReVID with additional SFT and TS-GRPO, so the remaining gap cannot be attributed specifically to the diffusion architecture. The absence of single-frame or text-only controls also limits the claim that the improvements reflect stronger reliance on temporal evidence. These claims require more comparable controls or qualification.

  4. The added methodological contribution needs stronger substantiation. The temporal success-rate criterion, trajectory shrinkage, and contrastive-logit formulation build on Video-R1 [1], TraceRL [2], and VCD [3], respectively. Such a combination can still provide valuable findings, but its contribution depends on demonstrating the added value of paired temporal alignment. The unresolved reward formulation and confounded architectural comparison currently limit that assessment.

[1] Kaituo Feng et al. “Video-R1: Reinforcing Video Reasoning in MLLMs.” NeurIPS, 2025. https://arxiv.org/abs/2503.21776

[2] Yinjie Wang et al. “Revolutionizing Reinforcement Learning Framework for Diffusion Large Language Models.” ICLR, 2026. https://openreview.net/forum?id=KNAyc9DMe3

[3] Sicong Leng et al. “Mitigating Object Hallucinations in Large Vision-Language Models through Visual Contrastive Decoding.” CVPR, 2024. https://arxiv.org/abs/2311.16922

Questions for authors
  1. How is the advantage in Eq. (5) standardized in the actual implementation: within each question-level rollout group, across the paired forward/backward groups, or over a larger batch? Could you provide a short derivation or example showing how rubr_{ub} changes the resulting advantages?

  2. Could you clarify the intended behavior of the temporal reliance reward in Eq. (4)? In particular, why is a sample rewarded when po=ps>0p_o = p_s > 0, or even in some cases when po<psp_o < p_s, despite the claim that the reward requires dependence on correct chronological order?

  3. Could you provide the exact evaluation counts and aggregation procedure used for Tables 2, 8, and 12, and reconcile the reported Overall accuracies with the category-level and TSS-decomposition statistics?

  4. What evidence isolates the benefit of the diffusion architecture itself from the effects of paired supervision, TS-GRPO, and TVCD? A comparison using an AR backbone with comparable post-training would help determine whether the claimed architectural advantage is specific to dVLMs.

  5. Since several ingredients build on prior temporal reward shaping, trajectory optimization, and contrastive decoding, what do the authors consider the main methodological contribution beyond combining these components, and which experiment most directly isolates the benefit of the proposed forward/backward symmetry mechanism?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.