- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper formulates spatiotemporal sensitivity as a counterfactual relational learning problem and proposes Counterfactual Relational Policy Optimization (CRPO), a dual-branch reinforcement learning framework. CRPO uses data augmentation to construct relational supervision, training models to answer both original video questions and auxiliary relation questions. This aims to improve sensitivity to spatiotemporal changes while preserving general video understanding. The paper also introduces DyBench, a paired diagnostic benchmark, and evaluates CRPO under controlled and out-of-distribution settings.
Critical strengths
-
The paper addresses Video LLMs’ limited sensitivity to temporal order and motion dynamics by framing spatiotemporal sensitivity as a counterfactual relational learning problem.
-
It proposes Counterfactual Relational Policy Optimization (CRPO), a dual-branch RL framework that uses data augmentation to jointly train models on original video questions and auxiliary relation questions, improving spatiotemporal sensitivity while preserving general video understanding.
-
It introduces DyBench, a paired diagnostic benchmark. Controlled and out-of-distribution evaluations show that auxiliary tasks involving reversible dynamics, motion direction, and event order consistently improve spatiotemporal sensitivity in both Qwen3-VL-4B and Qwen3-VL-8B.
Critical weaknesses
-
For dynamic questions, the counterfactual branch reward gives full credit to any response that differs from the original factual label. Rewarding any deviation does not encourage the policy to model the specific counterfactual physics or semantics—it simply rewards departing from the factual label. The authors argue that factual supervision grounds the policy, but might rewarding arbitrary alternative predictions risk encouraging unprincipled hallucinations or random drift under visual perturbations?
-
Could you clarify the train–test mismatch: the model is trained with a fifth option ("None of the above") but evaluated on standard benchmarks without it?
-
The paper claims that prior perturbation-based RL methods such as T-GRPO and ArrowRL suffer from "reward cancellation" under group-wise advantage normalization. In ArrowRL, however, the reverse reward is rollout-dependent rather than constant across the group, so its auxiliary signal does not cancel out under normalization.
-
Although performance on spatiotemporal benchmarks improves, the model performs on par with or worse than T-GRPO on general tasks such as VideoMME and MVBench. This raises the biggest concern: auxiliary tasks should help, or at least not hurt, performance on the main task.
Questions for authors
I would appreciate it if the authors could kindly address the concerns raised in the Critical Weaknesses section.
8 comments
Author Responses (2/4)
Training-time Null Option and evaluation choices (Weakness 2)
Thank you for raising the training–evaluation choice mismatch. We appreciate the opportunity to clarify why the option is introduced and to test whether it affects evaluation. The added Null Option provides a valid answer when a training transformation makes the original candidate set incomplete. For example, suppose a ball moves left and the choices are “left,” “up,” “down,” and “stationary.” A horizontal flip makes it move right, which none of those choices expresses. “None of the above” makes the transformed question answerable without assigning an incorrect original option.
During training, we transform videos while retaining their original questions and choices, so an additional option is needed to cover answers that the transformations move outside that choice set. It is included in the shared choices of both branches; the factual ground-truth label remains unchanged, and the added option is always incorrect on the factual branch.
Adding an extra “None of the above” option at benchmark evaluation is unnecessary, because the provided choices already include the annotated correct answer. This also holds for DyBench: each side of a pair has a verified correct answer among its provided choices. The added option therefore does not fill any gap in the candidate set. We retain each benchmark's original choices and evaluation protocol, including any none-like choices already present.
To directly address your concern, we also tested whether providing the additional option changes performance. We appended “None of the above” to every DyBench question and evaluated the same Qwen3-VL-4B base and CRPO checkpoints:
| Model | Answer choices | Acc (%) | O-Acc (%) | P-Acc (%) |
|---|---|---|---|---|
| Base | Original | 65.1 | 76.2 | 45.4 |
| Base | Original + Null | 65.0 | 76.2 | 45.2 |
| CRPO | Original | 70.3 | 79.5 | 54.8 |
| CRPO | Original + Null | 70.1 | 79.3 | 54.5 |
All score changes are at most 0.3 percentage points. CRPO's gains over Base are 5.2/3.3/9.4 points in Acc/O-Acc/P-Acc with the original choices and 5.1/3.1/9.3 with the added option. Thus, the additional option has a negligible effect in this evaluation, and CRPO's improvement remains nearly unchanged. The option addresses incomplete candidate sets created by training transformations; the observed DyBench gains do not depend on its omission at evaluation.
Manuscript changes: We clarified the training/default-evaluation option protocol and why an additional Null Option is unnecessary for benchmark evaluation in Appendix D.1. The additional-Null evaluation is reported in Appendix D.1, Table 12.
Author Responses (3/4)
ArrowRL and reward cancellation (Weakness 3)
We are grateful for this careful correction. You are right that ArrowRL's reverse reward is rollout-dependent and does not generally cancel under group-wise normalization. We apologize for the overly broad attribution in the original manuscript. Your observation has helped us correct the analysis and clarify the distinction between the methods.
ArrowRL Section 3.2 defines the reverse reward as
\qquad r_i=r_i^{\mathrm{fid}}+\alpha r_i^{\mathrm{rev}},$$ where the current policy generates the reverse-conditioned response $\tilde{o}$, and the reverse term is disabled when that response is sufficiently similar to the factual target. Because the reverse reward depends on each factual rollout, it can change relative rewards within the group. For a simple MCQ example, let the factual target be A, the reverse-conditioned prediction be B, and three factual rollouts answer A, B, and C. With the reverse term active, their rewards are $1 + \alpha$, $0$, and $\alpha$. The two incorrect answers receive different rewards, so the result is not a common rescaling of binary correctness and the auxiliary signal survives normalization. The cancellation derivation applies to the narrower case in which the shaped reward is a common positive rescaling of correctness, $R_i = (1 + C)c_i$, ignoring the format term and a negligible numerical stabilizer. T-GRPO's correctness-gated temporal bonus has this structure. ArrowRL can also exhibit cancellation in particular groups: if every factual answer differs from B, its reverse reward is constant and is removed by group centering. If the group contains only A and B, the total reward becomes $(1 + \alpha)c_i$, so the common factor cancels. **These are conditional cases, not a general property of ArrowRL.** We also removed the interpretation that ArrowRL's low, flat raw auxiliary-reward curve demonstrates normalization-induced cancellation. For our MCQ ArrowRL baseline, Similarity is exact answer matching, following ArrowRL's MCQ definition. The reference answer is generated by the current policy from the reversed video. No ground-truth answer for the reversed video or LLM similarity judge is needed. This implementation still produces rollout-dependent reverse rewards. The distinction we retain is about **which responses are used for training and how their rewards are normalized**. ArrowRL uses the reverse-video response to compute rewards for original-video responses. CRPO trains on responses to both the original and transformed videos. **CRPO uses a shared standard deviation for both branches.** Normalizing each branch separately would cancel its multiplicative CRR factor. The shared denominator preserves the branches' relative reward scales, allowing CRR to influence their relative contributions to optimization. We have revised Section 3.4 to explain both mechanisms without attributing generic reward cancellation to ArrowRL. **Manuscript changes:** We specified the common-factor condition in **Section 1**, narrowed the transition in **Section 3.3**, and revised **Section 3.4** to explain both the cancellation condition and its limited applicability to ArrowRL. **Appendix D.3** now gives ArrowRL's reward formula and MCQ examples with and without cancellation. We clarified our MCQ implementation in the ArrowRL description in **Appendix B.3** and removed the normalization-based interpretation of the raw-reward curves in **Section 5.3**. The experiments and reported baseline scores are unchanged.
Author Responses (1/4)
Arbitrary answer changes, hallucinations, and counterfactual correctness (Weakness 1)
Thank you for this thoughtful concern about counterfactual semantics and possible arbitrary answer changes. We agree that the dynamic behavioral reward alone does not specify the exact counterfactual physics or semantics. We appreciate your emphasis on validating actual correctness. CRPO combines this relational constraint with factual supervision through a shared policy and invariance supervision on static examples. Rather than infer correctness from the behavioral reward, we now measure it directly through a manual counterfactual-answer audit.
At 5%, 50%, and 95% of the total training steps, we randomly sampled 64 counterfactual video QA examples from one global batch across all devices. We manually annotated answers by inspecting the transformed videos and their questions and choices, annotating “None of the above” (the Null Option defined in the paper) as correct when none of the original answer choices was correct for the transformed video. We scored all eight counterfactual rollouts per example, giving 512 rollouts per stage. These labels are used only for analysis and never enter training.
The metrics have the following meanings:
All rollout-based metrics pool the corresponding counts across the sampled groups.
Dynamic reward precision rises from 71.7% to 89.9%, showing that approximately nine out of ten rewarded dynamic answers are exactly correct in the late-stage audit. Null precision reaches 89.8%, directly confirming that the added option is used appropriately in most audited selections.
Counterfactual accuracy increases from 39.3% to 63.7%, a gain of 24.4 percentage points. This includes all audited rollouts, including dynamic answers that fail to change. Together, the results show higher exact counterfactual correctness alongside increasingly reliable rewarded behavior.
Exact counterfactual answers are not provided by the existing training data, and obtaining them at scale would require additional annotation. CRPO derives a simple change-or-invariance signal from the transformation and the existing factual labels. The audit and paired benchmark results show that this simple relational signal, coupled with factual grounding, supports accurate counterfactual predictions without any additional counterfactual answer annotations for RL training. The manual labels are used only to assess the trained policy.
Manuscript changes: We added “Reliability of the Counterfactual Relation Reward” in Section 5.3, with the audit protocol, metric definitions, analysis, and results in Table 4. These details are consolidated in the main text. Appendix D.1 distinguishes Null usage frequency from the correctness of individual selections.