- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper studies autonomous answer verification and revision in video-language models. The proposed framework first constructs and cleans chain-of-thought demonstrations using multiple models for supervised initialization. It then retains rollout groups containing both correct and incorrect answers under the current policy and performs staged reinforcement learning. The initial turn is optimized with GRPO, while subsequent reflective turns jointly use GRPO and Correctness-Conditioned On-Policy Distillation (CCOPD). In CCOPD, a self-teacher additionally receives the gold-derived correctness of the preceding answer and provides detached token-level supervision along the student's reflective completion; the student does not receive this correctness label. At inference, the student alone performs up to three turns without oracle correctness feedback. Evaluation covers two model backbones and 13 benchmarks spanning video understanding, video reasoning, and temporal grounding.
Critical strengths
-
A relevant problem and a concrete training construction. Learning to detect and revise an earlier video answer without oracle feedback at deployment is practically useful. Conditioning the teacher only on the correctness of the preceding answer makes the training-time information advantage explicit, while the student must learn verification and revision from the video and interaction history. The distinction between preserving correct answers and correcting wrong ones is worth investigating.
-
Broad empirical evidence that the complete pipeline is useful. The paper evaluates two backbones on 13 benchmarks and reports main results across multiple training seeds. In Table 4, the five-benchmark average increases from 59.3 to 64.5 for Qwen2.5-VL-7B and from 63.5 to 69.5 for Qwen3-VL-8B. Table 9 also shows that adding a verification prompt to existing baselines does not consistently improve them, whereas the proposed model improves substantially from its first to second inference turn. These results support the usefulness of trained reflection beyond merely making another model call.
Critical weaknesses
-
The relative roles of GRPO and CCOPD in the joint objective lack direct evidence from training dynamics. Equation (5) specifies the joint objective, and Table 17 ablates the CCOPD weight, lambda. However, the paper does not separately report training curves or gradient contributions for the GRPO term and the weighted CCOPD term. It is therefore difficult to assess how the two signals shape optimization at different stages, or whether one dominates parameter updates. Neither lambda = 0.05 nor the raw loss magnitudes alone can establish this. This gap limits the explanation of why the joint optimization works.
-
A CCOPD-only reflective-training control is missing, leaving the additional benefit of GRPO at this stage unquantified. From the common SFT + Turn-1 GRPO initialization, Table 4 reports a Turn-2 average of 62.0 for GRPO alone and 64.3 for GRPO+CCOPD, demonstrating a 2.3-point gain from adding CCOPD to GRPO. The corresponding CCOPD-only result is absent. The ablations establish the benefit of adding CCOPD to GRPO, but do not directly quantify the benefit of adding GRPO when CCOPD is retained. This limits the evidence for the complementary roles of the two objectives.
Questions for authors
-
Could you report separate training curves for the GRPO loss and the weighted CCOPD loss to help clarify the relative roles of the two supervision signals during training? If feasible, an additional analysis of their gradient contributions could also help assess whether either objective dominates optimization.
-
Could you add a Turn-2 CCOPD-only control under the same initialization, sample-filtering rule, training budget, and inference settings, and compare it with GRPO-only and GRPO+CCOPD? This control would remove GRPO only from reflective training while retaining the common Turn-1 initialization, isolating its additional contribution during reflection.
-
Both CCOPD and SDPO use feedback-conditioned on-policy self-distillation. Could you further clarify what methodological differences arise from using the binary correctness of the preceding answer to supervise verification and revision in the next turn, relative to the existing framework? This would help clarify the paper's main incremental contribution. Reference: Hübotter et al., Reinforcement Learning via Self-Distillation, Section 2, https://arxiv.org/abs/2601.20802.
-
The introduction moves from an educational analogy to the three-step training procedure, then discusses verification difficulties for VLMs before introducing the overall framework. Could you more clearly explain the specific limitations of existing video-reasoning and self-correction methods and how they motivate the proposed training design, particularly CCOPD?
-
The current evaluation already covers a broad range of tasks. If resources permit, it would be interesting to add evaluations on a more challenging benchmark such as Video-MME-v2 to further assess how reflective training generalizes to complex video understanding and reasoning settings.
5 comments
Q3 — How CCOPD differs from SDPO
Thank you for pointing us to SDPO. The two methods differ in the feedback available for their target tasks, how they determine correctness, and which response they supervise.
-
Task setting. SDPO studies tasks such as coding and scientific question answering, where the training environment can return feedback ranging from execution diagnostics to scalar correctness scores. CCOPD addresses video reasoning, where a model generally cannot run an answer through an execution environment or obtain reliable external feedback at inference. It must assess whether the answer is supported by the video and interaction history.
-
Source of correctness. SDPO uses task feedback to inform its self-teacher. In the scalar-feedback setting, it can also use a successful rollout as a solution for the teacher to learn from. In CCOPD, we compare the preceding answer with the reference during training and give only the resulting binary correctness label to the teacher. The student learns to infer correctness from the video, question, and preceding response. At inference, it produces its own correctness judgment without access to the reference answer or an external verifier.
-
What the method supervises. In SDPO, the feedback-conditioned teacher re-evaluates tokens in the original sampled response, providing token-level credit assignment for that attempt. CCOPD supervises a separate, student-generated reflection turn. The student first outputs an explicit judgment of the preceding answer, then produces a final answer that can preserve or revise it. The teacher uses the preceding answer’s correctness label to provide token-level targets along this new trajectory. The intermediate judgment is therefore a behavior we explicitly train alongside answer generation. It teaches the model to assess an answer it has already produced and then decide how to proceed.
Table 4 supports the use of this training signal in our setting. The five-benchmark average is 64.3% with true correctness conditioning, 61.8% with the label removed, and 59.8% with randomized labels. The difference is revised in Section A of the Appendix.
Q4 — Motivation and organization of the Introduction
We agree that the technical problem should precede the educational analogy. We have reorganized the Introduction around the following progression: existing video-reasoning capabilities, the remaining verification problem, and the supervision needed to learn verification and revision.
The revised motivation states:
Existing video-reasoning methods improve answer generation, visual grounding, and evidence acquisition. A further challenge is deciding whether an already-generated answer should be retained or revised using the available video evidence. Additional self-feedback can miss errors or overturn correct answers, while reliable external correctness feedback is unavailable in our deployment setting. This motivates explicitly training verification and revision after initial answer generation.
In our reflective-training formulation, GRPO rewards the correctness judgment, output format, and final answer at the response level. These rewards do not specify token-level distributions for carrying out verification and revision. CCOPD adds such guidance on trajectories sampled from the current student policy. During training, a shared-parameter teacher receives the preceding answer's correctness as its only privileged input, while both teacher and student observe the video and interaction history. The resulting targets condition the reflective process on whether the previous answer is correct without supplying the reference answer. The student learns from these targets while operating without the privileged label, matching its inference-time information constraints.
The explanation is revised in the second paragraph of Section 1 (Main paper).
Q5 — Additional challenging benchmarks
Following the suggestion, we evaluated on Video-MME-v2 and additionally on Video-Holmes. The results are:
Table R2. Additional benchmark accuracy in percent.
| Model | Video-MME-v2 | Video-Holmes |
|---|---|---|
| Qwen2.5-VL-7B | 21.3 | 38.9 |
| Video-R1-7B | 20.7 | 41.9 |
| VideoRFT-7B | 21.7 | 41.4 |
| CCOPD | 24.3 | 42.5 |
CCOPD improves over the baseline by 3.0 points on Video-MME-v2 and 3.6 points on Video-Holmes. Relative to the strongest listed baseline on each benchmark, the gains are 2.6 and 0.6 points, respectively. These results extend the evaluation beyond the original benchmark suite and support the framework's applicability to additional video-understanding and reasoning tasks. The modest absolute accuracy on Video-MME-v2 also shows that substantial challenges remain. The results and evaluation settings are revised in Section D.3 and Table 12 of the Appendix.
Response to Reviewer fkDE
We thank the reviewer for the careful assessment and concrete suggestions. We agree that the original ablations did not fully isolate the two objectives or explain their training dynamics. We address these gaps below with separate loss and gradient diagnostics, a CCOPD-only control, a more precise comparison with SDPO, and additional benchmark results.
W1 and Q1 — Relative contributions of GRPO and CCOPD
Thank you for this suggestion. We have added separate training curves and a gradient analysis for the two terms in Eq. (5). The initial runs did not log gradients for each loss term, so we collected these diagnostics during the second turn of joint GRPO and CCOPD training on Qwen2.5-VL-32B. We conducted this larger-model experiment in response to Reviewer uEab’s question about scalability.
As shown in Appendix Figures 6 (a) and (b), over 3,000 training steps, the GRPO loss fell from about 0.04 to 0.02 and the weighted CCOPD loss fell from about 0.007 to 0.004. GRPO generally had the larger gradient norm, while the smoothed norm of the weighted CCOPD gradient was approximately 35–80% of GRPO’s. The two gradients were weakly aligned for much of training. Together, these observations suggest that CCOPD supplies a substantial, partly distinct learning signal for answer verification and revision. The norm ratio compares the two objective gradients. Their effects on the actual parameter updates also depend on the optimizer.
The analysis is included in the Section E.2 of the revised Appendix, with the new passages marked in red. Appendix Figures 6 (a) and (b) provide the full loss curves, gradient norms, norm ratios, and cosine similarities.
W2 and Q2 — CCOPD-only reflective training
We ran the missing control from the same Qwen2.5-VL-7B checkpoint after SFT and Turn-1 GRPO. The three reflective-training conditions use the same 38K filtered examples, sample-filtering rule, training budget, rollout settings, and 128-frame inference protocol. For CCOPD-only, we remove LGRPO from the Turn-2 objective and retain the same correctness-conditioned teacher. The experiment and description of CCOPD-only setting are revised in Table 4 and Section 3.3 of the main paper.
Table R1. Turn-2 objective ablation from a common initialization. Accuracy is reported in percent; Avg. is the unweighted mean over the five benchmarks.
The joint objective outperforms each individual objective on all five benchmarks. In particular, CCOPD-only performs better than GRPO-only on VideoMME, VideoMMMU and MMVU, but worse on the two video-understanding benchmarks. Combining the objectives improves both groups. This supports their complementary value under the tested protocol: GRPO supplies outcome-based supervision for judgment, format, and final-answer correctness, while CCOPD supplies correctness-conditioned token-distribution targets along the student's reflective trajectory. The result establishes the incremental benefit in both directions at the reported setting; it is not a claim that this combination is universally optimal across independently retuned objectives.