acceptodds
Under review as a conference paper at ICLR 2027

Correctness-Conditioned On-Policy Distillation for Reflective Multi-Turn Reasoning

25 views

Abstract

Reinforcement-learning-based post-training has improved the reasoning capabilities of vision-language models. Inspired by a teacher–student learning paradigm centered on questioning, receiving correctness feedback, and revising, we propose a Reflective Multi-Turn Video Reasoning method that enables models to verify and correct their own predictions without access to privileged feedback. First, Multi-Agent Dataset Filtering combines diverse reasoning trajectories generated by multiple agents with agent-based evaluation and human verification to construct high-quality supervision. Second, Reflective Multi-Turn Training iteratively retains mixed-outcome samples and jointly optimizes the model using Group Relative Policy Optimization (GRPO) and Correctness-Conditioned On-Policy Distillation (CCOPD), thereby improving answer verification and revision. During the second stage, CCOPD conditions the teacher model on the correctness as privileged information and provides token-level supervision along the student’s on-policy trajectory. We evaluate the proposed method on 13 widely used benchmarks spanning four categories: short-video understanding, long-video understanding, video reasoning, and temporal grounding. The results show that our method achieves broadly competitive performance across all 13 benchmarks.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Teacher-Student Learning, On-Policy Distillation, Multi-Turn Reasoning, Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fkDEOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies autonomous answer verification and revision in video-language models. The proposed framework first constructs and cleans chain-of-thought demonstrations using multiple models for supervised initialization. It then retains rollout groups containing both correct and incorrect answers under the current policy and performs staged reinforcement learning. The initial turn is optimized with GRPO, while subsequent reflective turns jointly use GRPO and Correctness-Conditioned On-Policy Distillation (CCOPD). In CCOPD, a self-teacher additionally receives the gold-derived correctness of the preceding answer and provides detached token-level supervision along the student's reflective completion; the student does not receive this correctness label. At inference, the student alone performs up to three turns without oracle correctness feedback. Evaluation covers two model backbones and 13 benchmarks spanning video understanding, video reasoning, and temporal grounding.

Critical strengths
  1. A relevant problem and a concrete training construction. Learning to detect and revise an earlier video answer without oracle feedback at deployment is practically useful. Conditioning the teacher only on the correctness of the preceding answer makes the training-time information advantage explicit, while the student must learn verification and revision from the video and interaction history. The distinction between preserving correct answers and correcting wrong ones is worth investigating.

  2. Broad empirical evidence that the complete pipeline is useful. The paper evaluates two backbones on 13 benchmarks and reports main results across multiple training seeds. In Table 4, the five-benchmark average increases from 59.3 to 64.5 for Qwen2.5-VL-7B and from 63.5 to 69.5 for Qwen3-VL-8B. Table 9 also shows that adding a verification prompt to existing baselines does not consistently improve them, whereas the proposed model improves substantially from its first to second inference turn. These results support the usefulness of trained reflection beyond merely making another model call.

Critical weaknesses
  1. The relative roles of GRPO and CCOPD in the joint objective lack direct evidence from training dynamics. Equation (5) specifies the joint objective, and Table 17 ablates the CCOPD weight, lambda. However, the paper does not separately report training curves or gradient contributions for the GRPO term and the weighted CCOPD term. It is therefore difficult to assess how the two signals shape optimization at different stages, or whether one dominates parameter updates. Neither lambda = 0.05 nor the raw loss magnitudes alone can establish this. This gap limits the explanation of why the joint optimization works.

  2. A CCOPD-only reflective-training control is missing, leaving the additional benefit of GRPO at this stage unquantified. From the common SFT + Turn-1 GRPO initialization, Table 4 reports a Turn-2 average of 62.0 for GRPO alone and 64.3 for GRPO+CCOPD, demonstrating a 2.3-point gain from adding CCOPD to GRPO. The corresponding CCOPD-only result is absent. The ablations establish the benefit of adding CCOPD to GRPO, but do not directly quantify the benefit of adding GRPO when CCOPD is retained. This limits the evidence for the complementary roles of the two objectives.

Questions for authors
  1. Could you report separate training curves for the GRPO loss and the weighted CCOPD loss to help clarify the relative roles of the two supervision signals during training? If feasible, an additional analysis of their gradient contributions could also help assess whether either objective dominates optimization.

  2. Could you add a Turn-2 CCOPD-only control under the same initialization, sample-filtering rule, training budget, and inference settings, and compare it with GRPO-only and GRPO+CCOPD? This control would remove GRPO only from reflective training while retaining the common Turn-1 initialization, isolating its additional contribution during reflection.

  3. Both CCOPD and SDPO use feedback-conditioned on-policy self-distillation. Could you further clarify what methodological differences arise from using the binary correctness of the preceding answer to supervise verification and revision in the next turn, relative to the existing framework? This would help clarify the paper's main incremental contribution. Reference: Hübotter et al., Reinforcement Learning via Self-Distillation, Section 2, https://arxiv.org/abs/2601.20802.

  4. The introduction moves from an educational analogy to the three-step training procedure, then discusses verification difficulties for VLMs before introducing the overall framework. Could you more clearly explain the specific limitations of existing video-reasoning and self-correction methods and how they motivate the proposed training design, particularly CCOPD?

  5. The current evaluation already covers a broad range of tasks. If resources permit, it would be interesting to add evaluations on a more challenging benchmark such as Video-MME-v2 to further assess how reflective training generalizes to complex video understanding and reasoning settings.

5 comments
AuthorsOpenReview ↗

Response to Reviewer fkDE

We thank the reviewer for the careful assessment and concrete suggestions. We agree that the original ablations did not fully isolate the two objectives or explain their training dynamics. We address these gaps below with separate loss and gradient diagnostics, a CCOPD-only control, a more precise comparison with SDPO, and additional benchmark results.

W1 and Q1 — Relative contributions of GRPO and CCOPD

Thank you for this suggestion. We have added separate training curves and a gradient analysis for the two terms in Eq. (5). The initial runs did not log gradients for each loss term, so we collected these diagnostics during the second turn of joint GRPO and CCOPD training on Qwen2.5-VL-32B. We conducted this larger-model experiment in response to Reviewer uEab’s question about scalability.

As shown in Appendix Figures 6 (a) and (b), over 3,000 training steps, the GRPO loss fell from about 0.04 to 0.02 and the weighted CCOPD loss fell from about 0.007 to 0.004. GRPO generally had the larger gradient norm, while the smoothed norm of the weighted CCOPD gradient was approximately 35–80% of GRPO’s. The two gradients were weakly aligned for much of training. Together, these observations suggest that CCOPD supplies a substantial, partly distinct learning signal for answer verification and revision. The norm ratio compares the two objective gradients. Their effects on the actual parameter updates also depend on the optimizer.

The analysis is included in the Section E.2 of the revised Appendix, with the new passages marked in red. Appendix Figures 6 (a) and (b) provide the full loss curves, gradient norms, norm ratios, and cosine similarities.

W2 and Q2 — CCOPD-only reflective training

We ran the missing control from the same Qwen2.5-VL-7B checkpoint after SFT and Turn-1 GRPO. The three reflective-training conditions use the same 38K filtered examples, sample-filtering rule, training budget, rollout settings, and 128-frame inference protocol. For CCOPD-only, we remove LGRPOL_{\mathrm{GRPO}} from the Turn-2 objective and retain the same correctness-conditioned teacher. The experiment and description of CCOPD-only setting are revised in Table 4 and Section 3.3 of the main paper.

Table R1. Turn-2 objective ablation from a common initialization. Accuracy is reported in percent; Avg. is the unweighted mean over the five benchmarks.

Turn-2 trainingVideoMMEMVBenchLongVideoBenchVideoMMMUMMVUAvg.
Turn-1 initialization63.465.360.253.266.161.6
GRPO only64.167.860.351.067.062.0
CCOPD only64.565.558.355.768.262.5
GRPO&CCOPD65.669.161.356.968.864.3

The joint objective outperforms each individual objective on all five benchmarks. In particular, CCOPD-only performs better than GRPO-only on VideoMME, VideoMMMU and MMVU, but worse on the two video-understanding benchmarks. Combining the objectives improves both groups. This supports their complementary value under the tested protocol: GRPO supplies outcome-based supervision for judgment, format, and final-answer correctness, while CCOPD supplies correctness-conditioned token-distribution targets along the student's reflective trajectory. The result establishes the incremental benefit in both directions at the reported setting; it is not a claim that this combination is universally optimal across independently retuned objectives.

Q3 — How CCOPD differs from SDPO

Thank you for pointing us to SDPO. The two methods differ in the feedback available for their target tasks, how they determine correctness, and which response they supervise.

  1. Task setting. SDPO studies tasks such as coding and scientific question answering, where the training environment can return feedback ranging from execution diagnostics to scalar correctness scores. CCOPD addresses video reasoning, where a model generally cannot run an answer through an execution environment or obtain reliable external feedback at inference. It must assess whether the answer is supported by the video and interaction history.

  2. Source of correctness. SDPO uses task feedback to inform its self-teacher. In the scalar-feedback setting, it can also use a successful rollout as a solution for the teacher to learn from. In CCOPD, we compare the preceding answer with the reference during training and give only the resulting binary correctness label to the teacher. The student learns to infer correctness from the video, question, and preceding response. At inference, it produces its own correctness judgment without access to the reference answer or an external verifier.

  3. What the method supervises. In SDPO, the feedback-conditioned teacher re-evaluates tokens in the original sampled response, providing token-level credit assignment for that attempt. CCOPD supervises a separate, student-generated reflection turn. The student first outputs an explicit judgment of the preceding answer, then produces a final answer that can preserve or revise it. The teacher uses the preceding answer’s correctness label to provide token-level targets along this new trajectory. The intermediate judgment is therefore a behavior we explicitly train alongside answer generation. It teaches the model to assess an answer it has already produced and then decide how to proceed.

Table 4 supports the use of this training signal in our setting. The five-benchmark average is 64.3% with true correctness conditioning, 61.8% with the label removed, and 59.8% with randomized labels. The difference is revised in Section A of the Appendix.

Q4 — Motivation and organization of the Introduction

We agree that the technical problem should precede the educational analogy. We have reorganized the Introduction around the following progression: existing video-reasoning capabilities, the remaining verification problem, and the supervision needed to learn verification and revision.

The revised motivation states:

Existing video-reasoning methods improve answer generation, visual grounding, and evidence acquisition. A further challenge is deciding whether an already-generated answer should be retained or revised using the available video evidence. Additional self-feedback can miss errors or overturn correct answers, while reliable external correctness feedback is unavailable in our deployment setting. This motivates explicitly training verification and revision after initial answer generation.

In our reflective-training formulation, GRPO rewards the correctness judgment, output format, and final answer at the response level. These rewards do not specify token-level distributions for carrying out verification and revision. CCOPD adds such guidance on trajectories sampled from the current student policy. During training, a shared-parameter teacher receives the preceding answer's correctness as its only privileged input, while both teacher and student observe the video and interaction history. The resulting targets condition the reflective process on whether the previous answer is correct without supplying the reference answer. The student learns from these targets while operating without the privileged label, matching its inference-time information constraints.

The explanation is revised in the second paragraph of Section 1 (Main paper).

Q5 — Additional challenging benchmarks

Following the suggestion, we evaluated on Video-MME-v2 and additionally on Video-Holmes. The results are:

Table R2. Additional benchmark accuracy in percent.

ModelVideo-MME-v2Video-Holmes
Qwen2.5-VL-7B21.338.9
Video-R1-7B20.741.9
VideoRFT-7B21.741.4
CCOPD24.342.5

CCOPD improves over the baseline by 3.0 points on Video-MME-v2 and 3.6 points on Video-Holmes. Relative to the strongest listed baseline on each benchmark, the gains are 2.6 and 0.6 points, respectively. These results extend the evaluation beyond the original benchmark suite and support the framework's applicability to additional video-understanding and reasoning tasks. The modest absolute accuracy on Video-MME-v2 also shows that substantial challenges remain. The results and evaluation settings are revised in Section D.3 and Table 12 of the Appendix.

Reviewer_uEabOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a reflective multi-turn video reasoning to allow VLM to verify and correct their answers. The motivation is that existing models often struggle to identify and correct their mistakes without external feedback, so this paper proposes Multi-Agent Dataset Filtering to construct high-quality training data, and Correctness-Conditioned On-Policy Distillation (CCOPD) to improve self-verification and correction by providing the teacher with correctness information during the training. Improvements were shown on 13 video understanding and reasoning benchmarks across 2 model backbones.

Critical strengths
  1. The motivation for improving video reasoning through self-verification and self-correction is intuitive and convincing. I appreciate that this motivation draws inspiration from the Tang dynasty, which is not something you see every day.
  2. I appreciate the experiments on 13 benchmarks across 2 model backbones to show the improvements.
  3. The ablation studies are helpful. E.g., Table 4 shows the effectiveness of CCOPD compared to standard GRPO and other distillation methods.
  4. The analysis of multi-turn reasoning is useful, showing that the model can correct incorrect answers while preserving most initially correct answers.
Critical weaknesses
  1. The method is built on existing GRPO and On-Policy Distillation of Language Models (ICLR 2024). Self-verification and multi-turn reasoning have been studied in LLaVA-SCo: Teach Vision Language Models to Self-Correct (CVPRW 2025). Thus, the technical contribution is weak.
  2. The proposed framework is complicated, e.g., it requires multi-agent data filtering, SFT, GRPO, and CCOPD. It is also slower due to multiple reasoning turns, e.g., Table 11 shows that the average inference latency increases from 8.25s to 15.05s (80+% slower).
  3. The experiments only cover models up to 8B parameters. Would it work on much larger models, e.g., 100B+, which may already have self-verification ability?
  4. The additional reasoning turns sometimes degrade performance, e.g., Table 4 shows that Turn 3 decreases performance on VideoMMMU (-1.1%) and MMVU (-0.8%) compared to Turn 2.
Questions for authors
  1. Would this method work with 100B+ models, e.g., Qwen3.5-122B-A10B (weakness 3)?
  2. Would the method still outperform existing approaches under the same inference latency budget (weakness 4)?
  3. Why does an additional reasoning turn sometimes decrease performance, as shown in Table 4 (weakness 5)? I appreciate the analysis in Appendix D.4, but it may not fully explain the root cause.
5 comments
AuthorsOpenReview ↗

Response to Reviewer uEab

We thank the reviewer for raising the questions of incremental contribution, practical cost, model scale, and the reliability of additional reflection. We clarify the methodological scope and report the new evidence below, including a larger-model experiment and a diagnostic of third-turn failures.

W1 — Technical contribution relative to GRPO, on-policy distillation, and LLaVA-SCo

Thank you for raising this point. CCOPD is designed to train a model to verify its own previous answer and use that judgment to preserve a correct answer or revise an incorrect one.

1. Differences from GRPO and on-policy distillation

Our work builds on ongoing efforts in the research community to address two challenges in reasoning-oriented post-training: sparse credit assignment in GRPO, and answer cheating and performance degradation during prolonged on-policy self-distillation. CCOPD addresses these challenges through dense token-level supervision conditioned on an intermediate correctness judgment.

Compared with GRPO, CCOPD provides dense supervision for verification and revision. In our GRPO formulation, the group-relative advantage evaluates the complete response, providing limited guidance about which individual steps contribute to successful verification or correction. Moreover, the reward-based advantage vanishes when all rollouts in a group receive the same reward. CCOPD complements this signal with token-level distributional supervision along the student’s own reflective trajectory, covering the explicit correctness judgment and the subsequent reasoning and answer generation.

Compared with answer-conditioned OPSD, CCOPD uses intermediate feedback while withholding the reference answer. In our OPSD baseline, the teacher receives the ground-truth answer, creating a risk of answer cheating: it may favor a plausible justification for the supplied answer without adequately grounding that justification in the video. CCOPD instead provides the privileged information, whether the student’s preceding answer was correct. This intermediate signal indicates whether preservation or correction is appropriate while leaving the model to work out the subsequent reasoning. The design encourages further examination of the video and interaction history, reducing the opportunity to learn answer-conditioned justifications or surface patterns. The student learns from these targets without receiving the privileged label and performs verification independently at inference.

Table 4 and Appendix Figure 5 provide empirical support for these design choices. GRPO alone achieves 62.0% average accuracy, GRPO with reference-answer-conditioned OPSD achieves 62.3%, and GRPO with CCOPD achieves 64.3%. Figure 5 further shows that the smoothed training reward of GRPO + OPSD declines after an early peak, whereas GRPO + CCOPD maintains a higher reward during later training.

2. Differences from LLaVA-SCo

  1. Source of supervision. LLaVA-SCo uses GPT-4o during offline data construction to evaluate initial responses, refine the reasoning behind correct answers, and help repair incorrect ones. During CCOPD’s reflective training, the student learns to judge its own answer from the video, question, and interaction history. The reference answer is used during training to derive the teacher’s binary correctness label. At inference, the student makes the judgment without an external evaluator or a correctness label.

  2. Training mechanism. LLaVA-SCo fine-tunes on a fixed set of offline refinement demonstrations. The corresponding stage in our pipeline is the initial SFT stage. During reflective training, CCOPD uses a teacher branch that shares the student’s parameters to provide token-level distribution targets on prefixes generated by the current student policy. The supervision therefore follows the policy as it evolves and explicitly trains the student to judge its preceding answer and then preserve or revise it.

  3. Evidence from ablations. In Table 4, direct SFT on correctness judgments and answers achieves 59.0%, GRPO with reference-answer-conditioned OPSD achieves 62.3%, and GRPO with CCOPD achieves 64.3%. These comparisons test supervision choices within our pipeline and support conditioning on the preceding answer’s correctness during on-policy reflective training.

The difference is revised in Section A of the Appendix.

W2 and Q2 — Pipeline complexity and inference latency

Thank you for raising these concerns. Multi-agent filtering, SFT, GRPO, and the CCOPD teacher are used only during training. At inference, a single model generates and revises its answers. The video is loaded and encoded once per example, and later turns reuse the visual features.

The cited 8.25 s and 15.05 s latencies are averages across five benchmarks. Table R3 compares accuracy and latency on MVBench, with an additional evaluation under a shared cumulative output budget of 4,096 tokens per example across all turns.

The original implementations allow up to 4,096 tokens per turn for VideoAuto-R1, 8,192 for VideoKR, and 2,048 for CCOPD. The shared cap therefore leaves VideoAuto-R1 and two-turn CCOPD unchanged, while restricting VideoKR and three-turn CCOPD.

Table R3. MVBench accuracy and inference latency.

Inference settingOriginal accuracy (%)Original mean latency (s)Accuracy under 4,096-token total cap (%)
Qwen2.5-VL-7B63.92.6463.9
VideoAuto-R170.27.6270.2
VideoKR68.94.5168.5
CCOPD, two turns69.15.1369.1
CCOPD, adaptive three turns71.35.8570.5

Under the original configurations, adaptive CCOPD exceeds VideoAuto-R1 by 1.1 while taking 1.77 s less on average. Compared with VideoKR, it gains 2.4 improvement for 1.34 s of additional latency. Under the shared token cap, VideoKR and three-turn CCOPD decrease by 0.4 and 0.8 respectively, while adaptive CCOPD retains the highest accuracy, exceeding VideoAuto-R1 by 0.3.

These results support a competitive accuracy–latency trade-off and show that the gains persist under a common generation budget. However, equal token budgets do not guarantee equal latency, and comparison after tuning each baseline to the full latency budget remains open. These results and evaluation settings have been added to Section D.7 and Table 15 of the revised Appendix.

W3 and Q1 — Scaling to larger models

Thank you for raising this question. A stronger model may already be able to verify some of its own answers, so we tested whether reflective training still helps beyond the 7B and 8B models in the original submission. Post-training Qwen3.5-122B-A10B was beyond our ability. We therefore ran the full training pipeline on Qwen2.5-VL-32B, including SFT, Turn-1 GRPO, and Turn-2 reflective training with GRPO + CCOPD. We evaluated all checkpoints with 128-frame inputs.

Table R4. Accuracy (%) on four benchmarks with Qwen2.5-VL-32B

Training stageVideoMMELVBenchVideoMMMUMMVU
Original checkpoint68.543.861.562.3
SFT67.145.963.166.9
Turn-1 GRPO68.148.364.768.6
Turn-2 GRPO + CCOPD69.849.168.070.4

After the reflective stage, average accuracy rose from 62.4% to 64.3%, with gains on all four benchmarks. To check whether reflection repaired more answers than it harmed, we tracked correctness changes from Turn 1 to Turn 2. On every benchmark, the model corrected more answers than it changed from correct to incorrect.

BenchmarkIncorrect → correctCorrect → incorrectNet gain in correct answers
VideoMME57 / 861 (6.62%)11 / 1,839 (0.60%)+46
LVBench18 / 801 (2.25%)5 / 748 (0.67%)+13
VideoMMMU36 / 318 (11.32%)6 / 582 (1.03%)+30
MMVU21 / 314 (6.69%)3 / 686 (0.44%)+18

The 32B results establish the benefit of the joint reflective stage at this larger scale. The experiments in 32B model is revised in Section D.2, Table 10, and Table 11 of the Appendix.

W4 and Q3 — Why a third turn can reduce accuracy

Thank you for asking us to clarify the drop observed in Table 4. We hypothesize that earlier reflection resolves some of the more obvious mistakes, leaving errors that are harder to detect and correct. As in reviewing an exam, another pass can miss remaining errors and sometimes turn a correct answer into an incorrect one. Our goal is for each reflection turn to correct more previously incorrect answers than it changes from correct to incorrect.

Across all 13 benchmarks, adding a third turn raises the average score from 57.9 to 58.5. Scores improve on eight benchmarks, so the gains elsewhere outweigh the declines on VideoMMMU, MMVU, and three other benchmarks.

BenchmarkTurn 2Turn 3
MVBench69.171.3
VideoMME65.666.2
MVP46.248.8
VSI43.544.7
LongVideoBench61.361.4
MLVU68.167.8
LVBench44.946.3
MMVU68.868.0
VideoMMMU56.955.8
Charades-STA58.060.3
ActivityNet52.651.9
NExT-QA36.437.8
TempCompass81.180.5
Average57.958.5

To test whether a third pass can still improve accuracy on the two benchmarks raised by the reviewer, we fixed the Turn-2 predictions, preserved all correct answers, and sent every incorrect answer to Turn 3. We used ground-truth labels only to select which answers to revisit. The model received no correctness label in its prompt.

Table R5. Oracle routing with fixed Turn-2 predictions

BenchmarkCorrect after Turn 2Incorrect answers routedErrors correctedFinal full-test accuracy
VideoMMMU512 / 900 (56.9%)38865 / 388 (16.75%)577 / 900 (64.11%)
MMVU688 / 1,000 (68.8%)31257 / 312 (18.27%)745 / 1,000 (74.50%)

Under oracle routing, the third pass improved full-test accuracy by 7.21 percentage points on VideoMMMU and 5.70 percentage points on MMVU.

Reviewer_5NXdOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a reflective multi-turn framework for video LLMs: the model answers, judges whether its answer is correct (True/False), and keeps or revises it. Its core training method, Correctness-Conditioned On-Policy Distillation (CCOPD), uses a teacher that shares the student's parameters but also sees one privileged bit, whether the previous answer was correct. The teacher gives token-level reverse-KL supervision on the student's own sampled trajectories. The pipeline also includes multi-agent CoT data curation, keeping only training samples whose rollouts mix correct and incorrect outcomes, and joint GRPO+CCOPD training. Experiments on Qwen2.5-VL-7B/32B and Qwen3-VL-8B across 13 video benchmarks show broadly competitive results.

Critical strengths
  • Giving the teacher only a correctness bit, rather than the reference answer, is a sensible middle ground: it supplies dense on-policy guidance without leaking the answer. Separating the judgment span from the revision span, and using an asymmetric judgment reward, makes verification an explicit learning target and discourages the model from always answering True.
  • The evaluation spans 13 benchmarks, multiple backbones and frame budgets, and a 32B scaling run. The ablations cover teacher conditioning, divergence type, teacher update strategy, data filtering, and inference cost.
  • The multi-agent pipeline withholds the reference answer during trace generation and sends conflicts with the original labels to human review, which guards against answer leakage and label noise.
Critical weaknesses
  • CCOPD is essentially privileged on-policy self-distillation (cf. OPSD/SDPO) with the privileged signal reduced to one bit, and its gain over GRPO alone (about 2 points) is close to seed variance. The learned gate is also weak (42% False-class recall, 68% balanced accuracy), so the self-verification claims are overstated.

  • The large SFT corpus distilled from six frontier models likely explains much of the gain, since GRPO without this CoT data already reaches 62.4, and baselines are not retrained on the same data. The multi-turn baselines were never trained to revise, and the roughly 2.2× inference cost is not compared against a compute-matched single-turn baseline.

  • Turn-2 gains in Tab. 9 do not match the net transitions in Tab. 17 (e.g., MMVU +8.3 vs. about +2.7). The Turn-1 accuracies implied by Tab. 17 match the Turn-1 GRPO checkpoint rather than the final model, contradicting the "fixed final checkpoint" claim. Configuration details also conflict between Tab. 7 and Sec. C.1.

Questions for authors

See weaknesses.

1 comment
AuthorsOpenReview ↗

Response to Reviewer 5NXd

Thank you for the time you have devoted to reviewing our paper and for your thoughtful comments. Given the limited rebuttal period, we focus on your main concerns and appreciate your understanding where we cannot address every point in full detail.

W1

We thank the reviewer for raising these questions about the contribution of CCOPD, seed variability, and self-verification. The main results in Tables 2 and 3 are averaged over three independent training seeds, with the minimum and maximum reported for each metric. Table 4 directly compares training objectives from the same checkpoint using the same 38K Turn-2 examples. GRPO + CCOPD improves over GRPO on all five benchmarks, raising the average from 62.0 to 64.3. Tables 2 and 3 thus quantify run-to-run variation in the main results, while Table 4 shows the size and cross-benchmark consistency of the objective-level gain.

The design choice concerns where the privileged information comes from and which decision it supervises. During training, we derive a correctness label from the model’s own preceding answer. Conditioned on that label, the teacher provides token-level guidance along a newly sampled reflection turn in which the student judges its answer and decides whether to retain or revise it. This avoids exposing the reference answer directly to the teacher and long-term training degradation (drawbacks of OPSD), and allows the student to revise without extra enviromental feedbacks at inference (drawback of SDPO). In Table 4, reference-answer-conditioned GRPO + OPSD scores 62.3, compared with 64.3 for CCOPD.

Consider a student checking an exam before handing it in. The easiest mistakes are usually caught first. As the review deepens, the remaining questions often require the student to retrace the underlying reasoning, making further corrections harder. In Table 16, the Turn-2 gate detects 42.00% of errors at 67.92% balanced accuracy, and 79.94% of the answers it flags are actually wrong. Table 17 reports 324 corrections and 38 regressions at Turn 2, a net gain of 286 correct answers, followed by 118 corrections and 31 regressions at Turn 3, a further net gain of 87. The later round yields a smaller gain, but corrections outnumber regressions in both rounds, raising overall accuracy at each step.

W2

We thank the reviewer for raising the question of attribution. Both the CCOPD training framework and the construction of the multi-teacher CoT corpus are main contributions of this paper. The 62.4% result already includes CCOPD: even when CoT SFT is omitted entirely, Turn-1 GRPO followed by Turn-2 GRPO + CCOPD reaches 62.4%, a 3.1-percentage-point gain over the 59.3% base model (Table 4). The CoT corpus contributes both reasoning traces and a structured response format. SFT alone raises accuracy from 59.3% to 59.9% (+0.6 points), and including six-teacher CoT SFT raises accuracy after reflective training from 62.4% to 64.3% (+1.9 points, Table 19).

CCOPD yields the larger incremental gain in the reported ablations. Starting from the same SFT + Turn-1 GRPO checkpoint and training on the same 38K revision examples, adding CCOPD to Turn-2 GRPO raises accuracy from 62.0% to 64.3% (+2.3 points, Table 4), compared with +1.9 points from adding CoT SFT to the reflective-training pipeline (Table 19). These two controls show the value of both contributions and directly address the role of revision training. For the inference-cost comparison, please see our response to Reviewer uEab, W2 and Q2.

W3

Thank you for pointing out the ambiguity in our description. Tables 17–18 provide the per-example transition counts for the training-stage comparisons and ablations in Table 4. Table 9 reports successive inference turns using the fixed final checkpoint. On MMVU, Table 17 compares the Turn-1 GRPO checkpoint (66.1%) with the Turn-2 GRPO + CCOPD checkpoint (68.8%). The 30 corrections and 3 regressions give a net gain of 27/1,000 = 2.7%, matching Table 4. In Table 9, the final checkpoint scores 60.2% on its first inference turn and 68.5% on its second, yielding +8.3 points. The reviewer is therefore correct that the starting accuracy in Table 17 matches the Turn-1 GRPO checkpoint. We will correct the “fixed final checkpoint” wording in the Appendix and state the distinction explicitly in the main text and table captions.

The configuration values also refer to different stages. Table 7 specifies the training limits of 64 video frames and 2,048 completion tokens. Section C.1 describes standard evaluation with 16, 128, or 768 frames for Qwen2.5-VL and a 4,096-token response limit. We will label these settings by stage and experiment in the manuscript.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We thank both reviewers for their constructive feedback. We summarize the main clarifications and additional evidence below.

1. What is the contribution, and how is it motivated? (Reviewer fkDE: Q3–Q4; Reviewer uEab: W1)

Our contribution is correctness-conditioned supervision of next-turn verification and revision. The shared-parameter teacher receives only the preceding answer’s correctness as privileged information and provides token-level targets along the student’s reflective trajectory. The student learns to judge, preserve, or revise its answer without external correctness feedback at inference. This complements GRPO’s response-level rewards, differs from answer-conditioned OPSD and offline refinement SFT in LLaVA-SCo, and specializes feedback-conditioned distillation to cross-turn verification relative to SDPO. Within our pipeline, true correctness conditioning achieves 64.3%, versus 61.8% without conditioning and 59.8% with randomized labels. This also indicates that if we can obtain the same environmental feedback as SDPO. Our test results will be much higher.

2. Do GRPO and CCOPD provide complementary benefits? (Reviewer fkDE: W1–W2, Q1–Q2; Reviewer uEab: W1)

Under matched initialization, data, training budget, and inference settings, joint training outperforms GRPO-only and CCOPD-only on all five benchmarks, increasing average accuracy from 62.0% with GRPO alone to 64.3%. The individual objectives have different strengths across benchmarks. Separate loss and gradient diagnostics on Qwen2.5-VL-32B further show that the weighted CCOPD gradient’s smoothed norm is approximately 35–80% of GRPO’s, with weak alignment for much of training, supporting a substantial, partly distinct learning signal.

3. Are the gains justified by the computational cost? (Reviewer uEab: W2, Q2)

Deployment uses one model with reused video features; filtering and the teacher branch are training components. Reflection adds computation, but on MVBench, CCOPD achieves 71.3% at 5.85 s, versus VideoAuto-R1’s 70.2% at 7.62 s. Under a shared 4,096-token total output cap, CCOPD retains 70.5%, versus 70.2% for VideoAuto-R1 and 68.5% for VideoKR. These results support a competitive trade-off.

4. Does the method scale to larger models? (Reviewer uEab: W3, Q1)

On Qwen2.5-VL-32B, the joint reflective stage improves the four-benchmark average from 62.4% to 64.3%, with gains and more corrections than corruptions on every benchmark. This supports scalability of joint reflective training to 32B.

5. Why can a third reflection turn reduce accuracy? (Reviewer uEab: W4, Q3)

Additional reflection provides an overall benefit, raising the average score across 13 benchmarks from 57.9 to 58.5, despite modest declines on VideoMMMU and MMVU. Earlier reflection resolves easier errors, while leaving increasingly difficult cases; subsequent reflection may then mistakenly reconsider some correct answers. More reliable correctness judgments could unlock further gains: with fixed Turn-2 predictions and oracle routing that preserves correct answers and revisits only errors, accuracy reaches 64.11% on VideoMMMU and 74.50% on MMVU.

6. Does the method generalize to harder benchmarks? (Reviewer fkDE: Q5)

CCOPD achieves 24.3% on Video-MME-v2 and 42.5% on Video-Holmes, exceeding the strongest listed baseline on each by 2.6 and 0.6 percentage points, respectively. These additional evaluations support applicability to more challenging video tasks.

Thanks again for the reviewers' effort. And these modifications have been revised in the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.