acceptodds
Under review as a conference paper at ICLR 2027

Beyond Entropy: Self-Diagnostic Multi-Role Token Optimization for Video Reasoning

Abstract

Reinforcement learning with verifiable rewards has substantially advanced multimodal reasoning, yet it remains fundamentally limited by ambiguous token-level credit assignment. While high-entropy token heuristics encourage possibility exploration, naively extending them to video reasoning tends to induce lengthy reasoning, as the model becomes overly reliant on high-entropy visual activations. Alternative approaches that rely on counterfactual-based visual token localization for credit assignment also tend to over-prioritize visual exploration at the expense of decisive reasoning cues for answer derivation, thereby exacerbating the interference from spurious visual nuances. Moreover, these methods employ static counterfactual strategies that fail to co-evolve with the policy during training. In this paper, we introduce DyCPO, a co-evolutionary framework that jointly optimizes reliable token selection and adaptive counterfactual intervention. It constructs a multi-role dependence metric to balance visual exploration and answer-relevance mining in token-wise contrastive learning, while suppressing exploration-only filler tokens and spurious visual noise. Rather than relying on static counterfactual priors, DyCPO dynamically derives counterfactual signals from the model's own successful and failed rollouts, enabling self-diagnostic analysis and co-evolution of the optimization objective with the policy. Extensive experiments on complex video reasoning and general video understanding benchmarks demonstrate consistent performance improvements, establishing DyCPO as a robust token-level credit assignment paradigm for multimodal reinforcement learning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
reinforcement learning, multi-role token, visual activation, self-diagnostic modeling, video reasoning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_pFy2OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes DyCPO, a self-diagnostic token-level optimization framework for video reasoning. One interesting aspect of the paper is its re-examination of the common assumption that high-entropy tokens are always the most important targets for reasoning optimization. The authors show that many vision-dependent and answer-dependent tokens can actually lie in the low-entropy region, and that optimizing only high-entropy tokens may over-emphasize exploration while missing stable visual or decision-critical cues. Based on this observation, DyCPO jointly measures visual and answer sensitivity through counterfactual interventions, selects multi-role tokens, and applies token-level contrastive optimization together with GRPO. The counterfactual visual signals are further adapted using successful and failed rollouts rather than relying only on static perturbations.

Critical strengths
  1. The analysis of entropy versus key tokens is interesting and well motivated. The paper provides empirical evidence that high entropy does not necessarily correspond to the most useful tokens for video reasoning, and that a substantial portion of visually grounded or answer-critical tokens can be missed by entropy-only selection.
  2. The distinction between vision-dependent and answer-dependent tokens is useful. The paper shows that visual sensitivity is concentrated more in earlier decoding stages, while answer sensitivity tends to appear later, supporting the motivation for multi-role token selection rather than a single criterion.
  3. The self-diagnostic counterfactual design is a meaningful component. Using successful and failed rollouts to adapt the counterfactual signal makes the intervention dependent on the evolving policy rather than completely static.
  4. The empirical results are strong and consistent across a broad range of video reasoning and understanding benchmarks.
Critical weaknesses
  1. The method incurs around 20–40% additional training cost per step compared with GRPO. While this overhead is non-trivial, the consistent gains across benchmarks make the trade-off reasonably acceptable.
  2. The self-diagnostic keyframe localization relies on response-to-frame attention. Although the paper provides layer-wise ablations and qualitative visualizations, a direct intervention test would make this design more convincing. For example, comparing the accuracy drop after masking the selected keyframes against masking random or low-attention frames could provide stronger evidence that the identified frames are causally important rather than merely correlated with model attention.
Questions for authors

see weakness

Reviewer_WatfOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes a DyCPO training framework for video reasoning that uses multi-role tokens (visual and answer sensitive) with robust token-level credit assignment and dynamic counterfactual signals to improve performance. The model is trained with 3K difficulty-filtered samples from Video-R1 initialized from VideoRFT SFT checkpoint trained from 102K CoT samples. Evaluations are conducted on 5 video reasoning and 2 video understanding benchmarks and compared with prior video reasoning training strategies. The ablation studies cover three models: Qwen2.5-VL-7B, Qwen3-VL-2B, and Qwen3-VL-4B.

Critical strengths
  • The motivation for the method is clearly stated in the introduction supported by Fig. 1 and 2. Scoring on tokens on both visual grounding and answer relevance instead of relying only on vision or text is a sensible extension to prior work on token-level credit assignment.

  • The gains are shown across three models (Table 2) and holds at different training frames (24 and 32) using only 3K samples.

  • Ablation studies broadly cover the different hyperparameters.

Critical weaknesses
  • Limited novelty relative to Video-KTR and PAPO. The multi-role token selection closely follows Video-KTR, which already combines counterfactual visual-masking, frame-shuffling, and entropy signals to select key tokens for selective RL. DyCPO’s main change is replacing the temporal signal with an answer-counterfactual signal. The contrastive objective is a margin-based variant of PAPO’s Implicit Perception Loss with an added positive view and restriction to selected tokens. PAPO is not cited in the references or discussed in related work despite being a baseline in Table 3. The answer-counterfactual signal is the novel contribution. The paper would be stronger if it framed itself around that and compared against Video-KTR and PAPO under identical settings.

  • The claim of needing less data is misleading since the model is initialized from SFT checkpoint trained with 102K CoT samples.

  • The effect of data curation over the method is not fully analyzed. GRPO trained with same data configurations already reaches 40.9 on Video-Holmes and 59.3 on VideoMME (Table 2). That is above Video-R1 and close to Video-KTR, so much of the Table 1 margin seems to come from initialization and data curation rather than the method. A fair comparison should show the closest method like Video-KTR, and an entropy-based method on the same SFT checkpoint and 3K data for a clear picture on the gains.

  • There is no baseline for high-entropy token selection in Table 1 or Table 2. The claim that high entropy token selection hurts video reasoning is not fully justified. Table 2 mentions an “entropy-only baseline” with “substantial degradation on reasoning benchmarks” (L365-371) but it is not shown.

  • The self-diagnostic counterfactual results are not convincing. Table 4 shows that selecting keyframes from correct rollouts only (R_r-focus) is within 0.2-0.9 of the full method and L395 states that the frames attended in incorrect rollouts “are not entirely noise”, which seems to be against the motivation for Eq. 13.

  • The method relies on several hyperparameters and requires effort in tuning them.

  • Clarity and presentation problems. Several figure references are wrong. Line 93 cites Fig. 1(a)(b) for the positional analysis, which is in Fig. 2(a)(b). Appendix D cites Fig. 2 for the training curves, which are in Fig. 6.

    • The curriculum-direction text contradicts the Table 4 label. The text says Narrow-->Broad is the reverse variant, but the table row is “Broad-->Narrow”.
    • Eq. 9 adds a loss to an objective that Eq. 1 maximizes, so the sign is unclear. It is unspecified whether gradients flow through the V+ and V− branches.
    • Eq. 8 is described as separating selected from unselected tokens, but the loss does not involve unselected tokens.
    • Algorithm 1 sets α\alpha=0.99, which makes the proposal model 99% the current policy each step. That contradicts the “slowly evolving anchor” description (L292).
    • The paper says it beats GPT-4o’s 42.0% on Video-Holmes, but Table 1 shows no GPT-4o number for that benchmark. Gemini-2.5-Pro is cited in the text but isn’t in the table either.
    • The method is called “PAPO” in places describing the papers’ own method (Appendix B and the overhead discussion)
    • The reproducibility statement refers to a “theoretical result” that does not exist.
Questions for authors
  • See weaknesses above.
  • Can you report Video-KTR and a high-entropy token baseline (e.g., top-20% entropy) trained from the same SFT checkpoint on the same 3K data and frame budget to see the actual gains of the proposed method over others?
  • Which model generates the rollouts used in the GRPO loss, πθπ_θ or πφπ_φ? If πφπ_φ, how is the importance ratio handled? What is the effect of removing the EMA or changing αα?
  • What is the actual definition and value of αvisα_{vis}/αansα_{ans} in the main experiments? Which backbone and benchmark do Tables 10 and 11 use?
  • Were hyperparameters chosen on a held-out validation set or on the reported benchmarks?
  • How is the answer a±a± inserted into the context for the answer-sensitivity forward pass?
Reviewer_AmzgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents a new credit assignment framework for RL with verifiable rewards for video understanding, including both video reasoning and generic video understanding. The main claim is that simple entropy-based credit assignment and counterfactual methods are not able to decouple reasoning and visual understanding. And one needs a multi-role metric that combines vision and answer-aware sensitivity in the credit assignment process for GRPO.

Critical strengths

The main results are strong. The paper has shown the comparison over video reasoning (R1) based post-trained baselines over several benchmarks. Despite the model size being small (<10B), the method does also show the effectiveness of the approach across different model size of the Qwen3/2.5-VL family.

Critical weaknesses

My main concern is on the ablation. First, it's unclear what wo Ht row stands for, in the main text it describes a baseline that uses simple token entropy-based subsequence credit assignment. Also, across VideoMMMU and Video-Holmes, the ablated version of wo Ht is largely comparable to that of the full model design. Thus, it raises the questions on the effectiveness of the claimed contributions.

Questions for authors

Also, the method is only shown over Qwen2.5/3-VL families, how does the approach generalize to other family of models such as Gemma4-12B?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.