- Rating
- 3(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes Memory-Grounded Video Latent Reasoning (MVLR), which interleaves textual reasoning with continuous latent steps that retrieve evidence from adaptive video memory. The memory retains detailed features from selected temporal intervals and compresses the remaining context. Training combines two-stage supervised learning, retrieval diversity regularization, and counterfactual-constrained reinforcement learning. On seven benchmarks, MVLR reports the best performance among compared methods on five, alongside 2.63$\times$ Video-o3’s throughput on Video-Holmes. Component ablations and temporal-range perturbations provide additional validation.
Critical strengths
-
A coherent solution to maintaining visual evidence during reasoning. Preserving selected frame details alongside compressed global context is well motivated.
-
Broad evaluation with useful interventions. Beyond seven-benchmark results, replacing latent states and perturbing temporal ranges consistently reduce performance. These interventions support the practical importance of learned latent content and temporal evidence selection.
Critical weaknesses
-
CCRL adds positive answer-token credit to correct trajectories, weighted by a frozen model using donor memory and its own generated reasoning prefix (§4.4). Thus, the comparison changes parameters and reasoning history as well as memory. Table 2 demonstrates improvement over standard GRPO, but does not distinguish counterfactual grounding from the benefit of additional correct-answer reinforcement. This limits the evidence for a central contribution.
-
MVLR receives substantial training from Video-o3 and OneThinker data, but no matched-data textual or agent baseline is reported (§5.1). Its advantages over Video-o3 are only 0.8–1.5 percentage points on the five winning benchmarks. These gaps make the accuracy advantages difficult to assess reliably.
-
The lack of certain implementation details reduces reproducibility; for specific details, see the Question section.
Questions for authors
-
Does CCRL outperform a comparable additional correct-answer bonus with constant weights or weights computed without donor-memory substitution? Does its advantage persist with matched reasoning prefixes?
-
How does MVLR compare with a textual or agent baseline trained on the same data and evaluated with matched visual budgets? Can you report uncertainty for the main gains?
-
What hardware, batch size, decoding limits, and frame/resolution budgets produced Table 4, and how much time is spent on feature caching and memory construction?
6 comments
Response to Reviewer txSD (Part 2/4)
(3) Updated reference with the same parameters as the rollout policy (Appendix C.3, Table 11). During the development of CCRL, we evaluated a reference that uses the current behavior-policy checkpoint at each rollout iteration. This variant retains donor-memory substitution, and its counterfactual scores are computed without gradients and held fixed during each policy update.
| Benchmark | GRPO | Updated reference | MVLR |
|---|---|---|---|
| Video-MME | 64.7 | 64.4 | 66.3 |
| MLVU | 68.2 | 67.7 | 71.9 |
| LVBench | 44.2 | 43.1 | 48.8 |
| LongVideoBench | 57.4 | 56.6 | 61.9 |
| Video-MMMU | 50.6 | 48.2 | 53.1 |
| MMVU | 62.7 | 60.1 | 68.0 |
| Video-Holmes | 38.1 | 36.6 | 48.0 |
The updated-reference variant underperforms standard GRPO across all seven benchmarks. Freezing the reference yields gains of 5.7 points on LVBench, 7.9 points on MMVU, and 11.4 points on Video-Holmes over this variant.
The explanation is that updating the reference changes counterfactual scoring across training iterations. The factual–counterfactual score margin can increase through weaker answer support under donor memory, without stronger answer support under factual memory. This reduces the contrastive loss and the additional credit assigned by CCRL, making the contrastive objective appear better satisfied without improving factual answer support. Freezing the Stage II reference anchors counterfactual scoring throughout training, providing a stable basis for credit allocation.
(4) Shared initial reasoning prefixes (Appendix C.4; Tables 11–13). During CCRL development, we initially explored a shared-prefix design pairing each factual rollout with a separate counterfactual continuation conditioned on the same initial textual reasoning prefix . However, we found that generating a separate counterfactual continuation for every factual rollout incurred substantially higher computational cost while achieving performance comparable to the independent-prefix approach. These observations motivated our current design. The following two comparisons explain this choice.
(a) Effect of prefix substitution on counterfactual scoring. We evaluated 500 questions. For each question, we compared the counterfactual reference scores obtained using the reference model’s own initial reasoning prefix with those obtained by replacing with each of eight factual reasoning prefixes, denoted by .
Following Eq. (8), the counterfactual score is the mean token log-likelihood of the same answer sequence:
where denotes the resulting counterfactual reasoning history before the answer, and corresponds to the reference model’s own initial prefix. We measure prefix sensitivity using the mean absolute score difference over the 500 questions:
The results are summarized below (in nats/token).
| Factual prefix | |
|---|---|
| 0.021 | |
| 0.014 | |
| 0.012 | |
| 0.017 | |
| 0.006 | |
| 0.014 | |
| 0.011 | |
| 0.005 |
The average distance across the eight prefix substitutions is 0.0125 nats/token. These results indicate that counterfactual scoring exhibits limited sensitivity to the choice of initial reasoning prefix. We further assessed the effect of prefix sharing through a complete RL training comparison.
(b) Performance and training efficiency with shared prefixes. We compared the original independent-prefix implementation with the shared-prefix variant under otherwise matched experimental settings. The reference model remains frozen in both variants.
| Metric | Independent | Shared |
|---|---|---|
| Video-MME | 66.3 | 66.1 |
| MLVU | 71.9 | 72.0 |
| LVBench | 48.8 | 48.6 |
| LongVideoBench | 61.9 | 61.6 |
| Video-MMMU | 53.1 | 53.2 |
| MMVU | 68.0 | 67.7 |
| Video-Holmes | 48.0 | 47.8 |
| Training throughput (items/s) ↑ | 0.0242 | 0.0097 |
These results indicate:
(I) Comparable benchmark performance. The differences between the two variants remain within 0.3 percentage points across all seven benchmarks, with no consistent performance advantage from sharing the initial reasoning prefix.
(II) Substantially higher training throughput. The independent-prefix implementation achieves 2.49× the measured training throughput of the shared-prefix variant.
Together, these comparisons support our decision to generate one independent counterfactual reference history per question and reuse it across the factual rollout group. This design preserves benchmark performance while amortizing reference generation across rollouts, making counterfactual feedback substantially more efficient to incorporate into training.
Response to Reviewer txSD (Part 3/4)
W2&Q2: “How does MVLR compare with a textual or agent baseline trained on the same data and evaluated with matched visual budgets? Can you report uncertainty for the main gains?”
Response: Thank you for highlighting the need for comparisons with matched training data and visual budgets. We added comparisons with matched training data and visual budgets in Appendix C.5 (Table 14). We clarify that the results for previously published baselines in the original Table 1 were taken from their respective reports. Following your recommendation, we trained a textual Qwen2.5-VL baseline, Video-o3, and vanilla video latent reasoning using the same training data as MVLR. We also matched the evaluation frame counts, resolutions, and visual budgets. For each method, we evaluated the same trained checkpoint three times and report the mean results below.
| Metric | Qwen2.5-VL | Qwen2.5-VL + data | Video-o3 | Vanilla latent | MVLR |
|---|---|---|---|---|---|
| Video-MME | 62.7 | 65.3 | 65.9 | 62.2 | 66.2 |
| MLVU | 69.6 | 70.1 | 71.3 | 65.7 | 72.0 |
| LVBench | 44.8 | 46.1 | 47.2 | 42.2 | 48.6 |
| LongVideoBench | 55.4 | 56.4 | 60.4 | 54.4 | 61.7 |
| Video-MMMU | 47.5 | 48.2 | 51.0 | 43.3 | 53.4 |
| MMVU | 61.2 | 61.9 | 66.5 | 52.4 | 68.3 |
| Video-Holmes | 33.9 | 35.3 | 46.1 | 29.6 | 48.4 |
| Inference throughput (items/s) ↑ | 0.0613 | 0.0569 | 0.0357 | 0.1170 | 0.1120 |
These results show:
(1) Improvements beyond the training data. Training Qwen2.5-VL on MVLR’s data improves its performance, while MVLR delivers further gains across all seven benchmarks. The improvements are particularly substantial on complex video reasoning: +5.2 points on Video-MMMU, +6.4 points on MMVU, and +13.1 points on Video-Holmes. These results demonstrate the contribution of memory-grounded latent reasoning beyond the benefit of additional training data.
(2) Effective video latent reasoning. Compared with vanilla video latent reasoning trained on the same data, MVLR improves Video-MMMU, MMVU, and Video-Holmes by 10.1, 15.9, and 18.8 points, respectively, while maintaining similar inference throughput. Together with the diagnostics and ablations in the paper, these results demonstrate the effectiveness of adaptive visual memory and the proposed training procedure in addressing visual forgetting and representation shift.
(3) Higher accuracy and inference throughput than Video-o3. Under matched training data and visual budgets, MVLR outperforms Video-o3 across all seven benchmarks, including gains of 2.4 points on Video-MMMU, 1.8 points on MMVU, and 2.3 points on Video-Holmes. It also achieves 3.14× the measured inference throughput. MVLR reuses cached visual features and retrieves evidence at each latent step, reducing the overhead associated with repeated video inspection while improving complex video reasoning performance.
Response to Reviewer txSD (Part 1/4)
Thank you for recognizing the strengths of our work and for your thoughtful suggestions. We address your concerns and provide the requested clarifications below.
Response: We appreciate your suggestions for more controlled comparisons of CCRL. In response, we added controlled comparisons in Section 5.5 and Appendices C.3–C.4 to examine how the weights in Eq. (9), counterfactual memory, reference parameters, and initial reasoning prefixes affect CCRL. We present the comparisons below.
(1) Constant and shuffled weights (Section 5.5, Table 6). We replaced the weights wni in Eq. (9) with either constant weights or shuffled weights. Both controls preserve CCRL’s eligibility rules and match its average additional credit, while leaving the remaining training settings unchanged.
Both controls perform similarly to standard GRPO, whereas CCRL consistently outperforms them across all seven benchmarks. These results demonstrate the effectiveness of computing the contrastive credit from factual and counterfactual answer scores. As defined in Eq. (9), correct trajectories receive larger weights when factual answer support is weaker relative to the reference. This directs greater optimization pressure toward strengthening their answer support under factual memory. Constant weights remove this distinction, while shuffled weights disrupt the correspondence between the weights and the trajectories receiving them.
(2) Factual memory instead of donor memory (Section 5.5, Table 6). We retained the same frozen reference checkpoint, contrastive credit formulation, and training settings, changing only the reference’s visual memory from donor memory to factual AVMC memory from the source video.
The factual-memory variant does not improve over standard GRPO and performs slightly worse on six benchmarks. Original CCRL outperforms this variant across all seven benchmarks, including gains of 5.2 points on LVBench, 6.2 points on MMVU, and 10.6 points on Video-Holmes.
These results demonstrate the benefit of counterfactual memory substitution beyond using a frozen reference alone. With factual memory on both sides, the score gap can reflect differences in general reasoning ability, so additional credit reinforce already-correct answers without strengthening visual-memory reliance. The donor-memory reference instead computes the contrastive credit against answer support under mismatched visual memory, encouraging the model to rely more strongly on factual visual memory during reasoning.