acceptodds
Under review as a conference paper at ICLR 2027

Memory-Grounded Video Latent Reasoning

Abstract

Long-video understanding and complex video reasoning require vision–language models (VLMs) to identify sparse visual cues amid redundant content and integrate evidence across distant moments through complex reasoning. Textual chain-of-thought (CoT) and agent-based approaches incur high inference costs; extended textual reasoning can lose focus on visual evidence, while repeated local inspection can reinforce errors. Although image-based visual latent reasoning reduces reliance on lengthy textual descriptions, we observed that directly transferring it to videos degrades performance, revealing visual forgetting and representation shift. We propose Memory-Grounded Video Latent Reasoning (MVLR), which interleaves textual reasoning with continuous visual latent reasoning grounded in adaptive visual memory. Guided by textual CoT, the memory preserves keyframe detail, compresses complementary context, and supplies relevant evidence at every latent step. Two-stage supervised training combines CoT and answer supervision with retrieval diversity regularization, learning latent representations without explicit visual-feature targets. Counterfactual-constrained reinforcement learning further assigns contrastive credit by comparing the model's support for correct answers under factual memory with a frozen reference model's support for the same answers under counterfactual memory. MVLR achieves state-of-the-art performance across 5 benchmarks spanning long-video understanding and complex video reasoning, while delivering 2.63x the inference throughput of strong baseline.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
VLM, Latent Reasoning, Video Understanding

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_txSDOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Memory-Grounded Video Latent Reasoning (MVLR), which interleaves textual reasoning with continuous latent steps that retrieve evidence from adaptive video memory. The memory retains detailed features from selected temporal intervals and compresses the remaining context. Training combines two-stage supervised learning, retrieval diversity regularization, and counterfactual-constrained reinforcement learning. On seven benchmarks, MVLR reports the best performance among compared methods on five, alongside 2.63$\times$ Video-o3’s throughput on Video-Holmes. Component ablations and temporal-range perturbations provide additional validation.

Critical strengths
  1. A coherent solution to maintaining visual evidence during reasoning. Preserving selected frame details alongside compressed global context is well motivated.

  2. Broad evaluation with useful interventions. Beyond seven-benchmark results, replacing latent states and perturbing temporal ranges consistently reduce performance. These interventions support the practical importance of learned latent content and temporal evidence selection.

Critical weaknesses
  1. CCRL adds positive answer-token credit to correct trajectories, weighted by a frozen model using donor memory and its own generated reasoning prefix (§4.4). Thus, the comparison changes parameters and reasoning history as well as memory. Table 2 demonstrates improvement over standard GRPO, but does not distinguish counterfactual grounding from the benefit of additional correct-answer reinforcement. This limits the evidence for a central contribution.

  2. MVLR receives substantial training from Video-o3 and OneThinker data, but no matched-data textual or agent baseline is reported (§5.1). Its advantages over Video-o3 are only 0.8–1.5 percentage points on the five winning benchmarks. These gaps make the accuracy advantages difficult to assess reliably.

  3. The lack of certain implementation details reduces reproducibility; for specific details, see the Question section.

Questions for authors
  1. Does CCRL outperform a comparable additional correct-answer bonus with constant weights or weights computed without donor-memory substitution? Does its advantage persist with matched reasoning prefixes?

  2. How does MVLR compare with a textual or agent baseline trained on the same data and evaluated with matched visual budgets? Can you report uncertainty for the main gains?

  3. What hardware, batch size, decoding limits, and frame/resolution budgets produced Table 4, and how much time is spent on feature caching and memory construction?

6 comments
AuthorsOpenReview ↗

Response to Reviewer txSD (Part 1/4)

Thank you for recognizing the strengths of our work and for your thoughtful suggestions. We address your concerns and provide the requested clarifications below.


W1&Q1: “Does CCRL outperform a comparable additional correct-answer bonus with constant weights or weights computed without donor-memory substitution? Does its advantage persist with matched reasoning prefixes?”

Response: We appreciate your suggestions for more controlled comparisons of CCRL. In response, we added controlled comparisons in Section 5.5 and Appendices C.3–C.4 to examine how the weights in Eq. (9), counterfactual memory, reference parameters, and initial reasoning prefixes affect CCRL. We present the comparisons below.

(1) Constant and shuffled weights (Section 5.5, Table 6). We replaced the weights wniw_{ni} in Eq. (9) with either constant weights or shuffled weights. Both controls preserve CCRL’s eligibility rules and match its average additional credit, while leaving the remaining training settings unchanged.

BenchmarkGRPOConstant weightsShuffled weightsMVLR (CCRL)
Video-MME64.764.864.766.3
MLVU68.268.168.371.9
LVBench44.244.544.548.8
LongVideoBench57.457.857.161.9
Video-MMMU50.650.250.553.1
MMVU62.762.162.668.0
Video-Holmes38.138.438.548.0

Both controls perform similarly to standard GRPO, whereas CCRL consistently outperforms them across all seven benchmarks. These results demonstrate the effectiveness of computing the contrastive credit from factual and counterfactual answer scores. As defined in Eq. (9), correct trajectories receive larger weights when factual answer support is weaker relative to the reference. This directs greater optimization pressure toward strengthening their answer support under factual memory. Constant weights remove this distinction, while shuffled weights disrupt the correspondence between the weights and the trajectories receiving them.

(2) Factual memory instead of donor memory (Section 5.5, Table 6). We retained the same frozen reference checkpoint, contrastive credit formulation, and training settings, changing only the reference’s visual memory from donor memory to factual AVMC memory from the source video.

BenchmarkGRPOFrozen ref. + factualMVLR
Video-MME64.764.766.3
MLVU68.267.971.9
LVBench44.243.648.8
LongVideoBench57.456.961.9
Video-MMMU50.649.853.1
MMVU62.761.868.0
Video-Holmes38.137.448.0

The factual-memory variant does not improve over standard GRPO and performs slightly worse on six benchmarks. Original CCRL outperforms this variant across all seven benchmarks, including gains of 5.2 points on LVBench, 6.2 points on MMVU, and 10.6 points on Video-Holmes.

These results demonstrate the benefit of counterfactual memory substitution beyond using a frozen reference alone. With factual memory on both sides, the score gap can reflect differences in general reasoning ability, so additional credit reinforce already-correct answers without strengthening visual-memory reliance. The donor-memory reference instead computes the contrastive credit against answer support under mismatched visual memory, encouraging the model to rely more strongly on factual visual memory during reasoning.

AuthorsOpenReview ↗

Response to Reviewer txSD (Part 2/4)

(3) Updated reference with the same parameters as the rollout policy (Appendix C.3, Table 11). During the development of CCRL, we evaluated a reference that uses the current behavior-policy checkpoint at each rollout iteration. This variant retains donor-memory substitution, and its counterfactual scores are computed without gradients and held fixed during each policy update.

BenchmarkGRPOUpdated referenceMVLR
Video-MME64.764.466.3
MLVU68.267.771.9
LVBench44.243.148.8
LongVideoBench57.456.661.9
Video-MMMU50.648.253.1
MMVU62.760.168.0
Video-Holmes38.136.648.0

The updated-reference variant underperforms standard GRPO across all seven benchmarks. Freezing the reference yields gains of 5.7 points on LVBench, 7.9 points on MMVU, and 11.4 points on Video-Holmes over this variant.

The explanation is that updating the reference changes counterfactual scoring across training iterations. The factual–counterfactual score margin can increase through weaker answer support under donor memory, without stronger answer support under factual memory. This reduces the contrastive loss and the additional credit assigned by CCRL, making the contrastive objective appear better satisfied without improving factual answer support. Freezing the Stage II reference anchors counterfactual scoring throughout training, providing a stable basis for credit allocation.

(4) Shared initial reasoning prefixes (Appendix C.4; Tables 11–13). During CCRL development, we initially explored a shared-prefix design pairing each factual rollout with a separate counterfactual continuation conditioned on the same initial textual reasoning prefix T0T_0. However, we found that generating a separate counterfactual continuation for every factual rollout incurred substantially higher computational cost while achieving performance comparable to the independent-prefix approach. These observations motivated our current design. The following two comparisons explain this choice.

(a) Effect of prefix substitution on counterfactual scoring. We evaluated 500 questions. For each question, we compared the counterfactual reference scores obtained using the reference model’s own initial reasoning prefix with those obtained by replacing T0T_0 with each of eight factual reasoning prefixes, denoted by F1,…,F8F_1,\ldots,F_8.

Following Eq. (8), the counterfactual score is the mean token log-likelihood of the same answer sequence:

bn,k−=1Ln∑u=1Lnlog⁡πω0(yn,u∣Hn,k−,yn,<u),b^-_{n,k} = \frac{1}{L_n} \sum_{u=1}^{L_n} \log \pi_{\omega_0} \left(y_{n,u}\mid H^-_{n,k},y_{n,\lt u}\right),

where Hn,k−H^-_{n,k} denotes the resulting counterfactual reasoning history before the answer, and k=0k=0 corresponds to the reference model’s own initial prefix. We measure prefix sensitivity using the mean absolute score difference over the 500 questions:

Dk=1500∑n=1500∣bn,k−−bn,0−∣.D_k = \frac{1}{500} \sum_{n=1}^{500} \left|b^-_{n,k}-b^-_{n,0}\right|.

The results are summarized below (in nats/token).

Factual prefixDkD_k
F1F_10.021
F2F_20.014
F3F_30.012
F4F_40.017
F5F_50.006
F6F_60.014
F7F_70.011
F8F_80.005

The average distance across the eight prefix substitutions is 0.0125 nats/token. These results indicate that counterfactual scoring exhibits limited sensitivity to the choice of initial reasoning prefix. We further assessed the effect of prefix sharing through a complete RL training comparison.

(b) Performance and training efficiency with shared prefixes. We compared the original independent-prefix implementation with the shared-prefix variant under otherwise matched experimental settings. The reference model remains frozen in both variants.

MetricIndependentShared
Video-MME66.366.1
MLVU71.972.0
LVBench48.848.6
LongVideoBench61.961.6
Video-MMMU53.153.2
MMVU68.067.7
Video-Holmes48.047.8
Training throughput (items/s) ↑0.02420.0097

These results indicate:

(I) Comparable benchmark performance. The differences between the two variants remain within 0.3 percentage points across all seven benchmarks, with no consistent performance advantage from sharing the initial reasoning prefix.

(II) Substantially higher training throughput. The independent-prefix implementation achieves 2.49× the measured training throughput of the shared-prefix variant.

Together, these comparisons support our decision to generate one independent counterfactual reference history per question and reuse it across the factual rollout group. This design preserves benchmark performance while amortizing reference generation across rollouts, making counterfactual feedback substantially more efficient to incorporate into training.

AuthorsOpenReview ↗

Response to Reviewer txSD (Part 3/4)

W2&Q2: “How does MVLR compare with a textual or agent baseline trained on the same data and evaluated with matched visual budgets? Can you report uncertainty for the main gains?”

Response: Thank you for highlighting the need for comparisons with matched training data and visual budgets. We added comparisons with matched training data and visual budgets in Appendix C.5 (Table 14). We clarify that the results for previously published baselines in the original Table 1 were taken from their respective reports. Following your recommendation, we trained a textual Qwen2.5-VL baseline, Video-o3, and vanilla video latent reasoning using the same training data as MVLR. We also matched the evaluation frame counts, resolutions, and visual budgets. For each method, we evaluated the same trained checkpoint three times and report the mean results below.

MetricQwen2.5-VLQwen2.5-VL + dataVideo-o3Vanilla latentMVLR
Video-MME62.765.365.962.266.2
MLVU69.670.171.365.772.0
LVBench44.846.147.242.248.6
LongVideoBench55.456.460.454.461.7
Video-MMMU47.548.251.043.353.4
MMVU61.261.966.552.468.3
Video-Holmes33.935.346.129.648.4
Inference throughput (items/s) ↑0.06130.05690.03570.11700.1120

These results show:

(1) Improvements beyond the training data. Training Qwen2.5-VL on MVLR’s data improves its performance, while MVLR delivers further gains across all seven benchmarks. The improvements are particularly substantial on complex video reasoning: +5.2 points on Video-MMMU, +6.4 points on MMVU, and +13.1 points on Video-Holmes. These results demonstrate the contribution of memory-grounded latent reasoning beyond the benefit of additional training data.

(2) Effective video latent reasoning. Compared with vanilla video latent reasoning trained on the same data, MVLR improves Video-MMMU, MMVU, and Video-Holmes by 10.1, 15.9, and 18.8 points, respectively, while maintaining similar inference throughput. Together with the diagnostics and ablations in the paper, these results demonstrate the effectiveness of adaptive visual memory and the proposed training procedure in addressing visual forgetting and representation shift.

(3) Higher accuracy and inference throughput than Video-o3. Under matched training data and visual budgets, MVLR outperforms Video-o3 across all seven benchmarks, including gains of 2.4 points on Video-MMMU, 1.8 points on MMVU, and 2.3 points on Video-Holmes. It also achieves 3.14× the measured inference throughput. MVLR reuses cached visual features and retrieves evidence at each latent step, reducing the overhead associated with repeated video inspection while improving complex video reasoning performance.

Reviewer_jgYwOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

MVLR alternates textual planning with continuous latent updates that retrieve video evidence from an adaptive memory. Predicted temporal intervals determine which frames retain detailed features and which are compressed. Training combines two-stage supervised fine-tuning with reinforcement learning augmented by contrastive credit from a frozen model using donor-video memory. Evaluation covers seven video understanding and reasoning benchmarks, with leading results on five and improved inference throughput.

Critical strengths
  • The memory keeps full features from selected intervals alongside a compressed representation of the rest of the video. I like that later latent steps can still retrieve evidence outside the current focus without encoding the video again.
  • On Video-Holmes, MVLR improves accuracy from Video-o3's 46.5% to 48.0% while reporting 2.63× its throughput. The accuracy gain is modest, but getting it together with faster inference is useful.
Critical weaknesses
  • The CCRL ablation does not distinguish the benefit of counterfactual weighting from additional supervision on correct answers. A constant-weight control is needed.
  • The experiments do not isolate recurrent latent reasoning from improved visual evidence access. A simpler baseline using the same adaptive memory and training data is missing.
Questions for authors
  • Have you compared CCRL with constant or shuffled weights while preserving the eligibility rules and matching the average additional credit? Does counterfactual weighting retain an advantage?
  • What happens when the frozen reference uses factual memory instead of donor memory for the CCRL contrastive score? I'm interested in this comparison because it would help me understand whether the gain comes from counterfactual memory or simply from using a frozen reference.
  • Can you report accuracy and throughput for a model trained with the same adaptive memory but without recurrent latent segments, using matched training data and visual evidence budgets?
3 comments
AuthorsOpenReview ↗

Response to Reviewer jgYw (Part 1/2)

We appreciate your careful review and constructive feedback. We respond to each of your comments and questions below.


W1&Q1: “The CCRL ablation does not distinguish the benefit of counterfactual weighting from additional supervision on correct answers... compared CCRL with constant or shuffled weights while preserving the eligibility rules and matching the average additional credit? Does counterfactual weighting retain an advantage?”

Response: We appreciate your recommendation to isolate the contribution of counterfactual weighting. To address this point, we added ablations in Section 5.5 (Table 6) in which CCRL’s counterfactual weights were replaced with either constant weights or shuffled weights. Both controls preserve CCRL’s eligibility rules and match its average additional credit, while leaving the remaining training settings unchanged. The results are summarized below.

BenchmarkGRPOConstant weightsShuffled weightsCCRL
Video-MME64.764.864.766.3
MLVU68.268.168.371.9
LVBench44.244.544.548.8
LongVideoBench57.457.857.161.9
Video-MMMU50.650.250.553.1
MMVU62.762.162.668.0
Video-Holmes38.138.438.548.0

We observe that both controls perform similarly to standard GRPO, whereas the CCRL consistently outperforms them across all seven benchmarks. These results support the effectiveness of trajectory-specific counterfactual weighting. By assigning greater credit to correct trajectories with smaller factual–counterfactual score margins, CCRL places greater optimization pressure on strengthening their answer support under factual memory.


W2&Q3: “The experiments do not isolate recurrent latent reasoning from improved visual evidence access. A simpler baseline using the same adaptive memory and training data is missing.”

Response: Thank you for suggesting a baseline that isolates the role of recurrent latent reasoning. Following your recommendation, we added an AVMC + Text-CoT baseline in Section 5.5 (Table 5), trained using the same training data, AVMC construction, visual evidence budgets, and training objectives as MVLR.

The baseline replaces recurrent continuous latent segments with textual reasoning. After T0T_0 predicts the initial temporal ranges and triggers memory construction, the reader integrates memory evidence into the decoder hidden states at all subsequent text-decoding positions. The current memory is reused until the next memory-read marker, when AVMC updates it using the latest predicted ranges. The results are summarized below.

MetricQwen2.5-VL-7BAVMC + Text-CoTMVLR
Video-MME65.165.966.3
MLVU70.270.871.9
LVBench45.346.548.8
LongVideoBench56.058.261.9
Video-MMMU47.449.153.1
MMVU61.363.568.0
Video-Holmes34.738.448.0
Inference throughput (items/s) ↑0.06210.05470.1040

MVLR outperforms AVMC + Text-CoT across all seven benchmarks, with particularly large gains on complex video reasoning: +4.0 points on Video-MMMU, +4.5 points on MMVU, and +9.6 points on Video-Holmes. It also achieves 1.9× the inference throughput.

These results suggest that MVLR’s learned visual latent tokens capture and integrate visual evidence more effectively than textual CoT, improving subsequent video understanding while enabling more efficient reasoning in latent space.

AuthorsOpenReview ↗

Response to Reviewer jgYw (Part 2/2)

Q2: “What happens when the frozen reference uses factual memory instead of donor memory for the CCRL contrastive score?”

Response: Thank you for raising this question about the visual memory supplied to the frozen reference. To address it, we added an ablation in Section 5.5 (Table 6) in which the frozen reference uses factual AVMC memory from the source video instead of donor memory. We retain the same frozen reference checkpoint, contrastive credit formulation, and training settings, changing only the visual memory supplied to the reference. The results are summarized below.

BenchmarkGRPOFrozen ref. + factualMVLR
Video-MME64.764.766.3
MLVU68.267.971.9
LVBench44.243.648.8
LongVideoBench57.456.961.9
Video-MMMU50.649.853.1
MMVU62.761.868.0
Video-Holmes38.137.448.0

We observe that the factual-memory variant does not improve over standard GRPO and performs slightly worse on six benchmarks. In comparison, the original CCRL consistently outperforms this variant, with gains of 5.2 points on LVBench, 6.2 points on MMVU, and 10.6 points on Video-Holmes.

These results support the value of counterfactual memory substitution beyond the use of a frozen reference alone. With factual memory on both sides, the score gap can reflect differences in general reasoning ability, making it difficult to distinguish these effects from stronger reliance on visual memory. Consequently, the additional credit reinforce already-correct answers based on general reasoning ability without strengthening visual-memory reliance, explaining the degradation relative to GRPO. The donor-memory reference instead calibrates credit against answer support under mismatched visual memory, encouraging the model to rely more strongly on factual visual memory during reasoning.


We appreciate your thoughtful feedback, which has helped us strengthen the experimental validation and clarify the presentation. We would be happy to address any further questions or concerns.

AuthorsOpenReview ↗

Thank you for your updated assessment!

Thank you for your updated assessment and for the time and care you have devoted to reviewing our work. Your suggestions have been valuable in guiding our revisions. Please let us know if any aspect would benefit from further explanation.

Reviewer_jy3KOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper aims to address the challenges of visual forgetting and representation shift when extending image-based visual latent reasoning to long-video understanding and complex video reasoning. Motivated by the limitations of lengthy textual CoT and costly agent-based video inspection, the paper argues for grounding continuous visual latent reasoning in adaptive visual memory to preserve relevant visual evidence throughout the reasoning process.

The author propose Memory-Grounded Video Latent Reasoning (MVLR), a framework that interleaves textual CoT with continuous visual latent reasoning. Specifically, it introduces Adaptive Visual Memory Construction (AVMC), which preserves fine-grained features from CoT-selected keyframes while compressing the remaining video context. During latent reasoning, each latent step retrieves relevant visual evidence from the memory through cross-attention. The model is trained through a two-stage supervised fine-tuning process with retrieval diversity regularization, followed by Counterfactual-Constrained Reinforcement Learning (CCRL), which encourages reliance on factual visual evidence.

The resulting model is built on Qwen2.5-VL-7B-Instruct. Evaluations are conducted on seven benchmarks, including Video-MME, MLVU, LVBench, LongVideoBench, Video-MMMU, MMVU, and Video-Holmes. MVLR achieves the best reported performance on five benchmarks and delivers 2.63× the inference throughput of Video-o3 on Video-Holmes.

Critical strengths
  • Overall, I appreciate the motivation of this paper and its emphasis on the visual forgetting and representation shift problem as they are practical challenges in extending visual latent reasoning to videos. The preliminary analysis provides useful insights into the limitations of directly transferring image-based latent reasoning methods.
  • The proposed memory-grounded latent reasoning framework is an interesting and intuitive design.
  • The paper provides relatively comprehensive experiments across seven benchmarks. The component ablations support the effectiveness of the proposed framework.
Critical weaknesses

My main concerns lie in the following aspects:

  • The effectiveness of the proposed CCRL in improving visual grounding is not sufficiently demonstrated. The counterfactual reference model uses not only different visual memory but also a separately generated reasoning prefix. Therefore, the likelihood difference may not be solely attributable to factual vs. counterfactual visual evidence.
  • The proposed method is only evaluated on Qwen2.5-VL-7B-Instruct. It remains unclear whether the memory-grounded latent reasoning framework and CCRL training strategy can generalize to other VLM architectures or larger model scales.
Questions for authors

Please see the weaknesses. For the first one, it would be helpful to include more controlled experiments that isolate the effect of counterfactual memory, as well as comparisons with alternative contrastive reward designs.

3 comments
AuthorsOpenReview ↗

Response to Reviewer jy3K (Part 1/2)

Thank you for your positive evaluation of our work and for the valuable suggestions provided. Below, we address your concerns point-by-point.


W1&Q1: “… The counterfactual reference model uses not only different visual memory but also a separately generated reasoning prefix. Therefore, the likelihood difference may not be solely attributable to factual vs. counterfactual visual evidence.”

Response: Thank you for highlighting the role of the initial reasoning prefix. To address this concern, we added the prefix-sensitivity analysis and shared-prefix training comparison in Appendix C.4 (Tables 12–13), with benchmark accuracy reported in Table 11. We initially explored a shared-prefix design during the development of CCRL, in which each factual rollout was paired with a counterfactual continuation conditioned on the same initial textual reasoning prefix T0T_0. However, we found that generating a separate counterfactual continuation for every factual rollout incurred substantially higher computational cost while achieving performance comparable to the independent-prefix approach. These observations motivated our current design. We present the following two comparisons to clarify the basis for this choice.

(1) Effect of prefix substitution on counterfactual scoring. We evaluated 500 questions. For each question, we compared the counterfactual reference scores obtained using the reference model’s own initial reasoning prefix with those obtained by replacing T0T_0 with each of eight factual reasoning prefixes, denoted by F1,…,F8F_1,\ldots,F_8.

Following Eq. (8), we define the counterfactual score as the mean token log-likelihood of the same answer sequence:

bn,k−=1Ln∑u=1Lnlog⁡πω0(yn,u∣Hn,k−,yn,<u),b^-_{n,k} = \frac{1}{L_n} \sum_{u=1}^{L_n} \log \pi_{\omega_0} \left(y_{n,u}\mid H^-_{n,k},y_{n,\lt u}\right),

where Hn,k−H^-_{n,k} denotes the resulting counterfactual reasoning history before the answer, and k=0k=0 corresponds to the reference model’s own initial prefix. We measure prefix sensitivity using the mean absolute score difference over the 500 questions:

Dk=1500∑n=1500∣bn,k−−bn,0−∣.D_k = \frac{1}{500} \sum_{n=1}^{500} \left|b^-_{n,k}-b^-_{n,0}\right|.

The results are summarized below.

Factual prefixDkD_k
F1F_10.021
F2F_20.014
F3F_30.012
F4F_40.017
F5F_50.006
F6F_60.014
F7F_70.011
F8F_80.005

The average distance across the eight prefix substitutions is 0.0125 nats/token. These results indicate that counterfactual scoring exhibits limited sensitivity to the choice of initial reasoning prefix. We further assessed the effect of prefix sharing through a complete RL training comparison.

(2) Performance and training efficiency with shared prefixes. We compared the original independent-prefix implementation with the shared-prefix variant under otherwise matched experimental settings. The reference model remains frozen in both variants.

Benchmark / metricIndependent prefixesShared prefixes
Video-MME66.366.1
MLVU71.972.0
LVBench48.848.6
LongVideoBench61.961.6
Video-MMMU53.153.2
MMVU68.067.7
Video-Holmes48.047.8
Training throughput (items/s) ↑0.02420.0097

These results indicate:

(a) Comparable benchmark performance. The differences between the two variants remain within 0.3 percentage points across all seven benchmarks, with no consistent performance advantage from sharing the initial reasoning prefix.

(b) Substantially higher training throughput. The independent-prefix implementation achieves 2.49× the measured training throughput of the shared-prefix variant.

Together, these comparisons support our decision to generate one independent counterfactual reference history per question and reuse it across the factual rollout group. This design preserves benchmark performance while amortizing reference generation across rollouts, making counterfactual feedback substantially more efficient to incorporate into training.

AuthorsOpenReview ↗

Response to Reviewer jy3K (Part 2/2)

Q1: “… it would be helpful to include more controlled experiments that isolate the effect of counterfactual memory, as well as comparisons with alternative contrastive reward designs.”

Response: We appreciate your suggestion to examine counterfactual memory and alternative contrastive reward designs. Following your recommendation, we report the counterfactual-memory control in Section 5.5 (Table 6) and compare CCRL with a direct score-difference reward baseline in Appendix C.3 (Table 11).

Specifically, this alternative incorporates the factual–counterfactual score difference into the task reward before GRPO normalization:

Rnidiff=Rnitask+αeni[sni+(ωold)−bni−],R^{\mathrm{diff}}_{ni} = R^{\mathrm{task}}_{ni} + \alpha e_{ni} \left[ s^+_{ni}(\omega_{\mathrm{old}}) - b^-_{ni} \right],

where enie_{ni} denotes the eligibility indicator for a correct, well-formed factual rollout and a valid reference prefix. We then apply standard GRPO group normalization to RnidiffR^{\mathrm{diff}}_{ni}, without adding CCRL’s answer-token credit.

BenchmarkGRPOScore-diff. rewardCCRL
Video-MME64.764.866.3
MLVU68.268.071.9
LVBench44.244.348.8
LongVideoBench57.457.161.9
Video-MMMU50.650.453.1
MMVU62.762.968.0
Video-Holmes38.138.348.0

We observe that direct score-difference reward shaping performs similarly to standard GRPO. In comparison, CCRL consistently improves performance across all seven benchmarks, outperforming this alternative by 4.5 points on LVBench, 5.1 points on MMVU, and 9.7 points on Video-Holmes.

Direct reward shaping subjects the factual–counterfactual score difference to GRPO group normalization. CCRL instead adds contrastive credit to final-answer tokens after task-advantage normalization, without further centering. For an eligible trajectory, this additional update implements descent on the contrastive objective at the behavior-policy parameters (Appendix G):

\left. \Delta\nabla_\omega J \right|_{\omega_{\mathrm{old}}} &= \left. \lambda_{\mathrm{cf}}w_{ni} \nabla_\omega s^+_{ni}(\omega) \right|_{\omega_{\mathrm{old}}} = -\left. \lambda_{\mathrm{cf}} \nabla_\omega\ell^{\mathrm{cf}}_{ni}(\omega) \right|_{\omega_{\mathrm{old}}}. \end{aligned}$$ This credit remains available even when equal task rewards make standard GRPO advantages vanish. --- > **W2: “The proposed method is only evaluated on Qwen2.5-VL-7B-Instruct. It remains unclear whether the memory-grounded latent reasoning framework and CCRL training strategy can generalize to other VLM architectures or larger model scales.”** **Response:** Thank you for encouraging us to evaluate MVLR beyond the original backbone. Following your suggestion, we extended our experiments to InternVL2.5-26B in **Appendix C.6 (Table 15)**, covering both a different VLM architecture and a larger model scale. The results are summarized below. | Benchmark | Qwen2.5-VL-7B | **Qwen2.5-VL-7B + MVLR** | InternVL2.5-26B | **InternVL2.5-26B + MVLR** | | --- | :---: | :---: | :---: | :---: | | Video-MME | 65.1 | **66.3** | 66.9 | **67.8** | | MLVU | 70.2 | **71.9** | 72.3 | **73.6** | | LVBench | 45.3 | **48.8** | 46.1 | **49.2** | | LongVideoBench | 56.0 | **61.9** | 59.9 | **63.2** | | Video-MMMU | 47.4 | **53.1** | 48.2 | **54.9** | | MMVU | 61.3 | **68.0** | 61.7 | **68.7** | | Video-Holmes | 34.7 | **48.0** | 35.9 | **48.8** | We observe that MVLR improves InternVL2.5-26B across all seven benchmarks, supporting its applicability beyond the original backbone, including at a larger model scale. The largest gains are on complex video reasoning benchmarks: +6.7 points on Video-MMMU, +7.0 points on MMVU, and +12.9 points on Video-Holmes. --- We sincerely thank you for your valuable suggestions, which have helped us refine and strengthen the paper. If you have any additional questions or concerns, please do not hesitate to let us know.
AuthorsOpenReview ↗

Please Feel Free to Share Any Further Questions

Dear Reviewer jy3K,

Thank you again for your thoughtful comments and constructive suggestions, which have guided our revisions and additional experiments. As the discussion period draws to a close, we remain available to provide any further clarification you may find helpful. If you have any remaining questions or concerns about our response, please feel free to let us know.

We sincerely appreciate the time and care you have devoted to reviewing our work.

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response

Dear AC and Reviewers,

We sincerely thank you for your careful evaluations and constructive feedback, which have helped us strengthen the experimental validation and improve the clarity of our work. In response to your suggestions, we have carefully revised the manuscript and highlighted all additions in red for your convenience.

In addition to addressing each reviewer’s comments point-by-point below, we would like to summarize the recognized strengths of our work and highlight the additional experiments and analyses incorporated during the rebuttal phase.


We are pleased that the reviewers recognized the following strengths and contributions:

  1. “I appreciate the motivation… and its emphasis… The preliminary analysis provides useful insights…proposed framework is an interesting and intuitive design… provides relatively comprehensive experiments… component ablations support the effectiveness” [Reviewer jy3K]
  2. “I like that later latent steps can still retrieve evidence outside the current focus without encoding the video again… MVLR improves accuracy… getting it together with faster inference is useful” [Reviewer jgYw]
  3. “A coherent solution… is well motivated… Broad evaluation with useful interventions… support the practical importance of learned latent content and temporal evidence selection.” [Reviewer txSD]

Following the reviewers’ suggestions, we have incorporated the following experiments and analyses:

Controlled Evaluation of CCRL and Reference Design (Reviewers jy3K, jgYw, and txSD)

We added controlled comparisons in Section 5.5 and Appendices C.3–C.4, covering constant and shuffled weights, direct score-difference reward, a frozen reference using factual memory, an updated reference using the rollout-policy parameters, and shared initial reasoning prefixes. CCRL consistently outperforms controls that alter its weights, reward formulation, reference memory, or reference parameters across all seven benchmarks, while independent prefixes preserve comparable accuracy and achieve 2.49× the training throughput of shared prefixes. Together, these results support the effectiveness of contrastive credit, donor-memory substitution, and frozen reference.

Controlled Comparisons of Latent Reasoning and Training Data (Reviewers jgYw and txSD)

We added controlled comparisons in Section 5.5 (Table 5) and Appendix C.5 against AVMC + Text-CoT, textual Qwen2.5-VL, Video-o3, and vanilla video latent reasoning, using the same training data and matched visual budgets. AVMC + Text-CoT additionally retains the same adaptive memory and training objectives while replacing recurrent continuous latent segments with textual reasoning. MVLR outperforms these baselines across all seven benchmarks and achieves 1.90× and 3.14× the inference throughput of AVMC + Text-CoT and Video-o3, respectively. Together, these results support the contribution of recurrent latent reasoning beyond the benefits of adaptive visual memory and additional training data.

Generalization across Backbones and Model Scales (Reviewer jy3K)

We extended MVLR to InternVL2.5-26B in Appendix C.6, covering a different VLM architecture and a larger model scale. MVLR improves performance across all seven benchmarks, including 12.9 points on Video-Holmes, supporting its applicability to a different VLM architecture and a larger model scale.

Implementation Details and Inference Efficiency (Reviewer txSD)

In Appendix A.7, we clarified the hardware, inference batch size, decoding limits, and visual budgets for Table 4. Feature caching and memory construction take 1.87 s and 0.64 s per question, respectively, both included in end-to-end throughput. Feature caching accounts for visual feature extraction already required for inference, while memory construction reuses cached features without re-encoding video frames.


We provide detailed responses to each reviewer’s comments and questions below and would be happy to address any further questions or concerns.

Best,
Authors

AuthorsOpenReview ↗

Summary of Reviewer Feedback and Rebuttal

Dear Area Chairs and Program Chairs,

Thank you for overseeing the review of our submission. We have addressed all concerns raised by the three reviewers, with every revision highlighted in red in the manuscript. All three reviewers recommend acceptance (Rating 3: Weak acceptance). Below, we summarize how each concern was resolved:

✅ 1. Reviewers jy3K/jgYw/txSD: CCRL Effectiveness — Addressed

  • Concern: whether CCRL's gains come from counterfactual visual grounding rather than extra correct-answer credit, the frozen reference, or the reference's separate reasoning prefix.
  • Resolution: controlled comparisons isolating each factor (Section 5.5; Appendices C.3–C.4). CCRL outperforms all alternative weighting, reward, and reference designs on all seven benchmarks.

✅ 2. Reviewers jgYw/txSD: Gains beyond Adaptive Memory and Training Data — Addressed

  • Concern: whether the gains come from recurrent latent reasoning, rather than from better visual-evidence access or additional training data.
  • Resolution: baselines retrained with matched training data and visual budgets, including a text-CoT variant with the same adaptive memory (Section 5.5; Appendix C.5). MVLR outperforms all of them on all seven benchmarks while running faster than the text-CoT variant and Video-o3.

✅ 3. Reviewer jy3K: Generalization to Other Backbones and Scales — Addressed

  • Concern: MVLR was evaluated only on Qwen2.5-VL-7B.
  • Resolution: MVLR also improves InternVL2.5-26B, a different architecture at a larger scale, on all seven benchmarks (Appendix C.6).

✅ 4. Reviewer txSD: Implementation Details — Addressed

  • Concern: missing implementation details behind the efficiency results (Table 4), including the cost of feature caching and memory construction.
  • Resolution: all settings and a per-question timing breakdown are now reported in Appendix A.7; both costs are included in the end-to-end throughput.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.