acceptodds
Desk rejected at ICLR 2027

VimPro: Long Video Understanding via Proxy-Guided Agentic Reasoning over Hierarchical Memory

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Long-video understanding remains challenging due to lengthy temporal contexts and sparsely distributed question-relevant evidence. Although memory-augmented methods organize long videos into structured memories to preserve and retrieve long-range information, they still face two major challenges. First, they use a static memory as the sole inference context, limiting their ability to adaptively refine or supplement question-specific evidence. Second, similarity-based retrieval may return semantically related yet answer-irrelevant or misleading memories, whereas LLM-based agentic retrieval incurs high inference cost and is difficult to control. To overcome these limitations, we propose VimPro, a proxy-guided framework that constructs and dynamically maintains a hierarchical memory for long-video understanding. During inference, the answer agent reasons over the evolving memory and requests supplementary evidence when needed, while a lightweight proxy, MemLens, filters noisy candidates into compact answer-supporting evidence. MemLens is trained through cold-start supervised fine-tuning followed by answer-gated reinforcement learning with tailored, fine-grained rewards. Extensive experiments demonstrate that VimPro consistently outperforms both closed-source and open-source SOTA methods. All datasets, code, and trained models will be publicly released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-form video understanding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_5dvuOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents VimPro, a proxy-guided reasoning framework for long-video question answering. It constructs episodic memory at multiple temporal scales offline and maintains question-specific evidence memory and working memory during inference. Its core component, MemLens, is based on Qwen3-VL-4B and trained through cold-start supervised fine-tuning followed by answer-gated GRPO reinforcement learning to select answer-supporting evidence from coarsely retrieved candidates. The answer agent reasons over the current memory and invokes textual retrieval or localized visual inspection tools as needed to gather additional evidence. The authors report strong accuracy on Video-MME, LongVideoBench, and LVBench. Ablations on memory components, MemLens, multi-turn reasoning, and training stages support the effectiveness of the overall approach. They also compare runtime and token consumption against two memory-based agents.

Critical strengths

1.The paper addresses the gap between semantic relevance and usefulness for answering by assigning fine-grained evidence selection to a lightweight, specifically trained model. This division of responsibilities helps reduce irrelevant or misleading context passed to the answer agent. With the candidate set and answer agent held fixed, adding MemLens improves accuracy on Video-MME Long from 76.6% to 81.0%, providing direct evidence for the value of this component. This is my main reason for supporting acceptance. 2.VimPro achieves strong results across three long-video QA benchmarks, and ablations on its components and training stages support the overall design. Comparisons with M3-Agent and WorldMM show potential reductions in runtime and token consumption. Combining lightweight evidence selection with a strong answer model offers a useful approach to building scalable long-video QA systems.

Critical weaknesses

1.VimPro relies on offline episodic memory and coarse retrieval to obtain evidence. Although visual inspection can recover details omitted from textual summaries, it requires the relevant time span to have already been located. If missing information prevents coarse retrieval from finding the supporting segment, subsequent reasoning may still lack the necessary evidence. The system’s performance is therefore constrained by the quality of the offline memory and retrieval coverage. 2.The paper presents answer gating and fine-grained rewards as important elements of MemLens training. However, the ablations mainly compare Base, SFT, RL, and SFT+RL, without comparing the full reward scheme against versions that remove answer gating or use simpler rewards. These results support the effectiveness of two-stage training, but leave unclear how much additional benefit the proposed reward design provides beyond reinforcement learning more generally. This makes the core training contribution difficult to assess. 3.The main accuracy results use GPT-5, whereas the efficiency experiments in Table 2 use a locally deployed Qwen3-32B without reporting each method’s accuracy under that configuration. It is therefore unclear whether the reported efficiency gains come with a loss in accuracy. The results do not fully support the claim that VimPro achieves both high accuracy and low computational cost.

Questions for authors

1.Visual inspection can recover missing details, but still relies on textual memory to identify the relevant time spans. The limitations section acknowledges that failures in coarse retrieval can make a question unanswerable, but does not quantify their impact. Could the authors provide an error analysis showing how many incorrect answers result from omissions in the offline memory or failures to retrieve supporting segments, to assess how much this bottleneck limits overall performance? 2.Table 6 demonstrates the benefits of SFT and RL, but does not isolate the contribution of answer gating. Could the authors compare the full reward scheme with a version without answer gating, using the same training data and budget, to establish whether this design provides an additional performance gain? 3.Could the authors report the accuracy of VimPro and the baselines under the configuration used in Table 2, and compare accuracy, runtime, and token consumption using the same answer model configuration, to determine whether the efficiency gains come at the expense of accuracy?

4 comments

We genuinely appreciate your recognition of our contributions and your positive feedback. Thank you for the time you invested in reviewing our paper and for your insightful questions. In our response, we will address each question individually, quoting them and providing our answers accordingly.

Q1: VimPro relies on offline episodic memory and coarse retrieval to obtain evidence. Although visual inspection can recover details omitted from textual summaries, it requires the relevant time span to have already been located. If missing information prevents coarse retrieval from finding the supporting segment, subsequent reasoning may still lack the necessary evidence. The system’s performance is therefore constrained by the quality of the offline memory and retrieval coverage.

A1: Thank you for highlighting this limitation. We agree that VimPro depends on the quality of its offline episodic memory and the coverage of coarse retrieval. MemLens can select useful evidence from the retrieved candidates, but cannot recover supporting segments that never enter the candidate set.

VimPro mitigates this bottleneck through multi-scale episodic memory and iterative retrieval. Memories at different temporal scales provide complementary retrieval cues, while the answer agent can issue supplementary queries based on its evolving reasoning state. Once a relevant span is located, visual inspection can recover details omitted or inaccurately represented in the textual memory. However, these mechanisms do not guarantee recovery when the supporting span remains unlocated.

This limitation is acknowledged in Appendix F. We will clarify it in the main text and distinguish evidence-selection improvements from retrieval-coverage limitations, avoiding any implication that visual inspection can compensate for all failures of offline memory or coarse retrieval.

Q2: The paper presents answer gating and fine-grained rewards as important elements of MemLens training. However, the ablations mainly compare Base, SFT, RL, and SFT+RL, without comparing the full reward scheme against versions that remove answer gating or use simpler rewards. These results support the effectiveness of two-stage training, but leave unclear how much additional benefit the proposed reward design provides beyond reinforcement learning more generally. This makes the core training contribution difficult to assess.

A2: Thank you for raising this point. We ablate the composite GRPO reward by removing one component at a time and retraining MemLens, keeping all other training settings unchanged. All variants are evaluated under identical inference settings. We keep RformatR_{\mathrm{format}} fixed as a structural-validity constraint and focus our ablation on the components directly related to evidence selection and downstream answerability. Specifically, we examine: (i) w/o ReviR_{\mathrm{evi}}, removing the signed evidence-ranking reward; (ii) w/o RansR_{\mathrm{ans}}, removing the auxiliary-answer reward; (iii) w/o RcalibR_{\mathrm{calib}}, removing confidence calibration; and (iv) w/o answer gate, disabling answer-gated reward modulation by setting g=1g=1 while leaving the other reward components unchanged.

Table: Ablation on the GRPO reward components. “✓” indicates that the component is used; each row except the last removes one component from the full objective.

ReviR_{\mathrm{evi}}RansR_{\mathrm{ans}}RcalibR_{\mathrm{calib}}Answer GateVideo-MME Long (%)
✓✓✓74.3
✓✓✓77.5
✓✓✓79.6
✓✓✓78.2
✓✓✓✓81.0

Each component contributes to the overall performance. The evidence-ranking reward ReviR_{\mathrm{evi}} and auxiliary-answer reward RansR_{\mathrm{ans}} yield the largest improvements (6.7 and 3.5 points), as they encourage MemLens to prioritize answer-supporting evidence and distinguish among candidate options. Meanwhile, the answer gate (2.8 points) promotes evidence selection aligned with downstream answerability, while RcalibR_{\mathrm{calib}} (1.4 points) encourages reliable confidence estimation. By integrating these complementary signals, the full objective achieves the highest accuracy of 81.0% on Video-MME Long.

Q3: The main accuracy results use GPT-5, whereas the efficiency experiments in Table 2 use a locally deployed Qwen3-32B without reporting each method’s accuracy under that configuration. It is therefore unclear whether the reported efficiency gains come with a loss in accuracy. The results do not fully support the claim that VimPro achieves both high accuracy and low computational cost.

A3: Thank you for raising this concern. We agree that the original efficiency study, without accuracy measured under the same setting, was insufficient to establish the accuracy–cost trade-off.

First, the original efficiency experiment used Qwen3-32B because M3-Agent relies on a fine-tuned checkpoint of this model. Using the same Qwen3-32B for the other methods was intended to reduce model-related differences when comparing computational costs.

Second, as reviewer nvPC noted, Qwen3-32B is a text-only model. To jointly evaluate accuracy and efficiency with multimodal answer models, we conducted additional comparisons with WorldMM using Qwen3.5-27B and GPT-5.

To address the accuracy–cost concern directly, we have expanded the evaluation from 10 videos to 30 randomly sampled videos from Video-MME Long, averaging 41.5 minutes and covering 90 questions.

MethodAnswer ModelAccuracy (%) ↑End-to-End Time (min) ↓Token Consumption (M) ↓
WorldMMQwen3.5-27B64.4174858.2
VimProQwen3.5-27B77.873417.1
WorldMMGPT-568.9154759.5
VimProGPT-582.261417.6

Accuracy is measured over all 90 questions. Time and token consumption are aggregated over the complete evaluation, including video processing, memory construction, evidence retrieval, and final answering.

As shown in the table, VimPro consistently outperforms WorldMM in both accuracy and efficiency. Under both Qwen3.5-27B and GPT-5, VimPro achieves an accuracy improvement of approximately 13.3 percentage points. Meanwhile, it reduces end-to-end runtime by 58.0% and 60.3%, and token consumption by 70.6% and 70.4%, respectively. These consistent improvements demonstrate a favorable accuracy–efficiency trade-off for VimPro under both answer-model configurations.

Q4: Visual inspection can recover missing details, but still relies on textual memory to identify the relevant time spans. The limitations section acknowledges that failures in coarse retrieval can make a question unanswerable, but does not quantify their impact. Could the authors provide an error analysis showing how many incorrect answers result from omissions in the offline memory or failures to retrieve supporting segments, to assess how much this bottleneck limits overall performance?

A4: Thank you for this suggestion. We analyzed all 171 incorrect answers on Video-MME Long using teacher-assigned evidence utility scores, treating items with u ≥ 50 as supporting evidence.

We distinguish three cases: supporting evidence is absent from the candidate pool, present but not passed to the answer model, or passed to the answer model but followed by an incorrect answer.

Error categoryPercentage
Memory/retrieval miss66.7%
Evidence-selection miss9.9%
Answering failure23.4%

The largest category involves missing supporting evidence in the candidate pool, indicating that memory quality and retrieval coverage remain a major bottleneck. However, this analysis cannot distinguish information omitted during memory construction from supporting memories that exist but were not retrieved. The remaining errors involve evidence selection or incorrect answering despite receiving supporting evidence.

These categories are based on teacher-estimated utility rather than definitive causal attribution. We will include this analysis in the revised paper and clarify that MemLens improves selection among retrieved candidates but cannot guarantee recovery of evidence absent from the candidate pool.

Q5: Table 6 demonstrates the benefits of SFT and RL, but does not isolate the contribution of answer gating. Could the authors compare the full reward scheme with a version without answer gating, using the same training data and budget, to establish whether this design provides an additional performance gain?

A5: Thank you for the suggestion. We have added an answer-gate ablation using the same training data and budget, as detailed in A2.

Q6: Could the authors report the accuracy of VimPro and the baselines under the configuration used in Table 2, and compare accuracy, runtime, and token consumption using the same answer model configuration, to determine whether the efficiency gains come at the expense of accuracy?

A6: Thank you for the suggestion. We have added joint accuracy, runtime, and token-consumption comparisons under the same answer-model configuration, as detailed in A3.

Thank you for your response. After considering your points, I have decided to keep my original score.

Reviewer_nvPCOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper focuses the problem of long-video understanding and proposes VimPro, a proxy-guided agentic framework built on a hierarchical memory architecture. It comprises a static episodic memory, a dynamically maintained evidence memory, and a working memory. A lightweight, option-aware proxy model that performs fine-grained evidence selection, which is trained via cold-start supervised fine-tuning (SFT) and reinforcement learning (GRPO). A reasoning agent that iteratively updates the hierarchical memory and invokes minimal multimodal tools (textual retrieval and visual inspection) when evidence is insufficient, before producing the final answer. VimPro is evaluated on three long-video QA benchmarks: Video-MME, LongVideoBench, and LVBench.

Critical strengths
  1. The paper identifies the bottleneck in long-video understanding and presents its decoupled, proxy-guided framework in a well-structured and easy-to-follow manner.
  2. Ablations demonstrate the effectiveness of the proposed hierarchical memory, MemLens, and the two-stage SFT + RL training strategy.
Critical weaknesses
  1. The main results (Table 1) rely on GPT-5 as the answer agent, which gives VimPro a significant advantage over baselines that use weaker or open-source models, making the reported gains potentially inflated rather than solely attributable to the proposed memory architecture.
  2. The reinforcement learning objective combines multiple hand-crafted components (format, auxiliary answer, signed NDCG evidence ranking, confidence calibration, and the answer gate) into a complex composite formula, yet no ablation isolates the contribution of each reward term, leaving it unclear which components are actually necessary.
  3. All benchmarks and the entire design assume a discrete option set, so the method's applicability to open-ended answering, temporal grounding, or dense captioning remains unverified.
  4. The efficiency evaluation (Table 2) swaps GPT-5 for a locally deployed Qwen3-32B answer model, but reports only runtime and token consumption without accuracy under the same setting, making the trade-off between speed and performance unclear. Moreover, Qwen3-32B is a text-only model, so it is unclear how it processes video content, which raises questions about whether the comparison truly measures video understanding.
  5. Some important baselines are missing. [1] MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism. arXiv preprint arXiv:2606.07512. [2] VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking. CVPR 2026.
  6. The multi-turn reasoning ablation (Table 5) only tests T=1,3,5, and accuracy is still improving at T=5, yet the authors stop there without exploring larger budgets or discussing the diminishing-returns point or latency cost.
  7. The qualitative comparison (Figure 9) does not show the actual video input frames, making it hard to intuitively verify the claims, and the paper only presents success cases while omitting failure cases of the proposed method, which weakens the credibility of the qualitative analysis.
Questions for authors

Please refer to the weaknesses.

5 comments

We sincerely thank you for your effort in reviewing our paper. We appreciate your recognition of the framework’s clear structure and the ablation evidence supporting the hierarchical memory, MemLens, and two-stage training strategy. Your constructive feedback helps us identify areas that require further clarification and validation. Below, we address each concern individually and present additional analyses and experiments.

Q1: The main results (Table 1) rely on GPT-5 as the answer agent, which gives VimPro a significant advantage over baselines that use weaker or open-source models, making the reported gains potentially inflated rather than solely attributable to the proposed memory architecture.

A1: Thank you for raising this concern. We agree that differences in answer-model capacity may affect cross-method comparisons. However, the improvements in Table 1 cannot be attributed solely to GPT-5. Specifically, we reproduce WorldMM using the same GPT-5 answer model as VimPro. Under this setting, VimPro outperforms WorldMM by 14.9, 11.6, and 22.1 percentage points on Video-MME Long, LongVideoBench Long, and LVBench, respectively.

To further isolate the influence of model capacity, we conduct additional comparisons with WorldMM, DVD, and VideoTree using Qwen3.5-27B as the shared reasoning backbone on the full benchmark splits.

MethodVideo-MME LongLongVideoBench LongLVBench
VideoTree63.354.350.3
DVD61.357.461.5
WorldMM58.757.648.7
VimPro77.666.363.6

Under this shared answer-model configuration, VimPro consistently achieves the highest accuracy across all three benchmarks, outperforming the strongest baseline by 14.3, 8.7, and 2.1 percentage points on Video-MME Long, LongVideoBench Long, and LVBench, respectively.

Furthermore, our MemLens ablation (Table 4) keeps the candidate set and answer agent fixed, yet improves Video-MME Long accuracy from 76.6% to 81.0%. This controlled ablation further confirms the effectiveness of learned evidence selection, independent of answer-model capacity.

Q2: The reinforcement learning objective combines multiple hand-crafted components (format, auxiliary answer, signed NDCG evidence ranking, confidence calibration, and the answer gate) into a complex composite formula, yet no ablation isolates the contribution of each reward term, leaving it unclear which components are actually necessary.

A2: Thank you for raising this point. We ablate the composite GRPO reward by removing one component at a time and retraining MemLens, keeping all other training settings unchanged. All variants are evaluated under identical inference settings. We keep RformatR_{\mathrm{format}} fixed as a structural-validity constraint and focus our ablation on the components directly related to evidence selection and downstream answerability. Specifically, we examine: (i) w/o ReviR_{\mathrm{evi}}, removing the signed evidence-ranking reward; (ii) w/o RansR_{\mathrm{ans}}, removing the auxiliary-answer reward; (iii) w/o RcalibR_{\mathrm{calib}}, removing confidence calibration; and (iv) w/o answer gate, disabling answer-gated reward modulation by setting g=1g=1 while leaving the other reward components unchanged.

Table: Ablation on the GRPO reward components. “✓” indicates that the component is used; each row except the last removes one component from the full objective.

ReviR_{\mathrm{evi}}RansR_{\mathrm{ans}}RcalibR_{\mathrm{calib}}Answer GateVideo-MME Long (%)
✓✓✓74.3
✓✓✓77.5
✓✓✓79.6
✓✓✓78.2
✓✓✓✓81.0

Each component contributes to the overall performance. The evidence-ranking reward ReviR_{\mathrm{evi}} and auxiliary-answer reward RansR_{\mathrm{ans}} yield the largest improvements (6.7 and 3.5 points), as they encourage MemLens to prioritize answer-supporting evidence and distinguish among candidate options. Meanwhile, the answer gate (2.8 points) promotes evidence selection aligned with downstream answerability, while RcalibR_{\mathrm{calib}} (1.4 points) encourages reliable confidence estimation. By integrating these complementary signals, the full objective achieves the highest accuracy of 81.0% on Video-MME Long.

Q3: All benchmarks and the entire design assume a discrete option set, so the method's applicability to open-ended answering, temporal grounding, or dense captioning remains unverified. A3: To evaluate the generalization of VimPro beyond multiple-choice QA, we conduct additional experiments on open-ended question answering and question-conditioned temporal grounding using CG-Bench. Importantly, we directly reuse the original MemLens trained on multiple-choice QA, without any additional training or parameter updates. We only adapt the inference prompt to accommodate the new task formats, while keeping the learned evidence-selection policy unchanged.

We randomly sample 100 videos and select one question per video for both tasks. We compare VimPro against direct Qwen3.5-27B inference, using Qwen3.5-27B as VimPro's reasoning model. Both methods are evaluated on identical samples under the same evaluation protocols.

TaskBenchmarkMetricQwen3.5-27BVimPro (Qwen3.5-27B)
Open-ended QACG-BenchAccuracy (%) ↑30.060.0
Temporal groundingCG-BenchmIoU (%) ↑0.0515.10

As shown in the table, VimPro improves open-ended QA accuracy by 30.0 percentage points and temporal grounding mIoU by 15.05 percentage points over direct Qwen3.5-27B inference. These results provide preliminary evidence of VimPro's generalizability and transferability beyond multiple-choice QA, particularly to open-ended answering and question-conditioned temporal grounding.

Q4: The efficiency evaluation (Table 2) swaps GPT-5 for a locally deployed Qwen3-32B answer model, but reports only runtime and token consumption without accuracy under the same setting, making the trade-off between speed and performance unclear. Moreover, Qwen3-32B is a text-only model, so it is unclear how it processes video content, which raises questions about whether the comparison truly measures video understanding.

A4: Thank you for raising this concern. We agree that the original efficiency study, without accuracy measured under the same setting, was insufficient to establish the accuracy–cost trade-off.

First, we clarify that Qwen3-32B did not directly process raw video frames. It reasoned over textual memories generated from the videos by the visual captioning model. Thus, the answer model operated on video-derived textual evidence, and this setting did not fully evaluate direct multimodal reasoning.

Second, we used Qwen3-32B in the original efficiency study because M3-Agent relies on a fine-tuned checkpoint of this model. Using Qwen3-32B-based answer models for the other methods was intended to reduce model-related differences in the computational cost comparison.

To address the accuracy–cost concern directly, we have expanded the evaluation from 10 videos to 30 randomly sampled videos from Video-MME Long, averaging 41.5 minutes and covering 90 questions. We additionally compare VimPro with WorldMM using Qwen3.5-27B and GPT-5 as multimodal answer models, reporting both accuracy and efficiency.

MethodAnswer ModelAccuracy (%) ↑End-to-End Time (min) ↓Token Consumption (M) ↓
WorldMMQwen3.5-27B64.4174858.2
VimProQwen3.5-27B77.873417.1
WorldMMGPT-568.9154759.5
VimProGPT-582.261417.6

Accuracy is measured over all 90 questions. Time and token consumption are aggregated over the complete evaluation, including video processing, memory construction, evidence retrieval, and final answering.

As shown in the table, VimPro consistently outperforms WorldMM in both accuracy and efficiency. Under both Qwen3.5-27B and GPT-5, VimPro achieves an accuracy improvement of approximately 13.3 percentage points. Meanwhile, it reduces end-to-end runtime by 58.0% and 60.3%, and token consumption by 70.6% and 70.4%, respectively. These consistent improvements demonstrate a favorable accuracy–efficiency trade-off for VimPro under both answer-model configurations.

Q5: Some important baselines are missing. [1] MemDreamer: Decoupling Perception and Reasoning for Long Video Understanding via Hierarchical Graph Memory and Agentic Retrieval Mechanism. arXiv preprint arXiv:2606.07512. [2] VideoSeek: Long-Horizon Video Agent with Tool-Guided Seeking. CVPR 2026.

A5: Thank you for pointing out these important baselines. We have further examined their methodological differences and reported results, and will expand the related-work discussion and experimental comparisons accordingly.

MemDreamer. We acknowledge that MemDreamer is a closely related and strong baseline. However, its official implementation was not publicly accessible when we prepared this response, preventing a controlled reproduction under our experimental settings. We therefore provide its reported results from the original paper for reference.

MethodReasoning ModelVideo-MME LongLongVideoBench LongLVBench
MemDreamerGemini-3.1-Pro92.192.990.7
MemDreamerQwen3-VL-235B-A22B-Thinking86.283.284.8
VimProGPT-581.073.474.7

These results are not directly comparable, as MemDreamer constructs hierarchical graph memory using Gemini-3.1-Pro and allows up to 12 tool calls, whereas VimPro uses Qwen3.5-27B for episodic memory construction and GPT-5 with a five-round reasoning budget. The two methods also differ in their memory representations and retrieval mechanisms.

Methodologically, while MemDreamer relies on agentic exploration over hierarchical graph memory, VimPro introduces a separately trained lightweight proxy, MemLens, to learn answer-oriented evidence selection and decouple fine-grained evidence filtering from answer reasoning. We will clarify these distinctions and include the reported results with their corresponding configurations in the paper. We also plan to conduct a controlled comparison once the official implementation becomes available.

VideoSeek. VideoSeek also uses GPT-5, making its reported results particularly relevant to our main evaluation. We have added the following comparison, with VideoSeek results taken from its original paper.

MethodVideo-MME LongLongVideoBench LongLVBench
VideoSeek70.173.568.4
VimPro81.073.474.7

VimPro achieves 10.9 and 6.3 percentage-point higher accuracy on Video-MME Long and LVBench, respectively, while obtaining a comparable score on LongVideoBench Long (73.4% vs. 73.5%).

Both methods use GPT-5, although their remaining pipeline configurations may differ. We will specify the evaluation splits and configurations in the revised paper to distinguish this comparison of reported results from a controlled reproduction.

Q6: The multi-turn reasoning ablation (Table 5) only tests T=1,3,5, and accuracy is still improving at T=5, yet the authors stop there without exploring larger budgets or discussing the diminishing-returns point or latency cost. A6: Thank you for this suggestion. To further investigate the effect of larger reasoning budgets, we extend the maximum number of reasoning rounds from T=5 to T=7 and T=10, keeping all other configurations unchanged.

Maximum Budget TVideo-MME LongLongVideoBench Long
179.370.4
379.673.0
581.073.4
781.474.1
1083.475.7

Increasing T from 1 to 5 improves accuracy by 1.7 and 3.0 percentage points on Video-MME Long and LongVideoBench Long, respectively. Further increasing T from 5 to 10 yields additional gains of 2.4 and 2.2 percentage points, indicating that performance has not yet saturated within the evaluated range. We retain T=5 as the default setting used in our main experiments, while larger reasoning budgets offer further accuracy improvements.

Reviewer_k2tcOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

The paper proposes VimPro for long-video understanding. It builds an episodic memory at several temporal scales, while an answer agent retrieves additional textual or visual information as needed. A lightweight model, MemLens, selects question-relevant evidence from the retrieved candidates to maintain an evolving evidence memory. MemLens is trained with supervised fine-tuning followed by reinforcement learning using answer-guided rewards. The authors evaluate VimPro on Video-MME, LongVideoBench, and LVBench, and provide component ablations and a runtime comparison.

Critical strengths
  1. Paper is well written and easy to follow.
  2. VimPro has lower latency and token consumption compared to M3-Agent and WorldMM.
Critical weaknesses
  1. The method appears to lack novelty, as its hierarchical memory design is similar to that of VideoARM [1].
  2. The experiments do not include a comparison with VideoARM, a strong baseline closely related to the proposed method. The VideoARM paper also reports results on Video-MME, LongVideoBench, and LVBench, making a direct comparison particularly relevant.
  3. During GRPO training, VIMPRO introduces multiple reward components and a final-answer gate, but the paper does not provide sufficient ablation studies to assess their individual contributions.
  4. The efficiency and generalization claims are broader than the evidence. The efficiency study covers only 10 videos and 30 questions, and uses a different answer model from the main accuracy evaluation. Without accuracy reported under the efficiency setting, the accuracy–cost trade-off is difficult to assess.

[1] Videoarm: Agentic reasoning over hierarchical memory for long-form video understanding. CVPR 2026.

Questions for authors
  1. The authors should compare their method with strong baselines that use similar approaches, particularly VideoARM.
  2. Can authors conduct ablation studies on the proposed reward components to clarify their individual contributions?
4 comments

We sincerely thank you for your effort in reviewing our paper. We greatly appreciate your valuable comments and insightful feedback. We also appreciate your positive assessment of the paper’s clarity and the lower latency and token consumption of VimPro compared with M3-Agent and WorldMM. In our response, we will address each question individually, quoting them and providing our answer accordingly.

Q1: The method appears to lack novelty, as its hierarchical memory design is similar to that of VideoARM [1].

A1: Thank you for raising this concern. The core methodological contribution of VimPro is a proxy-guided framework that separates persistent video-semantic storage, question-specific evidence selection, and answer reasoning through two interconnected designs:

  1. Hybrid static–dynamic memory. VimPro combines persistent, offline episodic memory with dynamically maintained evidence and working memories, separating reusable video-semantic storage from question-specific evidence maintenance and reasoning.

  2. Lightweight, learned evidence selection. VimPro explicitly formulates evidence selection as a learnable task performed by MemLens. Using purpose-built training data, evidence-utility supervision, and downstream answering feedback, MemLens learns to select answer-supporting evidence. With the candidate set and answer agent fixed, adding MemLens improves Video-MME Long accuracy from 76.6% to 81.0% (Table 4).

Unlike VideoARM’s dynamic memory construction during reasoning, VimPro maintains an offline episodic memory that can be reused across questions, while dynamically adapting evidence to each query. This enables reuse of video preprocessing without sacrificing question-specific evidence updates. Furthermore, whereas VideoARM emphasizes controller-driven exploration and evidence accumulation, VimPro explicitly learns an answer-oriented evidence-selection policy through MemLens. Our central novelty therefore lies not in hierarchical memory alone, but in this lightweight, learned selection mechanism, which decouples fine-grained evidence filtering from expensive answer reasoning and provides the answer agent with compact, filtered evidence.

Q2: The experiments do not include a comparison with VideoARM, a strong baseline closely related to the proposed method. The VideoARM paper also reports results on Video-MME, LongVideoBench, and LVBench, making a direct comparison particularly relevant.

A2: Thank you for this suggestion. We agree that VideoARM is a closely related and important baseline. We have conducted additional comparisons on Video-MME Long, LongVideoBench Long, and LVBench. To ensure a fair and controlled comparison, we evaluate both methods using Qwen3.5-27B as the controller/answer model on identical benchmark splits.

MethodVideo-MME LongLongVideoBench LongLVBench
VideoARM73.057.356.2
VimPro77.666.363.6

Under this shared controller/answer-model configuration, VimPro outperforms VideoARM by 4.6, 9.0, and 7.4 percentage points on Video-MME Long, LongVideoBench Long, and LVBench, respectively.

We will cite VideoARM in the main text and include the additional comparison and implementation details in the revised paper.

Q3: During GRPO training, VIMPRO introduces multiple reward components and a final-answer gate, but the paper does not provide sufficient ablation studies to assess their individual contributions.

A3: Thank you for raising this point. We ablate the composite GRPO reward by removing one component at a time and retraining MemLens, keeping all other training settings unchanged. All variants are evaluated under identical inference settings. We keep RformatR_{\mathrm{format}} fixed as a structural-validity constraint and focus our ablation on the components directly related to evidence selection and downstream answerability. Specifically, we examine: (i) w/o ReviR_{\mathrm{evi}}, removing the signed evidence-ranking reward; (ii) w/o RansR_{\mathrm{ans}}, removing the auxiliary-answer reward; (iii) w/o RcalibR_{\mathrm{calib}}, removing confidence calibration; and (iv) w/o answer gate, disabling answer-gated reward modulation by setting g=1g=1 while leaving the other reward components unchanged.

Table: Ablation on the GRPO reward components. “✓” indicates that the component is used; each row except the last removes one component from the full objective.

ReviR_{\mathrm{evi}}RansR_{\mathrm{ans}}RcalibR_{\mathrm{calib}}Answer GateVideo-MME Long (%)
✓✓✓74.3
✓✓✓77.5
✓✓✓79.6
✓✓✓78.2
✓✓✓✓81.0

Each component contributes to the overall performance. The evidence-ranking reward ReviR_{\mathrm{evi}} and auxiliary-answer reward RansR_{\mathrm{ans}} yield the largest improvements (6.7 and 3.5 points), as they encourage MemLens to prioritize answer-supporting evidence and distinguish among candidate options. Meanwhile, the answer gate (2.8 points) promotes evidence selection aligned with downstream answerability, while RcalibR_{\mathrm{calib}} (1.4 points) encourages reliable confidence estimation. By integrating these complementary signals, the full objective achieves the highest accuracy of 81.0% on Video-MME Long.

Q4: The efficiency and generalization claims are broader than the evidence. The efficiency study covers only 10 videos and 30 questions, and uses a different answer model from the main accuracy evaluation. Without accuracy reported under the efficiency setting, the accuracy–cost trade-off is difficult to assess.

A4: Thanks for bringing this to our attention. We agree that the original small-scale study, without accuracy measured under the same setting, was insufficient to establish the accuracy–cost trade-off.

First, we have expanded the evaluation from 10 videos to 30 randomly sampled videos from Video-MME Long, with an average duration of 41.5 minutes and a total of 90 questions.

Second, the original efficiency experiment used Qwen3-32B because M3-Agent relies on a fine-tuned checkpoint of this model. Using the same Qwen3-32B for the other methods was intended to reduce model-related differences when comparing computational costs.

Third, as reviewer nvPC noted, Qwen3-32B is a text-only model. To jointly evaluate accuracy and efficiency with multimodal answer models, we conducted additional comparisons with WorldMM using Qwen3.5-27B and GPT-5.

MethodAnswer ModelAccuracy (%) ↑End-to-End Time (min) ↓Token Consumption (M) ↓
WorldMMQwen3.5-27B64.4174858.2
VimProQwen3.5-27B77.873417.1
WorldMMGPT-568.9154759.5
VimProGPT-582.261417.6

Accuracy is measured over all 90 questions. Time and token consumption are aggregated over the complete evaluation, including video processing, memory construction, evidence retrieval, and final answering.

As shown in the table, VimPro consistently outperforms WorldMM in both accuracy and efficiency. Under both Qwen3.5-27B and GPT-5, VimPro achieves an accuracy improvement of approximately 13.3 percentage points. Meanwhile, it reduces end-to-end runtime by 58.0% and 60.3%, and token consumption by 70.6% and 70.4%, respectively. These consistent improvements demonstrate a favorable accuracy–efficiency trade-off for VimPro under both answer-model configurations.

Q5: The authors should compare their method with strong baselines that use similar approaches, particularly VideoARM.

A5: Thank you for the suggestion. We have added comparisons with VideoARM across all three benchmarks, as detailed in A2.

Q6: Can authors conduct ablation studies on the proposed reward components to clarify their individual contributions?

A6: Thank you for the suggestion. We have added reward-component and answer-gate ablations, as detailed in A3.

I thank the authors for their response. The additional experiments have addressed my concerns regarding the ablation studies; however, my reservations regarding the paper's novelty remain. Accordingly, I will maintain my original score.

Thank you for the follow-up and for acknowledging that the additional ablations address your concern.

However, the remaining concern to VimPro’s novelty is without basis.

First, VimPro’s memory architecture differs from VideoARM’s in how video content is stored and how question-specific evidence is maintained.

  1. Episodic memory. VimPro constructs a persistent, multi-scale representation of the video offline and reuses it across questions. VideoARM’s sensory memory holds frames and clips from the temporal regions currently being explored. These memories differ in their stored representations, construction procedures, and intended reuse.
  2. Evidence memory. VimPro maintains question-specific evidence selected by a separately trained model, MemLens. Retrieved candidates undergo learned filtering before entering this memory, allowing semantically relevant but misleading items to be excluded. VideoARM’s result memory records tool outputs and their temporal intervals as reasoning proceeds. The distinction concerns the mechanism that determines which evidence reaches the answer agent.
  3. Working memory. The two methods differ in how this memory is initialized and updated. VideoARM initializes working memory as empty and updates it by appending the controller’s reasoning trace at each iteration. VimPro initializes working memory using the question, answer options, and initial evidence selected by MemLens. After each retrieval step, it updates this state using the agent’s output, the tool request, and the updated evidence memory. VimPro’s working-memory update therefore explicitly depends on the evidence retained by MemLens, connecting learned evidence selection to subsequent retrieval and answer reasoning.

Second, the shared use of hierarchical memory and working memory is insufficient to establish a lack of novelty. Working memory dates back to Miller, Galanter, and Pribram (1960), and hierarchical memory had already been applied to long-video QA by Wang et al. (Hierarchical Memory for Long Video QA, 2024). These are established design principles. The novelty assessment must therefore examine the specific mechanisms for constructing and updating memory, particularly VimPro’s learned policy for selecting which retrieved evidence reaches the answer agent.

Finally, VimPro’s core methodological contribution is a proxy-guided framework built on two interconnected designs.

  1. Hybrid static–dynamic memory. VimPro combines persistent, offline episodic memory with dynamically maintained evidence and working memories, separating reusable video-semantic storage from question-specific evidence maintenance and reasoning.
  2. Lightweight, learned evidence selection. VimPro explicitly formulates evidence filtering as a learnable task performed by MemLens. Through purpose-built training data, evidence-utility supervision, and downstream answering feedback, MemLens learns to select answer-supporting evidence and suppress misleading candidates. This separates fine-grained evidence filtering from expensive answer reasoning and provides the answer agent with compact, filtered evidence.

The contribution of this learned selection mechanism is supported by a controlled ablation. With the candidate set and answer agent held fixed, adding MemLens improves Video-MME Long accuracy from 76.6% to 81.0% (Table 4).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.