acceptodds
Under review as a conference paper at ICLR 2027

VAMR: Multi-Question Agentic Reasoning for Efficient Long-Form Video Understanding

Abstract

Long-form video understanding often involves multiple questions about different aspects of the same recording. Yet existing video agents typically process each question through an isolated tool-use trajectory. This repeatedly restarts video exploration and memory construction, missing opportunities to acquire evidence jointly and progressively build a shared understanding that supports the complete question set. We introduce VAMR (Video Agent for Multi-Question Reasoning), which coordinates all questions about a video through one shared tool-use trajectory. At each round, a persistent policy model can invoke tools for one or more unresolved questions and submit answers for questions with sufficient evidence. Question-conditioned visual perception retrieves fine-grained clues for several questions in one call, while layered multi-question memory integrates reusable context into a shared video story and preserves separate evidence for individual questions. After supervised fine-tuning initializes this interaction protocol, we propose question-horizon policy optimization (\qhpo) to optimize shared trajectories in which questions progress and finish at different rounds. Specifically, a question-level critic estimates the value of each active question, while round alignment maps each question advantage to the rounds that directly serve it before the aligned advantages are aggregated to optimize the shared actor. Across LVBench, Video-Holmes, and LongVideoBench, VAMR achieves the highest accuracy overall and the fewest reasoning rounds among iterative methods. On LVBench, it reaches 62.1% accuracy, exceeding VideoARM by 4.3 points while reducing reasoning rounds and processed frames by 85.9% and 61.4%.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Long-form Video Understanding, Video Agents, Multimodal Reasoning, Reinforcement Learning

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 4 reviews, above 85% of the others.
Reviewer_hDM3OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work addresses the efficiency challenge of multi-question reasoning in long-form video understanding. Existing video agents typically process each question about the same video through separate retrieval, tool-calling, and memory-construction procedures, resulting in redundant computation and failing to fully exploit the visual evidence shared across different questions.

The paper propose VAMR (Video Agent for Multi-Question Reasoning), which organizes all questions about a video into one persistent agentic reasoning trajectory. Through a shared active question window, question-conditioned visual perception, and a layered multi-question memory structure, VAMR enables coordinated retrieval and asynchronous answering across questions. Furthermore, the paper introduces Question-Horizon Policy Optimization (QHPO), which uses a question-level critic to estimate the value of each question and applies round alignment to map question-level advantage signals to the tool calls and answer rounds that directly serve each question. Experiments across multiple benchmarks demonstrate that VAMR achieves state-of-the-art accuracy while significantly reducing the number of reasoning rounds.

Critical strengths
  1. Meaningful and practically relevant problem. VAMR organizes multiple questions into a single shared online reasoning trajectory, allowing different questions to reuse already-acquired visual evidence and to be answered at different times. This directly addresses the common problem of repeated video exploration faced by existing video agents when a video is associated with multiple questions.

  2. Sensible system design for shared perception and layered memory. The QA-inspector simultaneously produces a general description of the video segment and question-specific evidence, while the layered memory maintains the shared video context separately from the evidence belonging to each individual question. This design mitigates two potential pitfalls: the redundant video understanding caused by fully independent reasoning on the one hand, and the inter-question evidence interference caused by a fully shared context on the other.

  3. QHPO is well matched to the asynchronous-completion structure of multi-question tasks. It estimates an independent value for each question, then filters advantages according to the correspondence between actions and questions, and finally produces an update signal for the shared actor.

Critical weaknesses

1.The credit-assignment assumption in QHPO may conflict with the core mechanism of shared memory. QHPO assigns a question-level advantage only to the rounds that directly serve that question. The underlying logic is that a round should receive a learning signal from question qiq_i only if its action explicitly targets qiq_i. However, the central contribution of VAMR is precisely cross-question evidence reuse. This raises a fundamental question: if the value of shared information depends not only on the currently served question but also on how future questions utilize that information, can "direct action attribution" correctly represent the true credit assignment?

2.The current efficiency metrics are insufficient to support conclusions about end-to-end computational efficiency. The paper primarily relies on two efficiency metrics:

  • Rounds/Q: the average number of controller rounds required per question.
  • Frames/Q: the average number of video frames processed by the visual model per question.

These metrics quantify the number of interaction rounds and part of the visual processing workload, but they do not fully represent the actual inference cost. In particular, when multiple questions are consolidated into the same control round, the reduction in Rounds/Q may partially stem from action aggregation rather than a proportional saving in real computation. The authors should therefore report end-to-end latency, model input/output tokens, GPU time, or monetary cost under the same hardware and model-serving conditions, and compare the cost curves across different question densities.

Questions for authors
  1. The two questions raised in "Critical Weaknesses" should be addressed first. Resolving these is essential for evaluating the contribution.

  2. The strong external visual model may confound module contribution. It is unclear whether accuracy gains come from the shared reasoning and policy optimization, or from extra reasoning by the external visual model under targeted prompting.

  3. Sensitivity to question ordering and active-window scheduling is insufficiently analyzed. VAMR keeps a fixed set of five active questions, with the rest entering the backlog in input order. Since the shared story is persistent, earlier retrievals can influence later questions, so performance may be order-sensitive.

  4. Practical deployment boundaries are under-discussed. The setting assumes all questions belong to one video, but in reality question count, arrival time, and repeated-query patterns vary. VAMR may not suit streaming or incremental question arrival.

3 comments
AuthorsOpenReview ↗

Response to hDM3 (1/2)

We thank the reviewer for the insightful and positive comments. We sincerely hope that our response could properly address your concerns.

W1

The credit-assignment assumption in QHPO may conflict with the core mechanism of shared memory.

QHPO's conservative credit assignment and shared memory are not contradictory. The former provides an effective policy optimization strategy, while the latter enables collaborative multi-question reasoning and persistent evidence reuse. QHPO does not explicitly propagate rewards from future backlog questions to earlier actions. This is because tool observations accumulate in the shared story and undergo repeated consolidation, making it difficult to disentangle each original action's contribution to subsequent decisions and final rewards. Indiscriminate credit propagation would therefore introduce substantial attribution noise. Importantly, the absence of explicit credit does not eliminate these indirect benefits, as acquired evidence remains available to subsequent questions through the shared story.

We examine these two mechanisms through complementary ablations on LVBench. VAMR w/o round alignment retains shared memory but assigns each question's advantage to all its active rounds, testing broader credit propagation. VAMR w/o shared story removes persistent shared memory while retaining the trained policy, joint actions, and individual clue layers. An additional Qwen3.5-9B model routes visual summaries only to explicitly targeted questions and ASR information to each active question's clue layer, preserving question-specific evidence.

MethodAcc.Rounds/QFrames/Q
VAMR w/o round alignment60.01.093.7
VAMR w/o shared story57.91.8186.7
VAMR62.10.984.9

Removing round alignment reduces accuracy by 2.1 points, supporting conservative attribution over indiscriminate propagation. Separately, removing shared memory reduces accuracy by 4.2 points, confirming the substantial benefits of cross-question evidence reuse. Together, these results demonstrate the effectiveness of QHPO's credit assignment strategy alongside persistent shared memory in VAMR.

W2

The current efficiency metrics are insufficient to support conclusions about end-to-end computational efficiency.

We additionally measure end-to-end latency, token consumption, and API cost under identical serving conditions. Compared with the trained independent baseline on LVBench, VAMR reduces these costs by 58.4%, 57.1%, and 59.0%, respectively, with detailed baseline comparisons provided in our response to Reviewer CN9d (W3).

We further examine cost scaling on the same 48-video, 768-question subset as our question-density experiment.

Questions/LoopAcc.Tokens/QLatency/QAPI Cost/Q
158.143.4k61.8s$0.0132
259.430.2k46.9s$0.0094
460.224.1k39.1s$0.0077
861.720.0k34.2s$0.0067
1662.818.8k31.6s$0.0061

Increasing question density from 1 to 16 reduces token consumption by 56.7%, latency by 48.9%, and API cost by 53.8%. These results confirm substantial end-to-end computational savings beyond reductions in amortized reasoning rounds.

AuthorsOpenReview ↗

Response to hDM3 (2/2)

Q2

It is unclear whether accuracy gains come from the shared reasoning and policy optimization, or from extra reasoning by the external visual model under targeted prompting.

The QA-inspector performs question-conditioned visual analysis of selected video segments, while the backbone controller governs global reasoning and coordination, including where to inspect, which questions to serve, how to coordinate evidence retrieval, and when to answer. To assess the dependence on external model strength, we replace Qwen3.7-Plus with Qwen3.5-9B as the QA-inspector, keeping the trained controller, tool interface, and inference settings unchanged.

QA-inspectorAcc.Rounds/QFrames/Q
Qwen3.5-9B60.81.090.7
Qwen3.7-Plus62.10.984.9

The modest 1.3-point accuracy difference indicates that stronger visual analysis provides additional benefits, but VAMR's performance is not heavily dependent on the stronger external model. This supports the robustness of our collaborative reasoning framework across different QA-inspector strengths.

Q3

Sensitivity to question ordering and active-window scheduling is insufficiently analyzed.

To examine sensitivity to question ordering, we evaluate VAMR on LVBench using the original, reversed, and randomly shuffled question orders, while keeping the trained model and inference settings unchanged. Question IDs are reassigned after each permutation to avoid potential ordering biases.

Question OrderAcc.Rounds/QFrames/Q
Original62.10.984.9
Reversed62.71.092.5
Random62.20.986.4

Accuracy remains stable across different question orders, while reasoning rounds and processed frames exhibit modest variations. These results suggest that VAMR is robust in accuracy to question ordering. Active-window sensitivity is further examined in our response to Reviewer 3YL3 (Q2).

Q4

The setting assumes all questions belong to one video, but in reality question count, arrival time, and repeated-query patterns vary.

We appreciate the reviewer's suggestion. VAMR's active window, backlog, and persistent shared memory provide a natural basis for handling varying question counts and incremental arrivals. New questions can enter the active window as slots become available, while repeated or related questions can reuse evidence accumulated in the shared story. Moreover, the iterative reasoning framework offers a potential extension to streaming video understanding, where newly available video segments can be incorporated in subsequent reasoning rounds.

These extensions would require adaptations to online scheduling and termination and have not yet been experimentally validated. We will clarify the scope of the current evaluation in the revised paper and leave systematic investigation of these settings to future work.

Thank you very much for the authors' response. I will keep my recommendation score for this work.

Reviewer_CN9dOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies a central problem in answering multiple questions over the same long video: independent QA loops repeatedly perform video exploration, memory construction, and tool calls, while separate questions cannot jointly determine what evidence to acquire next. VAMR reframes this task as a shared sequential decision process: an active window and backlog manage unresolved questions, joint actions let multiple questions call tools or submit answers in a shared state, QA-inspector performs question-conditioned retrieval, layered memory separates a reusable video-level story from question-specific evidence, and QHPO uses question-level values and round alignment to handle asynchronous termination. Results on three held-out benchmarks, comparisons between the shared and independent loops, and question density and evidence overlap analyses support accuracy and amortized-efficiency gains under the authors’ protocol. The strongest contribution is the reformulation around a shared state and joint actions, together with the cross-question coordination mechanism, not an unconditional claim of accuracy superiority; the paper should draw a clearer boundary between this mechanism, matched cross-method performance, and applicability to more general online video agents.

Critical strengths
  1. The paper formulates multi-question video reasoning as a shared sequential decision process rather than parallel independent QA loops. The active window, backlog, joint actions, and asynchronous termination make cross-question state reuse explicit and directly address redundant exploration.
  2. QHPO handles credit assignment in shared trajectories by aligning question-level advantages with the rounds that serve each question. This design is tied to the proposed formulation with a shared state and joint actions rather than being a generic PPO variant.
  3. Comparisons among Independent loops, the Shared loop, PPO, and QHPO, together with question density and evidence overlap analyses, support the respective roles of shared-loop efficiency, QHPO training, and cross-question evidence reuse.
Critical weaknesses
  1. The unified comparison does not isolate method effects from interface effects. The paper reports 62.1 for VAMR versus 57.8 for VideoARM on LVBench (+4.3), but Appendix G restricts answers, tool calls, and memory updates for reproduced baselines to the current target question; VideoARM is one such baseline, whereas VAMR can specify multiple question identifiers in a visual call and use joint actions. The +4.3 therefore reflects the asymmetric unified protocol and cannot directly establish general performance superiority.
  2. The relationship between the training and evaluation setups remains underspecified. SFT trajectories are constructed with annotated evidence intervals, while evaluation exposes the complete question set in advance. It is therefore unclear whether QHPO learns general evidence coordination or a policy specific to this offline, supervised protocol.
  3. The efficiency claims rely on incomplete proxy metrics. Rounds/Q and Frames/Q amortize a shared round across questions, even though one round may issue multiple tool calls; they omit text tokens, audio processing, language-model computation, latency, parallelism, and provider cost. Appendix G reports Model Calls/Q, but the reported 85.9% and 61.4% reductions still do not establish lower end-to-end inference cost.
Questions for authors

Please clarify whether the +4.3 LVBench gain should be interpreted only as a result under the unified protocol, given that reproduced baselines are restricted to the current target question while VAMR can use cross-question joint actions.

1 comment
AuthorsOpenReview ↗

Response to CN9d

We thank the reviewer for the insightful and positive comments. We sincerely hope that our response could properly address your concerns.

W1 & Q1

The unified comparison does not isolate method effects from interface effects.

Please clarify whether the +4.3 LVBench gain should be interpreted only as a result under the unified protocol, given that reproduced baselines are restricted to the current target question while VAMR can use cross-question joint actions.

We acknowledge that the reported +4.3-point gain is measured under our unified evaluation protocol. To further examine interface differences, we introduce VideoARM-MR, which explicitly adapts VideoARM to VAMR's proposed shared-loop reasoning paradigm rather than merely exposing additional questions to its original controller. Specifically, we adapt its reasoning loop to support active-question scheduling, joint tool actions, and cross-question evidence reuse through HM³ memory updates. VideoARM-MR retains VideoARM's original training-free controller, underlying tools, and HM³ memory architecture, without adopting VAMR's QA-inspector, layered memory, or QHPO.

MethodAcc.Rounds/QFrames/Q
VideoARM57.86.4220.2
VideoARM-MR58.22.6168.3
VAMR62.10.984.9

On LVBench, VideoARM-MR improves inference efficiency but achieves only a modest accuracy gain. VAMR still outperforms this adapted baseline by 3.9 points, with substantially fewer reasoning rounds and processed frames. The remaining gap suggests that the shared-loop paradigm benefits from VAMR's specialized QA-inspector, layered memory, and QHPO optimization, rather than shared-loop adaptation alone.

W2

The relationship between the training and evaluation setups remains underspecified. SFT trajectories are constructed with annotated evidence intervals, while evaluation exposes the complete question set in advance. It is therefore unclear whether QHPO learns general evidence coordination or a policy specific to this offline, supervised protocol.

Our two-stage SFT design prevents privileged annotations from entering reasoning generation. Annotated evidence intervals and gold answers guide action selection in Stage 1, while Stage 2 generates reasoning solely from public states and selected actions. Although these annotations shape the supervised trajectories, they are not exposed to the student policy or subsequent on-policy QHPO rollouts.

Regarding question visibility, a predefined question set does not imply that all questions are exposed to the policy in advance. In our main evaluation, only up to five active questions are visible and actionable, while remaining questions are progressively introduced from the backlog. In contrast, reproduced independent baselines receive all question texts and options as read-only context, with actions restricted to the current target question. Thus, VAMR coordinates evidence based on the currently available questions and accumulated memory, rather than advance access to the complete question set.

VAMR is trained exclusively on CG-Bench and evaluated on three held-out benchmarks without privileged annotations, supporting cross-dataset generalization under the evaluated multi-question protocol.

W3

The efficiency claims rely on incomplete proxy metrics. Rounds/Q and Frames/Q amortize a shared round across questions, even though one round may issue multiple tool calls; they omit text tokens, audio processing, language-model computation, latency, parallelism, and provider cost.

We agree that Rounds/Q and Frames/Q alone do not fully capture computational cost. We therefore additionally measure token consumption, end-to-end latency, and API cost on LVBench under identical serving conditions.

MethodTokens/QLatency/QAPI Cost/Q
VideoARM35.2k53.7s$0.0100
Ind. Loops43.0k65.2s$0.0139
VAMR (W=1)31.5k48.2s$0.0099
VAMR (W=5)18.4k27.1s$0.0057

W denotes the active window size; W=1 processes questions sequentially while retaining shared memory.

Compared with the trained independent baseline, VAMR reduces tokens, latency, and API cost by 57.1%, 58.4%, and 59.0%, respectively. Moreover, Table 5 reports 1.2 model calls/Q for VAMR versus 2.4 for VideoARM. These results confirm substantial computational savings beyond reductions in reasoning rounds.

Reviewer_3YL3OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents VAMR, a multi-question agentic reasoning framework for efficient long-form video understanding. The key idea is to jointly reason over multiple questions about the same video, enabling shared perception and context reuse across questions instead of solving each question independently. The paper further introduces a layered shared/question-specific memory mechanism and Question-Horizon Policy Optimization (QHPO) to train the multi-question reasoning policy. Experiments on several long video benchmarks demonstrate improvements in both accuracy and inference efficiency.

Critical strengths

1.The most interesting technical contribution is the question-level advantage design in QHPO. Rather than assigning a single advantage to the entire multi-question reasoning trajectory, QHPO estimates question-specific advantages and associates them with the reasoning steps relevant to each question. This provides a more fine-grained credit assignment mechanism and is a technically meaningful extension of RL optimization for complex agentic reasoning.

Critical weaknesses
  1. The direct baseline uses only 32 uniformly sampled frames, whereas VAMR processes 84.9 frames on average, making it unclear how much of the gain comes from the reasoning framework versus increased visual input. Moreover, a simple baseline that feeds multiple questions jointly to the same model would help isolate whether the benefit comes from agentic multi-question reasoning or merely from cross-question information sharing.
  2. Table 6 suggests that with stronger perception and reasoning models, VAMR provides limited improvement over VideoARM even with additional ASR information. This raises questions about whether joint multi-question reasoning substantially improves video understanding itself. If its main benefit is efficiency through information reuse, this motivation may be more compelling in settings such as streaming video understanding, where questions arrive continuously and previously acquired evidence can be naturally reused.
Questions for authors
  1. The efficiency evaluation mainly reports reasoning rounds and processed frames. However, jointly reasoning over multiple questions may make each reasoning round more computationally expensive. Reporting wall-clock inference time or token consumption would provide a more direct comparison of the actual computational efficiency.

  2. The active question window is fixed to five, but the motivation for this choice is unclear. The question-density experiment varies the total number of questions rather than the active-window size, so it does not directly justify this design choice. An ablation over different window sizes, including making all questions simultaneously active, would help clarify its impact on both accuracy and efficiency.

  3. The SFT trajectories are generated using Qwen3-VL-32B, while the final policy model is Qwen3.5-9B, which already shows stronger performance on LVBench. What motivates this teacher choice? Would using a stronger model, such as Qwen3.5-27B, produce higher-quality SFT trajectories and further improve performance?

5 comments
AuthorsOpenReview ↗

Response to 3YL3 (1/2)

We thank the reviewer for the insightful and positive comments. We sincerely hope that our response could properly address your concerns. If so, we would deeply appreciate it if you could raise your score. If not, please let us know your further concerns, and we will continue actively responding to your comments and improving our submission.

W1

The direct baseline uses only 32 uniformly sampled frames, whereas VAMR processes 84.9 frames on average.

Moreover, a simple baseline that feeds multiple questions jointly to the same model would help isolate whether the benefit comes from agentic multi-question reasoning or merely from cross-question information sharing.

Existing independent baselines already receive the complete question set as read-only context, controlling for access to additional question texts and options. To isolate the effects of visual coverage and joint inference, we introduce two baselines on LVBench. Direct-85 increases uniformly sampled frames from the original 32 to 85. Joint-5 Direct answers five questions jointly using 425 shared frames in a single pass, matching VAMR's active window size (W=5) and per-question frame budget, but without tools or memory. Both use the same Qwen3.5-9B backbone.

MethodAcc.Frames/Q
Direct-8548.085.0
Joint-5 Direct56.785.0
VAMR62.184.9

At comparable frame budgets, VAMR outperforms Joint-5 Direct by 5.4 points, demonstrating benefits beyond increased visual coverage and one-shot joint inference. Moreover, Joint-5 Direct processes 54.2K input tokens per call on average, compared with 7.8K for VAMR.

W2

Table 6 suggests that with stronger perception and reasoning models, VAMR provides limited improvement over VideoARM even with additional ASR information.

Our controlled experiments demonstrate that VAMR's novel paradigm of collaborative multi-question reasoning improves both inference efficiency and answer accuracy. On LVBench, VAMR improves accuracy by 5.7 points over the matched trained independent baseline. More importantly, our additional active-window ablation further isolates the benefit of joint reasoning. VAMR (W=1) processes questions sequentially while retaining cross-question shared memory and QHPO training, whereas VAMR (W=5) jointly coordinates up to five active questions. Accuracy increases from 58.9% to 62.1%, demonstrating benefits beyond sequential evidence reuse. Detailed results are provided in our response to Reviewer 2w5F (W1 & Q1).

We acknowledge that the supplementary comparison in Appendix Table 6 shows less pronounced accuracy gains with stronger models. However, (1) VideoARM also uses ASR, so VAMR's advantage cannot be attributed simply to access to audio information; (2) the OpenAI o3 variant in this appendix is training-free and lacks dedicated adaptation to the shared-trajectory reasoning paradigm, potentially limiting its benefits; and (3) the other results, including VideoARM, are taken from the original papers rather than matched reproductions, limiting direct attribution of performance differences. We also appreciate the suggestion of streaming video understanding as a promising extension, while our current results establish the benefits of collaborative reasoning in the evaluated multi-question setting.

AuthorsOpenReview ↗

Response to 3YL3 (2/2)

Q1

Reporting wall-clock inference time or token consumption would provide a more direct comparison of the actual computational efficiency.

We additionally measure end-to-end latency and token consumption on LVBench under identical serving conditions. Ctrl. In/Out denote the local controller's input/output tokens, while Perception includes both visual and text tokens processed by external visual models. All token counts are normalized per question.

MethodCtrl InCtrl OutPercep.TotalTime/Q
VideoARM8.60.626.035.253.7s
Ind. Loops6.40.636.043.065.2s
VAMR (W=1)5.40.525.531.548.2s
VAMR (W=5)3.30.414.718.427.1s

Token counts are in k/Q. W denotes the active window size; W=1 processes questions sequentially while retaining cross-question shared memory.

Compared with the trained independent baseline, VAMR reduces total token consumption by 57.1% and end-to-end latency by 58.4%, confirming substantial computational savings beyond reductions in reasoning rounds.

Q2

The question-density experiment varies the total number of questions rather than the active-window size, so it does not directly justify this design choice.

To examine the choice of active window size, we use the same 48-video, 768-question test subset as the question-density experiment, with all 16 questions per video processed in every setting. We keep the trained VAMR model fixed and vary only the inference-time active window size (W).

Active Window (W)Acc.Rounds/QFrames/Q
159.43.4163.2
361.51.4103.6
562.80.987.7
860.90.779.8
1659.00.573.4

Accuracy peaks at W=5, while larger windows further reduce reasoning rounds and processed frames at the cost of accuracy. In particular, W=16 reduces Rounds/Q and Frames/Q to 0.5 and 73.4, respectively, but lowers accuracy by 3.8 points compared with W=5. This demonstrates the trade-off between coordination efficiency and reasoning accuracy, with W=5 providing a favorable balance.

Q3

What motivates this teacher choice? Would using a stronger model, such as Qwen3.5-27B, produce higher-quality SFT trajectories and further improve performance?

We appreciate the reviewer's suggestion. We selected Qwen3-VL-32B as an established open-weight teacher model adopted in multiple prior multimodal studies. Our SFT teacher primarily initializes the novel shared-trajectory reasoning protocol, with annotated evidence intervals guiding action selection. To assess teacher strength, we replace Qwen3-VL-32B with Qwen3.5-27B while keeping the student, training data, trajectory filtering criteria, and training settings identical.

SFT TeacherAcc.Rounds/QFrames/Q
Qwen3-VL-32B57.90.876.1
Qwen3.5-27B58.30.876.4

The modest +0.4 accuracy gain suggests that teacher strength is not the primary bottleneck, while QHPO provides substantially greater improvements.

AuthorsOpenReview ↗

Response to 3YL3

Thank you again for your feedback. We have added the experiments from our rebuttal to Appendix J of the revised manuscript, including direct and joint baselines at comparable frame budgets, a comparison of sequential and joint reasoning with shared memory, end-to-end latency and token measurements, and ablations on active-window size and SFT teacher choice. If you have a chance to review these updates before the discussion closes, we would appreciate your feedback and would be happy to clarify any remaining questions.

Reviewer_2w5FOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

1. This paper introduces VAMR, a video agent that answers several questions about the same video in one shared process.

It uses question-aware video tools, shared video memory, and separate notes for active questions. The paper also proposes QHPO, which estimates an advantage for each question and combines these values to train the agent.

2. Experiments on LVBench, VideoHolmes, and LongVideoBench show higher accuracy than the main compared methods.

VAMR also reduces the average number of reasoning rounds and processed frames compared with a trained agent that answers each question separately.

Critical strengths

1. The experiments include strong controls.

The trained independent baseline uses matched training data, tools, and training budgets. Other question texts are available as read-only context in the main baseline comparisons. The question-count experiment also keeps the visible question texts fixed. These choices help rule out access to more question information as the main reason for the gains.

2. The improvements are supported by several experiments.

On LVBench, VAMR improves accuracy from 56.4% to 62.1% over the trained independent baseline while reducing processed frames per question from 224.0 to 84.9. The component studies show benefits from both question-aware tools and layered memory. QHPO also improves accuracy over standard PPO under the same agent setup.

Critical weaknesses

1. The comparisons do not fully separate shared memory from joint scheduling.

The independent baseline resets its state between questions, while VAMR keeps useful information. The paper therefore leaves open how much of the gain could be obtained by a simpler agent that answers questions one at a time but keeps a shared video summary and previous tool results. The current component studies do not directly answer this question.

2. QHPO may miss useful effects across questions.

A video-tool action receives credit mainly from its explicitly targeted questions, even though its results may enter shared memory and help other active or later questions. ASR actions instead receive credit from all active questions. The comparison without alignment supports the chosen rule in practice, but it does not explain when this rule captures the value of shared information or when it misses it.

Questions for authors
  1. Could you compare VAMR with a sequential agent that keeps a shared video summary and reuses previous tool results across questions, using the same tools and training budget? This would help show the added benefit of joint scheduling.

  2. How often does a tool result help questions outside its explicit target set? Could you explain why QHPO uses direct targets for video tools but all active questions for ASR, and provide evidence that this choice handles shared information well?

1 comment
AuthorsOpenReview ↗

Response to 2w5F

We thank the reviewer for the insightful and positive comments. We sincerely hope that our response could properly address your concerns.

W1 & Q1

The comparisons do not fully separate shared memory from joint scheduling.

Could you compare VAMR with a sequential agent that keeps a shared video summary and reuses previous tool results across questions, using the same tools and training budget?

To disentangle the benefits of shared memory from joint scheduling, we set VAMR's active window size (W) to 1, so questions are processed sequentially while retaining the shared video story, ASR, and previously acquired visual evidence across questions. Tool actions serve only the current question, and the policy is retrained with QHPO using the same tools and training budget.

MethodAcc.Rounds/QFrames/Q
Independent Loops56.44.1224.0
VAMR (W=1)58.93.4165.8
VAMR (W=5)62.10.984.9

VAMR (W=1) improves accuracy by 2.5 points over independent loops, highlighting the benefits of persistent cross-question evidence reuse within our framework. Joint reasoning with W=5 brings a further 3.2-point gain, while reducing Rounds/Q and Frames/Q by 73.5% and 48.8%, respectively.

W2 & Q2

QHPO may miss useful effects across questions.

How often does a tool result help questions outside its explicit target set? Could you explain why QHPO uses direct targets for video tools but all active questions for ASR, and provide evidence that this choice handles shared information well?

QHPO adopts conservative credit assignment rather than exact causal attribution. Visual tools perform question-conditioned evidence extraction with explicit target IDs, providing a well-defined attribution set. In contrast, ASR generates question-agnostic transcripts shared across all active questions, without identifiable question-specific targets. We therefore attribute advantages to explicit targets for visual tools and to all active questions for ASR.

Each tool call contributes observations to shared memory, which is repeatedly consolidated to merge and compress evidence from multiple actions. Consequently, identifying individual actions' contributions to subsequent decisions, let alone quantifying their effects on question-level rewards, is highly challenging. Propagating advantages to indirectly related actions would therefore introduce substantial attribution noise. Importantly, the absence of explicit credit does not eliminate these indirect benefits. The acquired evidence remains accessible through the shared story, while subsequent rewards can still optimize decisions conditioned on this memory.

We examine these two aspects through complementary ablations on LVBench. VAMR w/o shared story removes persistent shared memory while retaining joint actions and individual clue layers. An additional Qwen3.5-9B model routes visual summaries only to explicitly targeted questions and ASR information to each active question's clue layer, preserving question-specific evidence. VAMR w/o round alignment retains shared memory but assigns each question's advantage to all its active rounds, testing broader credit propagation.

MethodAcc.Rounds/QFrames/Q
VAMR w/o shared story57.91.8186.7
VAMR w/o round alignment60.01.093.7
VAMR62.10.984.9

Removing shared memory reduces accuracy by 4.2 points, confirming the substantial benefits of cross-question information sharing. Separately, removing round alignment reduces accuracy by 2.1 points, supporting QHPO's conservative credit assignment over indiscriminate advantage propagation.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.