- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work addresses the efficiency challenge of multi-question reasoning in long-form video understanding. Existing video agents typically process each question about the same video through separate retrieval, tool-calling, and memory-construction procedures, resulting in redundant computation and failing to fully exploit the visual evidence shared across different questions.
The paper propose VAMR (Video Agent for Multi-Question Reasoning), which organizes all questions about a video into one persistent agentic reasoning trajectory. Through a shared active question window, question-conditioned visual perception, and a layered multi-question memory structure, VAMR enables coordinated retrieval and asynchronous answering across questions. Furthermore, the paper introduces Question-Horizon Policy Optimization (QHPO), which uses a question-level critic to estimate the value of each question and applies round alignment to map question-level advantage signals to the tool calls and answer rounds that directly serve each question. Experiments across multiple benchmarks demonstrate that VAMR achieves state-of-the-art accuracy while significantly reducing the number of reasoning rounds.
Critical strengths
-
Meaningful and practically relevant problem. VAMR organizes multiple questions into a single shared online reasoning trajectory, allowing different questions to reuse already-acquired visual evidence and to be answered at different times. This directly addresses the common problem of repeated video exploration faced by existing video agents when a video is associated with multiple questions.
-
Sensible system design for shared perception and layered memory. The QA-inspector simultaneously produces a general description of the video segment and question-specific evidence, while the layered memory maintains the shared video context separately from the evidence belonging to each individual question. This design mitigates two potential pitfalls: the redundant video understanding caused by fully independent reasoning on the one hand, and the inter-question evidence interference caused by a fully shared context on the other.
-
QHPO is well matched to the asynchronous-completion structure of multi-question tasks. It estimates an independent value for each question, then filters advantages according to the correspondence between actions and questions, and finally produces an update signal for the shared actor.
Critical weaknesses
1.The credit-assignment assumption in QHPO may conflict with the core mechanism of shared memory. QHPO assigns a question-level advantage only to the rounds that directly serve that question. The underlying logic is that a round should receive a learning signal from question only if its action explicitly targets . However, the central contribution of VAMR is precisely cross-question evidence reuse. This raises a fundamental question: if the value of shared information depends not only on the currently served question but also on how future questions utilize that information, can "direct action attribution" correctly represent the true credit assignment?
2.The current efficiency metrics are insufficient to support conclusions about end-to-end computational efficiency. The paper primarily relies on two efficiency metrics:
- Rounds/Q: the average number of controller rounds required per question.
- Frames/Q: the average number of video frames processed by the visual model per question.
These metrics quantify the number of interaction rounds and part of the visual processing workload, but they do not fully represent the actual inference cost. In particular, when multiple questions are consolidated into the same control round, the reduction in Rounds/Q may partially stem from action aggregation rather than a proportional saving in real computation. The authors should therefore report end-to-end latency, model input/output tokens, GPU time, or monetary cost under the same hardware and model-serving conditions, and compare the cost curves across different question densities.
Questions for authors
-
The two questions raised in "Critical Weaknesses" should be addressed first. Resolving these is essential for evaluating the contribution.
-
The strong external visual model may confound module contribution. It is unclear whether accuracy gains come from the shared reasoning and policy optimization, or from extra reasoning by the external visual model under targeted prompting.
-
Sensitivity to question ordering and active-window scheduling is insufficiently analyzed. VAMR keeps a fixed set of five active questions, with the rest entering the backlog in input order. Since the shared story is persistent, earlier retrievals can influence later questions, so performance may be order-sensitive.
-
Practical deployment boundaries are under-discussed. The setting assumes all questions belong to one video, but in reality question count, arrival time, and repeated-query patterns vary. VAMR may not suit streaming or incremental question arrival.
3 comments
Response to hDM3 (2/2)
Q2
It is unclear whether accuracy gains come from the shared reasoning and policy optimization, or from extra reasoning by the external visual model under targeted prompting.
The QA-inspector performs question-conditioned visual analysis of selected video segments, while the backbone controller governs global reasoning and coordination, including where to inspect, which questions to serve, how to coordinate evidence retrieval, and when to answer. To assess the dependence on external model strength, we replace Qwen3.7-Plus with Qwen3.5-9B as the QA-inspector, keeping the trained controller, tool interface, and inference settings unchanged.
| QA-inspector | Acc. | Rounds/Q | Frames/Q |
|---|---|---|---|
| Qwen3.5-9B | 60.8 | 1.0 | 90.7 |
| Qwen3.7-Plus | 62.1 | 0.9 | 84.9 |
The modest 1.3-point accuracy difference indicates that stronger visual analysis provides additional benefits, but VAMR's performance is not heavily dependent on the stronger external model. This supports the robustness of our collaborative reasoning framework across different QA-inspector strengths.
Q3
Sensitivity to question ordering and active-window scheduling is insufficiently analyzed.
To examine sensitivity to question ordering, we evaluate VAMR on LVBench using the original, reversed, and randomly shuffled question orders, while keeping the trained model and inference settings unchanged. Question IDs are reassigned after each permutation to avoid potential ordering biases.
| Question Order | Acc. | Rounds/Q | Frames/Q |
|---|---|---|---|
| Original | 62.1 | 0.9 | 84.9 |
| Reversed | 62.7 | 1.0 | 92.5 |
| Random | 62.2 | 0.9 | 86.4 |
Accuracy remains stable across different question orders, while reasoning rounds and processed frames exhibit modest variations. These results suggest that VAMR is robust in accuracy to question ordering. Active-window sensitivity is further examined in our response to Reviewer 3YL3 (Q2).
Q4
The setting assumes all questions belong to one video, but in reality question count, arrival time, and repeated-query patterns vary.
We appreciate the reviewer's suggestion. VAMR's active window, backlog, and persistent shared memory provide a natural basis for handling varying question counts and incremental arrivals. New questions can enter the active window as slots become available, while repeated or related questions can reuse evidence accumulated in the shared story. Moreover, the iterative reasoning framework offers a potential extension to streaming video understanding, where newly available video segments can be incorporated in subsequent reasoning rounds.
These extensions would require adaptations to online scheduling and termination and have not yet been experimentally validated. We will clarify the scope of the current evaluation in the revised paper and leave systematic investigation of these settings to future work.
Thank you very much for the authors' response. I will keep my recommendation score for this work.
Response to hDM3 (1/2)
We thank the reviewer for the insightful and positive comments. We sincerely hope that our response could properly address your concerns.
W1
QHPO's conservative credit assignment and shared memory are not contradictory. The former provides an effective policy optimization strategy, while the latter enables collaborative multi-question reasoning and persistent evidence reuse. QHPO does not explicitly propagate rewards from future backlog questions to earlier actions. This is because tool observations accumulate in the shared story and undergo repeated consolidation, making it difficult to disentangle each original action's contribution to subsequent decisions and final rewards. Indiscriminate credit propagation would therefore introduce substantial attribution noise. Importantly, the absence of explicit credit does not eliminate these indirect benefits, as acquired evidence remains available to subsequent questions through the shared story.
We examine these two mechanisms through complementary ablations on LVBench. VAMR w/o round alignment retains shared memory but assigns each question's advantage to all its active rounds, testing broader credit propagation. VAMR w/o shared story removes persistent shared memory while retaining the trained policy, joint actions, and individual clue layers. An additional Qwen3.5-9B model routes visual summaries only to explicitly targeted questions and ASR information to each active question's clue layer, preserving question-specific evidence.
Removing round alignment reduces accuracy by 2.1 points, supporting conservative attribution over indiscriminate propagation. Separately, removing shared memory reduces accuracy by 4.2 points, confirming the substantial benefits of cross-question evidence reuse. Together, these results demonstrate the effectiveness of QHPO's credit assignment strategy alongside persistent shared memory in VAMR.
W2
We additionally measure end-to-end latency, token consumption, and API cost under identical serving conditions. Compared with the trained independent baseline on LVBench, VAMR reduces these costs by 58.4%, 57.1%, and 59.0%, respectively, with detailed baseline comparisons provided in our response to Reviewer CN9d (W3).
We further examine cost scaling on the same 48-video, 768-question subset as our question-density experiment.
Increasing question density from 1 to 16 reduces token consumption by 56.7%, latency by 48.9%, and API cost by 53.8%. These results confirm substantial end-to-end computational savings beyond reductions in amortized reasoning rounds.