- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes NavIAgent, a coarse-to-fine framework for active frame selection in long-form video understanding. It first uses a multimodal event graph to retrieve relevant video segments. An RL-based agent called SMILE then selects informative frames within these segments. The authors introduce step-level rewards and dual-scale advantage estimation to improve active navigation. Experiments on three long-video benchmarks show improvements across multiple VLM backbones.
Critical strengths
- The paper addresses an important and meaningful problem in long-video understanding. The trade-off between global temporal coverage and fine-grained evidence localization is well motivated. The proposed coarse-to-fine framework is reasonable, and the overall method is clearly presented.
- The experimental results are promising. The method is evaluated on three benchmarks with multiple VLM backbones. Table 2 also shows that SMILE improves performance when the graph retrieval stage is fixed. The appendix provides additional training results and frame-budget analysis.
Critical weaknesses
- The proposed RL design is not sufficiently validated. Table 2 only evaluates SMILE as a whole, without isolating dual-scale advantage estimation or individual reward components. The reward contributions reported in Appendix E do not establish their effects on accuracy. Moreover, the evidence gain reward relies on a verifier score conditioned on the ground-truth answer. It is unclear whether this ratio reliably reflects evidence sufficiency. Comparisons with standard GRPO and targeted ablations would help establish the value of these design choices.
- The experimental gains are not fully explained, and some implementation details need clarification. The final VLM receives selected frames, graph memories, frame descriptions, and speech transcripts. It is unclear whether the baselines receive comparable information. The method also depends on coarse-stage retrieval, but the impact of retrieval errors is not analyzed. Besides, the supplementary code appears to differ from the paper in frame access, the navigation penalty in Eq. (9), and the evaluation dataset used during training. These differences raise questions about reproducibility.
Questions for authors
- How much does each reward component contribute to the final performance? Could the authors compare dual-scale advantage estimation with standard GRPO under the same settings?
- Could the authors explain why the ground-truth-conditioned verifier score ratio is a reliable measure of evidence gain? Is there any empirical evidence supporting this choice?
- Do the baselines receive the same auxiliary information as NavIAgent? How sensitive is the final performance to errors in coarse-stage retrieval?
- Could the authors provide a more complete efficiency analysis that includes graph construction and preprocessing costs?
- Is the supplementary code the same version used for the reported experiments? Could the authors clarify the differences in the RL implementation and explain how the held-out validation results were obtained?
9 comments
W2&Q3 Information Parity and Retrieval Errors
We thank the reviewer for this question. We first clarify that the paper's Table 1 reports NavIAgent under the same no-ASR setting.
Same setting in the paper's Table 1 and Fig. 4. All numbers in the paper's Table 1 for NavIAgent are measured in the no-subtitle setting. Fig. 4 is the frame-budget ablation on Video-MME Long without subtitles, and the paper's Table 1 uses exactly this setting for NavIAgent and for baselines. As defined in Eq. 15, the context is the concatenation of the segment memories , the frame descriptions and the transcripts . The memories and descriptions are always present because they are produced by the coarse stage and by Smile during navigation, while the transcripts are optional and are added only when the speech tool is enabled.
Same-input comparison with RL-based selectors. To check that the gain is not produced by the extra information of the coarse stage, we compare the three RL-based selectors under a shared front-end. The event-graph retriever first localizes four candidate segments for each Video-MME Long question, and ReFoCUS[R1], TSPO[R2] and NavIAgent then select 8 frames per segment from the same candidate frames and are answered by the same frozen Qwen3-VL-4B under identical prompts. Under this identical input and budget, NavIAgent reaches 59.7, against 55.3 for ReFOCUS and 56.9 for TSPO, so the gain of 4.4 and 2.8 points comes from the selection policy rather than from the front-end. The results are as follows.
Table 4 Comparison with RL-based frame selection under a shared front-end and a shared answering budget
| Method | Backbone | Video-MME Long |
|---|---|---|
| ReFoCUS [R1] | Qwen3-VL-4B | 55.3 |
| TSPO [R2] | Qwen3-VL-4B | 56.9 |
| NavIAgent | Qwen3-VL-4B | 59.7 |
Candidate set size. For the second question, we add a group of experiments. The coarse stage returns candidate chunks, and controls how much of the video the fine stage may inspect and how much it costs. We vary on Video-MME Long and report the accuracy and the recall of the answer-bearing chunk.
Table 5 Effect of the candidate set size (Qwen3-VL-8B, Video-MME Long)
| Candidate chunks | 2 | 4 | 8 | 16 |
|---|---|---|---|---|
| accuracy (%) | 57.2 | 60.9 | 61.4 | 61.6 |
| recall of the answer-bearing chunk (%) | 78.5 | 91.2 | 94.6 | 96.1 |
Analysis of results. Reducing the candidate set from four chunks to two costs 3.7 points, and the recall drops from 91.2 to 78.5 percent, so in many questions the answer-bearing evidence is simply missing from the set. Enlarging the set to eight and sixteen chunks adds only 0.5 and 0.7 points, and the recall rises to 94.6 and 96.1 percent, which means the graph retrieval has already placed the answer-bearing chunk in the top four in most cases and the extra chunks mostly add distractors, at a proportional cost in frames and latency. We therefore keep , where the accuracy has nearly saturated and the cost is still acceptable.
We thank the reviewer for this meaningful comment. We will add the candidate set analysis and Table 4 and 5 to the revision.
Reference
[R1] ReFoCUS: Reinforcement-guided Frame Optimization for Contextual Understanding - CVPR 26' Findings
[R2] TSPO: Temporal Sampling Policy Optimization for Long-form Video Language Understanding - AAAI 26'
W2&Q4 Full Efficiency Accounting
We thank the reviewer for this comment. Following the reviewer's suggestion, we add a full accounting that includes the coarse stage. Our design targets the setting of one video with multiple questions, so we report the cost as a function of the number of questions asked about the same video, and we report both the wall-clock time and the computation, measured as the number of processed tokens. The statistics use a 41-minute video from Video-MME Long.
Table 6 Coarse stage and online inference cost
| Stage | NavIAgent time (s) | NavIAgent tokens (K) | EGAgent time (s) | EGAgent tokens (K) |
|---|---|---|---|---|
| Coarse stage | 513 | 187 | 598 | 239 |
| Online inference, per question | 111 | 101 | 149 | 123 |
Table 7 Average cost per question as the number of questions grows
| Questions per video | 1 | 3 | 5 | 10 | 15 |
|---|---|---|---|---|---|
| NavIAgent, average time (s) | 624.0 | 282.0 | 213.6 | 162.3 | 145.2 |
| EGAgent, average time (s) | 747.0 | 348.3 | 268.6 | 208.8 | 188.9 |
| NavIAgent, average tokens (K) | 288.0 | 163.3 | 138.4 | 119.7 | 113.5 |
| EGAgent, average tokens (K) | 362.0 | 202.7 | 170.8 | 146.9 | 138.9 |
Analysis of the results. The coarse stage of NavIAgent is cheaper than the graph construction of EGAgent, at 513 s against 598 s and 187K against 239K tokens. And the online cost is lower as well, at 111 s against 149 s and 101K against 123K tokens per question. Besides, because both costs are one-time, the average cost per question falls as the number of questions grows, from 624.0 s and 288.0K tokens at one question to 145.2 s and 113.5K tokens at fifteen questions for NavIAgent, and from 747.0 s and 362.0K tokens to 188.9 s and 138.9K tokens for EGAgent.
Our advantage. We acknowledge that NavIAgent has a higher latency than the non-agent methods, but the accuracy improvement is significant, and the two-stage design can use different models to reduce the cost. The paper's Table 3 reports 117 s per query for LongVideo-R1 and 51.3 s for VideoBrain, and both are one-shot methods. For 15 questions NavIAgent takes 145.2 s, so it is slower than both, and we do not claim otherwise. However, this is a widely known property of agent-based video understanding rather than an artifact of our design. For a one-hour video the stored graph is a JSON file of about 1.6 MB. In addition, Figure 5 shows that the accuracy is not sensitive to the capacity of the indexing model, so the graph can be built with a cheaper model, which lowers the construction cost further. Since NavIAgent also brings a substantial accuracy improvement over the base model, we believe the design offers a clear advantage for long-video question answering with repeated queries.
Thanks for the reviewer’s comment and all the clarifications and modifications will be synchronized in the latest version of the manuscript.
W2&Q5 Code Consistency
We thank the reviewer for this question. The supplementary code is the same version used for the reported experiments, which we explain here and will correct in the revision.
Frame access convention. The action is a relative jump of 3 to 20 percent of the current time window, and the policy observation consists of the normalized position, the normalized displacement and the number of frames in the current window as context. The semantics of an action are therefore independent of the window length. During training the window is the sampled timeline of the whole video, and during inference Smile navigates inside a retrieved candidate chunk, where frames are passed per chunk and positions are normalized to . The phrase calibrated against the local candidate chunk in Section 3.3.1 describes the inference behavior, and we will state the training window explicitly.
Navigation penalty in Eq. 9. This point reflects an oversight on our side. The linear form in the paper was our earliest definition of the navigation cost, while all experiments reported in the paper were run with the code, in which the cost is the squared energy form. The CLIP novelty term is implemented as the L2 distance between two L2-normalized CLIP embeddings, which matches Eq. 9. We will correct Eq. 9 in the paper immediately so that the equation matches the code, and we apologize again for this oversight.
Evaluation data. Smile is trained on the NExT-QA train split. The curves in Fig. 6 and the values in Table 6 are obtained on the NExT-QA val split, with an evaluation every 30 optimization steps. The evaluation inside the training script is used only for monitoring and is not the setting behind Fig. 6 or Table 6.
Thanks for the reviewer’s comment and we will fix it in the revised version and add more explanations.
W1&Q1 RL Design Ablation
We thank the reviewer for this comment. We agree that Table 2 evaluates Smile as a whole, and we add the ablations that isolate the advantage estimator and each reward term below.
Experimental setup. All ablations use the same setup as the main training. And the two ablations change only the component under test, the balance ω in Eq. 12 for the advantage and the corresponding coefficient in Eq. 9 for the reward, and keep every other hyperparameter fixed. The results are as follows.
Table 1 Ablation of the advantage estimator
Table 2 Ablation of the reward terms
Analysis of results. The dual-scale advantage raises the held-out NExT-QA accuracy from 54.6 to 58.9, so the step-level term is what makes the trajectory-level signal usable for credit assignment. On the reward side, removing either the belief gain or the CLIP novelty term lowers the accuracy by a visible margin, removing the navigation cost has a smaller effect, and keeping only the terminal reward gives the lowest accuracy at 52.4. The RL design therefore gains from the advantage estimator and the dense rewards together, rather than from the terminal reward alone.
We thank the reviewer for this meaningful comment. We will add the two ablations and the analysis above to the revision.
W1&Q2 Verifier Ratio Reliability
We thank the reviewer for this question. We first define the ratio from the belief update, then show how it evolves during training together with the success rate.
What the ratio is. The belief update in Eq. 10 has two inputs. The first is the counterfactual ratio rt = p2 / max(p1, ϵeps), where p1 = J(Q, Mt) is the verifier's probability that the accumulated memory Mt is already sufficient to answer Q, and p₂ = J(Q, Mt, y∗) is the probability under the same memory when the ground-truth answer is also given as a reference. The second is the momentum coefficient ρ that smooths the recursion. We write the ratio as rt here to keep it distinct from ρ. Both terms come from the same verifier and the same memory, so anything that makes the context generally plausible, such as an on-topic scene description, raises p1 and p2 together and leaves the ratio near one. The ratio departs from one only when the memory makes the specific answer y∗ more credible than an arbitrary answer would be, which is the diagnosticity of the evidence rather than its topical relevance.
To make it clear, we track the mean ratio at STOP and the accuracy on NExT-QA, evaluated every 30 steps with the protocol of Fig. 6 and Table 6. Table 3 reports the checkpoints at steps 30 to 390.
Table 3 Mean ratio at STOP and accuracy during training (NExT-QA)
Analysis of results. Early in training the ratio stays below one, which means the memory at termination is not yet more supportive of the correct answer than of an arbitrary one, and the accuracy is correspondingly low. From step 300 onward the ratio moves above one while the accuracy reaches 53.0 and then 58.9, so the policy learns to stop on evidence that is diagnostic for the specific answer. The dip at step 180 shows that the trend is not monotone, which is expected because the ratio is a per-step signal and the accuracy is a trajectory-level outcome, but the two move upward together over training.
We thank the reviewer for this meaningful comment. We will add the tracking experiment and the analysis above to the revision.