- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TimeQuest trains a small Navigator to retrieve video segments for a frozen Oracle. The Oracle describes each segment, then answers using the original video and accumulated descriptions. Navigator training combines SFT and GRPO with format, boundary-distance, and termination rewards, without a final-answer reward. Experiments report improved QA and temporal grounding across several Oracle backbones.
Critical strengths
- Reusable navigation across frozen Oracles. The modular design improves several frozen Oracles without retraining them, making the learned navigation policy reusable. Supplementary code is provided.
- Substantial grounding improvements. On NExT-GQA with Qwen2.5-VL-7B, [email protected] rises from 23.3 to 60.4 and Acc@GQA from 41.0 to 47.0 (Table 2). Reporting localization and grounded answer accuracy helps assess whether retrieved evidence supports the task.
- Informative failure analysis. Appendix B.10 categorizes 50 selected failures into retrieval errors, information lost during description, and final-answer errors. Reporting category-level regressions alongside overall improvements helps identify the architecture's limitations, while leaving the causal attribution open to further clarification.
Critical weaknesses
- Efficiency is not established end to end. The reported 3.4x reduction concerns theoretical prefill FLOPs for a specified three-turn comparison with a monolithic agent (Sec. 4.3.3). It does not establish end-to-end speed after all Navigator, description, and final-answer calls. Measured latency would clarify the practical benefit.
- Alternative evidence paths are not clearly accommodated. Sec. 3.4 defines timestamp rewards against corresponding annotated windows and rewards termination at the annotated final step, without using final-answer correctness. The paper does not explain how reordered, alternative, or shorter valid evidence paths are scored. The reported QA gains do not resolve whether equally useful paths can be penalized by this supervision.
Questions for authors
- What end-to-end latency accompanies the FLOPs analysis, including all Navigator and Oracle calls? Please specify visual budgets and cache assumptions, and distinguish comparison with a single-pass Oracle from comparison with the monolithic agent.
- How are predicted windows matched to annotations when evidence is retrieved in a different order or through an alternative window? Does the termination reward penalize a shorter trajectory that already contains sufficient evidence?
- For failures attributed to description information loss, are the missing details visible in the final Oracle's sampled video input? If so, what distinguishes a text-compression effect from failure to use or integrate the available visual evidence?
3 comments
W2 & Q2: Alternative evidence trajectories and termination supervision
Our training objective uses annotated evidence trajectories as a structured supervision signal for sequential temporal localization. Predicted timestamp intervals are compared with their corresponding annotated intervals using step-aligned temporal rewards, while the termination reward encourages stopping at the annotated final step.
This formulation provides consistent localization and stopping supervision without relying on final-answer rewards. However, the annotated trajectory represents a reference evidence path rather than an exhaustive set of all semantically sufficient paths.
To further investigate how the final Oracle responds to alternative evidence trajectories, we conducted an additional diagnostic analysis using saved inference trajectories. We modified the retrieved evidence while keeping the final Oracle's sampled video input fixed and reran final answering under different evidence conditions.
| Evidence trajectory | Evaluated | Correct | Final QA accuracy |
|---|---|---|---|
| Original model-generated trajectory | 50 | 41 | 82.00% |
| Reordered evidence | 50 | 41 | 82.00% |
| Shorter evidence trajectory | 21 | 16 | 76.19% |
| Alternative evidence windows | 30 | 24 | 80.00% |
| Selected evidence removed | 30 | 21 | 70.00% |
Reordering the retrieved evidence preserved aggregate accuracy at 82% across the same 50 examples, indicating that the final Oracle can accommodate changes in evidence order in this setting. This does not imply identical predictions for every example.
The shortened and alternative-window conditions also supported correct answering in 16/21 and 24/30 cases, respectively. These interventions were evaluated on different subsets, so their aggregate accuracies should not be interpreted as directly comparable performance differences. Furthermore, because the final Oracle retains access to the original video, correct answers cannot be attributed exclusively to the modified descriptions.
Importantly, this inference-time behavior is distinct from the Navigator's training objective. Step-aligned temporal supervision and annotation-based termination do not explicitly recognize every alternative ordering, valid evidence window, or shorter sufficient trajectory. Thus, a semantically useful evidence path is not guaranteed to receive the same training reward as the annotated trajectory.
The additional analysis illustrates the Oracle's ability to use modified evidence paths, while clarifying the scope of our supervision design. We plan to investigate order-invariant matching and evidence-sufficiency-aware termination rewards to better accommodate multiple valid retrieval strategies.
Continued in Comment 3 of 3.
Q3: Description information loss versus visual-evidence integration
The reviewer raises an important distinction between information omitted during textual description and failure to utilize visual evidence already present in the final Oracle's input.
To investigate this, we performed an additional diagnostic analysis of the 23 cases categorized as textual information loss in Appendix B.10. We inspected the exact sampled video input provided to the final Oracle and found that the relevant detail remained visible in 19 cases, while it was absent or insufficiently resolved in four.
Keeping the final video input fixed, we evaluated three conditions: the original generated descriptions, corrected descriptions restoring omitted visual facts, and no descriptions.
| Visibility in final video input | Cases | Correct with original descriptions | Correct with corrected descriptions | Correct without descriptions |
|---|---|---|---|---|
| Relevant detail visible | 19 | 0 | 18 | 15 |
| Absent or insufficiently resolved | 4 | 0 | 4 | 1 |
| Total | 23 | 0 | 22 | 16 |
The original-description condition has zero correct answers by construction, since these 23 examples were selected from TimeQuest's failures.
Correcting the descriptions recovered 22 of the 23 cases. However, removing the descriptions also recovered 16 cases, including 15 of the 19 where relevant details were visible in the final Oracle's sampled video. This suggests that inaccurate or incomplete descriptions can sometimes interfere with the integration of available visual evidence, rather than failures arising exclusively from information omitted during textual conversion.
Conversely, among the four cases where the relevant information was absent or insufficiently resolved in the final video input, corrected descriptions recovered all four answers, whereas removing descriptions recovered only one. This supports the complementary value of localized textual evidence when the original video sampling does not adequately preserve the required information.
These findings suggest two distinct effects of the textual evidence interface: localized descriptions can recover information that is poorly represented in the original visual input, but inaccurate or incomplete descriptions may also interfere with the use of otherwise available visual evidence.
We will accordingly refine the original “information loss” characterization to the broader category of “description-related failures.” Since this diagnostic experiment focuses on 23 selected failure cases, its recovery rates should not be interpreted as the prevalence of these mechanisms across the full benchmark. The results motivate further investigation of confidence-aware evidence integration and hybrid visual-textual representations to preserve useful details while reducing interference from imperfect descriptions.
We thank the reviewer for the constructive assessment and for recognizing the benefits of reusable temporal navigation, substantial grounding improvements, and our failure analysis. Below, we clarify the scope of our computational analysis and provide additional diagnostic evidence concerning alternative retrieval trajectories and the integration of textual and visual evidence.
W1 & Q1: End-to-end inference cost and computational efficiency
Our computational analysis focuses on the cost of visual-context accumulation during multi-turn evidence acquisition, rather than establishing an end-to-end speedup over single-pass video answering.
Specifically, Section 4.3.3 compares a monolithic 7B architecture with our 3B Navigator + 7B Oracle under a three-turn configuration: a 200-frame high-resolution video, a 100-frame low-resolution proxy, three 90-frame retrieval cycles, and 784 pixels per visual token. Under these assumptions, theoretical prefill computation decreases from 6,656 to 1,935 TFLOPs, representing a 3.4× reduction.
The benefit arises from independently processing retrieved clips and retaining compact textual descriptions, rather than repeatedly accumulating high-resolution visual tokens across turns. This comparison isolates the computational implications of the evidence-management design.
Relative to a single-pass Oracle, TimeQuest intentionally introduces additional inference-time computation to acquire targeted evidence. Table 13 further characterizes the realized workload: the Adaptive configuration averages 2.2 retrieval turns and 3.2 Oracle calls per example, including final answering.
The reported FLOPs analysis concerns theoretical prefill computation and does not include all costs of preprocessing, autoregressive generation, or model execution. It also does not assume measured cross-call cache reuse. We therefore distinguish this architectural comparison from wall-clock efficiency. We plan to extend the evaluation with total inference FLOPs under explicitly specified visual budgets, including comparisons against the single-pass Oracle and multi-turn alternatives.
Continued in Comment 2 of 3.