acceptodds
Withdrawn from ICLR 2027

TimeQuest: Scaling Inference-Time Compute in Video-LLMs via Decoupled Temporal Navigation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Scaling inference-time compute has improved the performance of Large Language Models (LLMs), yet its application to video understanding is limited by a fundamental perception bottleneck. Existing Video-LLMs primarily scale reasoning in the text space; however, if a model misses brief but critical visual evidence during its initial pass, no amount of subsequent text-based "thinking" can recover it. In this paper, we introduce TimeQuest, a framework that decouples temporal evidence acquisition from reasoning via a novel Navigator–Oracle architecture. The Navigator, trained using Group Relative Policy Optimization (GRPO) with a smooth distance-based temporal reward, iteratively identifies informative video segments. Rather than naively stacking raw high-dimensional visual tokens (which rapidly exhausts context window), the Oracle translates these localized segments into dense textual descriptions. This accumulated context is appended to the initial query, enabling the Oracle to process the original video alongside an evidence-augmented prompt to deduce the final answer. This design fundamentally shifts inference-time compute from raw visual processing to active evidence distillation. Extensive evaluations on standard and temporally grounded Video QA benchmarks demonstrate consistent improvements of up to **+6.0** accuracy points, confirming that active evidence acquisition is a key factor in effective inference-time scaling for video understanding.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Video-LLM, T emporal Grounding, Long Video Understanding, GRPO

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_z8wpOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TimeQuest trains a small Navigator to retrieve video segments for a frozen Oracle. The Oracle describes each segment, then answers using the original video and accumulated descriptions. Navigator training combines SFT and GRPO with format, boundary-distance, and termination rewards, without a final-answer reward. Experiments report improved QA and temporal grounding across several Oracle backbones.

Critical strengths
  1. Reusable navigation across frozen Oracles. The modular design improves several frozen Oracles without retraining them, making the learned navigation policy reusable. Supplementary code is provided.
  2. Substantial grounding improvements. On NExT-GQA with Qwen2.5-VL-7B, [email protected] rises from 23.3 to 60.4 and Acc@GQA from 41.0 to 47.0 (Table 2). Reporting localization and grounded answer accuracy helps assess whether retrieved evidence supports the task.
  3. Informative failure analysis. Appendix B.10 categorizes 50 selected failures into retrieval errors, information lost during description, and final-answer errors. Reporting category-level regressions alongside overall improvements helps identify the architecture's limitations, while leaving the causal attribution open to further clarification.
Critical weaknesses
  1. Efficiency is not established end to end. The reported 3.4x reduction concerns theoretical prefill FLOPs for a specified three-turn comparison with a monolithic agent (Sec. 4.3.3). It does not establish end-to-end speed after all Navigator, description, and final-answer calls. Measured latency would clarify the practical benefit.
  2. Alternative evidence paths are not clearly accommodated. Sec. 3.4 defines timestamp rewards against corresponding annotated windows and rewards termination at the annotated final step, without using final-answer correctness. The paper does not explain how reordered, alternative, or shorter valid evidence paths are scored. The reported QA gains do not resolve whether equally useful paths can be penalized by this supervision.
Questions for authors
  1. What end-to-end latency accompanies the FLOPs analysis, including all Navigator and Oracle calls? Please specify visual budgets and cache assumptions, and distinguish comparison with a single-pass Oracle from comparison with the monolithic agent.
  2. How are predicted windows matched to annotations when evidence is retrieved in a different order or through an alternative window? Does the termination reward penalize a shorter trajectory that already contains sufficient evidence?
  3. For failures attributed to description information loss, are the missing details visible in the final Oracle's sampled video input? If so, what distinguishes a text-compression effect from failure to use or integrate the available visual evidence?
3 comments

We thank the reviewer for the constructive assessment and for recognizing the benefits of reusable temporal navigation, substantial grounding improvements, and our failure analysis. Below, we clarify the scope of our computational analysis and provide additional diagnostic evidence concerning alternative retrieval trajectories and the integration of textual and visual evidence.

W1 & Q1: End-to-end inference cost and computational efficiency

Our computational analysis focuses on the cost of visual-context accumulation during multi-turn evidence acquisition, rather than establishing an end-to-end speedup over single-pass video answering.

Specifically, Section 4.3.3 compares a monolithic 7B architecture with our 3B Navigator + 7B Oracle under a three-turn configuration: a 200-frame high-resolution video, a 100-frame low-resolution proxy, three 90-frame retrieval cycles, and 784 pixels per visual token. Under these assumptions, theoretical prefill computation decreases from 6,656 to 1,935 TFLOPs, representing a 3.4× reduction.

The benefit arises from independently processing retrieved clips and retaining compact textual descriptions, rather than repeatedly accumulating high-resolution visual tokens across turns. This comparison isolates the computational implications of the evidence-management design.

Relative to a single-pass Oracle, TimeQuest intentionally introduces additional inference-time computation to acquire targeted evidence. Table 13 further characterizes the realized workload: the Adaptive configuration averages 2.2 retrieval turns and 3.2 Oracle calls per example, including final answering.

The reported FLOPs analysis concerns theoretical prefill computation and does not include all costs of preprocessing, autoregressive generation, or model execution. It also does not assume measured cross-call cache reuse. We therefore distinguish this architectural comparison from wall-clock efficiency. We plan to extend the evaluation with total inference FLOPs under explicitly specified visual budgets, including comparisons against the single-pass Oracle and multi-turn alternatives.

Continued in Comment 2 of 3.

W2 & Q2: Alternative evidence trajectories and termination supervision

Our training objective uses annotated evidence trajectories as a structured supervision signal for sequential temporal localization. Predicted timestamp intervals are compared with their corresponding annotated intervals using step-aligned temporal rewards, while the termination reward encourages stopping at the annotated final step.

This formulation provides consistent localization and stopping supervision without relying on final-answer rewards. However, the annotated trajectory represents a reference evidence path rather than an exhaustive set of all semantically sufficient paths.

To further investigate how the final Oracle responds to alternative evidence trajectories, we conducted an additional diagnostic analysis using saved inference trajectories. We modified the retrieved evidence while keeping the final Oracle's sampled video input fixed and reran final answering under different evidence conditions.

Evidence trajectoryEvaluatedCorrectFinal QA accuracy
Original model-generated trajectory504182.00%
Reordered evidence504182.00%
Shorter evidence trajectory211676.19%
Alternative evidence windows302480.00%
Selected evidence removed302170.00%

Reordering the retrieved evidence preserved aggregate accuracy at 82% across the same 50 examples, indicating that the final Oracle can accommodate changes in evidence order in this setting. This does not imply identical predictions for every example.

The shortened and alternative-window conditions also supported correct answering in 16/21 and 24/30 cases, respectively. These interventions were evaluated on different subsets, so their aggregate accuracies should not be interpreted as directly comparable performance differences. Furthermore, because the final Oracle retains access to the original video, correct answers cannot be attributed exclusively to the modified descriptions.

Importantly, this inference-time behavior is distinct from the Navigator's training objective. Step-aligned temporal supervision and annotation-based termination do not explicitly recognize every alternative ordering, valid evidence window, or shorter sufficient trajectory. Thus, a semantically useful evidence path is not guaranteed to receive the same training reward as the annotated trajectory.

The additional analysis illustrates the Oracle's ability to use modified evidence paths, while clarifying the scope of our supervision design. We plan to investigate order-invariant matching and evidence-sufficiency-aware termination rewards to better accommodate multiple valid retrieval strategies.

Continued in Comment 3 of 3.

Q3: Description information loss versus visual-evidence integration

The reviewer raises an important distinction between information omitted during textual description and failure to utilize visual evidence already present in the final Oracle's input.

To investigate this, we performed an additional diagnostic analysis of the 23 cases categorized as textual information loss in Appendix B.10. We inspected the exact sampled video input provided to the final Oracle and found that the relevant detail remained visible in 19 cases, while it was absent or insufficiently resolved in four.

Keeping the final video input fixed, we evaluated three conditions: the original generated descriptions, corrected descriptions restoring omitted visual facts, and no descriptions.

Visibility in final video inputCasesCorrect with original descriptionsCorrect with corrected descriptionsCorrect without descriptions
Relevant detail visible1901815
Absent or insufficiently resolved4041
Total2302216

The original-description condition has zero correct answers by construction, since these 23 examples were selected from TimeQuest's failures.

Correcting the descriptions recovered 22 of the 23 cases. However, removing the descriptions also recovered 16 cases, including 15 of the 19 where relevant details were visible in the final Oracle's sampled video. This suggests that inaccurate or incomplete descriptions can sometimes interfere with the integration of available visual evidence, rather than failures arising exclusively from information omitted during textual conversion.

Conversely, among the four cases where the relevant information was absent or insufficiently resolved in the final video input, corrected descriptions recovered all four answers, whereas removing descriptions recovered only one. This supports the complementary value of localized textual evidence when the original video sampling does not adequately preserve the required information.

These findings suggest two distinct effects of the textual evidence interface: localized descriptions can recover information that is poorly represented in the original visual input, but inaccurate or incomplete descriptions may also interfere with the use of otherwise available visual evidence.

We will accordingly refine the original “information loss” characterization to the broader category of “description-related failures.” Since this diagnostic experiment focuses on 23 selected failure cases, its recovery rates should not be interpreted as the prevalence of these mechanisms across the full benchmark. The results motivate further investigation of confidence-aware evidence integration and hybrid visual-textual representations to preserve useful details while reducing interference from imperfect descriptions.

Reviewer_yjp3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes TimeQuest, a framework for inference-time scaling in long-video QA through temporal evidence acquisition. A lightweight navigator iteratively selects relevant video intervals from a low-resolution view, while a frozen oracle describes retrieved high-resolution clips in text. The captions are combined with the original video and question for final answer prediction, avoiding repeated accumulation of visual tokens across exploration turns. The Navigator is trained using supervised fine-tuning followed by GRPO with a temporal boundary reward and a termination reward. Experiments across multiple VLMs report consistent accuracy improvements and larger gains in temporal grounding.

Critical strengths
  1. Transferable architecture. The Navigator–Oracle design enables targeted evidence retrieval without fine-tuning the answer model. Improvements across Qwen, InternVL, and LLaVA-OneVision suggest practical transferability, making the framework potentially useful as a plug-and-play enhancement.
  2. Comprehensive experimental analysis. The paper includes informative analyses of grounding quality, retrieval strategy, GRPO training, and reasoning versus evidence acquisition. Substantial NExT-GQA grounding gains support the central claim that locating relevant evidence can improve video understanding.
Critical weaknesses
  1. Limited novelty over existing work. The claimed novelty of decoupling navigation from reasoning is insufficiently distinguished from prior agentic video systems, particularly Active Video Perception [1] and LongVideoAgent [2], which also separate evidence localization, observation, and answer generation. Without explicit architectural comparisons or direct experiments, it is difficult to identify the substantive contribution beyond combining existing design choices and reward shaping.
  2. Insufficient evidence for computational efficiency. TimeQuest adds Navigator inference and multiple Oracle calls, yet its primary accuracy comparisons use a single-pass baseline. Without matched-budget accuracy, end-to-end latency, and memory measurements against competitive agentic baselines, its claimed efficiency advantage remains insufficiently supported.

[1] Active Video Perception: Iterative Evidence Seeking for Agentic Long Video Understanding

[2] LongVideoAgent: Multi-Agent Reasoning with Long Videos

Questions for authors
  1. How does TimeQuest differ technically from Active Video Perception and LongVideoAgent, and can you provide matched-setting experimental comparisons? 2.What are the end-to-end latency, peak GPU memory, and accuracy-versus-FLOPs trade-offs under matched inference budgets across benchmarks?
2 comments

We thank the reviewer for the constructive assessment and for recognizing TimeQuest's transferability across frozen Video-LLMs and its substantial improvements in temporal grounding. Below, we clarify the technical distinctions from closely related agentic video-understanding frameworks and the scope of our computational efficiency analysis.

W1 & Q1: Novelty relative to Active Video Perception and LongVideoAgent

The central contribution of TimeQuest is not the general concept of modular video agents, but a dedicated, learned temporal-navigation policy optimized independently of final-answer generation and reusable across heterogeneous frozen Video-LLMs.

While Active Video Perception (AVP), LongVideoAgent, and TimeQuest all support iterative evidence acquisition, they differ in their control mechanisms, training objectives, and interaction with the answering model.

AspectActive Video PerceptionLongVideoAgentTimeQuest
Control structurePlanner–observer–reflector loopMaster agent coordinating grounding and vision agentsLearned Navigator proposing timestamps; frozen Oracle describing clips and answering
Policy trainingTraining-free agent orchestrationGRPO training of the master agent using structural and final-answer rewardsSFT+GRPO training of the Navigator using temporal localization and termination rewards
Evidence acquisitionObservation plans specifying temporal regions, sampling granularity, and observation goalsGrounding agent using the question and subtitles to locate relevant clipsNavigator predicting continuous temporal boundaries from a low-resolution video view and accumulated descriptions
Final answeringReflector answers using accumulated evidenceMaster agent reasons over accumulated contextFrozen Oracle processes the original sampled video and accumulated textual evidence

These distinctions have two important implications.

First, TimeQuest explicitly separates navigation optimization from answer generation. Unlike approaches that optimize an answering or coordinating agent using final-answer feedback, our GRPO objective directly supervises temporal localization and stopping decisions without relying on final-answer correctness. This enables the Navigator to specialize in evidence acquisition while leaving the Oracle unchanged.

Second, the learned navigation policy is transferable. The same 3B Navigator is reused across five frozen Oracle families, including Qwen2.5-VL, Qwen3-VL, InternVL3, InternVL3.5, and LLaVA-OneVision 1.5. This demonstrates that the learned navigation mechanism is not tied to a particular answering backbone.

Our ablations further support the effectiveness of these design choices:

  • Table 8: The trained Navigator improves EgoSchema accuracy from 60.2% to 61.6% over a prompted pretrained Navigator under the same retrieval protocol, with slightly fewer average retrieval turns (2.2 versus 2.3).
  • Table 7: Relevance-guided retrieval improves internal-test accuracy from 76.10% to 81.21% over random retrieval while controlling the number and duration of retrieved segments.
  • Table 4: Distance-based temporal supervision improves accuracy from 77.32% to 81.21% compared with an IoU-based reward.

Together, these results support the benefits of learned, relevance-guided temporal navigation and the proposed reward design within TimeQuest. They do not constitute a direct performance comparison against AVP or LongVideoAgent.

We therefore distinguish our contribution from the broader concept of agentic video exploration: TimeQuest learns a reusable temporal-localization policy independently of the answering model, with explicit temporal rewards and a compact textual evidence interface. We plan to strengthen the comparison with related agentic frameworks through direct evaluations under aligned backbones, visual budgets, and total inference costs.

Continued in Comment 2 of 2.

W2 & Q2: Computational efficiency and matched-budget evaluation

Our computational analysis investigates the cost of multi-turn visual-context accumulation, while the benchmark evaluations establish the accuracy benefits of augmenting frozen Oracles with targeted evidence acquisition. These address related but distinct questions.

Computational analysis. Section 4.3.3 considers a three-turn evidence-acquisition configuration with a 200-frame high-resolution video, a 100-frame low-resolution proxy, three 90-frame retrieval cycles, and 784 pixels per visual token. Under these assumptions, theoretical prefill computation decreases from 6,656 TFLOPs for a monolithic 7B architecture to 1,935 TFLOPs for our 3B Navigator + 7B Oracle design, a 3.4× reduction.

This comparison isolates the benefit of avoiding repeated accumulation of high-dimensional visual tokens. TimeQuest independently processes retrieved clips and retains their textual descriptions, whereas the modeled monolithic alternative carries an expanding visual context across turns. The result characterizes the theoretical prefill cost of these two configurations; it is not a measured end-to-end speedup or a direct efficiency comparison with AVP or LongVideoAgent.

Accuracy and inference workload. Our experiments show that targeted evidence acquisition consistently improves the evaluated frozen Oracles. The appendix provides additional analyses of how evidence selection and retrieval allocation affect performance:

  • Table 7: Relevance-guided retrieval outperforms random retrieval with matched segment counts and durations (81.21% versus 76.10%).
  • Table 8: Learned navigation improves accuracy over prompted navigation without increasing the average number of retrieval turns.
  • Table 6: Using 90 frames per retrieved segment achieves 50.4% accuracy, compared with 50.2% using 120 frames in the fixed-frame setting.
  • Table 13: The Adaptive configuration averages 2.2 retrieval turns and 3.2 Oracle calls, including final answering, providing a characterization of the realized inference workload.

These findings support the importance of retrieval quality and the design rationale for compact evidence accumulation. However, the primary Base-versus-TimeQuest evaluations are not matched in total inference compute: TimeQuest intentionally performs additional navigation and description calls to improve evidence acquisition.

A complete comparison of practical efficiency would require measuring end-to-end latency, peak GPU memory, throughput, and total FLOPs across all Navigator, description, and final-answer calls, under matched inference budgets and clearly specified caching assumptions. We plan to extend the evaluation accordingly, including direct comparisons with related agentic frameworks.

Thus, our current results establish accuracy improvements from learned evidence acquisition and a theoretical prefill-compute advantage over the specified monolithic design, while leaving the relative end-to-end accuracy–compute trade-off against other agentic approaches for further evaluation.

Reviewer_aLSaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents TimeQuest, a Navigator–Oracle framework for long-video QA that uses a separately trained Navigator to retrieve relevant clips and a frozen Oracle to convert them into textual evidence for final answering. The method consistently improves performance and substantially strengthens temporal grounding, supporting the value of decoupled, modular temporal evidence acquisition. However, its framing as test-time scaling is not fully convincing: TimeQuest also introduces dedicated SFT/GRPO supervision, additional model capacity, and multiple inference calls, so the gains may primarily arise from the specialized Navigator rather than from scaling inference-time computation itself. Final QA improvements are often modest despite much larger grounding gains. Overall, the framework remains a meaningful architectural contribution, but its stronger claim as a general inference-time scaling mechanism is not established without compute-matched and factorial ablations.

Critical strengths
  1. The paper clearly distinguishes a perception bottleneck from a reasoning bottleneck, and its argument that additional reasoning cannot recover visual evidence omitted during the initial video encoding is intuitive and important for multimodal inference-time scaling.
  2. Decoupling navigation from reasoning is an appealing and modular design: the Navigator can be paired with different frozen Video-LLMs, while converting retrieved clips into textual evidence avoids repeatedly accumulating high-dimensional visual tokens.
  3. The method is evaluated across multiple long-video benchmarks and five Oracle families, with improvements in all 20 Base-versus-TimeQuest comparisons in the main long-form VideoQA table. The transfer experiment pairing a 3B Navigator with a 32B Oracle further supports the claimed modularity.
  4. On NExT-GQA, temporal grounding improves substantially: Qwen2.5-VL rises from 23.3 to 60.4 in [email protected], while Qwen3-VL rises from 30.8 to 61.3. These results provide direct evidence that the Navigator retrieves more relevant temporal segments.
  5. The comparison among Qwen3-VL Base, Thinking, TimeQuest, and TimeQuest+Thinking supports the paper's main hypothesis: acquiring better visual evidence yields larger grounding gains than simply generating more reasoning tokens, while the two approaches remain complementary.
  6. The random-segment comparison shows that the gains come from segment relevance rather than merely adding visual input. The SFT/GRPO ablation identifies GRPO as the main contributor, while the manual failure analysis candidly reveals information loss caused by textual conversion.
Critical weaknesses
  1. The Base-versus-TimeQuest comparison changes multiple factors at once: TimeQuest introduces a separately SFT+GRPO-trained 3B Navigator, additional model capacity, a different multi-model architecture, and more inference-time computation. The gains therefore cannot be attributed specifically to inference-time scaling. The prompted-versus-trained Navigator and retrieval-budget ablations only partially disentangle these factors and are neither factorial nor compute-matched.
  2. Grounding comparisons are partly asymmetric because TimeQuest uses a Navigator explicitly trained for timestamp prediction, whereas several Base models rely on prompting or cannot reliably output timestamps. The large grounding gains therefore combine architecture, additional computation, and dedicated temporal supervision. Final QA accuracy improves much less, typically by approximately 0.6–3.3 points on NExT-GQA. The results demonstrate that the Navigator is better at temporal localization, but do not show that these localization gains translate sufficiently into final answer quality.
  3. Although TimeQuest improves the overall TVBench average by 0.9 points, it decreases performance by 14.5 points on Egocentric Sequence and 15.9 points on Unexpected Action. These substantial category-level regressions are obscured by the aggregate result and suggest that the method trades away certain fine-grained capabilities rather than uniformly improving video understanding.
  4. The distance-based temporal reward uses absolute errors in seconds rather than normalizing by video duration, so the same error has different significance for short and long videos. It is also unclear how predicted turns are matched to multiple ground-truth intervals, how alternative valid intervals are handled, or what happens when the number or order of predicted intervals differs from the annotations. Moreover, converting the mean nonlinear reward directly into an average boundary error is not justified without assumptions about the error distribution.
  5. The statistical reliability and reproducibility of the reported gains remain unclear. Improvements below two percentage points are not supported by multiple random seeds, confidence intervals, or significance tests. In addition, the internal test set and several key SFT/GRPO hyperparameters are insufficiently documented.
  6. Although Base and TimeQuest use the same final Oracle configuration, TimeQuest additionally invokes a 3B Navigator, performs a low-resolution full-video scan, and makes several Oracle calls over retrieved clips, so total compute is not matched. The reported 3.4x FLOP reduction is relative to a hypothetical monolithic three-turn architecture rather than the Base systems used in the accuracy comparisons, and the paper does not report end-to-end latency, peak memory, throughput, or total FLOPs on the actual benchmarks.
  7. The experiments show that active evidence acquisition is useful, but they do not establish a comprehensive performance-versus-compute scaling relationship. The retrieval-budget study covers only a narrow range and shows limited improvement from one to three maximum retrieval turns, making the results more convincing as test-time augmentation than as evidence of a general scaling law.
Questions for authors
  1. Can the authors provide a factorial ablation that separately controls (i) additional Navigator capacity, (ii) SFT/GRPO temporal supervision, and (iii) increased inference-time computation? At minimum, this should compare the Base Oracle, a prompted or untrained Navigator under matched compute, a trained Navigator at fixed compute, and the same trained system across multiple test-time budgets.
  2. Why is a separate lightweight 3B Navigator needed instead of allowing the Oracle to perform temporal navigation itself? Please compare the proposed decoupled architecture with an Oracle self-navigation baseline under matched temporal supervision and total inference compute, reporting temporal localization quality, final QA accuracy, inference cost, and latency.
  3. Can the authors provide a broader accuracy-versus-compute curve and compute-matched baselines that allocate the same total FLOPs to additional frames, repeated Oracle sampling, longer reasoning, or a larger visual context? Does TimeQuest continue to improve, saturate, or degrade as the retrieval budget increases?
  4. Can the authors report the total end-to-end inference cost of Base and TimeQuest, including the Navigator scan, all intermediate and final Oracle calls, total processed frames, FLOPs, latency, peak memory, and throughput?
  5. What explains the large regressions on Egocentric Sequence and Unexpected Action? Can a confidence-based fallback to the Base model or a hybrid visual-textual evidence representation mitigate these failures?
  6. How are predicted timestamp turns matched to multiple ground-truth intervals during GRPO, including different orders, missing or additional intervals, and alternative valid evidence segments? Why is the reward based on absolute seconds rather than duration-normalized error, and what assumptions justify mapping a mean nonlinear reward of 0.37 to approximately 1.7 seconds of average boundary error?
  7. Can the authors report multiple-seed results or bootstrap confidence intervals, release the complete SFT and GRPO hyperparameters, and provide a precise definition and composition of the internal test dataset used in the ablations?
  8. Please clarify the distinction between the fixed maximum-budget settings and the Adaptive setting in Table 13. Since the fixed-budget Navigator can apparently terminate early by emitting the termination token, in what sense is the final setting uniquely adaptive?
4 comments

We thank the reviewer for the constructive assessment of our work. We appreciate the recognition of TimeQuest's architectural contribution, cross-backbone transferability, and substantial improvements in temporal grounding. Below, we clarify the scope of our contributions, provide additional diagnostic analyses, and address the remaining methodological and experimental concerns.

W1 & Q1: Disentangling navigation, training, and inference-time computation

Our primary objective is to investigate whether a separately trained temporal-navigation module can improve frozen Video-LLMs through targeted evidence acquisition. Several existing ablations help distinguish the contributions of evidence relevance and learned navigation:

  • Table 7: TimeQuest-guided retrieval improves accuracy from 76.10% to 81.21% over random retrieval, with the same number and duration of retrieved segments. This isolates the importance of evidence relevance rather than simply retrieving additional visual content.
  • Table 8: The trained Navigator improves EgoSchema accuracy from 60.2% to 61.6% over a prompted pretrained 3B Navigator under the same retrieval protocol, with slightly fewer average retrieval turns (2.2 versus 2.3). This supports the benefit of learned navigation without increasing the number of retrieval steps.
  • Table 9: SFT+GRPO improves EgoSchema accuracy from 53.60% to 58.00% compared with SFT-only navigation, demonstrating the additional contribution of reinforcement learning in this ablation setting.

These experiments provide complementary evidence that retrieval relevance and learned temporal navigation contribute to TimeQuest's improvements. Importantly, the Navigator is trained once and transferred across five heterogeneous frozen Oracle backbones, rather than requiring each answering model to be fine-tuned.

The existing ablations isolate individual design choices but do not constitute a complete factorial comparison controlling model capacity, supervision, and total inference computation simultaneously. We plan to extend this analysis with compute-matched comparisons across prompted and trained Navigators, together with inference-budget variations applied to a fixed trained policy, to further quantify these contributions.

W2: Grounding supervision and final-answer accuracy

TimeQuest uses dedicated temporal supervision for the Navigator, whereas the Base Oracle models do not receive equivalent training. The grounding results should therefore be interpreted as system-level comparisons rather than identical-supervision comparisons.

Nevertheless, the evaluation extends beyond prompted Base models. Table 2 also includes temporally trained approaches such as Grounded-VideoLLM, TVG-R1, and VideoChat-R1, together with the evidence-acquisition framework Zoom-Zero. While these systems differ in their training and evaluation configurations, they provide additional context for TimeQuest's grounding performance.

More importantly, the gains are not limited to standalone localization metrics. On NExT-GQA, Acc@GQA, which jointly requires correct answering and temporal grounding, improves from 41.0% to 47.0% for Qwen2.5-VL and from 31.2% to 44.7% for Qwen3-VL. The corresponding Acc@QA improvements are smaller (73.9%→74.8% and 74.2%→75.0%, respectively).

These results indicate that the primary benefit is stronger evidence localization and grounded answering, accompanied by more modest improvements in raw QA accuracy. This distinction is important to our interpretation: improved grounding does not necessarily translate into an equally large increase in final-answer accuracy.

Continued in #2.

W3 & Q5: TVBench regressions and description-related failures

The regressions on Egocentric Sequence (−14.5 points) and Unexpected Action (−15.9 points) indicate that the current evidence representation is not uniformly beneficial across all temporal reasoning tasks. In particular, fine-grained temporal relationships may be difficult to preserve through textual descriptions.

Appendix B.10 examines 50 selected failures and identifies three categories: information lost during textual conversion (46%), incomplete or incorrect retrieval (32%), and incorrect final reasoning (22%).

To further investigate description-related errors, we conducted an additional diagnostic analysis of the 23 cases initially categorized as textual information loss. We inspected the exact sampled video input received by the final Oracle and compared three conditions while keeping that visual input fixed: the original descriptions, corrected descriptions restoring omitted information, and no descriptions.

Visibility in final video inputCasesCorrect: original descriptionsCorrect: corrected descriptionsCorrect: no descriptions
Relevant detail visible1901815
Absent or insufficiently resolved4041
Total2302216

The original-description condition has zero correct answers by construction, since these cases were selected from TimeQuest's failures. Correcting the descriptions recovered 22 of 23 cases, while removing descriptions recovered 16, including 15 cases where the relevant visual details were already available in the final Oracle's sampled input.

These findings suggest two related failure mechanisms. First, descriptions may omit relevant details that are not adequately preserved in the original sampled video. Second, when relevant details are already visible, inaccurate or incomplete descriptions may interfere with their integration during final answering.

This diagnostic analysis identifies plausible mechanisms but does not establish the specific causes of the two TVBench category regressions. It also concerns selected failures rather than the full evaluation distribution. We plan to investigate confidence-based fallback and hybrid visual-textual evidence aggregation to better handle cases where textual descriptions are insufficient or potentially misleading.

W4 & Q6: Temporal reward, interval correspondence, and alternative evidence paths

Our distance-based reward uses boundary errors measured in absolute seconds to encourage precise localization of brief events. This reflects the observation that even a small temporal offset can miss a decisive event, independently of the overall video duration. Unlike IoU-based rewards, the proposed formulation provides a nonzero localization signal for non-overlapping intervals. In Table 4, this formulation improves accuracy from 77.32% to 81.21% compared with the IoU-based alternative.

We recognize that duration-normalized or evidence-length-normalized rewards may offer advantages for videos or evidence intervals with substantially different temporal scales. The current ablation establishes the benefit of our reward over the evaluated IoU alternative, rather than its optimality across all settings.

For multi-interval evidence, the current training formulation uses step-aligned correspondence between predicted and annotated timestamps. The termination objective is defined relative to the annotated final step. This provides a consistent supervision signal for sequential localization but does not explicitly account for all alternative valid evidence trajectories.

Consequently, reordered or alternative evidence intervals, as well as shorter trajectories that already contain sufficient evidence, are not necessarily rewarded equivalently to the annotated sequence. The annotations serve as a practical supervision target rather than an exhaustive specification of all valid evidence paths. Future extensions could incorporate order-invariant interval matching and evidence-sufficiency-aware termination to make the objective more flexible.

Finally, we agree that a mean nonlinear distance reward of 0.37 cannot generally be interpreted as a measured average boundary error of approximately 1.7 seconds. That value corresponds only to an equivalent constant boundary error under a simplifying assumption, not the average error across heterogeneous predictions. We will correct this interpretation.

Continued in #3.

W5 & Q7: Statistical reliability and reproducibility

The reported improvements vary in magnitude, and small accuracy differences should be interpreted with appropriate statistical caution. Our empirical evidence includes positive improvements across all 20 Base-versus-TimeQuest comparisons in Table 1, together with substantially larger grounding gains on NExT-GQA. This consistency across different Oracle backbones and benchmarks supports the generality of the observed effect, although it does not replace formal uncertainty estimates.

We agree that confidence intervals and multiple-seed evaluations would help characterize the reliability of smaller improvements and the variability of Navigator training. A more complete statistical evaluation will therefore be an important part of extending this work.

Regarding reproducibility, the Navigator is trained on 15,453 samples derived from the Grounded-VideoLLM training dataset, comprising ActivityNet Captions and QVHighlights sources, using SFT followed by GRPO with LoRA adaptation. We also provide the implementation anonymously as supplementary material.

We plan to strengthen the experimental documentation with a complete account of SFT/GRPO hyperparameters, the construction and composition of the internal test set, and the evaluation configurations. These details, together with bootstrap confidence intervals and repeated-seed evaluations where feasible, would allow more precise assessment of the reported improvements.

W6 & Q4: End-to-end inference cost and computational efficiency

Our computational analysis focuses on the cost of accumulating visual context during multi-turn evidence acquisition, rather than comparing TimeQuest directly with a single-pass Oracle.

Under the specified three-turn configuration, the theoretical prefill cost decreases from 6,656 TFLOPs for a monolithic 7B architecture to 1,935 TFLOPs for our 3B Navigator + 7B Oracle design, representing a 3.4× reduction. The analysis assumes a 200-frame high-resolution video, a 100-frame low-resolution proxy, three 90-frame retrieval cycles, and 784 pixels per visual token.

This comparison isolates the computational benefit of independently processing retrieved clips and retaining textual evidence instead of repeatedly expanding a high-resolution visual context.

TimeQuest does introduce additional computation relative to single-pass Oracle answering through Navigator inference and intermediate Oracle calls. Table 13 characterizes part of this workload through realized retrieval turns and Oracle calls, but the theoretical FLOPs comparison should not be interpreted as a measured end-to-end speedup.

A complete practical efficiency evaluation would additionally account for all inference calls, visual preprocessing, generation costs, latency, peak memory, throughput, and cache assumptions. We plan to incorporate these measurements and compute-matched baselines in a more comprehensive efficiency study.

Continued in #4.

W7 & Q3: Evidence acquisition versus inference-time scaling

Our motivation is to explore a different allocation of inference-time computation: actively acquiring relevant visual evidence instead of exclusively increasing text-space reasoning.

Table 3 directly compares these approaches using Qwen3-VL 8B. TimeQuest produces larger temporal-grounding improvements than the evaluated Thinking configuration, while combining TimeQuest with Thinking achieves the strongest overall results. This supports the complementarity of visual evidence acquisition and text-space reasoning.

Table 13 provides additional insight into retrieval allocation. Policies trained with maximum allowances of one, two, and three retrieval turns achieve EgoSchema accuracies of 49.1%, 49.6%, and 49.7%, with average realized turns of 1.0, 1.7, and 1.9. The less restrictive Adaptive configuration achieves 50.4% accuracy with 2.2 average turns.

These results show diminishing improvements over the evaluated configurations. However, because the policies have different training-time retrieval allowances, this experiment is not equivalent to varying the inference budget of a single fixed policy.

We therefore distinguish two claims: our experiments demonstrate the effectiveness of allocating additional inference-time computation to targeted evidence acquisition, but do not establish a general scaling law or a compute-optimal performance frontier. The latter would require broader accuracy-versus-compute evaluations with fixed policies and matched total inference budgets, which we plan to investigate.

Q2: Why use a separate lightweight Navigator rather than Oracle self-navigation?

The separation is motivated by specialization and reusability. The lightweight Navigator can be trained specifically for temporal localization and reused across frozen answering models, avoiding the need to train each Oracle for interactive navigation.

It also separates two computational roles: low-resolution temporal exploration is handled by the Navigator, while high-resolution localized observation and final reasoning are handled by the Oracle. Retrieved clips are processed independently, with descriptions retained as compact evidence rather than continually expanding the multi-turn visual context.

Our results support the practical value of this separation. The same trained 3B Navigator improves performance across five heterogeneous frozen Oracle families. Additional transfer experiments with Qwen3-VL-32B-Instruct achieve 79.9% on Video-MME, compared with a published baseline of 76.6%. The latter result demonstrates compatibility with a substantially larger Oracle, although it is not a fully controlled matched-budget comparison.

A directly supervised Oracle self-navigation baseline would help isolate the benefit of architectural decoupling from the benefit of temporal specialization. We plan to examine this comparison under matched supervision and inference budgets.

Q8: Fixed versus Adaptive retrieval configurations

The distinction in Table 13 concerns the maximum retrieval allowance used during Navigator training, not whether early termination is supported at inference time.

The settings labeled 1, 2, and 3 correspond to separately trained policies with different maximum retrieval allowances. All support early termination through the <end> token. Their average realized inference-time retrieval turns are 1.0, 1.7, and 1.9, respectively.

The Adaptive configuration uses a less restrictive training-time allowance, resulting in an average of 2.2 retrieval turns and the highest reported accuracy of 50.4%.

Thus, the distinction lies in the training-time exploration allowance rather than the presence or absence of adaptive stopping. These results indicate that allowing more flexible retrieval trajectories during training can improve the resulting policy, although the differences cannot be attributed exclusively to additional inference-time computation because the policies are trained separately.

We will clarify the table headings and caption to distinguish training-time constraints from realized inference-time retrieval behavior.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.