acceptodds
Under review as a conference paper at ICLR 2027

Seeing the Small in the Long: Resolving Where as Well as When for Tiny Evidence in Long Videos

Abstract

Agentic long-video systems have learned to search the timeline, and left one regime unresolved: seeing the small in the long. Every one we evaluate exposes a single tool signature, given a time interval return more frames, so it re-samples when the model looks and never where in the frame. That is insufficient when the queried detail is illegible at the resolution those frames arrive in. To measure that regime we build FoveaBench: 1,044 open-ended questions over 454 videos whose answers turn on a target occupying a median 0.31% of the frame of a median 23-minute video. An item is admitted only if a gate model answers wrongly from a whole-frame span near the evidence moment and correctly from a native-resolution view of it. What the suite exposes is an inversion rather than uniformly low scores: the tool-augmented systems we evaluate score below the tool-free model they are built on, though all of them call their tool on every question. An oracle study over the same items prices what focus is worth, and corrects an intuition: the anchoring interval at native resolution is worth 4.2%; the ground-truth box cropped out is worth less than the frame it was cut from; that same box rendered as a tracked, dilated, context-preserving view is worth 19.6%. That last view changes several things at once, so it prices the rendering pipeline as a bundle rather than any ingredient of it. We then build FoveaSeek, which resolves when first and where in the frame second: it commits to an interval together with an object hypothesis, localises that hypothesis itself as a policy-emitted box rather than by a detector call, so where to look is an action that rollout-level reward can reach, and answers from native-resolution close-ups of the propagated tube. Fine-tuning an 8B backbone reaches 25.2%, against 18.2% for the strongest evaluated baseline; a single-seed reinforcement-learning run under a temporal process reward reaches 31.1%. FoveaSeek demonstrates that the regime is addressable; which ingredient of the rendering bundle carries the gain is left to future measurement.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
long video understanding, agentic reasoning, tool-augmented LLMs, spatio-temporal grounding

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_8uL5OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces FoveaBench, a benchmark targeting small visual evidence in long videos, and FoveaSeek, an agent combining temporal search with spatial localization and tracked close-up inspection. FoveaBench contains 1,044 questions over 454 videos, selected through model-based gates intended to identify cases where low-resolution whole-frame observations are insufficient but magnified evidence enables answering. An oracle study compares different evidence renderings. FoveaSeek predicts a temporal interval and target description, emits a bounding box, and uses SAM 2 to obtain tracked crops with contextual views. Supervised fine-tuning achieves 25.2% accuracy versus 18.2% for the backbone, while an exploratory single-seed RL run reaches 31.1%.

Critical strengths
  1. Important evaluation setting. Small evidence in long videos is a meaningful challenge that temporal resampling alone may not resolve. Releasing target boxes and annotated instants makes the benchmark potentially useful for studying both temporal search and spatial grounding.
  2. Informative oracle study. Comparing whole videos, evidence windows, individual frames, tight crops, and context-preserving tracked views demonstrates that locating evidence and presenting it effectively are different problems. The finding that a tight crop can underperform its source frame highlights the importance of context.
  3. Meaningful empirical gains. The supervised model improves over its own backbone by 7.0 percentage points, providing evidence that the setting is addressable. The fixed-weight observation ablations further demonstrate that performance depends on access to the tool’s returned evidence.
  4. Transparent reporting. The paper carefully discloses unmatched budgets, benchmark-selection effects, unavailable spatial baselines, and the preliminary status of RL. Paired statistical tests, cross-family judge agreement, a human audit, and documented training–evaluation separation strengthen the experimental presentation.
Critical weaknesses
  1. Benchmark validity. The admission gates do not consistently isolate spatial legibility from temporal evidence availability. For 16.5% of egocentric items, the whole-frame gate excludes the annotated evidence frame. Failure in this condition can therefore reflect missing evidence rather than insufficient resolution. Moreover, the whole-frame and magnified conditions differ in resolution, spatial guidance, and presentation, so passing the gates establishes sensitivity to these particular renderings rather than the necessity of spatial magnification alone. The human audit is useful, but covers only 100 construction candidates, uses one reviewer, and assesses legibility at a different resolution. Re-running admission with guaranteed evidence-frame coverage and matched contextual information would strengthen the benchmark’s central premise.
  2. Incomplete controls. No spatially aware or query-adaptive resolution baseline is evaluated. The lack of public weights for the closest agents is understandable, but simpler controls remain important: high-resolution keyframe selection, static dilated crops with context, or query-aware allocation of the same visual-token budget. The oracle rendering simultaneously changes dilation, context, frame count, tracking, and budget. Similarly, the temporal-only ablation evaluates a tube-trained policy under different observations, which measures dependence on its trained interface rather than the best achievable temporal-only alternative. The authors acknowledge these limitations, but they leave the necessity and distinctive benefit of the proposed procedure insufficiently established.
  3. Missing grounding evidence. Although policy-emitted boxes are central to the method, spatial localization is not evaluated quantitatively. Answer accuracy alone cannot establish whether training improves box selection, temporal search, or answer generation. This is particularly important for RL, whose explicit process reward supervises time rather than space. There is also ambiguity in the reward reference: Appendix F distinguishes generator-produced anchoring intervals from certified evidence instants, with the annotated instant inside the shipped interval for only 56.9% of egocentric items. This does not directly establish the quality of RL training windows, which needs separate validation. Aggregate localization metrics, reward ablations, and repeated RL runs would clarify what is learned.
  4. Limited coverage. Of the 1,044 questions, 939 concern colour, substantially narrowing the benchmark’s coverage of fine-grained perception. Both domains also come from the Ego4D/Ego-Exo4D ecosystem. The question-only construction filter does not establish that language priors are negligible: it explicitly permits abstention, and the gate model returns UNKNOWN on 982 retained items. A forced-answer text-only baseline on the final benchmark is needed to measure residual predictability. Furthermore, general-video results omit the actual Qwen3-VL-8B backbone and use unmatched configurations, so they cannot quantify the specialization cost relative to the model’s starting point.
  5. Presentation. The manuscript is overly narrative and repetitive, repeatedly restating the motivation and caveats while scattering essential implementation and evaluation details across multiple appendices. A concise algorithm and a clearer separation of benchmark construction, method design, and experimental findings would substantially improve readability. Despite the manuscript’s length, the decisive empirical evidence remains limited: the main results rely on one custom benchmark and one backbone, the central tool ablations use a single exploratory RL checkpoint, and several general-benchmark comparisons quote external results. Important spatial baselines, quantitative grounding evaluations, and controlled rendering ablations are missing. Consequently, the work feels preliminary, with the breadth of the narrative exceeding the completeness of the experimental validation. Explicitly acknowledging these gaps does not substitute for addressing them.
Questions for authors
  1. Validity: Can you re-run the whole-frame gate with the annotated evidence frame guaranteed to be included, and report results separately for the currently affected 16.5% of egocentric items?
  2. Controls: How does FoveaSeek compare with high-resolution keyframe selection or static multi-frame crops with context under matched total visual-token budgets? Can you separate the contributions of dilation, contextual views, tracking, and additional frames?
  3. Grounding: Can you report temporal coverage of certified instants, box IoU or target-hit rates, and tracking success before and after RL? How are the reward’s reference intervals validated, and how does answer-only RL compare with temporal shaping?
  4. Generality: What accuracy do forced-answer text-only models achieve on the released questions? Please also provide non-colour results and evaluate Qwen3-VL-8B on the general-video benchmarks under the same configuration as FoveaSeek.
Reviewer_QZKpOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper introduces FoveaBench, a benchmark of 1,044 open-ended questions whose visual evidence occupies a median 0.31% of the frame, making it very difficult for current video models to perceive. Without any fine-tuning, every evaluated video model scores below 20% on it. The paper then presents FoveaSeek, a three-stage training framework that resolves both when and where to look, reaching 31.1%.

Critical strengths
  • The paper proposes a genuinely hard and interesting benchmark in which the visual evidence is difficult to retrieve, and on which most VLMs score below 20%. The benchmark propose a relatively new and interesting idea of testing the capabilities of VLMs to answer about very small details in the video just like human.
  • The paper proposes a training framework that achieves strong performance relative to other models with limited training.
Critical weaknesses
  • General video understanding remains weak relative to prior methods, even though the model is built on Qwen3-VL-8B while the baselines are mostly Qwen2.5-VL-7B.
Questions for authors

The model's general video understanding appears notably weaker than that of the baselines. If possible a budget-matched comparison against Qwen3-VL-8B on these suites would separate a genuine specialisation cost from a mere allocation choice.

Reviewer_UBntOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work points out that existing agentic long-video systems primarily address "searching along the timeline" but fail to resolve the challenge of "identifying small objects within long videos." Typically, these tools merely return additional frames from specific time intervals, focusing on when to look, rather than where to look within a frame. The authors introduce FoveaBench, a benchmark comprising 1,044 open-ended questions across 454 videos. A "gate model" is employed to filter for questions that are answered incorrectly when the full frame is viewed, but correctly when a cropped view at native resolution is provided. Evaluations reveal that tool-augmented systems underperform compared to their base models (without tools); the authors further utilize an "oracle" study to analyze the costs and benefits of the focus-and-render pipeline.

Critical strengths

Targeted problem definition and benchmark design: FoveaBench uses a gate model to ensure that questions necessitate the use of fine-grained, native-resolution details, effectively isolating the specific, unresolved challenge of "identifying small objects in long videos."

Valuable, counterintuitive findings: The study shows that tool-augmented systems, even when invoking tools for every question, perform worse than the underlying base models. This suggests that simply increasing frame sampling or temporal searching is insufficient to address the problem of spatial focus.

"Where to look" as a learnable action: FoveaSeek does not rely on external object detectors; instead, its policy outputs bounding boxes to locate objects. This allows spatial localization to be optimized via rollout-level rewards, leading to quantitative results that significantly outperform the strongest baselines.

Critical weaknesses

No baseline uses spatial cropping or inputs at different resolutions. So it is hard to know whether “stating a region” is really needed.

There is no single combined localization score. So it is unclear whether RL improved spatial localization, temporal search, or both.

The gate uses a model from the same family. The final selection keeps only questions that native-resolution crops can solve. So the benchmark is not fully neutral about spatial interfaces.

All results use one 8B backbone and two video datasets from the same family. This limits how far the findings can be generalized.

The control for “answer priors” is a filter at build time, not a measurement. So the relative contribution of visual evidence and language priors is still not cleanly measured.

Questions for authors

Please refer to the weaknesses.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary for the Discussion Period

We thank the AC and the reviewers for their time. This comment summarizes the discussion, and our responses under each review give the details.

What the reviews agree on. All three reviews value the problem and the benchmark. Reviewer UBnt notes "valuable, counterintuitive findings" and results that "significantly outperform the strongest baselines". Reviewer QZKp calls the benchmark "genuinely hard and interesting". Reviewer 8uL5 names an "important evaluation setting", an "informative oracle study" and "meaningful empirical gains".

Six new results. The weaknesses ask for controls and measurements. The results below answer each weakness, and they support the claims of the paper.

  1. The benchmark needs spatial magnification (UBnt W3, 8uL5 W1). With a gate model of another family (GPT-5.6), magnification adds 21.8 points. With the context matched and the crop as the one difference, the crop still adds 23 to 29 points for the original gate model and 9.7 points for GPT-5.6. Without the items whose whole-frame span excludes the annotated frame, FoveaSeek-SFT is 7.3 points above its backbone (+7.0 on all items). Other interfaces also gain (point 3), so the suite does not require our tool.
  2. The video carries the gain, not answer priors (UBnt W5, 8uL5 W4). When no system may abstain, FoveaSeek-SFT is the one system of six with a significant gain from the video (+6.8 points). On the 591 items that no question-only run answers, it is 7.1 points above its backbone. On the 105 non-colour items, it gains 8.6 points.
  3. A stated region is what helps (UBnt W1, 8uL5 W2). At a matched visual budget on Qwen3-VL-8B, the best region-free control gains 6.5 points. A region from a detector, with the same frames and budget, adds 3.5 points to it. FoveaSeek-SFT+RL, whose own policy states the region, gains 12.9 points, with no frame selector and no detector. A trained temporal-only agent gains 5.3 points, below FoveaSeek-SFT (+7.0).
  4. RL learns when to look (UBnt W2, 8uL5 W3). After RL, the median distance from the window centre to the annotated instant falls from 220 to 167 s, on a benchmark that shares no video with the RL items. The window also shows the object more often: 74.0% of the items, against 70.5%. The reward intervals contain the annotated instant for 99.5% of the RL items.
  5. Dilation carries the rendering gain (8uL5 W2). In the oracle setting, with one factor changed at a time, a crop with its surroundings is 9.9 points above the tight crop of the same size, with the visual tokens matched. The revised abstract reports this result.
  6. The recipe transfers at no general-video cost (UBnt W4, QZKp W1, 8uL5 W4). On Qwen2.5-VL-7B, the same supervised recipe gains 10.3 points over its backbone, and the three tool-augmented systems on it are 0.6 to 3.1 points below it. On the seven general-video suites with the same configuration, FoveaSeek-SFT+RL is within 0.2 points of Qwen3-VL-8B on average.

The revised paper (8uL5 W5). Section 5 gives the agent loop as Algorithm 1. The Limitations paragraph reports the main measurements, and the new last appendix (Appendix P, Tables 14 and 15) gives all of them. Every change is in blue. No number of the submission changes, and the section, table and figure numbers stay the same.

Discussion status. Reviewer QZKp has no further question and keeps the positive score. Reviewers UBnt and 8uL5 did not reply yet, and their reviews came before these results. Reviewer 8uL5 names three missing items: "spatial baselines, quantitative grounding evaluations, and controlled rendering ablations". Points 3, 4 and 5 give all three. We would be glad to answer any further question.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.