acceptodds
Withdrawn from ICLR 2027

GazeSieve: Gaze-Guided Keyframe Selection for Egocentric Video Question Answering

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Egocentric video question answering often depends on a few evidence moments: an attended object, a brief state change, or an earlier observation that explains the query. The bottleneck is therefore selecting the few frames that contain sufficient evidence under a small visual budget, yet downstream answer accuracy alone cannot reveal whether selection succeeded. Recent video-language agents can perform query-conditioned search, but remain gaze-free and may miss moments humans naturally attend to. We present GazeKeyBench, a direct keyframe-selection benchmark that annotates acceptable evidence windows, required evidence groups, and alternative sufficient evidence sets, judging whether selected frames cover the evidence rather than land near one timestamp. We also present GazeSieve, a training-free gaze-guided selector that turns noisy gaze into fixation proposals, extracts query-relevant concepts, searches high-resolution gaze-centered crops under a gaze-informed temporal prior, and returns a relation-aware evidence set to a frozen MLLM. Under a unified controlled protocol that freezes the answerer, decoding, prompt, rendering, and frame budget, replacing uniform sampling with GazeSieve raises StreamGaze Macro-8 from 58.0 to 60.8 (+2.8, p < 0.01) on a frozen 4,700-unit roster. On GazeKeyBench's 2,607 human-annotated cases from 99 videos, GazeSieve reaches 56.0% Evidence Recall@4, outperforming a strong gaze-free Agent(VLM) baseline at 52.0%. A paired timing control further shows that preserving measured gaze timestamps adds 1.9 points (p = 0.0247). These results suggest that measured gaze provides complementary evidence-selection signal for egocentric video QA beyond query-conditioned video search alone.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CV: Language, Vision & Multi-modal

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_tYutOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses gaze-guided keyframe selection in egocentric video question answering. The proposed method leverages human gaze signals to identify potentially relevant moments, performs query-conditioned visual grounding, and selects a compact set of evidence frames based on temporal relations. The paper also introduces GazeKeyBench, a new benchmark that directly evaluates keyframe selection through human-annotated evidence windows and sufficient evidence sets. Experiments demonstrate some improvements over uniform sampling and several naive gaze-based selection methods.

Critical strengths
  1. I find the idea of incorporating human gaze signals into egocentric video understanding relatively novel and interesting. Using gaze to guide evidence selection offers a promising direction for improving the efficiency and effectiveness of video understanding.
  2. The paper introduces GazeKeyBench, a new benchmark for directly evaluating keyframe selection. This is a valuable contribution because it enables the evaluation of evidence selection with gaze signals.
  3. The experimental results provide some evidence that GazeSieve improves upon simple uniform sampling and naive gaze-based selection strategies. These improvements offer empirical support for the effectiveness of the proposed approach.
Critical weaknesses
  1. The paper appears to be unfinished. In particular, Appendices D and E are empty. This affects the completeness and soundness of the submission.
  2. The overall presentation could be improved. The GazeSieve pipeline is relatively complicated, but the explanations are not always sufficiently clear. In particular, Figure 2 is poorly organized, and the text within its subfigures is too small to read comfortably, making it difficult to understand the overall workflow and assess the proposed method.
  3. The experiments primarily rely on Qwen2.5-VL-7B-Instruct, which is a relatively outdated vision-language model. Evaluating the proposed method with more recent and capable MLLMs would provide stronger evidence of its effectiveness and generalizability.
  4. The evaluated baseline, such as uniform sampling, is too simple. Maybe some learnable frame selection methods should be compared.
  5. The AI Use Statement (L479) still contains unfilled template placeholders rather than an actual disclosure. This further suggests that the manuscript is incomplete and raises concerns about whether the authors have adequately addressed the conference's AI disclosure requirements.
Questions for authors
  1. Could the authors explain why the submission is incomplete? (see weaknesses 1 & 5)
Reviewer_FZkSOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents GazeKeyBench and GazeSieve. GazeKeyBench is a benchmark that directly evaluates keyframe selection by annotating 2,607 QA cases from 99 StreamGaze videos with a three-level evidence structure (equivalent windows → required groups → sufficient sets) and scoring whether the selected frames cover the evidence needed to answer each question via Evidence Recall@K, All-Group Coverage@K, and Precision@K. GazeSieve is a training-free gaze-guided keyframe selector that, through fixation filtering and concept extraction (Phase 1), gaze-prior-driven foveal temporal search (Phase 2), and a relation-aware evidence finalizer, selects up to 4 frames for a frozen MLLM. Under a controlled protocol (frozen Qwen2.5-VL-7B answerer, fixed decoding and frame budget), GazeSieve raises StreamGaze Macro-8 from 58.0 (uniform sampling) to 60.8 (+2.8, p<0.01) and achieves 56.0% Evidence Recall@4 on GazeKeyBench, above the strongest gaze-free baseline Agent(VLM) at 52.0%.

Critical strengths
  1. The three-level evidence annotation is well thought out. The "equivalent window → required group → sufficient set" design (§4, Fig. 3) captures the fact that a question can have multiple alternative evidence routes, each needing several complementary pieces. This is clearly more realistic than single-interval labels and is the paper's most substantive contribution.

  2. The controlled evaluation protocol is solid. Freezing the answerer, decoding, candidate order, prompt, rendering, and frame budget so that only the selector varies (§5) makes the gains cleanly attributable. The source-video-clustered bootstrap and sign-flip tests (§5.1) are sound.

  3. The negative result about naive gaze use is genuinely informative. Gaze-weighted uniform (58.04), Gaze Top-K (58.32), and plain uniform (57.99) are statistically indistinguishable (p>0.7, §5.1, Fig. 4). This shows that simply having gaze is not enough — it takes structured exploitation to make a difference.

Critical weaknesses
  1. GazeKeyBench is largely a re-annotation of StreamGaze, and its design choices are not well justified. All 2,607 cases come from the same 99 StreamGaze videos and the same 10 task types (§4). The budget is fixed at K=4 with the argument that StreamGaze is four-way MCQ, but this reasoning conflates the number of answer options with the amount of evidence needed — and current MLLMs can process far more than 4 frames (GPT-4o uses 16 in Table 3). No results at K=8/16/32 are provided, so it is unclear whether GazeSieve's advantage persists under a larger budget. The concept of "minimal sufficient evidence set" is central to the benchmark's design, yet the paper does not explain how annotators determine minimality or sufficiency, and basic annotation statistics — number of annotators, inter-annotator agreement, average window width, average groups per case — are all absent. Additionally, the benchmark is evaluated with GazeSieve's own baselines only; there are no results from existing frame-selection methods (e.g., SeViLA, VideoTree) or other MLLMs beyond the three ≤8B models tested.

  2. The benchmark has no gaze-specific evaluation dimension, and the "no direct selection benchmark" framing is misleading. Despite being motivated by gaze-guided selection, GazeKeyBench's annotation does not record any gaze-related information — there is no way to tell, for instance, what fraction of evidence windows overlap with fixation events. The benchmark could in principle be used with any selector, gaze-based or not, which is fine for generality, but undermines the paper's positioning of it as a gaze-guided evaluation tool. More importantly, the introduction claims "downstream answer accuracy alone cannot reveal whether selection succeeded" (§1, lines 079–081), implying no direct selection evaluation existed. This is not accurate: NExT-GQA provides QA evidence intervals, Ego4D NLQ offers egocentric GT temporal intervals, and the authors' own cited GeLM (Chen et al., 2024) annotates scattered evidence intervals for multi-hop egocentric QA.

  3. The six relation templates in the finalizer appear overfitted to StreamGaze's task taxonomy, and the integration of gaze with the query is unclear. The templates map nearly one-to-one onto StreamGaze tasks. If task-type labels are used to select the template, this is benchmark-specific hand-design, and calling the method "training-free / general" is misleading — it also makes the comparison with Agent(VLM) uneven, since that baseline has no task-tailored composition rules. More broadly, how gaze signals are combined with the query throughout the pipeline is not clearly explained.

  4. The evaluation is narrow, key metrics are missing, and the ablations are incomplete. All evaluation uses a single benchmark source (StreamGaze/GazeKeyBench, 99 videos). EgoGazeVQA (Peng et al., 2025), cited in the introduction (line 055), is not tested. Only three answerers are used, all open-source ≤8B, and the gain is significant on only one (Qwen2.5-VL-7B; the other two are p>0.1, §6). AGC@4 is defined in §4 but never reported, so the reader cannot judge how often evidence is fully covered. Appendix D (per-task absolute scores) and Appendix E (external systems breakdown) contain only headings with no content. On the ablation side, Table 2 (Appendix B) removes grounding and the five relation templates, but does not ablate the two components claimed to distinguish GazeSieve from T*: there is no "remove gaze prior (use uniform prior)" or "remove foveal cropping (use whole-frame grid)" condition (§3.2, §3.4). The timing control (Appendix C) perturbs both simultaneously, so it cannot separate their individual contributions.

Questions for authors
  1. Which specific models are used for concept extraction (VLM) and open-vocabulary detection? What model and call budget does Agent(VLM) use? What is the end-to-end selector-side latency per unit? All core hyperparameters (λ₁, λ₂, N, g, crop size, wₜ, ε, Bₛ, detection threshold) are missing — could these be provided?

  2. How was the annotation conducted — how many annotators, was double-annotation used, what is the IAA? What are the average number of routes and groups per case, and the average window width? Will the annotations themselves (not just the schema and scoring code) be publicly released?

  3. If the frame budget is increased to K=8 or K=16, does GazeSieve's advantage over Agent(VLM) persist? And can you report Evidence Recall and AGC@4 on GazeKeyBench for Gaze Top-K, gaze-weighted uniform, and the timing-reassigned variant, to give a fuller picture of how simpler gaze strategies perform on the benchmark?

Reviewer_NcNtOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GazeSieve, a training-free framework for gaze-guided keyframe selection in egocentric video QA, and GazeKeyBench, a benchmark for directly evaluating evidence selection. GazeSieve first identifies fixation proposals from gaze signals and extracts query-relevant visual concepts. It then performs gaze-guided temporal search with object grounding and relation-aware frame selection. Experiments show that GazeSieve improves downstream QA accuracy from 58.0% to 60.8% over uniform sampling on StreamGaze. It also achieves 56.0% evidence recall on GazeKeyBench, compared with 52.0% for the gaze-free Agent(VLM) baseline.

Critical strengths
  1. Using human gaze to guide keyframe selection is a reasonable and well-motivated idea for egocentric video QA, especially when important evidence appears only briefly.

  2. GazeKeyBench provides a useful benchmark for directly evaluating evidence selection instead of relying only on downstream QA accuracy. The annotation design considers multiple acceptable evidence windows and alternative sufficient evidence sets.

  3. The proposed method is training-free and combines gaze-guided temporal search, semantic grounding, and relation-aware evidence selection without additional task-specific training.

  4. The paper provides controlled comparisons and ablation studies. The experiments use the same answerer, prompt, and frame budget across different selection strategies, and show improvements in both downstream QA accuracy and evidence recall.

Critical weaknesses
  1. The paper does not clearly explain how raw gaze signals are synchronized or resampled to video frames, or how fixations and saccades are distinguished when gaze-state labels are unavailable. Important implementation details, such as the velocity thresholds, temporal windows, and temporal NMS parameters, are also missing.

  2. In Figure 1, the question "After seeing the trash can, what did I cut?" appears to use gaze fixation as a temporal anchor. However, people can recognize objects through peripheral vision or brief visual scanning without directly fixating on them. It is unclear how the benchmark defines the moment of "seeing" an object and whether this ambiguity affects the annotated evidence windows.

  3. The paper reports better evidence recall than Agent(VLM), but does not include Agent(VLM) in the downstream QA accuracy comparison. Since Agent(VLM) is the strongest gaze-free baseline, a comparison under the same answerer and frame budget is needed to better demonstrate the effectiveness of GazeSieve.

  4. The annotation quality of GazeKeyBench is not sufficiently validated. The paper does not clarify how many annotators independently label each question or report inter-annotator agreement. It is also unclear how the authors verify that the annotated evidence sets are sufficient to answer the questions and that all frames within an evidence window provide equivalent evidence.

  5. The experiments are limited to relatively small MLLMs (3B–8B), and the improvements are statistically significant only for the primary 7B model. Evaluation on larger models would help determine whether gaze-guided keyframe selection remains effective with stronger MLLMs.

Questions for authors
  1. Could the authors report the downstream QA accuracy of Agent(VLM) under the same experimental settings?

  2. Could the authors provide more details about gaze synchronization, fixation detection, and temporal NMS parameters?

  3. How are temporal anchors such as "seeing an object" defined and annotated when direct fixation may not be necessary?

  4. Could the authors clarify the number of independent annotators, inter-annotator agreement, and how evidence sufficiency and window equivalence are validated?

  5. Could the authors comment on the generalization to larger MLLMs or provide additional results if available?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.