acceptodds
Under review as a conference paper at ICLR 2027

GazeLens for Efficient Egocentric Video Question Answering with Simple Gaze-Driven Adaptive Token Reduction

Abstract

Smart glasses can help users understand their activities and surroundings, but small vision-language models must preserve useful evidence within limited computation. This is difficult because redundant frames consume the visual budget, while aggressive spatial compression can discard relevant details. Although gaze identifies the wearer's focus, the question may require evidence from the surrounding scene. To address these constraints, we propose GazeLens, which combines frame selection with gaze- and question-guided token reduction. First, HOG transitions and image sharpness select frames before visual encoding. For each selected frame, Gaussian pooling then forms gaze-focused and complementary summaries. Because their relevance depends on the question, a lightweight gate blends them into a primary token. Together with a global scene token, this retains both focused and contextual summaries within two tokens per frame. On StreamGaze, GazeLens achieves higher macro and micro accuracy than the evaluated full-grid and token-reduction baselines, with lower input-forward FLOPs than the shared-backbone token-reduction baselines.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric video question answering, gaze-guided visual token compression, vision-language models, adaptive frame selection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Phv8OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

Paper proposes a novel method to incorporate effecient gaze information in various ego-centric VQA task. The central claim is that effecient modelling of gaze information can reduce both compute cost (tokens) and increase task accuracy.

The multi-stage pipeline as follows:

  1. selects useful frames dynamically with HOG transition cues and Laplacian sharpness scores to identify frames changes.
  2. generate a anisotropic Gaussian map central at the gaze point to incorporate gaze information (proxy for visual attention)
  3. condition a semantic gate on the question to control the balance between source 1 and 2

LoRA was trained on a frozen light-weight VLM (SmolVLM2-500M) with the selected benchmark: StreamGaze.

The proposed method was benchmarked against two groups of baseline: (i) small VLMs without token reduction or gazetuning; for full token grids and standard inference pipelines comparison (ii) token-reduction methods re-implemented on the SmolVLM2-500M framework on identical hardware; for vis-a-vis comparison

Proposed method achieves the lowest TFLOPS while outperforming SOTA accuracy. A CROSS-DATASET GENERALIZATION test was also done to evaluate the domain-transferability of the approach. Ablation study was also done to assess the contributions of the stages in the pipeline.

Critical strengths
  1. Paper proposes a approach to incorporate effecient gaze information for ego-centric VQA
  2. Method is somewhat novel.
  3. Experimental results reasonably support the claim.
Critical weaknesses
  1. Method is incrementally novel. The use of gaussian mask to represent visual attention has been long used in many prior works. The use of a MLP to weight the 2 different sources conditioned on the question text embedding is also not novel.
  2. Experiment on a single dataset is weak evidence for generalization of the performance. This is especially the case as there is a training phase involved.

The claim for evaluation against the small VLMs without token reduction or gazetuning is so that the full set of visual information is available to the VLM. But the experiment result show that the proposed method outperforms these full information design. This is contrary to the experiment design. Please explain.

Questions for authors

Refer to Weaknesses comment and explain and/or elaborate if the weaknesses identify are correct or reasonable.

1 comment
AuthorsOpenReview ↗

Follow up on Response

Dear Reviewer Phv8,

Thank you again for your review. We have addressed your concerns. We would love to hear your feedback, especially if there are any remaining questions. As your evaluation weighs significantly in the final decision, we would greatly appreciate your thoughts before the discussion period closes on November 18.

Best regards,

Authors

Reviewer_pm24OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

GazeLens combines frame selection before visual encoding with compact gaze-guided spatial pooling for egocentric video QA. HOG transitions and image sharpness select a variable number of frames. Each retained frame produces gaze-focused, complementary, and global summaries. A question-conditioned scalar blends the focused and complementary summaries, and this mixture is paired with the global summary to provide two tokens per frame.

Critical strengths
  • the combination of temporal reduction before encoding with spatial reduction before language-model processing is a useful practical design. It addresses both redundant encoder inputs and the downstream visual-token budget so it's a 2-in-1 solution. Preserving complementary and global context alongside gaze-focused information is also well motivated.

  • The results show a promising accuracy–computation tradeoff on small VLMs. The 500M configuration achieves 53.94% Macro and 64.01% Micro accuracy on StreamGaze, and lower FLOPs than the evaluated shared-backbone token-reduction baselines. The appendix also provides useful spatial-budget controls: increasing to four or sixteen grid tokens gives only small aggregate accuracy gains.

Critical weaknesses
  • The comparisons do not sufficiently isolate the contribution of the proposed integration. GazeVLM is discussed as a closely related method but is absent from the main experimental comparison. One thing to try is to add an off the shelf frame selection algorithm to GazeVLM and see its performance as a baseline.

  • The evidence for question-adaptive gating is incomplete. The gate depends only on the question and shares one coefficient across the clip and it does not translate its mixture to individual frames. Figure 5 also doesn't show the full story; we don't get to know the question or gaze conditioned on it would be nice if we can see the selected gaze tokens as well as add the textual prompt or question. A separately trained constant-gate baseline near the learned mean would better establish whether question conditioning contributes beyond a generally useful gaze/context mixture.

Questions for authors
  • How does GazeLens compare with GazeVLM, or another gaze-guided baseline, when both use the same frame selector and comparable spatial-token budgets?

  • Does the question-conditioned gate outperform a separately trained constant mixture, particularly one initialized near the observed mean of 0.65? Which question types benefit most from the variation in the learned gate?

Reviewer_31RcOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes GazeLens, an efficient framework for egocentric video QA that combines HOG- and sharpness-based frame selection with gaze-guided spatial pooling and question-conditioned gating to reduce visual inputs to two tokens per frame. Experiments on StreamGaze show improved accuracy and lower model-forward FLOPs compared with the evaluated baselines. The authors also provide component ablations and cross-dataset evaluation on EgoGazeVQA.

Critical strengths
  1. The paper addresses a practical problem of reducing visual computation for egocentric video QA. Combining gaze-focused and peripheral information within only two visual tokens per frame is a simple and potentially useful design for resource-constrained VLMs.

  2. The method achieves promising accuracy-efficiency trade-offs on StreamGaze. GazeLens outperforms the evaluated token-reduction baselines in Macro and Micro accuracy while reporting lower model-forward FLOPs. The paper also provides several ablation studies and cross-dataset experiments to evaluate the proposed components.

Critical weaknesses
  1. The contributions of the proposed components are not sufficiently isolated. In particular, Figure 6 shows that the question-conditioned gate varies very little across questions (mean 0.6509, standard deviation 0.0253), but Table 4 does not compare it with a fixed gate using the learned mean. It is unclear whether the improvement comes from meaningful question adaptation or simply a better fixed mixing ratio. Moreover, the individual contributions of frame selection and gaze-guided spatial compression are not fully separated. These limitations make it difficult to assess the actual contribution of the proposed design.

  2. The efficiency improvements are not evaluated under fully comparable settings. GazeLens processes fewer frames than the baselines (12.72 on average vs. 16), while the reported FLOPs exclude frame selection, preprocessing, and answer generation. It is therefore unclear how much of the computational savings comes from spatial token reduction rather than selecting fewer frames. Without matched-budget comparisons and end-to-end latency measurements, the claimed efficiency advantage is not sufficiently supported.

Questions for authors
  1. Could the authors provide a more controlled ablation to separate the contributions of dynamic frame selection and gaze-guided spatial token reduction under matched frame and token budgets?

  2. Would a fixed gate of s = 0.65 or shuffled question-gate assignments achieve similar accuracy to the learned gate? This would help determine whether the improvement actually comes from question adaptation.

  3. How much of the reported FLOPs reduction remains when GazeLens and the baselines process the same number of frames? Could the authors also report end-to-end latency including frame selection and preprocessing?

  4. Could the authors clarify whether frame selection uses only the video history available at each query timestamp, particularly for present and proactive questions?

  5. Why is GazeVLM, which is discussed in the related work, not included in the experimental comparison? Could the authors clarify how GazeLens compares with this closely related method?

1 comment
AuthorsOpenReview ↗

Follow up on Response

Dear Reviewer 31Rc,

Thank you again for your review. We have addressed your concerns. We would love to hear your feedback, especially if there are any remaining questions. As your evaluation weighs significantly in the final decision, we would greatly appreciate your thoughts before the discussion period closes on November 18.

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.