- Rating
- 3
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The manuscript investigates the utilization of human eye-gaze data for egocentric video question answering. To address this challenge without relying on manual annotations of gaze usefulness, the authors introduce Reliance on Gaze (ReGaze), an adaptive frame selection framework driven by the proposed Adaptive Gaze Reliance Estimation (AGRE) module. Optimization of the AGRE architecture is achieved without explicit supervisory labels of gaze utility. Instead it is supervised by questions paired with their relevant intervals from gaze-recorded videos which are generated from the introduced Gaze-aware QA Curation (GQC) pipeline. ReGaze is evaluated against three groups of frame sampler baselines across four benchmarks of EgoExoLearn, EgoEverything, EgoGazeVQA, and WearerQA using InternVL3.5-8B and Qwen3.5-9B backbones.
Critical strengths
- The primary strength of this work lies in its systematic deconstruction of the prevailing assumption that human visual attention is universally beneficial for egocentric video understanding. To support the claim, the paper introduces Reliance on Gaze (ReGaze) to learn adaptive gaze reliance from egocentric videos. The design of the AGRE module to train the light-weight gaze reliance estimator without manual labels of gaze usefulness is also interesting and insightful.
- Another substantive strength resides in the comprehensive evaluation of ReGaze on four benchmarks. In particular, the authors collected WearerQA to bridge the domain gap between synthetic benchmarks and natural user behavior. The model implicitly captures the underlying behavior of gaze, learning to downweight reliance for rapidly shifting gaze or gaze that lies in the periphery.
Critical weaknesses
- The major weakness impacting the paper's significance is the narrow quantitative margin separating the proposed framework from simple, training-free heuristics. While ReGaze achieves the highest aggregate scores across benchmarks, its empirical advantage over elementary combination strategies—specifically a fixed reliance coefficient (Fixed ) and a parameter-free maximum similarity selector (Maxsim)—is exceptionally narrow, especially when scaling to stronger Video LLM backbones. The manuscript provides no standard deviations or significance tests to confirm whether these fractional improvements exceed stochastic evaluation noise.
- Another critical limitation stems from severe temporal aliasing caused by subsampling high-frequency gaze dynamics down to 1 frame per second. The wearable sensors utilized in this work record gaze coordinates between 30 Hz and 120 Hz. The proposed implementation subsamples video frames to 1 FPS and assigns each second a single gaze sample that is temporally nearest to the video timestamp. This nearest-neighbor discretization introduces temporal aliasing: a wearer fixating on an object for 950 milliseconds who initiates a saccadic glance during the remaining 50 milliseconds risks having that momentary glance registered as the representative gaze point for the entire second.
Questions for authors
- Statistical Verification of Performance Gains: Given that ReGaze outperforms the unparameterized Maxsim baseline marginally or even underperforms it on EgoEverything (43.79% vs. 44.07%), can the authors provide multi-seed evaluation runs, standard errors, or paired significance tests to demonstrate that the performance advantages over Maxsim are statistically meaningful rather than stochastic fluctuations?
- Impact of Native Oculomotor Temporal Pooling: Subsampling high-frequency gaze to an instantaneous point at 1 FPS introduces severe saccadic aliasing. How does model performance change if the gaze feature is computed by aggregating all native gaze coordinates occurring within a temporal window (e.g., computing a Gaussian mixture density or fixation cluster over the interval) prior to patch pooling, thereby filtering out momentary saccades?
- Category-Specific Breakdown on WearerQA: Figure 11 breaks down WearerQA into diverse question categories, including single-moment object attributes (22.9%) and multi-moment temporal order (5.2%) or state conditions (8.0%). Can the authors provide a per-category performance breakdown comparing ReGaze against Frame-only, Gaze-only, and Maxsim to clarify whether the advantages of adaptive reliance are concentrated in specific cognitive task types?
- Computational Latency and Deployment Profile: The AGRE framework requires extracting local context windows of preceding frames (Transformer) over all candidate frames in a video. What is the precise wall-clock latency, parameter overhead, and peak GPU memory footprint introduced by AGRE during inference on hour-long videos compared to standard cosine similarity selection and the Maxsim heuristic?
We sincerely thank you for the positive assessment and the concrete questions. We address each below.
W1/Q1. Statistical verification of the gains
We re-ran the two best methods of each backbone in Table 1 with three seeds (full table in our response to Reviewer DnSS, W1, and App. E). Against Maxsim with Qwen3.5-9B:
The average margin with the stronger backbone is small but consistent: ReGaze is higher in every seed, and a paired t-test gives p = 0.009. Pooling both backbones, ReGaze outperforms the strongest baseline in all six backbone–seed pairs (p = 0.002; p = 0.096 for InternVL3.5-8B alone). On EgoEverything, Maxsim remains slightly better (−0.17 pp, not significant). Its questions are generated around the wearer's gaze, so gaze is rarely noise and Maxsim's rule already suffices; the gain appears where gaze is often noise, as on EgoExoLearn.
W2/Q2. Temporal aliasing and native-rate pooling
We agree that assigning the nearest sample to each 1-fps frame can register a momentary glance. Following your suggestion, we aggregated all gaze samples in [t − 0.5 s, t + 0.5 s) before patch pooling, keeping all other settings. We evaluated on EgoExoLearn, whose eye tracker records gaze at 120 Hz, and on WearerQA:
None of the aggregations improved accuracy in this setting. We conjecture that each frame is a single instant, so the sample closest in time is the one best aligned with its content, whereas samples up to 0.5 s away may point to locations that have shifted with head motion. In addition, AGRE lowers its reliance on gaze with large shifts between frames (Fig. 5), which can limit the effect of such glances. This does not show that aliasing is absent; modeling gaze dynamics together with denser frames remains future work (App. D).
Q3. WearerQA by category
Per-category accuracy, re-aggregated from the runs in Table 3 (InternVL3.5-8B):
ReGaze achieves the best overall accuracy, and is above Maxsim in 6 of 8 categories and above the frame-only sampler in 5. However, each category is small, so most differences correspond to only a few questions, and none is significant after Holm correction. We therefore report these results as descriptive and do not claim that the advantage concentrates in a specific task type.
Q4. Computational cost
We measured a 1-hour video (3,600 frames at 1 fps), averaged over 5 runs:
The time includes EVA-CLIP image encoding, but excludes frame decoding and preprocessing (62.3 s) and question encoding (0.2 s), which are shared by all methods. AGRE adds 7.49M parameters and runs in 7.9 ms for the whole video, including building the local windows. Its weights take about 29 MiB in FP32, about 1% of the sampler's peak GPU memory (2,718 MiB). The overhead of ReGaze over the training-free samplers is thus negligible.