acceptodds
Under review as a conference paper at ICLR 2027

When to Trust the Eyes: Learning Adaptive Gaze Reliance in Egocentric Video

Abstract

Gaze recorded by wearable devices reveals where the wearer attended, and prior work exploits it for egocentric video understanding under the implicit assumption that it is useful regardless of the task. However, we show that not all gaze is useful for question answering, the core function of a wearable assistant. Rather, its usefulness depends on the question and the scene context, so the same fixation can be useful for one question yet noise for another. Accordingly, how much to rely on gaze should be decided for each question and frame, trusting gaze where it is useful and suppressing it where it acts as noise. To this end, we propose **Re**liance on **Gaze** (ReGaze), a method that learns this reliance. Specifically, we design the Adaptive Gaze Reliance Estimation (AGRE) module, a lightweight network trained without any label of gaze usefulness, using a ranking loss that favors reliance on gaze where it helps localize question-relevant frames and a noise suppression loss that penalizes it where it hurts. To train the AGRE module, we further build a Gaze-aware QA Curation (GQC) pipeline that generates questions paired with their relevant intervals from gaze-recorded videos. On three egocentric QA benchmarks, ReGaze outperforms state-of-the-art frame samplers and naive uses of gaze. Beyond performance, ReGaze captures the underlying behavior of gaze without being trained on it: its reliance decreases as the gaze shifts rapidly or lies in the periphery. It also excels on WearerQA, our evaluation set of questions written by the wearers themselves.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Video Understanding, Gaze, Egocentric Question Answering

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_DhtNOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The manuscript investigates the utilization of human eye-gaze data for egocentric video question answering. To address this challenge without relying on manual annotations of gaze usefulness, the authors introduce Reliance on Gaze (ReGaze), an adaptive frame selection framework driven by the proposed Adaptive Gaze Reliance Estimation (AGRE) module. Optimization of the AGRE architecture is achieved without explicit supervisory labels of gaze utility. Instead it is supervised by questions paired with their relevant intervals from gaze-recorded videos which are generated from the introduced Gaze-aware QA Curation (GQC) pipeline. ReGaze is evaluated against three groups of frame sampler baselines across four benchmarks of EgoExoLearn, EgoEverything, EgoGazeVQA, and WearerQA using InternVL3.5-8B and Qwen3.5-9B backbones.

Critical strengths
  1. The primary strength of this work lies in its systematic deconstruction of the prevailing assumption that human visual attention is universally beneficial for egocentric video understanding. To support the claim, the paper introduces Reliance on Gaze (ReGaze) to learn adaptive gaze reliance from egocentric videos. The design of the AGRE module to train the light-weight gaze reliance estimator without manual labels of gaze usefulness is also interesting and insightful.
  2. Another substantive strength resides in the comprehensive evaluation of ReGaze on four benchmarks. In particular, the authors collected WearerQA to bridge the domain gap between synthetic benchmarks and natural user behavior. The model implicitly captures the underlying behavior of gaze, learning to downweight reliance for rapidly shifting gaze or gaze that lies in the periphery.
Critical weaknesses
  1. The major weakness impacting the paper's significance is the narrow quantitative margin separating the proposed framework from simple, training-free heuristics. While ReGaze achieves the highest aggregate scores across benchmarks, its empirical advantage over elementary combination strategies—specifically a fixed reliance coefficient (Fixed α\alpha) and a parameter-free maximum similarity selector (Maxsim)—is exceptionally narrow, especially when scaling to stronger Video LLM backbones. The manuscript provides no standard deviations or significance tests to confirm whether these fractional improvements exceed stochastic evaluation noise.
  2. Another critical limitation stems from severe temporal aliasing caused by subsampling high-frequency gaze dynamics down to 1 frame per second. The wearable sensors utilized in this work record gaze coordinates between 30 Hz and 120 Hz. The proposed implementation subsamples video frames to 1 FPS and assigns each second a single gaze sample that is temporally nearest to the video timestamp. This nearest-neighbor discretization introduces temporal aliasing: a wearer fixating on an object for 950 milliseconds who initiates a saccadic glance during the remaining 50 milliseconds risks having that momentary glance registered as the representative gaze point for the entire second.
Questions for authors
  1. Statistical Verification of Performance Gains: Given that ReGaze outperforms the unparameterized Maxsim baseline marginally or even underperforms it on EgoEverything (43.79% vs. 44.07%), can the authors provide multi-seed evaluation runs, standard errors, or paired significance tests to demonstrate that the performance advantages over Maxsim are statistically meaningful rather than stochastic fluctuations?
  2. Impact of Native Oculomotor Temporal Pooling: Subsampling high-frequency gaze to an instantaneous point at 1 FPS introduces severe saccadic aliasing. How does model performance change if the gaze feature ztz_t is computed by aggregating all native gaze coordinates occurring within a temporal window (e.g., computing a Gaussian mixture density or fixation cluster over the [t−0.5s,t+0.5s][t-0.5\text{s}, t+0.5\text{s}] interval) prior to patch pooling, thereby filtering out momentary saccades?
  3. Category-Specific Breakdown on WearerQA: Figure 11 breaks down WearerQA into diverse question categories, including single-moment object attributes (22.9%) and multi-moment temporal order (5.2%) or state conditions (8.0%). Can the authors provide a per-category performance breakdown comparing ReGaze against Frame-only, Gaze-only, and Maxsim to clarify whether the advantages of adaptive reliance are concentrated in specific cognitive task types?
  4. Computational Latency and Deployment Profile: The AGRE framework requires extracting local context windows of preceding frames (Transformer) over all candidate frames in a video. What is the precise wall-clock latency, parameter overhead, and peak GPU memory footprint introduced by AGRE during inference on hour-long videos compared to standard cosine similarity selection and the Maxsim heuristic?
1 comment

We sincerely thank you for the positive assessment and the concrete questions. We address each below.


W1/Q1. Statistical verification of the gains

We re-ran the two best methods of each backbone in Table 1 with three seeds (full table in our response to Reviewer DnSS, W1, and App. E). Against Maxsim with Qwen3.5-9B:

Qwen3.5-9BEgoExoLearnEgoEverythingEgoGazeVQAAvg
Maxsim43.05 ± 0.2344.04 ± 0.0357.66 ± 0.1048.25 ± 0.05
ReGaze43.86 ± 0.3743.87 ± 0.1257.85 ± 0.2948.53 ± 0.02

The average margin with the stronger backbone is small but consistent: ReGaze is higher in every seed, and a paired t-test gives p = 0.009. Pooling both backbones, ReGaze outperforms the strongest baseline in all six backbone–seed pairs (p = 0.002; p = 0.096 for InternVL3.5-8B alone). On EgoEverything, Maxsim remains slightly better (−0.17 pp, not significant). Its questions are generated around the wearer's gaze, so gaze is rarely noise and Maxsim's rule already suffices; the gain appears where gaze is often noise, as on EgoExoLearn.


W2/Q2. Temporal aliasing and native-rate pooling

We agree that assigning the nearest sample to each 1-fps frame can register a momentary glance. Following your suggestion, we aggregated all gaze samples in [t − 0.5 s, t + 0.5 s) before patch pooling, keeping all other settings. We evaluated on EgoExoLearn, whose eye tracker records gaze at 120 Hz, and on WearerQA:

Gaze per frameEgoExoLearnWearerQA
Nearest sample (paper)38.9139.56
Window mean38.2138.35
Fixation pooling38.4038.35
Gaussian-mixture heatmap37.4638.76

None of the aggregations improved accuracy in this setting. We conjecture that each frame is a single instant, so the sample closest in time is the one best aligned with its content, whereas samples up to 0.5 s away may point to locations that have shifted with head motion. In addition, AGRE lowers its reliance on gaze with large shifts between frames (Fig. 5), which can limit the effect of such glances. This does not show that aliasing is absent; modeling gaze dynamics together with denser frames remains future work (App. D).


Q3. WearerQA by category

Per-category accuracy, re-aggregated from the runs in Table 3 (InternVL3.5-8B):

CategoryFrame-onlyGaze-onlyMaxsimReGaze
Object attribute45.6141.2343.8644.74
Object location17.0018.0022.0023.00
Action verification68.5768.5768.5767.14
Counting34.8534.8531.8237.88
Interaction purpose27.8736.0729.5132.79
State condition42.5047.5045.0042.50
Temporal order26.9226.9230.7734.62
Others19.0523.8114.2923.81
All37.1537.9537.7539.56

ReGaze achieves the best overall accuracy, and is above Maxsim in 6 of 8 categories and above the frame-only sampler in 5. However, each category is small, so most differences correspond to only a few questions, and none is significant after Holm correction. We therefore report these results as descriptive and do not claim that the advantage concentrates in a specific task type.


Q4. Computational cost

We measured a 1-hour video (3,600 frames at 1 fps), averaged over 5 runs:

MethodSampler time
Frame-only (cosine)16.22 s
Maxsim16.31 s
ReGaze16.31 s (+6.6 ms vs. Maxsim)

The time includes EVA-CLIP image encoding, but excludes frame decoding and preprocessing (62.3 s) and question encoding (0.2 s), which are shared by all methods. AGRE adds 7.49M parameters and runs in 7.9 ms for the whole video, including building the local windows. Its weights take about 29 MiB in FP32, about 1% of the sampler's peak GPU memory (2,718 MiB). The overhead of ReGaze over the training-free samplers is thus negligible.

Reviewer_Uej2OpenReview ↗
Rating
2(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper argues that egocentric gaze is not uniformly useful for video question answering and proposes ReGaze, which learns a per-question, per-frame reliance alpha t on gaze for question-aware frame sampling. The frame score is the cosine similarity between the question and a convex combination of a frame feature and a gaze feature from a frozen EVA-CLIP. alpha t is predicted by a small transformer over a 5-frame window (AGRE), trained with a Bradley–Terry ranking loss and a "noise suppression" loss on questions and relevant intervals generated by an LLM pipeline (GQC) from HD-EPIC and SuperMemory-VQA. The top-32 frames are fed to a Video LLM. Results are reported on EgoExoLearn, EgoEverything, EgoGazeVQA and a new 498-question wearer-authored set (WearerQA).

Critical strengths
  • The question of when gaze helps egocentric QA is relevant, and Fig. 1 is a clear motivation that naive gaze reliance can hurt.
  • The method is simple, lightweight, and evaluated with two Video LLM backbones under a fixed frame budget, so differences come only from frame selection.
  • Collecting WearerQA is a valuable direction for evaluating wearable assistants.
Critical weaknesses
  • The training signal is not what the main text describes, and the gains may come from distilling an external retriever rather than from learning gaze reliance.

    • Sec. 4.1 states that "frames inside R are positives" (L239) and averages over R in Eqs. 4-5. App. B.3 (L855-857) reveals that only one positive is used per question (|R| = 1), selected as the frame most similar to the question according to Qwen3-VL-Embedding-8B. The AGRE module is therefore trained to reproduce the ranking of a much stronger embedding model.
    • Without this selection, ReGaze no longer beats the naive baselines: using all interval frames gives 38.30 on EgoExoLearn and 41.25 on EgoEverything (L858-860), below both Fixed α (38.54 / 41.37) and Maxsim (38.42 / 41.38) in Table 1.
    • The relevant intervals themselves are produced by GPT-5.4 from descriptions generated with the gaze marker overlaid on each frame (L288-290). The claim of learning "without any label of gaze usefulness" (L21-22, L97-98, L264-266) is therefore overstated: the supervision is gaze-informed VLM output.
    • The necessary baseline is missing: Qwen3-VL-Embedding-8B used directly as the (frame-only) sampler, and AGRE trained with that model's similarity as a distillation target but without gaze.
  • The evidence that the learned reliance depends on gaze and on the question is insufficient.

    • There is no control for the gaze signal itself. Egocentric gaze has a strong center bias (Tatler, 2007, cited in L405). AGRE should be compared with (i) a center-pooled pseudo-gaze feature and (ii) temporally shuffled gaze from the same video. If either gives similar gains, the improvement comes from a learned re-weighting of a local and a global feature, not from gaze.
    • There is no question-agnostic ablation (f_θ(v_t, z_t) without q), although question dependence is the central claim (L16-18, L113-114). The only evidence is two qualitative examples (Fig. 8).
    • Fig. 5 is presented as AGRE learning gaze behavior "without being trained on it" (L26-28, L403-413), but this is expected from the inputs: ViT patch features carry positional information, and the 5-frame window sees consecutive z_t, so eccentricity and displacement are directly inferable. A rule-based baseline (α as a function of eccentricity and displacement) is needed to show that AGRE learns more than this. Also, at 1 fps with the nearest gaze sample (L851-852), "gaze speed" is the displacement between samples one second apart, not saccadic speed (as acknowledged in App. D), so it does not support the claims about "rapidly shifting" gaze.
    • The optimization does not define the intended solution. With uniform initialization, the losses drive α_t to ~0.95, i.e. the worst-performing gaze-only sampler (Fig. 3, L227-230). The desired behavior is obtained only through zero initialization. Even with L_NS, 22.93% of EgoExoLearn frames have α_t ≥ 0.95 (L873-875).
  • Hyperparameters and baselines are tuned on the test benchmarks, and the margins are within this tuning noise.

    • σ and λ are swept on EgoExoLearn and EgoEverything, and the selected values (σ = 1.5, λ = 1.0) give exactly the numbers reported in Table 1 (Fig. 4 vs Table 1: 38.91 / 41.82). Fixed α uses "the α that performs best on each benchmark" (L338-339), also chosen on the test set, while FOCUS, AKS and Nar-KFC use default hyperparameters (L865).
    • The resulting margins are small: on average +0.54 over Fixed α and +0.67 over Maxsim with InternVL3.5-8B, and +0.23 over Maxsim with Qwen3.5-9B, where Maxsim is better on EgoEverything and tied on EgoGazeVQA. The σ/λ sweep in Fig. 4 moves accuracy by 0.5-1.0 points.
  • WearerQA does not support the real-world claims.

    • The guideline shown to participants contains the paper's hypothesis: "Because human recall is driven by where attention was allocated, the resulting questions are implicitly grounded in gaze rather than in uniform frame sampling" (Table 5, L1047-1050). This introduces a demand effect and contradicts L461-462, which states that "no question in WearerQA is generated around the gaze".
    • The set is small (498 questions from five participants). The gap between ReGaze and Fixed α (39.56 vs 38.76) corresponds to 4 questions, and Fixed α is again tuned on the evaluation set. The answers are judged by an LLM, and only InternVL is evaluated, with no frame-sampler baselines.
    • Participant recruitment (e.g. whether participants are the authors or lab members), compensation and ethics-board approval are not described.
  • Limited novelty. A learned gate that mixes a local and a global feature per frame is a standard fusion design, and the ranking loss is a standard pairwise objective. Gaze-aware baselines from the cited literature (e.g. Gaze-VLM, Voila-A) are not compared as samplers, so it is unclear whether AGRE improves over existing ways of using gaze.

Some minor issues:

  • Several numbers and statements are inconsistent or unsupported:
    • L363-366 "the gaze-only sampler slightly exceeds the frame-only sampler on EgoEverything" → the difference is 0.05 points with InternVL and should not be interpreted.
    • L461-462 "no question in WearerQA is generated around the gaze" → see the Table 5 instruction above.
    • L403 "takes neither gaze position nor gaze speed as input" → both are implicitly available through the patch features and the temporal window.
    • Fig. 5 uses truncated y-axes (α_t from ~0.55 to ~0.68), which visually exaggerates the trend.
  • Table 2 compares fine-tuned models (Low-/High-α_t) against a model without fine-tuning ("None", L423). A model fine-tuned on the same data without gaze regularization and one with gaze regularization on a random quarter of frames are needed to attribute the effect to α_t.
  • B.7 (Fig. 10): the single-frame variant trained on SuperMemory-VQA (45.02) outperforms the local-window variant on the same data (44.96), and on both datasets it falls below the frame-only sampler (44.35 vs 44.41). This does not support the robustness claim in L958-963.
  • The unit of σ (patches or normalized coordinates) is not specified (L209, L349).
  • The tanh output is not constrained to [0, 1]. The convexity of Eq. 2 holds only empirically (L231-234, L889). A sigmoid with a bias initialized near zero would be equivalent and properly bounded.
  • The 40K GQC training questions (L307) are not human-verified, and the relevant intervals are snapped to LLM-defined events. The rate of incorrect intervals should be estimated on a sample.
  • Eye-tracking data is biometric. The plan to release WearerQA (L496-497) should describe how gaze data will be protected beyond EgoBlur face anonymization.
  • The authors should tone down the language:
    • L26-28 "ReGaze captures the underlying behavior of gaze without being trained on it" → see the comments on Fig. 5 above.
    • L28-29 "It also excels on WearerQA" → +0.8 points over Fixed α on 498 questions.
    • L481-482 "this judgment can be learned without any label, only from where the answers lie" → the intervals and positives come from GPT-5.4 and Qwen3-VL-Embedding-8B with gaze-overlaid descriptions.
Questions for authors
  • Each main argument above is followed by supporting evidence; please address each main argument and the evidence behind it.
  • Please report (i) Qwen3-VL-Embedding-8B as a frame sampler, (ii) AGRE with center-pooled and temporally shuffled gaze, and (iii) AGRE without the question input.
  • Please select σ, λ and Fixed α on held-out data (e.g. a split of the GQC data), and report results over multiple training seeds with significance tests.
  • Please clarify how WearerQA participants were recruited and whether the guideline in Table 5 was shown to them as written.
4 comments

We sincerely appreciate your detailed and valuable feedback. Several of your points are correct, and we revised the paper accordingly (changes in red). We address each point below.

W1/Q2. Training signal

You are right that the main text did not state that a single positive per question is selected with Qwen3-VL-Embedding-8B. We revised Sec. 4.1 to describe it. To test whether the gain comes from this model, we ran the two baselines you requested:

SamplerEgoExoLearnEgoEverythingEgoGazeVQAAvg
Qwen3-VL-Embedding-8B (frame-only)38.8941.7054.1744.92
Distilled from Qwen3-VL-Embedding-8B (no gaze)38.9541.2054.2944.81
ReGaze38.9141.8254.9745.23

Neither the 8B model used directly as a sampler nor distilling its similarity into a gaze-free sampler reaches ReGaze. The selector denoises the event-level intervals during training, which is why using all interval frames hurts, but it does not explain the gain. We also agree that "without any label of gaze usefulness" was overstated: the intervals come from gaze-aware descriptions. We now write "without explicit labels of gaze usefulness" and state this dependence in Sec. 4.1.


W2/Q2. Dependence on gaze and on the question

All variants below use the same supervision as ReGaze:

VariantEgoExoLearnEgoEverythingEgoGazeVQAAvg
Frame-only sampler (no training)38.0541.2353.9444.41
Center-pooled pseudo-gaze38.5840.9553.6344.39
Temporally shuffled gaze38.2441.4053.3644.33
Without the question input38.4341.3353.8044.52
ReGaze38.9141.8254.9745.23

Center-pooled and shuffled gaze give no gain over the frame-only sampler, so the improvement is not a learned re-weighting of a local and a global feature; it requires real gaze. Removing the question also loses most of the gain. As you note, eccentricity and displacement are inferable from the inputs, so this question-agnostic model can capture a rule on them; its gap to ReGaze shows that the reliance depends on the question beyond these statistics. Accordingly, we toned down the Fig. 5 claims (Abstract, Sec. 1, Sec. 5.3) and define "gaze speed" as the displacement between frames one second apart. On the optimization, we now present the zero initialization in Sec. 4.1 as a prior toward the frame-only sampler, from which the losses raise the reliance only where gaze helps.


W3/Q3. Hyperparameters

We agree that reporting the values that are best in Fig. 4 amounts to selecting them on the test benchmarks. Fig. 4 was intended as a sensitivity analysis of how σ\sigma and λ\lambda affect the model behavior, but we now also select them on held-out data. We use a split of the GQC data whose videos are not used for training as a validation set (3.8K QA pairs), and repeat the sweep of Fig. 4 with three seeds, measured by QA accuracy on the curated answers (Sec. 4.2).

Averaged over the three seeds, σ\sigma = 1.5 and λ\lambda = 1.0 remain the best. λ\lambda = 1.0 is the best for all three seeds, and σ\sigma = 1.5 for two of them, while the third slightly prefers σ\sigma = 0.5. The held-out protocol thus selects the same values as reported, so the results in Table 1 do not rely on tuning on the test sets. For Fixed α\alpha, the per-benchmark value is an oracle chosen on each test set, which favors this baseline (now noted in Sec. 5.1).

For multiple seeds and significance, we re-ran the two best methods of each backbone in Table 1 with three seeds (full table in our response to Reviewer DnSS, W1). ReGaze outperforms the strongest baseline on average in all six backbone–seed pairs; a paired t-test pooling both backbones gives pp = 0.002.

W4/Q4. WearerQA

We agree that L461–462 was misleading. We meant that participants were not asked to write questions about where they looked; they wrote questions freely from memory, and recall itself is shaped by attention. The sentence now reads "even though WearerQA questions are written from memory rather than generated around the gaze."

Regarding the participants, master's and Ph.D. students in our research group were recruited, and the guideline in Table 5 was shown as written. Its premise may have encouraged questions about attended content. We disclose this in Sec. 5.5 and Table 5.

WearerQA is small because each question must be written by the wearer from their own memory and then verified by the same wearer against the recording, so it cannot be scaled like benchmarks whose questions are generated by LLM agents. In exchange, every question is one a wearer actually asked about their own experience, which is closer to how a wearable assistant is used.

Regarding the experiments on WearerQA, we did not include the state-of-the-art frame samplers (FOCUS, AKS, Nar-KFC) on WearerQA because they bring at most marginal gains over the frame-only sampler on egocentric videos (Table 1). As discussed in App. B.2, they spread the frame budget for temporal coverage, whereas a wearer typically asks about a specific moment, and most WearerQA questions are of this kind (App. C.2). Nevertheless, we agree that including them, together with Qwen3.5-9B, makes the evaluation more complete, and we will add both to the final version.


W5. Novelty

We agree that the gate and the pairwise loss are standard components, and we do not claim them as contributions. Our contribution is the question and its formulation. Prior work uses gaze as recorded, or filters it by signal quality such as stability, implicitly assuming that the remaining gaze helps regardless of the task. We instead ask whether all gaze is useful for answering a question, show that its usefulness depends on the question and the frame, and formulate learning this reliance from relevance supervision alone.

Regarding Gaze-VLM and Voila-A, they address a different problem: they feed raw gaze into the model, e.g., by regularizing its attention toward the gaze, and use all gaze as given. They do not select frames, so they cannot be compared directly as samplers. Our work instead controls how much to rely on gaze, and we study it in frame sampling, where its effect can be isolated. The two are complementary: in Sec. 5.3 (Table 2), we apply Gaze-VLM's attention regularization only to the gaze selected by our learned reliance, and the same amount of gaze supervision improves accuracy when placed on high-reliance gaze rather than low-reliance gaze.


Minor issues

We appreciate your feedback and revised the manuscript accordingly, e.g., we no longer interpret the 0.05 pp gap (L363–366), clarify that gaze position and speed are only implicitly available to AGRE (L403), and tone down the language you pointed out (L26–29, L481–482).

I thank the authors for the detailed response and for acknowledging several issues openly. The new controls for "W2/Q2. Dependence on gaze and on the question" are convincing, and I have raised my rating from 1 to 2. Some concerns remain:

  • The center-pooled, shuffled-gaze and question-free controls support dependence on both gaze and the question. They are single-seed, however, and most of them are not yet in the revised paper.

  • The average gain is 0.30 pp, and the per-backbone result for InternVL is not significant (p = 0.096). The held-out selection of σ and λ should be included in the paper.

  • I appreciate the disclosure about recruitment and the guideline. The demand effect, the small margin, and the missing baselines remain, and IRB approval and the protection of the gaze data are still not described.

I will discuss with other reviewers to finalize the score in post-rebuttal discussion phase.

We sincerely appreciate your thoughtful follow-up, and thank you for raising your score. Your review has substantially improved our work, especially through the control experiments you suggested, which turned out to be the most direct evidence for our central claim that the gain requires both the real gaze and the question. We address the remaining points below.

  • Controls and held-out selection. Within the short discussion period, we ran all the experiments you listed in Q2 and, from Q3, the held-out selection of λ\lambda and σ\sigma. We have added these experiments to App. G of the revised paper. Because of the limited time, the controls were run with a single seed. We will repeat them with multiple seeds and report the mean and standard deviation in the final version.

  • Margin and significance. We agree that the average gain is modest and that the InternVL3.5-8B result alone is not significant with three seeds. We will increase the number of seeds for both backbones and report per-backbone tests.

  • WearerQA. We will add the frame-sampler baselines and Qwen3.5-9B, and annotate whether the evidence of each question lies inside or outside the gazed region to quantify the possible demand effect. Regarding ethics review, WearerQA was collected from members of our research group who gave written informed consent, following the privacy checklist and anonymization in App. C.1. We will consult our institution's IRB and release WearerQA only after its review and under its conditions. In particular, we plan to release only per-frame gaze coordinates aligned to the 1-fps frames, without eye images or pupil data, under a research-only data-use agreement, and we will describe this in the ethics statement.

Lastly, we'd like to emphasize the value of this work. To our knowledge, it is the first to show that gaze is useful only for some questions and frames, whereas prior work uses it as given, and to learn how much to rely on it without explicit labels. With the controls you suggested, the gain largely disappears when gaze or the question is removed, and it holds on average across benchmarks recorded by different wearers and mostly different eye trackers than the training data. We hope this view of human signals, as evidence to be weighed rather than ground truth to follow, will be useful to the community, and we would be grateful if you would take this into account in the final discussion.

Reviewer_DnSSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes ReGaze, which adaptively learns how much to rely on gaze in egocentric video question answering. There is good motivation that the usefulness of gaze depends on the context, but the method has limited novelty, and its effectiveness needs some more validation (see weaknesses). The authors present a clear, coherent method with extensive evaluation across multiple benchmarks using real gaze data. The WearerQA dataset is also a meaningful contribution but limited to five participants.

Critical strengths

The paper is clearly written, with extensive evaluation across three benchmarks using real gaze data and two Video LLM backbones.

The proposed AGRE module is straightforward and relatively well motivated.

Critical weaknesses

The main weakness is that the improvement over strong baselines is incremental and inconclusive (0.23–0.54 percentage points on average). There is also no analysis of statistical significance. ReGaze also does not consistently outperform Maxsim on individual benchmarks.

Additionally, the experiments do not sufficiently demonstrate whether the improvement arises from learning how much to rely on gaze, or simply from extra supervision during training. For example, the proposed AGRE module is trained on roughly 40K GQC-curated questions, but Maxsim does not receive this supervision. The encoder fine-tuning ablation in Appendix B.6 is helpful but does not address the issue. I recommend a comparison to a similarly trained gaze-free baseline (or shuffled/fixed gaze) to isolate the contribution of the proposed method and justify the improvement.

If the authors can address the above two weaknesses, and provide statistically significant improvement over baseline methods under the same level of supervision, I am willing to raise my score to 3. Otherwise I am not confident that the paper demonstrates sufficient incentive for learning adaptive gaze reliance.

Questions for authors

None

3 comments

We sincerely appreciate your valuable feedback. We address both points with new experiments.

W1. Statistical significance

We re-ran the top-2 methods of each backbone in Table 1 with three random seeds (mean ± std):

BackboneMethodEgoExoLearnEgoEverythingEgoGazeVQAAvg
InternVL3.5-8BFixed α38.49 ± 0.0441.40 ± 0.0354.09 ± 0.0644.66 ± 0.03
ReGaze38.69 ± 0.2041.55 ± 0.2354.72 ± 0.2244.99 ± 0.21
Qwen3.5-9BMaxsim43.05 ± 0.2344.04 ± 0.0357.66 ± 0.1048.25 ± 0.05
ReGaze43.86 ± 0.3743.87 ± 0.1257.85 ± 0.2948.53 ± 0.02

ReGaze outperforms the strongest baseline on average in all six backbone–seed pairs. A paired t-test pooling both backbones gives p = 0.002 (per backbone: p = 0.096 for InternVL3.5-8B and p = 0.009 for Qwen3.5-9B).

On individual benchmarks, ReGaze is clearly better than Maxsim on EgoExoLearn and on par on EgoEverything (−0.17 pp, not significant). EgoEverything's questions are generated around the wearer's gaze, so gaze is rarely noise and Maxsim's naive rule already suffices; learning the reliance pays off where gaze is often noise, as on EgoExoLearn. We added this discussion to Appendix E.


W2. Gain from gaze vs. extra supervision

We trained two controls on the same GQC data with the same training setup as ReGaze:

  • Gaze-free: the same AGRE architecture with the 5-frame context, but scoring frames from the frame and question features only. The noise-suppression loss, which requires gaze, is removed.
  • Shuffled gaze: ReGaze trained with gaze shuffled over time within each video.
MethodGazeGQC trainingEgoExoLearnEgoEverythingEgoGazeVQAAvg
Frame-only sampler✗✗38.0541.2353.9444.41
Gaze-free✗✓37.8141.2852.5143.87
Shuffled gazeshuffled✓38.2441.4053.3644.33
ReGaze✓✓38.9141.8254.9745.23

With the same supervision, neither the gaze-free model nor the shuffled-gaze model improves over the frame-only sampler. This agrees with App. B.6, where fine-tuning the encoder on the same data also leaves the frame-only sampler unchanged: the GQC supervision adds little beyond the frozen frame features, and the gain requires the information that gaze carries. Real gaze alone is not enough either: without the noise-suppression loss (λ\lambda = 0, Fig. 4), ReGaze does not improve over the frame-only sampler. In summary, these results indicate that ReGaze improves only when it uses real gaze and carefully controls its noise, i.e., the gain comes from learning how much to rely on gaze, not from the extra supervision.

We added these results to the revised paper (Appendix F). We hope they address both of your concerns, and would be grateful if you would reconsider your score.

I thank the authors for their additional experiments and clarifications. While they address my concerns to some extent, the improvements remain small and the statistical significance is mixed. I decided to maintain my original score.

We sincerely appreciate your valuable feedback, which has helped us improve the quality of our work.

We acknowledge that the improvement of ReGaze over the baselines is modest. During the rebuttal, we found that controlling gaze noise while leveraging its benefits is truly challenging, because the AGRE module is trained on gaze from other wearers (HD-EPIC and SuperMemory-VQA), whose gaze distribution differs from that of the wearers in EgoExoLearn, EgoEverything, and EgoGazeVQA. Moreover, the three benchmarks differ from one another in their gaze distributions and properties, which makes it difficult to improve performance on all of them.

Nevertheless, we would like to emphasize that, to our knowledge, this is the first work to show that the usefulness of gaze depends on the user's question and the frame context, and that ReGaze addresses this challenge, improving performance on average over naive uses of gaze as well as state-of-the-art frame samplers. We believe this work opens up this direction, and that future research can further address this problem.

We would be grateful if you would take this into account.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.