- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
The paper proposes SAGE, a single model that jointly performs gaze detection, action/HOI recognition, gaze anticipation and action anticipation. Two modules link gaze and action: GCSA uses gaze as an attention bias over human-object tokens, and GCTP uses predicted future gaze to anticipate future actions. The authors also build Exo-Cook, an exocentric benchmark derived from Ego-Exo4D cooking videos with projected gaze and 10 action categories. Experiments are on VidHOI, EGTEA Gaze+ and Exo-Cook.
Critical strengths
- Joint recognition and anticipation of gaze and HOI in exocentric video is a relevant and under-explored setting. Exo-Cook could be a useful benchmark if released with documented annotation quality.
- The ablation design (Tables 1, 14-17) and multi-seed results (Tables 11-12) are the right kind of evidence for the paper's thesis.
Critical weaknesses
- The claimed bidirectional benefit between gaze and action is weakly supported. Gains from gaze are small (+0.8 action recognition, +0.4-0.5 gaze anticipation F1). For action anticipation, SAGE-124, which replaces future gaze with learnable tokens, already reaches 56.3 vs. 58.4 for full SAGE (Table 14). If I understand correctly, this means that only ~2 points are attributable to anticipated gaze. There are no shuffled/random/ground-truth gaze controls showing that the gains come from gaze content.
- The SOTA comparisons are not under a consistent protocol. The VidHOI best run (38.65 mAP) is incompatible with the reported three seeds (37.22 ± 0.42), and the seed mean is below ST-Gaze (38.46). ST-Gaze numbers in Table 4 are lower than in its original paper. On EGTEA (Table 5), S2 is omitted and baseline S1 values are left blank, although SAGE's S1 (61.8) is below GC-TSM, SAP and R34-2Stream.
Questions for authors
- Why do the other changes bring such a large improvement, while anticipated gaze contributes so little? This deserves further analysis.
- Please add controls with ground-truth, random and shuffled gaze to show the gains come from gaze content.
6 comments
Response to Reviewer AkF3 [2/2]
W2: Experiment Protocol - We would like to clarify the experimental protocols and reporting conventions for VidHOI and EGTEA Gaze+.
- Best-run performance versus three-seed statistics on VidHOI: Following the best-run reporting practice of prior methods ST-Gaze (Ni et al., (2023)), we report SAGE’s best-performing run (38.65 mAP) in the main paper. We additionally provide results from three independently trained runs in the appendix (37.22 ± 0.42 mAP) to quantify training variability. These additional runs exclude the best-performing run; including it yields a four-run mean and sample standard deviation of approximately 37.58 ± 0.79 mAP. We will clarify this distinction in the manuscript. Note that ST-Gaze (Ni et al. (2023)) do not report a corresponding mean and standard deviation across seeds, so their training variability is unavailable for comparison.
- ST-Gaze baseline results: The ST-Gaze results reported in our tables were obtained by retraining the model, rather than directly copying the best-performing results from the original publication. We retrained ST-Gaze for two reasons. First, SAGE refers to the human–object feature construction strategy from ST-Gaze, making it an important reference baseline which requires retraining for fair comparison. Second, to specifically evaluate the effectiveness of GCSA (comparing to use frozen gaze feature as an input in ST-Gaze), we introduce a spatial-only variant of ST-Gaze by removing its temporal layer, as reported in Table 10. The differences between our reproduced results and the originally reported results in thier paper illustrate that retraining a model may not reproduce its best-run performance. This observation also provides context for the difference between SAGE's best-performing run and its independent three-seed statistics.
- EGTEA Gaze+ evaluation metrics: Table 5 includes results reported using different evaluation metrics following prior studies. Specifically, the S1 column reports mean-class accuracy, following MCN (Huang et al., 2020) and EGTEA (Li et al., 2021), whereas the S3 column reports Top-1 accuracy, as reported by I3D-2Stream, R34-2Stream, SAP, and GC-TSM. We additionally report the average Top-1 accuracy across all three splits to provide a more comprehensive evaluation of action recognition performance. Since prior methods differ in their reported metrics and available split-wise results, we retain their original evaluation protocols. To clarify and provide complete split-wise results, we also report Top-1 accuracy for S1, S2, and S3 in the above gaze control ablation table. Note that providing S1 (Top-1) and S2 (Top-1) results does not change the overall comparison, since Table 5’s (main paper) average already includes all three splits. SAGE scores 66.4/65.4/63.1 versus GC-TSM’s 66.5/66.1/62.6, remaining competitive at 65.0 versus 65.1 on average. Unlike us, GC-TSM does not report across-seed mean and standard deviation for these results, so we cannot determine whether the 0.1-point average gap persists across seeds or reflects training variability.
Q1: Anticipated Gaze's Contribution - Anticipating future gaze is inherently challenging because future video frames are unavailable: SAGE must infer future gaze distributions solely from observed video features and gaze information. Despite this, the tables below show consistent improvements over SAGE-124 in both egocentric and exocentric settings, and across multiple anticipation horizons on EGTEA Gaze+. These gains are additional to the observed-gaze conditioning already provided by GCSA, and we also include the margin between leading action-anticipation baselines to put the magnitude of anticipated gaze's contribution in context.
Contribution of anticipated gaze and comparison with prior baselines
| Comparison | Dataset / viewpoint | Action anticipation accuracy (%) | Gain (percentage points) | Reference |
|---|---|---|---|---|
| SAGE-124 → Full SAGE | EGTEA Gaze+ / Egocentric | 56.3 → 58.4 | +2.1 | Appendix Table 15 |
| SAGE-124 → Full SAGE | Exo-Cook / Exocentric | 52.5 → 54.8 | +2.3 | Appendix Table 19 |
| ORViT-MF → InAViT | EGTEA Gaze+ / Egocentric | 57.2 → 58.2 | +1.0 | Table 6 |
The anticipated-gaze ablation shows consistent benefits in both viewpoints. For context, its 2.1-point gain on EGTEA Gaze+ exceeds the 1.0-point gap between the two strongest prior action-anticipation baselines reported in Table 5.
Anticipated-gaze contribution across horizons on EGTEA Gaze+ (Appendix Table 16)
| Anticipation horizon | Gain over SAGE-124 (percentage points) |
|---|---|
| 0.5 seconds | +2.1 |
| 1.0 seconds | +1.7 |
| 1.5 seconds | +1.7 |
| 2.0 seconds | +3.0 |
Gains remain positive at every evaluated horizon, with the largest improvement at 2.0 seconds, where observed interaction cues become less informative.
Q2: Gaze Control Ablation - Provided in the above table (Response [1/2])
Follow-up on Rebuttal
Thank you again for your thoughtful questions and comments! We have added control experiments using ground-truth, random, and shuffled gaze, clarified the evidence for gaze–action coupling and the contribution of anticipated gaze, addressed the SOTA comparisons, split-wise results, and seed statistics.
We would greatly appreciate your feedback on whether these additions address your main concerns or if anything remains unclear.
Response to Reviewer AkF3 [1/2]
Thank you for recognizing the importance of studying gaze and human–object interactions together in exocentric video, and Exo-Cook’s potential to support research in this setting. We also appreciate your acknowledgment that the ablations and multi-seed experiments provide an appropriate basis for evaluating our approach. Below, we address your concerns and clarify the supporting evidence and annotation procedures. We also incorporate revisions in the paper based on the reviewer's suggestion.
W1: Gaze-Action Bidirectional Coupling- The cited SAGE-123/124 comparisons with SAGE only measure incremental future-task benefits, since both already include current gaze–action conditioning through GCSA. To assess the overall benefits of gaze and action coupling, Table 1 (main paper) provides two complementary comparisons: full SAGE against single-task models, and against joint four-task training without GCSA/GCTP. We summarize these existing results below; each cell shows baseline → full SAGE, with the gain in bold.
The single-task baselines are SAGE-1, SAGE-2, SAGE-3, and SAGE-4, respectively. The first row captures the combined benefit of joint training and gaze–action coupling, showing improvements across all four tasks. The second holds four-task supervision unchanged, demonstrating that GCSA/GCTP improve both gaze and action prediction beyond simply combining task losses. The final 14.4-point action anticipation gain reflects the combined contribution of GCSA and GCTP towards it, consistent with Equation 1 (main paper). These comparisons assess the broader benefits of coupling that the incremental SAGE-123/124 ablations do not capture.
As requested by the reviewer, we additionally provide random, shuffled, predicted, and ground-truth gaze controls below on EGTEA Gaze+ dataset. Predicted gaze improves recognition and anticipation over random gaze and shuffled gaze. Ground-truth gaze further improves recognition and anticipation, providing an oracle reference. These controls support the importance of informative gaze input and complement Table 1’s (main paper) evidence for the coupling mechanisms.
\hline & \text{Action Detection (Top-1 Acc.)} & \text{Action Anticipation} \\\\ \begin{array}{l} \strut \text{Gaze Condition} \\\\ \hline \strut \text{Random gaze} \\\\ \strut \text{Shuffled gaze} \\\\ \strut \text{Predicted gaze (SAGE)} \\\\ \hline \strut {\color{gray}{\text{Ground truth gaze (Oracle)}}} \end{array} & \begin{array}{cccc} \strut \text{S1} & \text{S2} & \text{S3} & \text{Avg.} \\\\ \hline \strut 61.3 & 60.4 & 59.8 & 60.5 \\\\ \strut 61.5 & 59.6 & 58.4 & 59.8 \\\\ \strut \mathbf{66.4} & \mathbf{65.4} & \mathbf{63.1} & \mathbf{65.0} \\\\ \hline \strut {\color{gray}{69.2}} & {\color{gray}{68.3}} & {\color{gray}{66.0}} & {\color{gray}{67.8}} \end{array} & \begin{array}{c} \strut \text{Avg. mean cls acc} \\\\ \hline \strut 47.3 \\\\ \strut 44.2 \\\\ \strut \mathbf{58.4} \\\\ \hline \strut {\color{gray}{60.5}} \end{array} \\\\ \hline \end{array}$$ We have incorporated this additional gaze control ablation study into Section 5.1 and Table 2 of the revised manuscript.