acceptodds
Under review as a conference paper at ICLR 2027

SAGE: Synchronized Action-Gaze Recognition and Anticipation for Human Behavior Understanding

Abstract

Human object interaction (HOI), gaze pattern, and their anticipation are intricately linked, providing valuable insights into cognitive processes, intentions, and behavior. However, most existing models handle gaze and actions separately, missing both their interdependence and the advantages of a unified solution. This paper presents a novel unified framework, SAGE (Synchronized Action-GazE), which integrates simultaneous recognition and anticipation of both HOI and human gaze into a single unified end-to-end trainable model. Our approach leverages a transformer-based architecture and incorporates gaze data into spatiotemporal attention mechanisms to simultaneously predict current and future human actions and gaze behavior. We explore this bidirectional relationship between gaze and actions under different scenarios, whether requiring a close-up, detailed view (egocentric) or a wider, more contextual view (exocentric), making our framework versatile for various applications. Additionally, due to lack of datasets for comprehensive analysis of both HOI and gaze in exocentric videos, we establish a new benchmark Exo-Cook using the Ego-Exo4D dataset to facilitate further research in this domain. Experiments on three benchmark datasets—VidHOI, EGTEA Gaze+, and Exo-Cook—show that jointly modeling gaze and actions across current and future frames achieves consistently strong results, often surpassing specialized state-of-the-art models tailored to individual tasks. Our experiments on three benchmark datasets: VidHOI, EGTEA Gaze+ and Exo-Cook, demonstrate that the synergy between gaze and actions in the current and future frames compares favorably and even outperforms individual task specialized state-of-the-art models. By unifying actions and attention in a comprehensive way, our work lays the groundwork for more intuitive human-machine interaction.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Action Recognition, Action Anticipation, Gaze Detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_AkF3OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper proposes SAGE, a single model that jointly performs gaze detection, action/HOI recognition, gaze anticipation and action anticipation. Two modules link gaze and action: GCSA uses gaze as an attention bias over human-object tokens, and GCTP uses predicted future gaze to anticipate future actions. The authors also build Exo-Cook, an exocentric benchmark derived from Ego-Exo4D cooking videos with projected gaze and 10 action categories. Experiments are on VidHOI, EGTEA Gaze+ and Exo-Cook.

Critical strengths
  1. Joint recognition and anticipation of gaze and HOI in exocentric video is a relevant and under-explored setting. Exo-Cook could be a useful benchmark if released with documented annotation quality.
  2. The ablation design (Tables 1, 14-17) and multi-seed results (Tables 11-12) are the right kind of evidence for the paper's thesis.
Critical weaknesses
  1. The claimed bidirectional benefit between gaze and action is weakly supported. Gains from gaze are small (+0.8 action recognition, +0.4-0.5 gaze anticipation F1). For action anticipation, SAGE-124, which replaces future gaze with learnable tokens, already reaches 56.3 vs. 58.4 for full SAGE (Table 14). If I understand correctly, this means that only ~2 points are attributable to anticipated gaze. There are no shuffled/random/ground-truth gaze controls showing that the gains come from gaze content.
  2. The SOTA comparisons are not under a consistent protocol. The VidHOI best run (38.65 mAP) is incompatible with the reported three seeds (37.22 ± 0.42), and the seed mean is below ST-Gaze (38.46). ST-Gaze numbers in Table 4 are lower than in its original paper. On EGTEA (Table 5), S2 is omitted and baseline S1 values are left blank, although SAGE's S1 (61.8) is below GC-TSM, SAP and R34-2Stream.
Questions for authors
  1. Why do the other changes bring such a large improvement, while anticipated gaze contributes so little? This deserves further analysis.
  2. Please add controls with ground-truth, random and shuffled gaze to show the gains come from gaze content.
6 comments
AuthorsOpenReview ↗

Response to Reviewer AkF3 [1/2]

Thank you for recognizing the importance of studying gaze and human–object interactions together in exocentric video, and Exo-Cook’s potential to support research in this setting. We also appreciate your acknowledgment that the ablations and multi-seed experiments provide an appropriate basis for evaluating our approach. Below, we address your concerns and clarify the supporting evidence and annotation procedures. We also incorporate revisions in the paper based on the reviewer's suggestion.

W1: Gaze-Action Bidirectional Coupling- The cited SAGE-123/124 comparisons with SAGE only measure incremental future-task benefits, since both already include current gaze–action conditioning through GCSA. To assess the overall benefits of gaze and action coupling, Table 1 (main paper) provides two complementary comparisons: full SAGE against single-task models, and against joint four-task training without GCSA/GCTP. We summarize these existing results below; each cell shows baseline → full SAGE, with the gain in bold.

ComparisonGaze detection F1Action recognition accuracy (%)Gaze anticipation F1Action anticipation accuracy (%)Reference
Overall benefit: single-task models → full SAGE44.8 → 46.8 (+2.0)63.1 → 65.0 (+1.9)35.4 → 37.0 (+1.6)47.3 → 58.4 (+11.1)Main paper, Table 1
Benefit beyond joint supervision: four-task training without GCSA/GCTP → full SAGE44.8 → 46.8 (+2.0)63.1 → 65.0 (+1.9)34.5 → 37.0 (+2.5)44.0 → 58.4 (+14.4)Main paper, Table 1

The single-task baselines are SAGE-1, SAGE-2, SAGE-3, and SAGE-4, respectively. The first row captures the combined benefit of joint training and gaze–action coupling, showing improvements across all four tasks. The second holds four-task supervision unchanged, demonstrating that GCSA/GCTP improve both gaze and action prediction beyond simply combining task losses. The final 14.4-point action anticipation gain reflects the combined contribution of GCSA and GCTP towards it, consistent with Equation 1 (main paper). These comparisons assess the broader benefits of coupling that the incremental SAGE-123/124 ablations do not capture.

As requested by the reviewer, we additionally provide random, shuffled, predicted, and ground-truth gaze controls below on EGTEA Gaze+ dataset. Predicted gaze improves recognition and anticipation over random gaze and shuffled gaze. Ground-truth gaze further improves recognition and anticipation, providing an oracle reference. These controls support the importance of informative gaze input and complement Table 1’s (main paper) evidence for the coupling mechanisms.

\hline & \text{Action Detection (Top-1 Acc.)} & \text{Action Anticipation} \\\\ \begin{array}{l} \strut \text{Gaze Condition} \\\\ \hline \strut \text{Random gaze} \\\\ \strut \text{Shuffled gaze} \\\\ \strut \text{Predicted gaze (SAGE)} \\\\ \hline \strut {\color{gray}{\text{Ground truth gaze (Oracle)}}} \end{array} & \begin{array}{cccc} \strut \text{S1} & \text{S2} & \text{S3} & \text{Avg.} \\\\ \hline \strut 61.3 & 60.4 & 59.8 & 60.5 \\\\ \strut 61.5 & 59.6 & 58.4 & 59.8 \\\\ \strut \mathbf{66.4} & \mathbf{65.4} & \mathbf{63.1} & \mathbf{65.0} \\\\ \hline \strut {\color{gray}{69.2}} & {\color{gray}{68.3}} & {\color{gray}{66.0}} & {\color{gray}{67.8}} \end{array} & \begin{array}{c} \strut \text{Avg. mean cls acc} \\\\ \hline \strut 47.3 \\\\ \strut 44.2 \\\\ \strut \mathbf{58.4} \\\\ \hline \strut {\color{gray}{60.5}} \end{array} \\\\ \hline \end{array}$$ We have incorporated this additional gaze control ablation study into Section 5.1 and Table 2 of the revised manuscript.
AuthorsOpenReview ↗

Response to Reviewer AkF3 [2/2]

W2: Experiment Protocol - We would like to clarify the experimental protocols and reporting conventions for VidHOI and EGTEA Gaze+.

  • Best-run performance versus three-seed statistics on VidHOI: Following the best-run reporting practice of prior methods ST-Gaze (Ni et al., (2023)), we report SAGE’s best-performing run (38.65 mAP) in the main paper. We additionally provide results from three independently trained runs in the appendix (37.22 ± 0.42 mAP) to quantify training variability. These additional runs exclude the best-performing run; including it yields a four-run mean and sample standard deviation of approximately 37.58 ± 0.79 mAP. We will clarify this distinction in the manuscript. Note that ST-Gaze (Ni et al. (2023)) do not report a corresponding mean and standard deviation across seeds, so their training variability is unavailable for comparison.
  • ST-Gaze baseline results: The ST-Gaze results reported in our tables were obtained by retraining the model, rather than directly copying the best-performing results from the original publication. We retrained ST-Gaze for two reasons. First, SAGE refers to the human–object feature construction strategy from ST-Gaze, making it an important reference baseline which requires retraining for fair comparison. Second, to specifically evaluate the effectiveness of GCSA (comparing to use frozen gaze feature as an input in ST-Gaze), we introduce a spatial-only variant of ST-Gaze by removing its temporal layer, as reported in Table 10. The differences between our reproduced results and the originally reported results in thier paper illustrate that retraining a model may not reproduce its best-run performance. This observation also provides context for the difference between SAGE's best-performing run and its independent three-seed statistics.
  • EGTEA Gaze+ evaluation metrics: Table 5 includes results reported using different evaluation metrics following prior studies. Specifically, the S1 column reports mean-class accuracy, following MCN (Huang et al., 2020) and EGTEA (Li et al., 2021), whereas the S3 column reports Top-1 accuracy, as reported by I3D-2Stream, R34-2Stream, SAP, and GC-TSM. We additionally report the average Top-1 accuracy across all three splits to provide a more comprehensive evaluation of action recognition performance. Since prior methods differ in their reported metrics and available split-wise results, we retain their original evaluation protocols. To clarify and provide complete split-wise results, we also report Top-1 accuracy for S1, S2, and S3 in the above gaze control ablation table. Note that providing S1 (Top-1) and S2 (Top-1) results does not change the overall comparison, since Table 5’s (main paper) average already includes all three splits. SAGE scores 66.4/65.4/63.1 versus GC-TSM’s 66.5/66.1/62.6, remaining competitive at 65.0 versus 65.1 on average. Unlike us, GC-TSM does not report across-seed mean and standard deviation for these results, so we cannot determine whether the 0.1-point average gap persists across seeds or reflects training variability.

Q1: Anticipated Gaze's Contribution - Anticipating future gaze is inherently challenging because future video frames are unavailable: SAGE must infer future gaze distributions solely from observed video features and gaze information. Despite this, the tables below show consistent improvements over SAGE-124 in both egocentric and exocentric settings, and across multiple anticipation horizons on EGTEA Gaze+. These gains are additional to the observed-gaze conditioning already provided by GCSA, and we also include the margin between leading action-anticipation baselines to put the magnitude of anticipated gaze's contribution in context.

Contribution of anticipated gaze and comparison with prior baselines

ComparisonDataset / viewpointAction anticipation accuracy (%)Gain (percentage points)Reference
SAGE-124 → Full SAGEEGTEA Gaze+ / Egocentric56.3 → 58.4+2.1Appendix Table 15
SAGE-124 → Full SAGEExo-Cook / Exocentric52.5 → 54.8+2.3Appendix Table 19
ORViT-MF → InAViTEGTEA Gaze+ / Egocentric57.2 → 58.2+1.0Table 6

The anticipated-gaze ablation shows consistent benefits in both viewpoints. For context, its 2.1-point gain on EGTEA Gaze+ exceeds the 1.0-point gap between the two strongest prior action-anticipation baselines reported in Table 5.

Anticipated-gaze contribution across horizons on EGTEA Gaze+ (Appendix Table 16)

Anticipation horizonGain over SAGE-124 (percentage points)
0.5 seconds+2.1
1.0 seconds+1.7
1.5 seconds+1.7
2.0 seconds+3.0

Gains remain positive at every evaluated horizon, with the largest improvement at 2.0 seconds, where observed interaction cues become less informative.

Q2: Gaze Control Ablation - Provided in the above table (Response [1/2])

AuthorsOpenReview ↗

Follow-up on Rebuttal

Thank you again for your thoughtful questions and comments! We have added control experiments using ground-truth, random, and shuffled gaze, clarified the evidence for gaze–action coupling and the contribution of anticipated gaze, addressed the SOTA comparisons, split-wise results, and seed statistics.

We would greatly appreciate your feedback on whether these additions address your main concerns or if anything remains unclear.

Reviewer_ysLoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents SAGE (Synchronized Action-GazE), a transformer-based framework designed to jointly recognize and anticipate human-object interactions (HOI) and human gaze fixations across both egocentric (first-person) and exocentric (third-person) video domains. The framework models the spatial and temporal dependencies between gaze and actions using two core modules: Gaze-Conditioned Spatial Attention (GCSA), which uses predicted gaze heatmaps as attention biases for current HOI detection, and Gaze-Conditioned Temporal Prediction (GCTP), which conditions future action anticipation on predicted future gaze sequences. In addition, the authors curate Exo-Cook, a dataset consisting of 32,050 clips derived from Ego-Exo4D by projecting 3D Aria eye-tracking data into exocentric camera views and clustering textual narrations into 10 action classes. SAGE is evaluated on EGTEA Gaze+, Vid-HOI, and Exo-Cook across gaze detection, action recognition, gaze anticipation, and action anticipation.

Critical strengths
  1. Unified multi-task framework: Rather than treating gaze and actions as isolated multi-task outputs, the formulation systematically factorizes the joint action-gaze distribution over observed and future horizons. The method at a high level is conceptually coherent. Modules such as GCSA and GCTP provide a structured, principled mechanism to study the value of visual attention priors in forecasting future human behavior.
  2. Third-person gaze-action dataset (Exo-Cook): By computing 3D gaze ray intersections from wearable Project Aria devices and projecting them via calibrated extrinsics into external cameras, the paper provides a benchmark addressing the gap in third-person gaze anticipation and HOI forecasting.
Critical weaknesses
  1. Severe Oversimplification of Exo-Cook: The newly proposed Exo-Cook benchmark reduces complex cooking procedures across 32,050 video clips into only 10 coarse clusters. This significantly limits the benchmark's value and diagnostic capability for fine-grained action anticipation, making the high classification accuracy on this benchmark somewhat artificial compared to standard benchmarks like EGTEA Gaze+ (86 classes) or EPIC-KITCHENS.
  2. Lack of method clarity: Key parts of the architecture are too ambiguously described in the main paper, making it difficult to confidently assess the paper's correctness and reproducibility. Important implementation details are deferred to the supplementary, and even then some notation remains inconsistent. For example:
    2.1 Construction of the HOI tokens remains too vague. The paper states that TPV tokens encode human appearance, location, and human–object relations, whereas FPV tokens encode hand–object relations with a single implicit human. However, it does not explain how these quantities are extracted or projected into tokens.
    2.2 The role of GCSA in Figure 1 seems confusing. Figure 1 may need be re-designed.
    2.3 The future gaze decoding equations are under-specified. In Equation 16, the right-hand side of g^m\hat{g}_m does not explicitly depend on m.
  3. Marginal improvement: On the EGTEA Gaze+ dataset, SAGE’s mean action recognition accuracy (65.0%) underperforms GC-TSM (65.1%). For action anticipation, the mean class accuracy (58.4%) exceeds InAViT (58.2%) by only 0.2 percentage points.
Questions for authors
  1. Several issues exist in references. The reference entry for Zhao et al. (2024) (AntGPT) fabricates volume and pagination metadata (volume 2024, pp. 56677–56697). Please note that ICLR papers are open-access electronic conference proceedings without pagination.
  2. Please clarify how the probabilistic formulation maps to the actual architecture implementation. For example, if MC sampling produces gaze trajectories as sequences of points, how are these used by the action branches when GCSA/GCTP appear to condition on gaze heatmaps rather than sampled points?
  3. Given that feature construction, gaze backbones, and inputs are entirely distinct between FPV and TPV, is there any shared parameterization or genuine cross-view generalization between egocentric and exocentric settings?
3 comments
AuthorsOpenReview ↗

Response to Reviewer ysLo [1/2]

Thank you for recognizing the rationale behind SAGE’s joint probabilistic formulation and the value of Exo-Cook for studying gaze and action anticipation in third-person videos. We appreciate your assessment that the modeling approach and benchmark together provide a useful basis for understanding how visual attention informs future human behavior. We address your questions and concerns below. In the meantime, we have incorporated corresponding revisions in the paper.

W1: Oversimplification of Exo-Cook - We acknowledge that Exo-Cook uses broader action categories than EGTEA Gaze+ or EPIC-KITCHENS; however, we do not present it as a benchmark for fine-grained action recognition or anticipation. Its 10 categories are systematically derived through semantic clustering of verb–noun descriptions and annotation consistency checks (Appendix A.3.3, Figures 7–8). Exo-Cook is a first step toward establishing a benchmark for joint gaze and action recognition and anticipation in exocentric videos, a setting existing egocentric benchmarks do not directly support—so its contribution should not be judged by action granularity alone. Furthermore, its smaller vocabulary does not produce near-saturated performance: SAGE achieves 60.2% HOI detection accuracy and 54.8% HOI anticipation accuracy, compared with 65.0% Top-1 recognition accuracy and 58.4% mean-class anticipation accuracy on EGTEA Gaze+. Although the evaluation settings prevent a direct ranking of dataset difficulty, these results leave substantial room for improvement on Exo-Cook; category count alone does not establish that the reported performance is artificial. We view finer-grained annotations as an opportunity to extend the benchmark, which would only make the results on the benchmark even more competitive. Established egocentric benchmarks provide precedent for expanding action vocabularies: GTEA Gaze evaluated 25 classes after filtering rare actions (Fathi et al., ECCV 2012), GTEA Gaze+ established 44 classes using a minimum-frequency criterion (Li et al., CVPR 2015), and EGTEA Gaze+ later expanded to 106 classes (Li et al., ECCV 2018).

W2: Lack of Method Clarity

  • HOI token construction -. Following STTran/ST-Gaze, SAGE constructs pairwise HOI tokens from human/object appearance and spatial features. Specifically, human and object bounding boxes are encoded using a ResNet backbone followed by fully connected layers to obtain 512-dimensional appearance features, while bounding-box spatial coordinates are projected into 256-dimensional spatial features through linear layers. These features are concatenated to construct pairwise HOI tokens. For TPV videos, tokens are constructed for each human–object pair, allowing the model to represent interactions involving multiple subjects and objects. For FPV videos, where the acting subject is implicit, the construction is simplified to hand–object pairs: the human representation is derived from the hand's appearance and spatial features. We will explicitly describe these feature extraction and projection operations in the main paper.

(We have expanded the description of HOI token construction in the revised Section 4.1 (highlighted in blue), providing a clearer explanation of how SAGE constructs HOI tokens differently for egocentric (FPV) and exocentric (TPV) views.)

  • Role of GCSA -. GCSA is our proposed gaze-conditioned replacement for standard multi-head self-attention (MHSA) within the Transformer encoder. Rather than directly selecting objects based on gaze scores, GCSA computes object-level gaze relevance from the predicted gaze heatmap and introduces it as an additive bias to the self-attention logits.
  • Future gaze decoding (Eq.16) - We clarify that mm denotes the temporal index of the future gaze prediction. In Eq.16, the decoder Dϕ\mathcal{D}_{\phi} produces a tensor of future gaze representations in RM×H^×W^×D′\mathbb{R}^{M\times\hat H\times\hat W\times D'}, which is decoded into MM future gaze heatmaps in parallel. Thus, g^m\hat g_m denotes the (m)(m)-th temporal slice of the decoded output tensor. We will revise the notation accordingly in Eq.16.

(We have revised the formulation and accompanying explanation of Eq. (16) in blue color to better clarify the future gaze prediction process.)

References:

  • Fathi, Alireza, Yin Li, and James M. Rehg. "Learning to recognize daily actions using gaze." European Conference on Computer Vision. Berlin, Heidelberg: Springer Berlin Heidelberg, 2012.
  • Li, Yin, Zhefan Ye, and James M. Rehg. "Delving into egocentric actions." Proceedings of the IEEE conference on computer vision and pattern recognition. 2015.
  • Li, Yin, Miao Liu, and James M. Rehg. "In the eye of beholder: Joint learning of gaze and actions in first person video." European Conference on Computer Vision. Cham: Springer International Publishing, 2018.
AuthorsOpenReview ↗

Response to Reviewer ysLo [2/2]

W3: Marginal Improvement - SAGE jointly addresses gaze detection, action recognition, gaze anticipation, and action anticipation; achieving the highest score on every metric is not its sole objective. Even remaining competitive with specialized baselines while providing all four predictions together is valuable for applications such as proactive human–robot collaboration and AR/VR (Line 41-43). SAGE also improves performance on several other metrics: +2.0/+2.7 F1, +0.9/+6.4 Rec, +1.4/+3.1 Prec for EGTEA gaze detection/anticipation (Table 5); +7.0 F1 over the best adapted gaze-anticipation baseline and +2.1 mAP/+2.5 F1 over ST-Gaze for Exo-Cook HOI detection (Table 2); and +2.00/+2.03 mAP over ST-Gaze for VidHOI anticipation at one/three seconds (Table 4). Thus, the two cited action metrics capture only part of the framework’s benefits. SAGE additionally reduces computation from 601.5 to 533.4 GFLOPs compared with running four separate task-specific models—an 11.3% reduction (Table 7).

Q1: Reference Metadata - The volume and pagination information in the reference to Zhao et al. (2024), AntGPT, was obtained from Google Scholar’s citation export and was not fabricated. The export includes both fields (shown below). Similar metadata also appears in Google Scholar’s citation exports for other ICLR papers.

@inproceedings{zhao2024antgpt,
  title={Antgpt: Can large language models help long-term action anticipation from videos?},
  author={Zhao, Qi and Wang, Shijie and Zhang, Ce and Fu, Changcheng and Do, Minh Quan and Agarwal, Nakul and Lee, Kwonjoon and Sun, Chen},
  booktitle={International Conference on Learning Representations},
  volume={2024},
  pages={56677--56697},
  year={2024}
}

Q2: Probabilistic Framing & MC Sampling - During Monte Carlo sampling, gaze locations are drawn from these distributions to obtain sampled gaze trajectories across the observed and anticipated time steps. Each sampled gaze location can be represented as a spatial indicator map aligned with the corresponding heatmap resolution, allowing the sampled gaze information to be processed by the gaze-conditioned action modules. For current action recognition, GCSA computes object-level gaze relevance by integrating the gaze representation over detected object bounding boxes and incorporates the resulting scores into transformer self-attention. For action anticipation, the sampled future gaze information is encoded and incorporated into GCTP alongside the current gaze-conditioned interaction features. Thus, the probabilistic gaze variables and the gaze-conditioned transformer modules operate on compatible spatial representations, rather than representing unrelated mechanisms.

We have incorporated a more detailed explanation of how SAGE performs MC sampling in Section 5, Section 5.4, and Appendix A.2.4, with all revisions highlighted in blue in the revised paper.

Q3: Shared Parameterization for FPV/TPV - SAGE uses a shared parameterized formulation for its core gaze–action modeling: both FPV and TPV follow the same probabilistic factorization and GCSA/GCTP functional forms (Appendix A.2.2). Here, the sharing concerns the model structure; the modules are instantiated with separately learned weights for each viewpoint, as stated in Lines 77–79. Viewpoint-specific features and gaze backbones accommodate different visual observations while preserving these core mechanisms. We therefore claim a common modeling formulation across viewpoints, without claiming weight sharing or cross-view generalization. This broader applicability is a substantive contribution: to our knowledge, prior joint gaze–action methods have not demonstrated these tasks in both egocentric and exocentric settings, even through separate training.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Thank you again for your helpful suggestions! We have added information about the Exo-Cook dataset, architectural clarity and the implementation of gaze sampling, performance comparisons, reference metadata, and cross-view parameter sharing and generalization.

We would greatly appreciate your feedback on whether these additions address your main concerns or if anything remains unclear.

Reviewer_Pr7hOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents SAGE, an end-to-end Transformer framework that jointly performs gaze detection, action recognition, gaze anticipation, and action anticipation across both egocentric and exocentric videos. The method decomposes joint gaze-action modeling into four sequential stages and introduces two core modules: Gaze-Conditioned Spatial Attention, which injects object-level gaze probabilities as an additive attention bias, and Gaze-Conditioned Temporal Prediction, which conditions future action anticipation on predicted future gaze via cross-attention. The authors also curate Exo-Cook, a third-person benchmark derived from Ego-Exo4D, and demonstrate strong empirical performance across EGTEA Gaze+, Vid-HOI, and Exo-Cook.

Critical strengths
  1. Integrating present and future gaze-action modeling into a single end-to-end pipeline is well-motivated. Using object-level gaze probability as a soft attention bias in GCSA preserves global scene context better than prior pixel-masking approaches.

  2. Experiments clearly separate the gains of architectural design from multi-task supervision. Under identical losses, adding GCSA and GCTP boosts action anticipation accuracy on EGTEA Gaze+ from 44.0% to 58.4%.

  3. SAGE reduces total GFLOPs by 11.3% compared to running four separate state-of-the-art models while enabling interpretable uncertainty analysis via Monte Carlo gaze sampling.

Critical weaknesses
  1. The framework assumes frame-wise conditional independence when sampling gaze trajectories, ignoring smooth temporal eye movement dynamics. Moreover, training uses single-sample approximation while multi-sample marginalization is only qualitatively shown at inference.

  2. Despite supporting both egocentric and exocentric views, SAGE requires separate feature extractors, different gaze backbones, and independent training for each view rather than learning a unified cross-view representation.

  3. Gaze is predicted on a 64x64 grid. When multiple small kitchen objects overlap or sit close together, how does GCSA avoid assigning high gaze scores to adjacent distractor boxes?

  4. The related work omits several recent studies exploring the synergy between human gaze and HOI understanding, such as "Learning from observer gaze: Zero-shot attention prediction oriented by human-object interaction recognition" (CVPR 2024).

Questions for authors

Please refer to the above Weaknesses.

3 comments
AuthorsOpenReview ↗

Response to Reviewer Pr7h [1/2]

Thank you for recognizing the motivation behind SAGE and the evidence supporting its design. We appreciate your positive assessment of how joint gaze–action modeling improves prediction while offering practical benefits in efficiency and understanding model uncertainty. Your suggestions motivated additional experiments and clarifications that strengthened our evaluation, which we have incorporated in the revised paper. We address your questions below:

W1: Temporal Dependence - We would like to clarify that frame-wise independent sampling does not imply that SAGE’s gaze and action representations are independent across frames. We factorize observed-gaze sampling (Eq. 23, Appendix A.2.4), following the mean-field principle (Blei et al., 2017), to make marginalization over the complete gaze sequence G={g0:t+M}G = \{g_{0:t+M}\} (Eq. 1) computationally manageable. This approximation does not imply that human gaze is temporally independent, and is a widely adopted strategy in computer vision, e.g. Girdhar et al. (2024) independently perturbs video frames while using temporal convolution and attention to model cross-frame relationships. SAGE models temporal context through self-attention over observed gaze and cross-attention with human–object interaction features for gaze anticipation (Eqs. 4 and 25), followed by temporal modeling in GCTP for action anticipation.

We have further clarified the conditional independence assumption in the blue-highlighted text before and after Eq. (23) in Appendix A.2.4.

W1: Single-Sample Approximation - We have added a quantitative comparison of K=1 and K=3 during both training and inference on EGTEA Gaze+ below, addressing the concern that multi-sample inference was demonstrated only qualitatively. Each sample represents a complete gaze sequence rather than one gaze point. Increasing to K=3 yields modest, mixed changes: gaze anticipation F1 improves from 37.0 to 37.8 and action anticipation accuracy from 58.4 to 58.8, while gaze detection F1 changes from 46.8 to 46.5 and action recognition accuracy from 65.0 to 64.9. Meanwhile, training-step time increases by approximately 22.5% (0.600 to 0.735 seconds), peak GPU memory by 51.0% (10.269 to 15.509 GiB), and extrapolated training cost from 10.46 to 12.81 GPU-hours. These results support K=1 as a practical balance between performance and computational cost, consistent with single-sample Monte Carlo training in prior work (Kingma et al. (2014), Section 2.3), while providing quantitative evaluation of the multi-sample configuration. At inference, multiple samples enable closer approximation of the gaze-marginalized predictive distribution and analysis of action uncertainty.

We have clarified the Monte Carlo sampling strategies and sample counts used during training and inference in the blue-highlighted text in Sections 4, Section 5 and Section 5.4 of the revised paper. The corresponding performance comparison table is included in Appendix A.5.7.

\hline & \text{Gaze Detection} & \text{Action Recognition} & \text{Gaze Anticipation} & \text{Action Anticipation} & \text{GPU Efficiency} \\\\ \begin{array}{l} \strut \text{MC Samples} \\\\ \hline \strut K=1 \\\\ \strut K=3 \end{array} & \begin{array}{ccc} \strut \text{F1} & \text{Rec} & \text{Prec} \\\\ \hline \strut \mathbf{46.8} & \mathbf{62.1} & 36.8 \\\\ \strut 46.5 & 61.8 & \mathbf{37.0} \end{array} & \begin{array}{c} \strut \text{Acc} \\\\ \hline \strut \mathbf{65.0} \\\\ \strut 64.9 \end{array} & \begin{array}{ccc} \strut \text{F1} & \text{Rec} & \text{Prec} \\\\ \hline \strut 37.0 & \mathbf{54.9} & \mathbf{32.7} \\\\ \strut \mathbf{37.8} & 54.5 & 32.5 \end{array} & \begin{array}{c} \strut \text{Acc} \\\\ \hline \strut 58.4 \\\\ \strut \mathbf{58.8} \end{array} & \begin{array}{ccc} \strut \text{Step Time (s)} & \text{GPU Mem. (GiB)} & \text{GPU-hours} \\\\ \hline \strut \mathbf{0.600} & \mathbf{10.269} & \mathbf{10.46} \\\\ \strut 0.735 & 15.509 & 12.81 \end{array} \\\\ \hline \end{array}$$ **References:** - David M Blei, Alp Kucukelbir, and Jon D McAuliffe. Variational inference: A review for statisticians. Journal of the American statistical Association, 112(518):859–877, 2017. - Rohit Girdhar, Mannat Singh, Andrew Brown, Quentin Duval, Samaneh Azadi, Sai Saketh Rambhatla, Akbar Shah, Xi Yin, Devi Parikh, and Ishan Misra. Factorizing text-to-video generation by explicit image conditioning. In European Conference on Computer Vision, pp. 205–224. Springer, 2024 - Diederik P Kingma and Max Welling. Auto-encoding variational bayes. In International Conference on Learning Representations, 2014.
AuthorsOpenReview ↗

Response to Reviewer Pr7h [2/2]

W2: Cross-View Representation - We acknowledge that SAGE does not learn a unified cross-view representation, but we never claimed it as a contribution of our work. The manuscript explicitly states that egocentric and exocentric settings use separate training (Lines 77–79). In our paper, “unified” means jointly learning gaze detection, action recognition, gaze anticipation, and action anticipation within one framework. Our contribution is that the proposed framework supports both egocentric and exocentric views using the same underlying probabilistic formulation and GCSA/GCTP mechanisms (Eqs 1-6), whose effectiveness we demonstrate in both settings. We use viewpoint-specific feature extractors and gaze backbones only to accommodate the substantial differences in visual observations, while keeping the core methodology unchanged. Cross-view representation learning and transfer are beyond the scope of this work. Nevertheless, supporting both viewpoints is a meaningful extension: to our knowledge, no prior gaze–action frameworks has demonstrated joint recognition and anticipation of gaze and actions in both egocentric and exocentric settings, even with separate training.

W3: 64x64 grid - The predicted 64$\times$64 gaze heatmap is first normalized into a spatial probability distribution, and object bounding-box coordinates are transformed into the same normalized coordinate space. Gaze–object score is then computed by integrating the gaze probability mass within each object's bounding box. Now adjacent or overlapping objects can receive similar gaze relevance scores, but these scores do not determine GCSA’s attention weights on their own. GCSA computes gaze relevance by aggregating gaze probability mass within each object’s bounding box and introduces it as an additive bias to the learned attention logits (Eq. 2). Attention therefore depends on both gaze relevance and the visual and spatial information encoded in the human–object interaction tokens. These tokens capture appearance and human–object spatial relationships, including hand–object relationships in egocentric videos, while a subsequent temporal encoder integrates the resulting representations across observed frames (Eq. 11). The action recognition branch jointly considers gaze attention and human–object spatial–temporal relationships to distinguish the interacting object. For example, an active object typically moves together with the human hand over time, while adjacent inactive objects often remain stationary. Such temporal interaction cues can help the model identify the correct object during training.

We have added further explanations of gaze heatmap and bounding-box normalization in the blue-highlighted text preceding Eq. (2) in the revised paper.

W4: Missing Related Work - Thank you for highlighting this relevant work. We will include Zhou et al. (CVPR 2024) in our related-work discussion. While both methods leverage the relationship between attention and HOI understanding, their objectives and mechanisms differ substantially. First, the gaze target differs: Zhou et al. predict an external observer’s attention when interpreting a specified interaction in a static image, whereas SAGE predicts the actor’s own gaze during an interaction. Second, the tasks differ: their IA attention predictor receives the interaction label as input, while SAGE jointly performs gaze detection, action recognition, gaze anticipation, and action anticipation from video; future-gaze and future-action prediction are not addressed by Zhou et al. Third, gaze is integrated differently: their downstream HOI models use human-provided or IA-generated attention maps as targets for an attention-alignment loss. SAGE directly incorporates predicted actor gaze into interaction representations through GCSA and conditions future action prediction on anticipated gaze through GCTP.

We have also incorporated the suggested work into the Related Work section (Section 2.3) of the revised paper, with additional discussion to clarify its relationship to and distinctions from our approach.

AuthorsOpenReview ↗

Follow-up on Rebuttal

Thank you again for your encouraging assessment and constructive feedback! We wanted to kindly follow up on the new experiments with multiple Monte Carlo samples and our clarifications on handling egocentric and exocentric views, the missing related work, and GCSA scoring.

We would greatly appreciate your thoughts on whether these results and clarifications address your main concerns, or whether any questions remain.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response to Reviewers

We thank all three reviewers. We added gaze-control and multi-sample experiments and clarified the methodology. We first highlight SAGE’s broader value.

SAGE unifies gaze detection, action recognition, gaze anticipation, and action anticipation for proactive human–robot collaboration and AR/VR, which need attention, current interactions, and future behavior together. Its value extends beyond leading every metric on every task and dataset: SAGE matches or outperforms specialized baselines while delivering all four predictions with 11.3% fewer GFLOPs (533.4 vs. 601.5) than four task-specific models.

ConcernResponse and evidence
Gaze–action couplingSAGE-123/124 already use current gaze through GCSA. Over single-task models, full SAGE gains 2.0/1.6 gaze detection/anticipation F1 and 1.9/11.1 recognition/anticipation accuracy points. With four-task supervision fixed, the corresponding GCSA/GCTP gains are 2.0/2.5 F1 and 1.9/14.4 accuracy points (Table 1). These gains reflect both modules, not future gaze alone.
Anticipated gazeWithout future frames, anticipated gaze adds 2.1 anticipation points on EGTEA and 2.3 on Exo-Cook beyond observed gaze, with gains across evaluated horizons (Appendix Tables 15–16, 19).
Temporal independenceObserved-gaze factorization makes marginalization tractable, without making all representations independent. Temporal attention captures cross-frame context but does not enforce smooth sampled trajectories.
Architecture and 64×64 gazeAdjacent boxes may receive similar gaze scores; GCSA uses these as an additive bias alongside visual/spatial cues and subsequent temporal modeling. We detailed human–object/hand–object tokens, sampled-point spatial maps, and decoding of M future maps indexed by m.
Cross-view scope“Unified” means four tasks. Both views use the same probabilistic formulation and GCSA/GCTP mechanisms, with separate weights/backbones; cross-view transfer is never claimed.
Exo-Cook granularityThe 10 systematically clustered categories establish an initial exocentric four-task benchmark (Appendix A.3.3). Despite broader labels, HOI accuracy (detection/anticipation: 60.2%/54.8%) is numerically close to fine-grained EGTEA (65.0%/58.4%). Gaze detection/anticipation F1 (57.7/49.2) also leaves headroom. Finer annotations offer an extension.
Reporting consistencyVidHOI’s best run (38.65 mAP) is excluded from the three additional runs (37.22 ± 0.42). ST-Gaze was retrained. EGTEA’s 61.8 is S1 mean-class accuracy, not Top-1; S1/S2/S3 Top-1 scores are 66.4/65.4/63.1%.
Related work and referencesWe added Zhou et al. (CVPR 2024), distinguishing observer attention/alignment supervision in images from SAGE’s direct actor-gaze conditioning across four video tasks. Questioned metadata came from Google Scholar.

New gaze controls — EGTEA Gaze+ (revised Section 5.1, Table 2)

Gaze inputRecognition: avg. Top-1 (%)Anticipation: avg. mean-class accuracy (%)
Random60.547.3
Shuffled59.844.2
Predicted (SAGE)65.058.4
Ground truth (oracle)67.860.5

Predicted gaze improves recognition by 4.5/5.2 points and anticipation by 11.1/14.2 points over random/shuffled gaze, supporting gaze’s value.

New Multi-sample evaluation — EGTEA Gaze+ (Appendix A.5.7)

\hline &\text{Gaze Detection}&\text{Action Recognition}&\text{Gaze Anticipation}&\text{Action Anticipation}&\text{GPU Efficiency} \\\\ \begin{array}{l} \strut \text{MC Samples} \\\\ \hline \strut K=1 \\\\ \strut K=3 \end{array} & \begin{array}{ccc} \strut \text{F1}&\text{Rec}&\text{Prec} \\\\ \hline \strut \mathbf{46.8}&\mathbf{62.1}&36.8 \\\\ \strut 46.5&61.8&\mathbf{37.0} \end{array} & \begin{array}{c} \strut \text{Acc} \\\\ \hline \strut \mathbf{65.0} \\\\ \strut 64.9 \end{array} & \begin{array}{ccc} \strut \text{F1}&\text{Rec}&\text{Prec} \\\\ \hline \strut 37.0&\mathbf{54.9}&\mathbf{32.7} \\\\ \strut \mathbf{37.8}&54.5&32.5 \end{array} & \begin{array}{c} \strut \text{Acc} \\\\ \hline \strut 58.4 \\\\ \strut \mathbf{58.8} \end{array} & \begin{array}{ccc} \strut \text{Step Time (s)}&\text{GPU Mem. (GiB)}&\text{GPU-hours} \\\\ \hline \strut \mathbf{0.600}&\mathbf{10.269}&\mathbf{10.46} \\\\ \strut 0.735&15.509&12.81 \end{array} \\\\ \hline \end{array}$$ K applies in training and inference; each sample is a complete gaze sequence. Step time and peak allocated GPU memory measure training efficiency; *GPU-hours are extrapolated to the full schedule.* K=3 gives modest, mixed changes at **22.5% higher training-step time** and **51.0% higher peak GPU memory**, supporting K=1’s performance–cost balance.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.