acceptodds
Under review as a conference paper at ICLR 2027

VANTAGE: Temporally Structured Predictive Uncertainty for Probabilistic Egocentric Gaze Anticipation

Abstract

Egocentric gaze anticipation is inherently uncertain, yet existing methods predominantly predict a single deterministic future sequence, leaving plausible future variations unmodeled, the structure of predictive uncertainty uncharacterized, and its reliability information unused. Our key insights are that (i) the conditional variability of future gaze is temporally structured: the uncertainty varies along the anticipation horizon, and (ii) predictive uncertainty manifests in trajectory-level and gaze heatmap-level forms. We therefore reformulate the task as conditional probabilistic forecasting and propose **VANTAGE**, a temporally structured variational framework that unifies latent modeling and two-level uncertainty estimation over a shared interval-based horizon decomposition, with the estimated uncertainty directly guiding temporal attention during decoding. Experiments on three datasets demonstrate the effectiveness of the proposed temporally structured modeling and uncertainty estimation, while revealing different roles of the two forms. These results support predictive uncertainty as both a measure of prediction reliability and a signal for temporal reasoning.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Egocentric Vision, Gaze Anticipation, Probabilistic Forecasting, Uncertainty Estimation, Generative Models

Reviews

Among:
Shape:
1234this paper, Bottom 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.25 over 4 reviews, above 49% of the others.
Reviewer_kUGbOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes VANTAGE for probabilistic egocentric gaze anticipation from past video, with optional gaze history. It uses interval-specific latent variables to generate future gaze heatmaps, estimates trajectory- and heatmap-level uncertainty, and uses the trajectory-level estimate to guide temporal attention. Across three datasets, the paper reports sampled-prediction, distributional, and single-prediction results; it analyzes uncertainty reliability on Ego-Exo4D.

Critical strengths

Forecasting future gaze as a distribution is well motivated: the same observed scene can lead to more than one plausible gaze path. The paper evaluates both the quality of sampled futures and whether estimated uncertainty tracks prediction error, rather than relying only on point-prediction accuracy. The error-association and selective-prediction analyses on Ego-Exo4D add evidence about whether the estimated uncertainty identifies unreliable forecasts, beyond what point-prediction accuracy can show.

Critical weaknesses

1. Uncertainty-guided attention adds little beyond plain attention. In Table 4, plain cross-time attention already reaches 17.06 minADE and 20.29 CRPS. Adding trajectory-level uncertainty brings these to 16.58 and 19.91, but NLL rises from 10.10 to 10.74. Heatmap-level uncertainty, alone or combined, does not improve minADE or CRPS over trajectory-level uncertainty alone. Table 3 is also uneven across the number of intervals: R=3 has the worst NLL (27.03), and the capacity-matched R=1 variant has better meanADE than R=4. These ablations cover only Ego-Exo4D, and no variation across training seeds is reported. The gain from uncertainty guidance is small, while the interval-latent results are mixed; neither ablation reports whether these differences persist across datasets or training runs.

2. Probabilistic comparisons are narrow, and the accuracy cost is understated. The probabilistic baselines in Table 1 are MC DropPath and two CVAE variants on the same backbone; the paper does not adapt an existing gaze or trajectory generator, or explain why such a comparison is infeasible. In Table 5, Raw-σ is the only alternative uncertainty estimator used for the fidelity comparison; ensembles and MC dropout are absent. Even within Table 1, TS-CVAE has better NLL than VANTAGE on all three datasets with gaze history. In Table 2, VANTAGE with gaze history beats the same-input MViT in F1 only on Ego-Exo4D (0.614 vs. 0.607). With video alone, its F1 is below MViT and GLC on all three datasets (e.g., 0.231 vs. 0.278 for MViT on EGTEA Gaze+). Calling this a "modest trade-off" understates the cost.

Questions for authors

Needs further clarification

  1. How sensitive are the NLL rankings in Tables 1 and 3 to fitting a diagonal Gaussian to only 20 samples (Appendix B.1)? How was the spatial-softmax temperature of 10 chosen, given that the error correlation of MC heatmap uncertainty changes sign across temperatures in Table 6?
  2. Which Ego-Exo4D subset was used? The supplied split files contain 862 training and 218 test takes. How were these takes selected, and is this the same split used for the GLC and CSTS comparisons?
  3. Were all Table 2 baselines retrained and evaluated on these splits, or were any results taken from prior work?
  4. How much do the differences in Tables 3 and 4 vary across training seeds? The take-level bootstrap intervals reported for the uncertainty analysis capture variation over test samples, not over training runs.
  5. What are the inference-time and computational overheads of the uncertainty estimator and UCTA relative to plain cross-time attention? Please report separately the time for one uncertainty estimate, one prediction, and 20 sampled predictions, and compare uncertainty estimation without stochastic decoding with estimation from Monte Carlo samples.

Suggestions

  1. A task-adapted external generative baseline would help establish the probabilistic contribution. Comparing uncertainty estimates with alternatives such as ensembles or MC dropout would also help.
  2. Figure 1 is the only qualitative forecasting example and shows one sequence per method. Its caption should explain what the colored heatmaps and green dots represent, and whether VANTAGE shows a sampled or representative prediction. Showing multiple sampled futures, interval-wise uncertainty, and failure cases across datasets would make its probabilistic behavior easier to assess.

Minor: "Latnt" in Fig. 2(b), "We uses" in Appendix A, inconsistent spelling of Ego-Exo4D, and a repeated sentence in Appendix C.2.

3 comments

We thank the reviewer for the feedback. We address CW1–CW2 and Q1–Q5 below.

1. UCTA and Temporal Structuring (CW1, Q4)

To assess robustness, we conducted additional experiments across three seeds; details are in Tables 9–10. Pred-Traj UCTA consistently improves minADE, meanADE, and CRPS over CTA, while CTA retains a lower NLL. These results demonstrate the consistent benefits of uncertainty guidance across training seeds. Regarding temporal structuring, R=4 consistently outperforms the capacity-matched R=1 variant in minADE, NLL, and CRPS, though R=1 has lower meanADE. We acknowledge R=3's higher NLL. However, increasing R changes the temporal granularity and structure of the latent space, so improvements are not necessarily monotonic.

Furthermore, Pred-Traj achieves the best overall performance among the uncertainty signals (Table 4), supporting trajectory-level uncertainty as the most effective guidance signal. Importantly, the improvement from TS-CVAE to VANTAGE reflects the combined effects of cross-time attention and uncertainty guidance, whereas Table 4 isolates the incremental benefit of uncertainty guidance over ordinary CTA. This distinction clarifies the respective contributions of the two components.

2. Probabilistic Baselines and Uncertainty Reliability (CW2)

Our baselines prioritize controlled comparisons with identical backbones, inputs, and protocols. ScanGAN360, ScanTD, and ScanDiff primarily target image-conditioned scanpath generation, not ego gaze anticipation without future observations. Deep Future Gaze generates video but doesn't model gaze-traj distributions; physical trajectory methods use different modalities and outputs. Adapting them requires substantial changes to their conditioning and prediction formulations. In contrast, Global CVAE and TS-CVAE isolate our proposed mechanisms under controlled settings.

Regarding uncertainty estimation, Table 5 shows improved temporal MC-target fidelity over Raw-σ. MC DropPath, a structured variant of MC Dropout, is already included in Table 1. We also compare direct MC dispersion and single-forward heatmap confidence in Table 11: Pred-Traj yields better AURC vs. MC-Traj, entropy, and peak uncertainty. These results support the estimate as a more effective signal for ranking forecasting reliability.

3. NLL and Deterministic Accuracy Trade-offs (CW2)

We appreciate your concern regarding NLL and deterministic accuracy. As reported in Tables 1–2, TS-CVAE has lower NLL, whereas VANTAGE improves minADE and CRPS. So we further evaluate nonparametric joint-trajectory Energy Score in Table 12: VANTAGE achieves 18.11 vs. 27.82 for TS-CVAE, indicating that its higher NLL does not necessarily indicate poorer joint predictive distribution quality.

We further clarify the distinction between single-sample deterministic accuracy and probabilistic forecasting quality, as VANTAGE primarily targets probabilistic forecasting, while F1 is included for comparability with prior methods. We additionally evaluate Mean@20 predictive heatmaps in Table 8. On Ego-Exo4D, Mean@20 improves F1 and recall. Recall also improves on Ego4D and EGTEA, although F1 does not consistently improve.

4. NLL and Temperature Sensitivity (Q1)

We evaluated K={10,20,30,50} stochastic samples (Table 6). The model rankings remain stable, including the relatively higher R=3 NLL. Nevertheless, NLL relies on a moment-matched diagonal-Gaussian approximation; complementary CRPS and joint-trajectory Energy Score provide a broader assessment of predictive distribution quality.

We use spatial-softmax temperature T=10 following CSTS, without tuning it for error correlation. Temperature directly affects heatmap concentration and thus within-heatmap spatial variance, explaining its sensitivity. Trajectory uncertainty remains positively correlated with error across temperatures in Table 7, supporting our distinction between uncertainty forms.

5. Dataset Construction and Baseline Provenance (Q2, Q3).

We use Ego-Exo4D's official EgoPose subset and take-level split of 862 train/218 test takes, without random reselection. Clip construction follows CSTS. All Table 2 baselines MViT, GLC, CSTS were retrained on the same constructed datasets and splits using released implementations and original method-specific settings; no results were copied from prior papers. Only dataset interfaces were adapted.

6. Inference-Time Analysis (Q5)

As shown in Table 13, VANTAGE introduces only 0.7% (41.54 vs. 41.25 ms/clip) inference overhead over CTA, while its uncertainty estimator achieves a 64× speedup over MC-Traj (K=20, 639.18). The table also reports separate timings for cached-feature estimation, end-to-end uncertainty estimation, single prediction, and K=20 sampling. CTA is evaluated using an independent checkpoint with uncertainty estimation and bias disabled. We also clarify the GT points and predicted heatmaps in Fig1, and provide low/high-uncertainty visualizations (Fig2).

Thanks to the authors for the additional experiments and clarifications. The rebuttal addresses some of my concerns, especially regarding the robustness of the results. However, the improvements from uncertainty-guided attention remain relatively small, and the comparisons with existing probabilistic methods are still limited. I also remain concerned about the trade-off in deterministic prediction accuracy. Overall, I appreciate the authors' efforts, but I will keep my original score.

Thank you for your follow-up and for recognizing the improved robustness of our results. We would like to further clarify three points.

  • Contribution of uncertainty-guided attention: Importantly, CTA is itself part of our proposed temporal reasoning mechanism and technical contribution, which yields substantial improvements, rather than an off-the-shelf baseline. The improvements from TS-CVAE to VANTAGE reflect the combined benefits of CTA and uncertainty guidance, whereas the CTA-to-UCTA comparison isolates only the incremental contribution of uncertainty guidance. Although this incremental improvement is moderate, it is supported by our multi-seed experiments. More importantly, our contribution is not limited to accuracy improvements, but introduces predictive uncertainty as an explicit signal for temporal decoding. The learned estimator also achieves better uncertainty-based error ranking than direct MC trajectory dispersion, with approximately 64× faster uncertainty estimation.

  • Probabilistic comparisons. The limited availability of directly comparable probabilistic baselines reflects the current state of egocentric gaze anticipation, where existing methods predominantly produce deterministic predictions. Extending probabilistic methods from related domains would require substantial adaptations due to both task mismatch and domain shift. We recognize the value of broader comparisons, but emphasize that the lack of established probabilistic baselines is also part of the research gap that motivates our work.

  • Deterministic accuracy. Deterministic metrics assess individual prediction accuracy rather than the quality of the predictive distribution, which is the primary objective of our work. Although a trade-off remains on some datasets, VANTAGE achieves substantial improvements in distributional quality, including a 34.9% reduction in joint-trajectory Energy Score. We believe probabilistic forecasting models should be assessed primarily by their distributional performance, rather than solely by deterministic accuracy.

We respectfully believe that focusing primarily on the incremental gains of uncertainty guidance and deterministic prediction accuracy does not fully reflect the methodological contributions of VANTAGE. Our central contributions lie in probabilistic forecasting, temporally structured uncertainty modeling, and uncertainty-aware temporal reasoning, which are supported by extensive additional experimental evidence. We hope these contributions and findings will receive appropriate consideration in the final assessment.

Reviewer_tKsuOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper reformulates egocentric gaze anticipation as conditional probabilistic forecasting over future gaze heatmap sequences. VANTAGE extends an MViT backbone with interval-specific latent variables. It adds a learned estimator that predicts interval-wise trajectory-level and heatmap-level uncertainty before decoding; the estimator is supervised by Monte Carlo references, and the two uncertainty forms are linked by a law-of-total-variance decomposition. It also adds an uncertainty-aware cross-time attention module (UCTA) that uses the predicted trajectory uncertainty as a key-side attention bias. The evaluation covers three datasets with sample-based metrics, single-prediction F1, MC-reference fidelity, selective prediction, and a spatial-softmax temperature sweep.

Critical strengths

The probabilistic formulation is well motivated, and moving gaze anticipation beyond single-future evaluation is useful for the community. The decomposition of output-space uncertainty into between-trajectory and within-heatmap terms is clean and interpretable. The reliability analysis is more careful than is typical. It uses interval-stratified quantiles, paired take-level bootstrap, and a temperature sweep on fixed decoder outputs, and it explicitly separates fidelity to MC references from association with realized error. The capacity-matched R=1 control supports the claim for temporally structured latents. The paper also openly reports that TS-CVAE attains the best NLL.

Critical weaknesses

1.Diversity loss vs. calibration. The diversity loss pushes sampled trajectories apart, which directly helps minADE@20 when there is only one ground-truth future. The margin and weight of this loss are not reported, and there is no ablation without it. TS-CVAE’s meanADE is about twice that of Global CVAE (43.97 vs 20.74 on Ego-Exo4D), which suggests the samples are very spread out. Spread-out samples can also give a lower NLL under the diagonal-Gaussian fit. Without calibration checks (e.g., empirical coverage, PIT histograms, spread–skill), I cannot tell whether the distributions are actually better or just wider.

2.Drop in recall. In the same-input comparison, F1 is lower than MViT on Ego4D (0.518 vs 0.532) and EGTEA (0.444 vs 0.474). Recall drops a lot (0.459 vs 0.629 on Ego4D, 0.379 vs 0.608 on EGTEA) while precision goes up. This looks like the predicted heatmaps are more concentrated or shifted, not just sampling noise. How is the single prediction for F1 obtained? F1 of the mean heatmap over multiple samples should also be reported.

  1. UCTA. It is not clear why future positions with higher predicted spread should get lower attention weight when decoding other positions. With 8 frames and MViT’s temporal downsampling, each decoder stage may have only a few temporal tokens, maybe one per interval when R=4, so the bias may act as a fixed reweighting. The gains in Table 4 are small (minADE 17.06→16.58, CRPS 20.29→19.91), NLL gets worse (10.10→10.74), and there is only one run. Please report the number of temporal tokens per stage and results over multiple seeds.

  2. Uncertainty estimator baselines. Selective prediction is only compared with Raw-σ. Two baselines are missing: MC trajectory dispersion computed directly, which is what the estimator tries to approximate, and a simple confidence from a deterministic heatmap (peak value or entropy). The AURC gain over Raw-σ (16.18 vs 16.63) is small compared with the gap to the oracle (10.04). The paper also gives no runtime comparison with K-sample MC, so the efficiency argument is not supported. Table 1 should also include the same-input MViT with its heatmaps sampled under the same budget.

Questions for authors

1.What are the diversity-loss margin and weight, and how do results change without this loss? 2.How is the single F1 prediction obtained, and why does recall fall so sharply? 3.How many temporal tokens are present at each decoder stage, and how are intervals aligned to them? 4.What AURC do direct MC dispersion and a heatmap-confidence baseline achieve, and what is the estimator’s actual inference saving? 5.Can multi-seed results be provided for Tables 3 and 4?

1 comment

We thank the reviewer for the constructive feedback. We address CW1–4 and Q1–Q5 with additional experiments and analysis.

1. Diversity Loss and Predictive Calibration (CW1, Q1).

We provide additional evaluations to examine whether the improvements arise merely from increased dispersion. Our diversity regularizer uses weight λdiv=0.5\lambda_{\mathrm{div}}=0.5 and margin mdiv=0.04m_{\mathrm{div}}=0.04. During training, we observed that removing it led to stochastic-sample collapse, with different latent samples producing nearly identical trajectories and minADE approaching meanADE. The regularizer therefore helps preserve distinct plausible futures rather than encouraging arbitrarily wide predictions. Notably, all variants including TS-CVAE and VANTAGE employ this regularizer, yet VANTAGE produces less dispersed predictions and achieves a substantially lower joint-trajectory Energy Score (18.11 vs. 27.82; Table 12) and better CRPS.

We additionally examine empirical coverage (Fig.6). Although all three models exhibit undercoverage, with TS-CVAE closer to nominal levels, VANTAGE achieves better distributional scores with less dispersed predictions. These findings suggest that VANTAGE's improvements cannot be explained simply by increased predictive dispersion, supporting the effectiveness of temporally structured modeling and UCTA.

2. Single-Sample F1 and Mean-Heatmap Evaluation (CW2, Q2).

Table 2 evaluates 1 stochastic prediction using a fixed eval seed. In a multimodal model, the latent mean can lie between plausible modes; an individual sample instead commits to one future. The observed higher precision and lower recall are consistent with a sharper single-sample heatmap covering less of the plausible gaze support. Importantly, VANTAGE targets probabilistic forecasting rather than deterministic point prediction, for which single-sample F1 serves as a complementary evaluation metric to compare with the existing methods. Following your suggestion, we additionally evaluate the mean heatmap over 20 stochastic samples using the same protocol (Table 8). Mean@20 improves F1 and recall on Ego-Exo4D, with recall also increasing on Ego4D and EGTEA Gaze+. This demonstrates that aggregating multiple plausible futures can recover broader gaze coverage compared with a single stochastic realization, although F1 does not improve uniformly across datasets.

3. UCTA Mechanism, Temporal Alignment, and Robustness (CW3, Q3, Q5).

All decoder stages have Ts=4T_s=4 temporal positions, corresponding to 8 future frames. With R=4R=4, each interval spans two future frames and maps to one temporal position at each stage, following the same partition used for latent injection.

The uncertainty bias encodes interval-wise reliability, downweighting more uncertain keys while preserving attention. Although each interval corresponds to one temporal token and the uncertainty bias is shared across queries, the bias is input-dependent, and the final attention weights remain query-dependent through the content logits. Thus, UCTA incorporates predictive reliability alongside content relevance: a temporally relevant feature may still be uncertain, and the uncertainty bias modulates its contribution without replacing query-dependent attention.

We further conducted multi-seed ablations (Tables 9–10). Pred-Traj UCTA consistently improves minADE, meanADE, and CRPS over CTA, while CTA retains lower NLL. R=4 also outperforms the capacity-matched R=1 in minADE, NLL, and CRPS, although R=1 achieves lower meanADE. Pred-Traj provides the best overall forecasting performance among the uncertainty signals (Table 4). These Results demonstrate the consistent benefits of uncertainty guidance and temporally structured modeling across runs, complementing the take-level bootstrap analysis.

4. Estimator Baselines, Efficiency, and MViT Comparison (CW4, Q4).

We further evaluate direct MC-Traj dispersion and single-forward heatmap entropy/peak baselines under the same risk–coverage protocol (Table 11). Pred-Traj achieves the lowest AURC (16.18), outperforming MC-Traj (16.75) and both heatmap-confidence baselines. Together with its improved temporal fidelity to MC targets (Table 5), they demonstrate that the uncertainty estimator provides a more effective signal for ranking forecasting reliability than direct MC dispersion or simple heatmap confidence.

For efficiency (Table 13), learned uncertainty estimation is approximately 64× faster than K=20 MC-Traj estimation, while VANTAGE introduces only 0.7% measured single-prediction overhead over CTA. This comparison concerns uncertainty-estimation cost, not total forecasting speed.

Regarding MViT, Table 2 provides same-input deterministic comparisons, while our CVAE baselines enable controlled evaluation of joint probabilistic forecasting, which independent per-frame heatmap sampling does not explicitly model.

Reviewer_3MrrOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes VANTAGE, a probabilistic framework for egocentric gaze anticipation. Given historical video and, optionally, historical gaze, the model predicts a distribution over future gaze heatmap sequences. Its central idea is to partition the forecast horizon into intervals and use the same partition for three components: interval-specific latent variables, interval-wise uncertainty estimation, and uncertainty-guided temporal attention.

The framework distinguishes two output-space dispersion measures: trajectory-level uncertainty, describing variation among sampled gaze centers, and heatmap-level uncertainty, describing spatial spread within individual predicted heatmaps. A learned estimator predicts these quantities before decoding, using Monte Carlo predictions as training targets. The estimated trajectory uncertainty then biases cross-time attention toward relatively more reliable intervals.

Experiments on Ego-Exo4D, Ego4D, and EGTEA Gaze+ show improvements in best-of-20 displacement error and coordinate-wise CRPS over the evaluated probabilistic baselines in the video-plus-gaze setting. Additional analyzes suggest that trajectory uncertainty is more useful for ranking forecast reliability than heatmap dispersion, whose behavior depends strongly on spatial-softmax temperature.

The paper offers a coherent modeling approach and useful empirical observations. However, its current evidence more clearly establishes improved sampled accuracy and marginal distribution quality than the quality of the full distribution over future trajectories. The specific contribution of uncertainty guidance also requires stronger isolation from ordinary temporal attention.

Critical strengths

1, The problem is well motivated and relevant. I quite like this idea, which is very timely and have good taste, which can be used in gaze interaction, and Gaze in HRI. i.e. CSUR paper, end to end gaze estimation in handheld mobile devices, there are many gaze interactions of mobile, if the device can sense the potential gaze path, then, the system could be better response to users. And this paper addresses meaningful limitation of deterministic gaze anticipation: a single historical observation can be compatible with several plausible future gaze sequences. Modeling this variability is relevant to applications that must reason about future visual attention. 2, The paper also makes a useful distinction between representing an individual gaze prediction with a spatial heatmap and modeling a distribution over possible future heatmap sequences. Although I think the heatmap is too simple for this, but it is the must direct show the gaze activities. 3, Using one temporal partition for latent injection, uncertainty estimation, and attention modulation creates a clear connection among the proposed components. 4, With matched video-plus-gaze inputs, VANTAGE achieves the best reported minADE@20 and CRPS on all three datasets. But i am quite interesting the performance in some of datasets, i.e. the gaze following datasets.

Critical weaknesses

1, The distributional evaluation measures a narrower object than the claimed predictive distribution. The most important evaluation concern is the definition of NLL. Appendix B.1 computes this metric by fitting a diagonal Gaussian to the 20 sampled gaze centers at each future time point. It therefore evaluates a Gaussian approximation based on sampled means and variances, rather than the likelihood under VANTAGE’s potentially multimodal distribution over heatmap sequences. So, I'd suggest authors to change the releted claims. And, This approximation discards multimodality, dependence between horizontal and vertical coordinates. 2, The coordinate-wise CRPS is a useful marginal distribution score, but it similarly does not assess the joint spatial or temporal structure. 3, The ablations do not fully isolate the central mechanisms. The improvement specifically attributable to uncertainty guidance is smaller than the full comparison against TS-CVAE might suggest. Can you explain? 4, The uncertainty estimator is trained on a different decoding configuration from the final predictor. During Monte Carlo target construction, uncertainty feedback is neutralized, recovering uncertainty-free cross-time attention. At inference, estimated uncertainty changes the attention weights and therefore can change the output distribution. 5, The novelty and performance claims should be framed more precisely. Although i enjoy read this paper, but some of contribution and the novelty claims are too big.

Questions for authors

1, What predictive distribution is the reported NLL intended to evaluate? 2, Does the learned uncertainty describe the final uncertainty-guided predictor? 3, How robust is the incremental benefit of uncertainty guidance over ordinary attention? 4, How exactly is the capacity-matched global-latent control implemented?

1 comment

We thank the reviewer for the thoughtful and constructive feedback. We clarify the evaluation scope, separate the contributions of attention and uncertainty guidance, and provide additional evidence on robustness and joint trajectory quality.

1. Scope of NLL/CRPS and Joint Distribution Evaluation (CW1-2, Q1).

We agree that the reported NLL should not be interpreted as the exact likelihood of VANTAGE's multimodal distribution over future gaze sequences. It evaluates a moment-matched diagonal-Gaussian approximation computed from sampled gaze centers at each future step. Likewise, coordinate-wise CRPS evaluates marginal predictive quality rather than joint spatial and temporal dependencies. We will clarify these scopes in Sec. 4.2 and Appendix B.1 and refine the corresponding claims.

To complement these marginal metrics, we additionally evaluate joint-trajectory Energy Score using complete sampled future trajectories (Table 12), thereby assessing multivariate forecasting quality without fitting a diagonal-Gaussian approximation. VANTAGE achieves a substantially lower ES than TS-CVAE (18.11 vs. 27.82; 34.9% reduction), despite its higher NLL. Moreover, NLL rankings remain stable across sampling budgets of K=10–50 (Table 6). Together, these results highlight the importance of evaluating probabilistic forecasts through complementary metrics, with ES assessing multivariate trajectory quality beyond the marginal evaluations provided by NLL and CRPS.

2. Attribution and Robustness of UCTA (CW3, Q3).

We clarify that the improvement from TS-CVAE to VANTAGE reflects the combined contributions of cross-time attention and uncertainty guidance, rather than uncertainty guidance alone. So in Table 4, we isolate the latter by comparing CTA with Pred-Traj UCTA under the same architecture. UCTA improves minADE, meanADE, and CRPS, while CTA retains lower NLL.

To assess robustness, we additionally conduct multi-seed experiments (Table 10). The gains in minADE, meanADE, and CRPS remain consistent across seeds, supporting the effectiveness of uncertainty guidance beyond plain content-based attention. These results quantify training-run variability, whereas take-level bootstrap intervals quantify test-sample variability.

3. MC Targets and the Final Uncertainty-Guided Predictor (CW4, Q2).

The uncertainty estimator and UCTA-equipped decoder are jointly trained, with uncertainty guidance active during regular training and inference. Feedback is neutralized only during MC target construction to avoid self-referential supervision, while joint training allows the estimator and guided decoder to adapt together.

Importantly, we validate the learned uncertainty on the final UCTA predictions rather than relying solely on agreement with MC targets. Pred-Traj achieves the lowest AURC (16.18), outperforming direct MC-Traj dispersion (16.75) and heatmap-confidence baselines (Table 11). Moreover, Pred-Traj UCTA consistently improves forecasting performance over CTA across three seeds (Table 10). These results support the practical usefulness of the estimator for the final uncertainty-guided predictor despite the decoupled target construction.

4. Capacity-Matched Global-Latent Control (Q4).

The capacity-matched R=1 (4-MLP) control uses four distinct global latents, each processed by an independently parameterized MLP and conditioned on the entire forecasting horizon. In contrast, R=4 assigns four distinct latents to separate temporal intervals. All other architectural components are identical, isolating global versus interval-specific latent modeling. Across three seeds (Table 9), R=4 achieves better minADE, NLL, and CRPS, while R=1 retains lower meanADE. These results support the benefit of temporal structuring rather than increased MLP capacity.

5. Scope of Novelty and Performance Claims (CW5).

Our central novelty lies in incorporating predicted interval-wise uncertainty directly into the generative process, instead of treating uncertainty solely as a post-hoc output. Specifically, VANTAGE couples temporally structured latent modeling with pre-decoding uncertainty estimation, which guides cross-time attention during forecasting. Unlike MC-based uncertainty estimation from completed stochastic predictions, this mechanism allows uncertainty to influence the prediction process itself. We will refine our performance claims to emphasize the experimentally demonstrated improvements in forecasting accuracy and distributional quality.

Reviewer_pKkWOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a framework that can predict not only future egocentric gaze but also the uncertainty of gaze trajectories. Gaze prediction uncertainty is indeed crucial in the eye gaze community, as uncertainty can be used in downstream human intent inference and decision-making.

Critical strengths

Uncertainty is a crucial topic in the eye gaze community. The design of UCTA is reasonable and easy to understand. The idea of the uncertainty estimator is interesting and necessary to make the framework run in real time.

Critical weaknesses
  1. Modeling the uncertainty of eye gaze and gaze prediction is not new. There are many papers that have modeled uncertainty in eye gaze, but they are not referenced in this paper. Therefore, the contribution is not well differentiated from previous work. Especially in the related work section, more uncertainty-aware eye gaze modeling or anticipation studies should be discussed.

  2. More analysis of the results is needed. Rather than just showing numbers, more heatmap figures showing how the method improved performance would be helpful.

Questions for authors
  1. Figure 1 doesn't show too much information. What is the uncertainty level? What is the trajectory? A figure showing how 20 trajectories differ from each other and influence the uncertainty would be helpful. Additional visualizations showing how UCTA influences the heatmaps and trajectories would also be useful. The heatmap ground truth in Figure 1 seems not to consider temporal decay, but temporal decay can be seen in the predicted results. Why is that? Please explain why gaze fixation locations are not predicted. Do the gaze fixation locations in GLC and VANTAGE appear exactly the same as the GT?

  2. What are the references for Equations 5 and 6? Are these uncertainty measures reliable? Eye gaze trajectory uncertainty is different from other trajectory uncertainty, as human intent is ambiguous. These two uncertainty measures need to be evaluated if there are no references to support them. In the related work section, many papers about trajectory uncertainty modeling are referenced, but they are not gaze-related.

  3. Are the performance improvements reported in Table 4 statistically significant? The results seem to show that UCTA didn't improve the performance of the framework. However, the idea of UCTA is reasonable.

  4. A video could help readers better understand the differences between the proposed method and other methods. As the eye gaze is dynamic and in Ego 4D and EGTEA Gaze+ the human move quite fast.

3 comments

We thank the reviewer for the constructive feedback. We clarify our distinction from prior gaze uncertainty studies, provide additional evidence for uncertainty modeling and UCTA, clarify the interpretation of Fig1, and expand the qualitative analysis with representative prediction cases.

1. Relation to Prior Gaze Uncertainty Modeling (CW1).

We agree that gaze-specific uncertainty modeling deserves broader discussion. We will incorporate Gaze360 (ICCV 2019), UnReGA (CVPR 2023), and Uncertainty Modeling for Gaze Estimation (TIP 2024). These works respectively study uncertainty in gaze estimation, domain adaptation, and input/annotation modeling, rather than probabilistic anticipation of gaze trajectories over unobserved future frames.

We also draw methodological inspiration from established probabilistic trajectory forecasting, as both settings involve multiple plausible futures arising from partially observed human intent.

Our contribution is therefore not gaze uncertainty estimation itself, but temporally structured probabilistic gaze anticipation, where interval-specific predictive uncertainty guides cross-time attention during generation. We further distinguish trajectory-level and heatmap-level uncertainty to characterize complementary aspects of the predictive distribution. We will clarify these distinctions and expand the related work accordingly.

2. Interpretation and Validation of the Uncertainty Measures (Q2).

Eqs. 5–6 are standard second-moment statistics of the model's output-space predictive distribution, rather than new definitions of human-intent uncertainty. Specifically, Eq. 5 is the square root of the between-trajectory variance of sampled 2D gaze centers, while Eq. 6 is the square root of the expected within-heatmap spatial variance. Their squared forms correspond to the two components of the law of total variance. This distinction is important because multiple plausible future fixation targets can produce high between-trajectory uncertainty even when each predicted heatmap is spatially concentrated.

We further validate their usefulness for egocentric gaze anticipation. Table 5 shows that the learned estimators correlate with their MC uncertainty targets (ρ=0.632 for trajectory and 0.575 for heatmap uncertainty). Beyond target agreement, Fig. 4 demonstrates that Pred-Traj uncertainty effectively ranks realized forecasting errors, improving AURC over Raw-σ (16.18 vs. 16.63). Fig. 6 further evaluates empirical coverage calibration of sampled gaze trajectories. Together, these results support the mathematical motivation and practical relevance of the proposed uncertainty measures.

3. Effectiveness and Robustness of UCTA (Q3).

Table 4 isolates uncertainty guidance by comparing Pred-Traj UCTA against ordinary CTA under the same architecture. UCTA improves minADE, meanADE, and CRPS, while CTA retains lower Gaussian-approximated NLL.

We further conduct three-seed experiments (Table 10). UCTA achieves lower minADE (16.59±0.07 vs. 17.18±0.13) and CRPS (19.79±0.23 vs. 20.39±0.15), alongside improved meanADE. These results provide evidence of consistent improvements with low seed-to-seed variability, complemented by take-level bootstrap confidence intervals over test samples.

4. Qualitative Interpretation and Visualization (CW2, Q1, Q4).

In Fig. 1, the green gaze markers denote ground-truth gaze locations overlaid on all methods as a common reference, rather than predicted fixations from GLC or VANTAGE. VANTAGE predicts probabilistic gaze heatmaps, from which gaze centers and sampled trajectories can be extracted. No explicit temporal-decay operation is applied to either ground-truth or predicted heatmaps; the apparent differences reflect changes in predicted spatial concentration across future horizons.

We additionally provide qualitative results (Fig. 2) illustrating low-uncertainty, high-uncertainty, and overconfident failure cases. Each example visualizes sampled gaze predictions across four future frames and five trajectories selected from 20 stochastic samples, alongside ground-truth fixations. These examples illustrate how predicted gaze trajectories vary under different uncertainty levels.

I thank the authors for the reply, but I don't think my main concerns are fully addressed. A demo video would be needed to illustrate the difference of the proposed method and others.

Thank you for your follow-up. As requested, we have prepared an anonymous demo video featuring four examples with synchronized comparisons of GT gaze, GLC, and VANTAGE predictions, complementing Fig. 1.

Demo video: https://anonymous.4open.science/r/anonymous-demo-vantage-6E1F/

Together with the additional experiments and analyses provided during the rebuttal, we hope this addresses your remaining concerns. We would greatly appreciate your reconsideration.

Comments on the paper

1 comment

Summary of Revisions

We thank all reviewers for their constructive feedback. We have expanded the experiments, manuscript, and Appendix with the following additional analysis:

  • Robustness: Multi-seed ablations of temporal structuring and CTA/UCTA (Tables 9–10).
  • Distributional evaluation: NLL sample-budget sensitivity, joint-trajectory Energy Score, and empirical coverage calibration (Tables 6, 12; Fig. 6).
  • Uncertainty reliability: Direct MC-Traj and heatmap-confidence baselines (Table 11).
  • Deterministic evaluation: Mean predictive heatmap aggregation across three datasets (Table 8).
  • Efficiency: Inference-time comparison between uncertainty estimator and MC-based uncertainty estimation (Table 13).
  • Qualitative visualization: Sampled gaze trajectories under low/high predictive uncertainty and overconfident failure cases (Fig. 2).

We have also clarified the scope of NLL and CRPS, refined the discussion of deterministic accuracy trade-offs, and expanded the related work on gaze uncertainty. Detailed responses to individual concerns are provided under the corresponding reviews.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.