- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes VANTAGE for probabilistic egocentric gaze anticipation from past video, with optional gaze history. It uses interval-specific latent variables to generate future gaze heatmaps, estimates trajectory- and heatmap-level uncertainty, and uses the trajectory-level estimate to guide temporal attention. Across three datasets, the paper reports sampled-prediction, distributional, and single-prediction results; it analyzes uncertainty reliability on Ego-Exo4D.
Critical strengths
Forecasting future gaze as a distribution is well motivated: the same observed scene can lead to more than one plausible gaze path. The paper evaluates both the quality of sampled futures and whether estimated uncertainty tracks prediction error, rather than relying only on point-prediction accuracy. The error-association and selective-prediction analyses on Ego-Exo4D add evidence about whether the estimated uncertainty identifies unreliable forecasts, beyond what point-prediction accuracy can show.
Critical weaknesses
1. Uncertainty-guided attention adds little beyond plain attention. In Table 4, plain cross-time attention already reaches 17.06 minADE and 20.29 CRPS. Adding trajectory-level uncertainty brings these to 16.58 and 19.91, but NLL rises from 10.10 to 10.74. Heatmap-level uncertainty, alone or combined, does not improve minADE or CRPS over trajectory-level uncertainty alone. Table 3 is also uneven across the number of intervals: R=3 has the worst NLL (27.03), and the capacity-matched R=1 variant has better meanADE than R=4. These ablations cover only Ego-Exo4D, and no variation across training seeds is reported. The gain from uncertainty guidance is small, while the interval-latent results are mixed; neither ablation reports whether these differences persist across datasets or training runs.
2. Probabilistic comparisons are narrow, and the accuracy cost is understated. The probabilistic baselines in Table 1 are MC DropPath and two CVAE variants on the same backbone; the paper does not adapt an existing gaze or trajectory generator, or explain why such a comparison is infeasible. In Table 5, Raw-σ is the only alternative uncertainty estimator used for the fidelity comparison; ensembles and MC dropout are absent. Even within Table 1, TS-CVAE has better NLL than VANTAGE on all three datasets with gaze history. In Table 2, VANTAGE with gaze history beats the same-input MViT in F1 only on Ego-Exo4D (0.614 vs. 0.607). With video alone, its F1 is below MViT and GLC on all three datasets (e.g., 0.231 vs. 0.278 for MViT on EGTEA Gaze+). Calling this a "modest trade-off" understates the cost.
Questions for authors
Needs further clarification
- How sensitive are the NLL rankings in Tables 1 and 3 to fitting a diagonal Gaussian to only 20 samples (Appendix B.1)? How was the spatial-softmax temperature of 10 chosen, given that the error correlation of MC heatmap uncertainty changes sign across temperatures in Table 6?
- Which Ego-Exo4D subset was used? The supplied split files contain 862 training and 218 test takes. How were these takes selected, and is this the same split used for the GLC and CSTS comparisons?
- Were all Table 2 baselines retrained and evaluated on these splits, or were any results taken from prior work?
- How much do the differences in Tables 3 and 4 vary across training seeds? The take-level bootstrap intervals reported for the uncertainty analysis capture variation over test samples, not over training runs.
- What are the inference-time and computational overheads of the uncertainty estimator and UCTA relative to plain cross-time attention? Please report separately the time for one uncertainty estimate, one prediction, and 20 sampled predictions, and compare uncertainty estimation without stochastic decoding with estimation from Monte Carlo samples.
Suggestions
- A task-adapted external generative baseline would help establish the probabilistic contribution. Comparing uncertainty estimates with alternatives such as ensembles or MC dropout would also help.
- Figure 1 is the only qualitative forecasting example and shows one sequence per method. Its caption should explain what the colored heatmaps and green dots represent, and whether VANTAGE shows a sampled or representative prediction. Showing multiple sampled futures, interval-wise uncertainty, and failure cases across datasets would make its probabilistic behavior easier to assess.
Minor: "Latnt" in Fig. 2(b), "We uses" in Appendix A, inconsistent spelling of Ego-Exo4D, and a repeated sentence in Appendix C.2.
3 comments
Thanks to the authors for the additional experiments and clarifications. The rebuttal addresses some of my concerns, especially regarding the robustness of the results. However, the improvements from uncertainty-guided attention remain relatively small, and the comparisons with existing probabilistic methods are still limited. I also remain concerned about the trade-off in deterministic prediction accuracy. Overall, I appreciate the authors' efforts, but I will keep my original score.
Thank you for your follow-up and for recognizing the improved robustness of our results. We would like to further clarify three points.
-
Contribution of uncertainty-guided attention: Importantly, CTA is itself part of our proposed temporal reasoning mechanism and technical contribution, which yields substantial improvements, rather than an off-the-shelf baseline. The improvements from TS-CVAE to VANTAGE reflect the combined benefits of CTA and uncertainty guidance, whereas the CTA-to-UCTA comparison isolates only the incremental contribution of uncertainty guidance. Although this incremental improvement is moderate, it is supported by our multi-seed experiments. More importantly, our contribution is not limited to accuracy improvements, but introduces predictive uncertainty as an explicit signal for temporal decoding. The learned estimator also achieves better uncertainty-based error ranking than direct MC trajectory dispersion, with approximately 64× faster uncertainty estimation.
-
Probabilistic comparisons. The limited availability of directly comparable probabilistic baselines reflects the current state of egocentric gaze anticipation, where existing methods predominantly produce deterministic predictions. Extending probabilistic methods from related domains would require substantial adaptations due to both task mismatch and domain shift. We recognize the value of broader comparisons, but emphasize that the lack of established probabilistic baselines is also part of the research gap that motivates our work.
-
Deterministic accuracy. Deterministic metrics assess individual prediction accuracy rather than the quality of the predictive distribution, which is the primary objective of our work. Although a trade-off remains on some datasets, VANTAGE achieves substantial improvements in distributional quality, including a 34.9% reduction in joint-trajectory Energy Score. We believe probabilistic forecasting models should be assessed primarily by their distributional performance, rather than solely by deterministic accuracy.
We respectfully believe that focusing primarily on the incremental gains of uncertainty guidance and deterministic prediction accuracy does not fully reflect the methodological contributions of VANTAGE. Our central contributions lie in probabilistic forecasting, temporally structured uncertainty modeling, and uncertainty-aware temporal reasoning, which are supported by extensive additional experimental evidence. We hope these contributions and findings will receive appropriate consideration in the final assessment.
We thank the reviewer for the feedback. We address CW1–CW2 and Q1–Q5 below.
1. UCTA and Temporal Structuring (CW1, Q4)
To assess robustness, we conducted additional experiments across three seeds; details are in Tables 9–10. Pred-Traj UCTA consistently improves minADE, meanADE, and CRPS over CTA, while CTA retains a lower NLL. These results demonstrate the consistent benefits of uncertainty guidance across training seeds. Regarding temporal structuring, R=4 consistently outperforms the capacity-matched R=1 variant in minADE, NLL, and CRPS, though R=1 has lower meanADE. We acknowledge R=3's higher NLL. However, increasing R changes the temporal granularity and structure of the latent space, so improvements are not necessarily monotonic.
Furthermore, Pred-Traj achieves the best overall performance among the uncertainty signals (Table 4), supporting trajectory-level uncertainty as the most effective guidance signal. Importantly, the improvement from TS-CVAE to VANTAGE reflects the combined effects of cross-time attention and uncertainty guidance, whereas Table 4 isolates the incremental benefit of uncertainty guidance over ordinary CTA. This distinction clarifies the respective contributions of the two components.
2. Probabilistic Baselines and Uncertainty Reliability (CW2)
Our baselines prioritize controlled comparisons with identical backbones, inputs, and protocols. ScanGAN360, ScanTD, and ScanDiff primarily target image-conditioned scanpath generation, not ego gaze anticipation without future observations. Deep Future Gaze generates video but doesn't model gaze-traj distributions; physical trajectory methods use different modalities and outputs. Adapting them requires substantial changes to their conditioning and prediction formulations. In contrast, Global CVAE and TS-CVAE isolate our proposed mechanisms under controlled settings.
Regarding uncertainty estimation, Table 5 shows improved temporal MC-target fidelity over Raw-σ. MC DropPath, a structured variant of MC Dropout, is already included in Table 1. We also compare direct MC dispersion and single-forward heatmap confidence in Table 11: Pred-Traj yields better AURC vs. MC-Traj, entropy, and peak uncertainty. These results support the estimate as a more effective signal for ranking forecasting reliability.
3. NLL and Deterministic Accuracy Trade-offs (CW2)
We appreciate your concern regarding NLL and deterministic accuracy. As reported in Tables 1–2, TS-CVAE has lower NLL, whereas VANTAGE improves minADE and CRPS. So we further evaluate nonparametric joint-trajectory Energy Score in Table 12: VANTAGE achieves 18.11 vs. 27.82 for TS-CVAE, indicating that its higher NLL does not necessarily indicate poorer joint predictive distribution quality.
We further clarify the distinction between single-sample deterministic accuracy and probabilistic forecasting quality, as VANTAGE primarily targets probabilistic forecasting, while F1 is included for comparability with prior methods. We additionally evaluate Mean@20 predictive heatmaps in Table 8. On Ego-Exo4D, Mean@20 improves F1 and recall. Recall also improves on Ego4D and EGTEA, although F1 does not consistently improve.
4. NLL and Temperature Sensitivity (Q1)
We evaluated K={10,20,30,50} stochastic samples (Table 6). The model rankings remain stable, including the relatively higher R=3 NLL. Nevertheless, NLL relies on a moment-matched diagonal-Gaussian approximation; complementary CRPS and joint-trajectory Energy Score provide a broader assessment of predictive distribution quality.
We use spatial-softmax temperature T=10 following CSTS, without tuning it for error correlation. Temperature directly affects heatmap concentration and thus within-heatmap spatial variance, explaining its sensitivity. Trajectory uncertainty remains positively correlated with error across temperatures in Table 7, supporting our distinction between uncertainty forms.
5. Dataset Construction and Baseline Provenance (Q2, Q3).
We use Ego-Exo4D's official EgoPose subset and take-level split of 862 train/218 test takes, without random reselection. Clip construction follows CSTS. All Table 2 baselines MViT, GLC, CSTS were retrained on the same constructed datasets and splits using released implementations and original method-specific settings; no results were copied from prior papers. Only dataset interfaces were adapted.
6. Inference-Time Analysis (Q5)
As shown in Table 13, VANTAGE introduces only 0.7% (41.54 vs. 41.25 ms/clip) inference overhead over CTA, while its uncertainty estimator achieves a 64× speedup over MC-Traj (K=20, 639.18). The table also reports separate timings for cached-feature estimation, end-to-end uncertainty estimation, single prediction, and K=20 sampling. CTA is evaluated using an independent checkpoint with uncertainty estimation and bias disabled. We also clarify the GT points and predicted heatmaps in Fig1, and provide low/high-uncertainty visualizations (Fig2).