- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes GazeGGT, which solves a meaningful 3D gaze estimation problem: uncalibrated multi-view gaze estimation from arbitrary number of synchronized face crops. GazeGGT takes multiple face images as input and learns view-specific feature for each image, which are fused through alternating multi-view attention. The author further utilize a Geometry-Grounded Supervision Paradigm that jointly supervises gaze and camera pose predictions by constraining the relative rotations among different views. Then separate decoders estimate 3D gaze directions and camera poses for each view. Two multi-view datasets ETH-XGaze and GazeGene are evaluated through in-domain and cross-domain experiments.
Critical strengths
- The perspective of "arbitrary uncalibrated views" for 3D gaze estimation is interesting and this paper makes a good foundation and baseline in this domain.
- Comprehensive experimental results are provided across five gaze datasets and achieves improvement in most single view datasets and multi-view datasets. The cross-dataset evaluation validates the effectiveness of multi-view inference.
Critical weaknesses
- Although I like the idea of unconstrained, calibration-free multi-view gaze estimation, the Geometry-Grounded Supervision Paradigm is not novel to me. Similar relative rotation constraints have been proposed in [1], which is not cited as a related work in this paper. The central idea of rotation consistency in [1] is constraining a invariant gaze direction in the same global coordinate system between image pairs under different camera poses. Correspondingly, the ablation study in Table 6 shows that GazeGGT without Geometry-Grounded Supervision term only performs slightly worse than the full model on both two datasets. This raises the question of the contribution of GGSP theoretically and experimentally.
[1] Bao, Y., Liu, Y., Wang, H. and Lu, F., 2022, June. Generalizing gaze estimation with rotation consistency. In 2022 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (pp. 4197-4206). IEEE.
- Although the paper is targeting at unconstrained multi-view gaze estimation from uncalibrated images, but the relative camera angles of ETH-XGaze dataset are fixed (which means ETH-XGaze use pre-defined 18 camera orientations, I'm not sure about if GazeGene randomize camera angle for each subject). Based on this limitation, the model is trained and evaluated within the same camera pose distribution. It is unclear how the model performs in real-world multi-view uncalibrated images which are out of the pose distribution of ETH-XGaze and GazeGene. Since the author also mentioned that at test time, view order is not shuffled. The model might already have the prior knowledge of the pose distribution in the test set.
Questions for authors
In Table 6, the random initialized model perform much worse than Pi3X initialized model. At the same time, the DINOv2 initialization performs slightly worse than the geometry-aware model in both N =1 and N = 9 cases. The results in Table 6 raises three essential questions:
-
The architecture of the GazeGGT model largely builds upon the Visual Geometry Grounded Transformer and the Pi3X model. What is the technical novelty of GazeGGT beyond adapting these two models to 3D gaze estimation task?
-
How much does GazeGGT rely on pret-rained geometry models like Pi3X? Since Pi3X already have abilities for geometry-aware representation learning. What will be the gaze detection performances using frozen Pi3X weights?
-
The improvements after introducing GGSP is subtle, the contribution of GGSP is unclear to me.
In Section 4.3, please fix the text “Appendix Table ??”
2 comments
Follow-up response
Dear Reviewer,
Thank you again for your careful review. As the discussion period ends today, we would like to briefly follow up on our response. Please let us know if you have any further questions; we would be happy to address them before the deadline. If our responses have resolved your concerns, we would be grateful if you would consider reflecting this in your final assessment.
Thank you and best wishes!
Dear Reviewer VY9f,
Thank you for your careful review and constructive feedback. And we address each concern as follows.
Q1. Contribution beyond applying Pi3X to gaze. Thank you for recognising our conceptual contribution: uncalibrated, arbitrary-view 3D gaze estimation, which provides a foundation and a baseline for this setting.
Technically, we contribute three things:
Q2. Reliance on Pi3X; frozen Pi3X. Without Pi3X's pretraining on large-scale view combinations data, cross-domain transfer does not hold (initialisation ablation, Table 5 in the paper). Freezing the backbone and training only the gaze head is not sufficient either. Ref-Avg (°) error on ETH-XGaze:
The frozen model also improves with N, but its error stays far above the full model.
Q3. Contribution of GGSP. GGSP is what makes calibration-free use possible: at inference the model relies on the camera rotations it predicts itself. The relevant metric is Ref-Avg. With GGSP, the model predicts gaze and rotation well together, so even cross-domain, uncalibrated Ref-Avg error falls below single-view at every N. Without GGSP it does not. Ref-Avg (°), trained on ETH-XGaze, tested on GazeGene:
Without GGSP the error is above the single-view value at N=2 and N=9, and reaches 13.44° at N=9. In-domain (ETH-XGaze, 3 training runs each), Ref-Avg at N=9 is 2.47 ± 0.03 (full) vs 4.24 ± 0.14 (w/o GGSP).
W1. GGSP vs. [1] (Bao et al., CVPR 2022). [1] uses known camera rotations during training to make single-view predictions rotation-consistent. In GazeGGT the relative rotation is a prediction target of the camera head, estimated from images alone at test time, which is what removes calibration at inference. We will cite and discuss [1] in Related Work.
W2. Does ETH-XGaze ↔ GazeGene transfer show uncalibrated performance? The two datasets define cameras very differently. ETH-XGaze is a real-world capture with a fixed 18-camera rig. GazeGene is a synthetic dataset rendered from 9 cameras per frame. Measured from the released calibration and labels:
How much do the camera configurations overlap? Take the GazeGene-trained model tested on ETH-XGaze. For each camera combination in ETH-XGaze, we find the closest combination in the GazeGene rig, in terms of the relative rotations between its cameras. Share of ETH-XGaze combinations with a GazeGene match within the given error:
Even for pairs, only 2.0% of ETH-XGaze combinations have a GazeGene counterpart within 3°, and for three views the overlap is essentially zero: ETH-XGaze has wide baselines up to 125° that the 75° GazeGene rig does not cover. So GazeGene → ETH-XGaze is evaluated on camera configurations unseen in training, without any calibration. Ref-Avg (°), random camera subsets:
Error decreases with N in both directions.
Minor errors. Thank you for pointing out the unresolved references. We will fix it and proofread the full text in the revised version.