- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
Paper proposes a novel method to incorporate effecient gaze information in various ego-centric VQA task. The central claim is that effecient modelling of gaze information can reduce both compute cost (tokens) and increase task accuracy.
The multi-stage pipeline as follows:
- selects useful frames dynamically with HOG transition cues and Laplacian sharpness scores to identify frames changes.
- generate a anisotropic Gaussian map central at the gaze point to incorporate gaze information (proxy for visual attention)
- condition a semantic gate on the question to control the balance between source 1 and 2
LoRA was trained on a frozen light-weight VLM (SmolVLM2-500M) with the selected benchmark: StreamGaze.
The proposed method was benchmarked against two groups of baseline: (i) small VLMs without token reduction or gazetuning; for full token grids and standard inference pipelines comparison (ii) token-reduction methods re-implemented on the SmolVLM2-500M framework on identical hardware; for vis-a-vis comparison
Proposed method achieves the lowest TFLOPS while outperforming SOTA accuracy. A CROSS-DATASET GENERALIZATION test was also done to evaluate the domain-transferability of the approach. Ablation study was also done to assess the contributions of the stages in the pipeline.
Critical strengths
- Paper proposes a approach to incorporate effecient gaze information for ego-centric VQA
- Method is somewhat novel.
- Experimental results reasonably support the claim.
Critical weaknesses
- Method is incrementally novel. The use of gaussian mask to represent visual attention has been long used in many prior works. The use of a MLP to weight the 2 different sources conditioned on the question text embedding is also not novel.
- Experiment on a single dataset is weak evidence for generalization of the performance. This is especially the case as there is a training phase involved.
The claim for evaluation against the small VLMs without token reduction or gazetuning is so that the full set of visual information is available to the VLM. But the experiment result show that the proposed method outperforms these full information design. This is contrary to the experiment design. Please explain.
Questions for authors
Refer to Weaknesses comment and explain and/or elaborate if the weaknesses identify are correct or reasonable.
Follow up on Response
Dear Reviewer Phv8,
Thank you again for your review. We have addressed your concerns. We would love to hear your feedback, especially if there are any remaining questions. As your evaluation weighs significantly in the final decision, we would greatly appreciate your thoughts before the discussion period closes on November 18.
Best regards,
Authors