- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether predicted human visual attention can replace the model-internal signals used by training-free attention steering against VLLM hallucination. Contributions: (i) VLFeedback-ETC, Tobii eye-tracking from 21 participants judging response pairs on 30 VLFeedback VQA triplets (Section 3; Fleiss kappa 0.453); (ii) QaSalFormer, VisSalFormer fine-tuned on VOILA-COCO mouse traces, benchmarked against Gazeformer, VisSalFormer and MD-SEM (Section 4, Table 1); (iii) a comparison of LLaVA-1.5-7B/13B attention (rollout at the first generated token, and per layer) with human saliency, CC about 0.09 (Table 2, Figure 7); (iv) GazeSteer (Eq. 3), adding a scaled saliency map to pre-softmax image-token scores, alone or on top of the consensus term of Jiang et al. (2025), evaluated with CHAIR on 500 COCO images, greedy decoding, layers 5-18 (Section 6, Tables 7-10). The authors conclude that QaSalFormer maps beat a constant map of equal mass, that caption length does not explain the gain, that gaze agreement does not predict usefulness, and that saliency is comparable to, and possibly complementary to, consensus.
Critical strengths
- Controls many steering papers omit, and calibrated claims. The mass-matched constant map (Section 6.1), greedy captions truncated to equal length (Appendix E.3, Figures 8-15) and matching at equal F1/recall rather than equal beta address the obvious confounds. The text states CS has about two points of uncertainty with 500 images and calls the combination "comparable" to consensus, not better (Section 6.2). The constant control shows added image attention alone helps (7B, Table 8, lower CS is better: Constant 34.8 vs greedy 53.2).
- A released gaze corpus plus an informative negative result. Ranking by human-gaze agreement (MD-SEM best, Table 1) differs from ranking as a steering signal (QaSalFormer best, Section 6.2), a useful caution against using gaze alignment as a proxy. Consent is described (Appendix B.3); raw data and code are promised.
Critical weaknesses
- That human-like saliency is the useful signal is not isolated, and margins are small relative to the stated noise. The only structure-free control is a constant map. I found no control that keeps spatial structure but breaks image correspondence (a map from another image, a centre-bias prior); centre bias appears only in the specificity definition (Section 4.1). QaSalFormer covers about 43% of the image versus 21% for humans (Table 1), the better-aligned MD-SEM is less useful, and the authors say the sources "cannot establish their respective contributions" (Section 6.2). On LLaVA-1.5-7B (CS; greedy 53.2), QaSalFormer is 30.0 vs Constant 34.8 at the 1-point F1 margin (Table 8) and 30.0 vs 32.4 at the 3-point margin (Table 10): a 2.4-point gap against the roughly two-point uncertainty. Consensus on the same model is 27.4 (Table 8) and 14.8 (Table 10), so saliency alone is clearly weaker. On 13B it is comparable (30.8 vs 35.4 in Table 8; 30.8 vs 30.0 in Table 10). Each number is one greedy run; I found no confidence intervals or significance tests, and Tables 7-10 report the best configuration per family from sweeps of different sizes. Partly fixable with added controls, bootstrap intervals and a common beta grid.
- Narrow evaluation and a weak link between gaze data and intervention. Only CHAIR and only LLaVA-1.5 (7B, 13B) are used (Section 6.1, Limitations); I found no other benchmark or model family. QaSalFormer's question conditioning is not exercised: every caption uses "What is in the image?", picked from a 10-prompt ablation on 7B (Table 6, CS 30.0 to 41.4) that appears to use the same CHAIR setup. QaSalFormer is trained on mouse traces, and the corpus has 30 stimuli from a preference task, not VQA answering (Appendix B.2). The "grounding gap" (Table 2: CC 0.090 ± 0.274 on 7B) rests on first-token rollout, as Appendix F acknowledges. Fixable only with new experiments.
Questions for authors
- (Clarification / new experiment) At matched F1, what happens with QaSalFormer maps shuffled across images, or replaced by a centre-bias or blurred model-attention map?
- (Clarification) Can you give bootstrap intervals over the 500 images for CS and CI in Tables 7-10, with paired QaSalFormer vs Constant comparisons? How was the "around two points" uncertainty (Section 6.2) computed?
- (Clarification) Was the Table 6 prompt chosen on the same 500 images used for Figures 4 and 8-15? If so, does the QaSalFormer-vs-Constant gap hold on held-out images?
- (Clarification) With std 0.27-0.29 over 30 stimuli (Table 2), is the layer contrast (about 0.3 in layers 20-25 vs near zero in layers 3-6, Section 5) statistically supported?
- (Future work) Does the effect transfer to another VLLM family and to a question-driven benchmark that uses QaSalFormer's conditioning?
- (Clarification) Please name the "ethical board" (Ethical Considerations) when de-anonymised, and confirm the licence covers raw gaze recordings.
Answer to Reviewer 1CZ2
We agree that shuffled maps are an important control, and we have started running them together with RGVq’s suggestion (Q1). We also thank the reviewer for the bootstrap suggestion: we have run a paired bootstrap over the 500 images, comparing methods over full strength sweeps at matched F1 and recall, and most of the intervals include zero. The “around two points” in Section 6.2 was obtained by comparing the values in the current tables; it was not a formal estimate of uncertainty (Q2). Regarding the prompt, it was chosen on the same 500 images (Q3). Adding another model family and a question-driven benchmark is part of our future work (Q5).
We thank the reviewer for the detailed questions. We want to analyse some of these new results in more detail and revise our claims accordingly; for this reason, we will withdraw the paper.