- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes Text-Conditioned Visual Steering (TVS), a training-free method for mitigating hallucinations in vision-language models. Building on the visual steering vector from VISTA, TVS augments the global image-versus-text-only direction with a prompt-conditioned correction derived from attention over visual tokens. The proposed correction uses zero-sum coefficients to emphasize relevant visual content. The resulting direction is cached after two prefills and applied during generation with one model forward per generated token. The paper evaluates LLaVA-1.5-7B and Qwen2.5-VL-7B on POPE, alongside caption hallucination and general capability evaluations. It also presents direction-component ablations and an efficiency analysis. However, discrepancies between the stated method, released implementation, and reported experimental results currently limit my confidence in the conclusions.
Critical strengths
A simple and potentially useful extension of visual steering. TVS introduces prompt-conditioned visual-token weighting into the existing VSV framework while retaining a cached steering direction. This is a plausible way to improve the specificity of visual guidance without introducing an additional decoding branch. The relationship to the steering component of VISTA is explicitly acknowledged.
Useful controls for interpreting hallucination reductions. The direction-component experiments include spatial-only, shuffled, and norm-matched uncentered controls. The captioning experiments also report length and object recall, helping assess whether lower hallucination rates arise from reduced output coverage. These controls address meaningful alternative explanations for the observed improvements.
Critical weaknesses
The released implementation does not match the central zero-sum construction. Equation (5) defines , ensuring and consequently . However, qwen_compute_vsv_att in the released Qwen evaluation code implements alpha * saliency + max(0.0, 1.0 - alpha) * uniform. At the reported default , this gives , so the correction coefficients sum to , rather than . This changes the method and removes the stated shared-offset cancellation property. The batched evaluation script calls this function, making the discrepancy directly relevant to the experimental pipeline. Additionally, essential steering routines are imported from a local pade package that is absent from the released repository. It is therefore unclear whether the reported experiments validate the method described in the paper.
The released experimental records do not support several reported LLaVA results. Table 7 and the repository’s three-dataset results summary give substantially different mean accuracies:
| Dataset | Method | Table 7 | Released records |
|---|---|---|---|
| A-OKVQA | VSV | 82.21 | 82.54 |
| A-OKVQA | TVS | 87.91 | 82.24 |
| GQA | VSV | 83.33 | 82.67 |
| GQA | TVS | 83.78 | 81.34 |
The launcher identifies vista as the global-direction baseline and psv as the attention-conditioned variant, with the latter using the stated , , final-layer attention, and object-token query. Thus, the released records indicate decreases on both datasets, whereas Table 7 reports improvements. Different experiment versions could explain these discrepancies, but no sufficient mapping is provided. This materially affects the claimed generality of the method’s benefits.
Limited methodological novelty beyond VISTA’s visual steering component. TVS inherits the image-versus-text-only direction extraction, cached steering framework, and decoding-time intervention from VSV. Its main addition is an attention-weighted correction to the existing direction. This is a reasonable extension, but the current evidence does not establish a sufficiently substantial advance: in the matched component study, TVS improves mean accuracy over VSV by only percentage points, while centering contributes just points over the norm-matched uncentered variant. Although the paper appropriately acknowledges the limited evidence for centering, the incremental construction and modest gains in this controlled comparison weaken the case for a significant methodological contribution.
Questions for authors
Clarification of the evaluated algorithm. Which implementation generated the reported Qwen results: the signed weighting in Equation (5), or the clipped expression in the released code? Please identify the corresponding code version and provide the missing steering dependency. If the clipped expression was used, does the improvement persist with the stated zero-sum formulation? A representative matched comparison would be sufficient.
Clarification of experimental provenance. Please reconcile the Table 7 results with the released LLaVA three-dataset records, particularly the reversed TVS–VSV comparisons on A-OKVQA and GQA. Which configurations and artifacts reproduce the table, and do any aggregate results or conclusions require correction?
Limited additional analysis. In the matched component study, TVS improves mean accuracy over VSV by approximately percentage points and over the norm-matched uncentered variant by only points. Could you provide a paired uncertainty estimate using the existing predictions, preferably accounting for questions sharing an image? This would help determine how much evidence supports the practical benefit of the added correction.