- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper asks whether internal interventions in a VLM can reproduce the change in its answer score when visual evidence is masked. It compares evidence masks with same-area controls, then swaps internal states between clean and masked runs in both directions. On selected OK-VQA examples, the tested sparse reconstructions, selected channels, and PCA subspaces fail their respective criteria across Gemma3-4B, Qwen2.5-VL-7B, and LLaVA-1.5-7B. A full hidden-state intervention at LLaVA layer 30 passes a separate confirmation on 60 reviewed masks, recovering 95.8% and removing 95.0% of the mean clean-to-mask score change.
The evaluation is careful and the effect-matching criterion is useful. I lean toward rejection because the strongest confirmed finding is a full-state match at one late LLaVA layer, while missing reference results limit the broader compact-versus-complete comparison. The confirmation also leaves the broader evidence-specific interpretation unresolved. I could move toward acceptance if the missing results substantiate the compact-versus-complete comparison. A control analysis on the confirmation set would strengthen the broader evidence-specific interpretation.
Critical strengths
-
The evaluation separates selection from confirmation. The LLaVA intervention and decision rule are frozen before testing on 60 masks reviewed without model outputs. All four bootstrap bounds pass (Table 10), and a post-hoc tolerance sweep passes at the tested tolerances from 15% to 30%, but fails at 10% (Appendix D.6).
-
The tests check whether interventions reproduce the target effect rather than simply change the answer score. Gemma's positive-only channel selection substantially overshoots the full-channel reference (Table 8). The reconstruction checks also help explain the sparse tools' failures, and the authors appropriately limit their conclusions to the tools tested.
Critical weaknesses
-
Table 6 omits the complete feed-forward effect at the same intervention sites. All Qwen and LLaVA sparse and residual effects are within ±0.022 nats, but without this reference it is unclear whether they miss a substantial answer effect. In Table 8, the positive-and-negative groups have mean effects within the matching interval. Gemma fails the local-link test, but Qwen passes it; the quantitative results explaining Qwen’s remaining effect-matching and outside-channel failures are missing. The initial full-state references for Gemma and Qwen themselves fail the matching test (Table 9), limiting what the subspace failures at those interfaces establish. Table 15 strengthens the PCA result by repeating the tests at matched late-layer interfaces. LLaVA remains the clearest case supporting the compact-versus-complete comparison.
-
The early experiments compare evidence masks with same-area controls, while the later matching tests use the clean-to-evidence-mask change (Appendix C.3). The abstract describes the 95% result correctly, but that result does not show whether the intervention reproduces the additional effect of masking evidence rather than comparable image content. A control-mask comparison on the 60 confirmation examples would connect these two parts of the paper.
Questions for authors
-
Could you report the complete feed-forward effects at Table 6's exact sites and the estimates and uncertainty behind Table 8's failed criteria? Please also give the mean clean-to-mask score changes underlying Tables 3 and 9 so the reported ratios can be assessed.
-
Is the patch applied at the input or output of layer 30? For the four pre-answer positions in Table 10, how much of the natural score change, restoration and writeback comes from the first answer token? Do the same four positions pass at earlier layers?
-
On the same 60 examples, could you compare evidence-minus-control restoration and writeback with evidence-minus-control score loss, using paired uncertainty?