- Rating
- 3
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether visual-token representations already predict downstream functional sensitivity before language-decoder processing. Across four 7B vision-language models, it observes decreased fixed-top-coordinate concentration within tokens but strong reuse of dominant coordinate identities across tokens (Section 2, Figure 3). Default binary PRISM uses image-wise tails of the pre/post-interface concentration difference as calibration anchors, then greedily learns a compact disjunction of coordinate-rank predicates; downstream labels and decoder attention are not used to learn this rule (Section 3, Equations 2-8). Matched tokens are called contributive and the unmatched complement residual, based on subsequent removal experiments. Validation includes role removal across four backbones, count-matched interventions within attention quartiles on two backbones, scalar and dense-probe controls, and image-mask reuse across questions (Tables 1-4). I favor weak acceptance for the useful empirical finding, while treating the attention-independence and generalization claims more cautiously.
Critical strengths
-
The central distinction has substantial intervention evidence rather than only representational visualization. Complete contributive removal is more damaging than residual removal on all three primary metrics across all four backbones (Table 1A); the retention curves provide additional context rather than equating residual with useless (Appendix C, Table 8). The count-matched, attention-stratified comparison further shows large gaps despite lower mean attention for the selected contributive tokens (Appendix G, Table 12). PRISM also substantially exceeds direct concentration-difference ranking on POPE and MME at the original top-decile budget, with three-seed variation reported (Table 2; Appendix F.2, Table 10). These controls make the finding consequential: a simple pre-decoder representation contains predictive structure that is not captured adequately by the tested scalar ordering.
-
The image-level prior has a concrete, reusable application. Appendix I fixes both the role map and retained mask per image across 5,000 questions on 883 images. The gains over the attention-based comparator persist across the reported question categories (Table 14), supporting useful reuse rather than merely asserting query independence from the method's definition. Restricting the candidate pool while keeping the selection budget and ranking criterion fixed improves aggregate retained performance on all four backbones (Table 4B; Appendix K.1, Table 16). The complementary diversity-selector experiment is also informative, although its improvements are smaller and VILA's two role-biased variants tie (Appendix K.2, Table 17).
Critical weaknesses
-
The attention control supports a narrower conclusion than exact attention matching. Appendix G matches counts within four attention-mass quartiles using decoder layers 10-30. The matched contributive tokens have 47.2% and 55.9% lower mean attention than residual tokens (Table 1B). This deliberately reversed ordering convincingly challenges a simple explanation based on larger average attention; it tests aggregate attention rather than establishing independence from all attention structure. The matched intervention is reported for LLaVA-1.5 and VILA (Table 12). Unlike the three-seed standard deviations for fixed-budget selection in Table 10, Table 12 presents point estimates without repeated-sampling uncertainty or image-level confidence intervals. Thus, Section 5.2's conclusion should be scoped to the tested attention statistic and backbones. This is fixable through tighter matching, uncertainty estimates, and more qualified wording.
-
Calibration independence and cross-interface reproducibility need narrower clarification. Appendix H specifies MME as the default calibration source for each backbone, and Appendix A.1 gives 1,187 images for primary LLaVA-1.5 calibration. Appendix H also reports positive removal gaps after TextVQA and POPE recalibration on two backbones (Table 13), providing genuine cross-source evidence. However, an explicit image-level calibration/evaluation split or overlap accounting is not given; Appendix F.3's anchor-region analysis is explicitly on calibration images. The remaining issue is separating calibration-image behavior from unseen-image behavior, not absence of cross-dataset controls. Also, Section 2 indexes paired pre/post representations by the same token index, but the merger-based model's construction of corresponding pre-interface representations is not specified in Appendices A/B. These are fixable reproducibility clarifications, potentially requiring image-disjoint results.
Questions for authors
-
[Clarification] Appendix A.1 specifies uniform retention sampling within roles. For Appendix G/Table 12, how are the count-matched subsets selected within each quartile, and what are their per-image counts and attention distributions? Do the gaps persist under tighter matching, with image-level uncertainty estimates?
-
[Clarification] For Appendix A.1 and Appendix H, which images overlap calibration and evaluation for each backbone and benchmark? Can you provide image-disjoint intervention results and clarify whether the three seeds in Table 10 vary only token sampling or also calibration?
-
[Clarification] For Equations 1-3 and the Qwen2.5-VL experiments, how is each merged post-interface token paired with its pre-interface representation? Given Appendix H's default MME source, do the other backbones use the same calibration image set as LLaVA-1.5?
Thank you for your detailed responses and additional experiments.
I appreciate the substantial effort you have made to address my concerns. After reviewing your responses, I believe that my major concerns have been adequately addressed, and I will therefore raise my rating to 3.