- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper models one attention query as a few evidence tokens with a margin over many Gaussian noise tokens. It shows that attention mass on the evidence decays as 1/N, while top-1 retrieval fails only at an exponentially larger token count, leaving a window where attention is diffuse but the answer token is still the argmax. The predictions are checked with Monte Carlo, a CLIP proxy, and frozen Qwen3-VL and LLaVA-1.5 models.
Critical strengths
-
The separation between attention mass and argmax retrieval gives a simple explanation for why a diffuse attention map can coexist with a correct answer. It also connects several known observations about token pruning, such as random dropping being competitive.
-
The margin is estimated from logits on a held-out split and fixed before evaluation, so the comparison with real models is not a curve fit to the outcome. The Monte Carlo results match the stated laws closely.
Critical weaknesses
-
The model is oversimplified. It describes one frozen attention operation with i.i.d. Gaussian logits, while real multimodal models have many layers and heads, and the paper does not show how this single-softmax statistic determines the model output.
-
The experimental setup does not correspond to the theory. The theory concerns one needle token, but the probe is image-level retrieval among several images, and Appendix F states both that the needle image is placed before the distractors and that it is shuffled, with no analysis of needle position.
Questions for authors
-
Where is the needle image placed in the Qwen3-VL prompts? Please report results with the needle at different positions.A single-image probe with a small target region, such as OCR or referring tasks, would match the single-needle setting of the theory more directly.
-
The attention is read from the last layer and averaged over heads. Do the two laws hold for individual layers and heads, and how do they combine across layers?
I would consider raising my score if the authors clarify the needle placement protocol, report results across needle positions, and provide per-layer or per-head evidence that the two laws hold in real models.