- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper is mainly about whether VLM can locate a graphical cue with the relation that cue expresses.Across 450 paired conditions from 150 scenes, five open VLMs locate arrow endpoints 8.5–13.1 points more accurately than they attribute speakers and stating that "the arrow points to the speaker" does not narrow the gap.The paper give two training-free functions: prompt-effect decoding (PED) and a complete K = 3 operator orbit.These raise relation accuracy by 7.2 to 22.9 points.The complete three-target operator orbit recovers relation-sensitive responses while the raw answers remain
Critical strengths
The experiment in the paper is rigorous enough,which align the models' input para meters and have six renderings.The experiment is sufficiently and convincingly done.
The issue framework in the paper has general-purposing value where reference grounding,sensitivity to the operator and the use of the relation to select an answer significantly contribute to understanding the issue.
Critical weaknesses
The model tested in the paper are all open-source models equal or under 8B ,lacking the test for closed-source or more recent models which is disadvantage to the significance of the paper. Newer models may even have solved this issue.
The paper only phenomenologically describes its findings ,where,"distinguish reference grounding, sensitivity to the operator, and use of the relation to select an answer.“ Adding some analysis of the reason might help readers better understand the mechanistic explanation at the principle level.
The PED function in the paper seems have limited relationship with its finding of the gap between the use of the relation to select an answer and reference grounding and sensitivity to the operator
The paper's logic is not clear enough, and many expressions are hard to understand.Less self-coined terms might be better for readers to follow.
Questions for authors
Can author show the experiment result on models which are more advanced ,newer or merely have more parameters?
Can author explain mechanistic reasoning on why Q3 didn‘t help .If not, hoping for explaining why not investigating further?
Can further explain the relationship between the function mentioned in the paper and the issue and observation that the paper offered?