- Rating
- 2
- Confidence
- 3
- Informativeness
- 3
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This work investigates how vision–language models obtain the spatial ordering information needed to associate objects with their properties. The main idea is to combine linear probing with activation interventions that transfer representations between examples. The authors conduct experiments to identify spatial information distributed across visual object and background tokens, as well as ordering representations formed within the language backbone. They then use probe-derived directions to modify natural-image representations and test whether these interventions repair spatial-reasoning errors. These analyses partly explain how visual and language components contribute to spatial variable binding and how the identified representations can influence model behavior.
Critical strengths
- Interesting problem setup. I think this setting is indeed worth a deeper study as our understanding regarding the proposed issue (spatial variable binding) was mostly treating it as a blackbox problem. In this light this study has its worthwhile points.
- The anaylsis in my opionin is of sufficient depth.
Critical weaknesses
- The main conclusion overgeneralizes across architectures. The paper presents the vision encoder as the primary source of ordering information. However, Appendix C.3 attributes Pixtral’s ordering signal largely to the language backbone. Hence, in my opinion the findings support architecture-dependent mechanisms but not a consistent hierarchy. I encourage the authors to reflect this distinction in the abstract and main results.
- The interventions do not fully isolate spatial information. Strip-level patching changes both the number of replaced tokens and their spatial and contextual information. Following this, the behavioral changes cannot be attributed specifically to spatial ordering, which somewhat weakens the paper’s central mechanistic interpretation. I believe the authors should add equal-token-count controls and interventions restricted to the identified spatial component.
- The headline repair gains overstate net accuracy improvements. The main results use per-example best coefficients and retest only previous failures, yielding an upper bound on repair performance. However, in contrast full-set evaluation with a fixed coefficient improves Qwen from 60.0% to 70.9%, but Gemma only from 52.7% to 53.6%. Given this I encourage the authors to foreground full-set results with coefficients selected on a separate validation set, accounting for both repaired and newly introduced errors.
Questions for authors
- How should the central account of vision-encoder dominance be revised to accommodate Pixtral’s different mechanism?
- Do the intervention effects persist with equal-token-count controls and patching restricted to the identified spatial component?
These are mainly derived from weakness, please refer to weakness for full info.
Dear Reviewers,
Although the initial review scores for this submission are somewhat negative, it is important that we thoroughly discuss our perspectives before reaching a final recommendation.
Now that the authors' rebuttal is available, I kindly ask each of you to respond in the discussion and clarify whether your original concerns have been addressed or still remain.
Thank you in advance for your timely engagement before the discussion period ends.
Best, Your AC