- Rating
- 1
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
I cannot accurately provide a summary of the technical contributions and main research questions of this paper, as it reads incomprehensible to me after almost an hour of parsing.
Critical strengths
N/A.
Critical weaknesses
Unclear main contribution and positioning
It is hard to differentiate what the main finding of this paper is and how the authors are positioning it in the abstract and intro: is it a mechanistic interpretability problem, a capability/computer vision problem, or a failure mode they are proposing solutions for? The reader should know what to expect within the first 2 pages of reading.
Abstract and introduction are difficult to interpret and lose the reader quickly
I frankly do not understand the abstract, which is the most important summary of the main claims of this work - it is incoherent, introduces a lot of undefined terminology, and it starts with deep findings rather than a framing of what the problem studied even is, or why it matters
Related work lacks proper synthesis, and contains untraceable citations and an unclear relationship to prior work
The related work section cites papers from 2015, as well as a large number of recent papers - it feels like it oscillates all over the place, without actually providing context for how each
There are also two Anonymous citations in this submission:
Anonymous. Sophon: A procedural diagnostic for spatial reasoning in vision-language models, 2026a. Accepted at NeurIPS 2026. Benchmark and generator provided as anonymized supplementary material. Anonymous. Gain and alignment at the readout layer of vision-language models, 2026b. under review.
The first is a paper that has recently been accepted to NeurIPS 2026: https://neurips.cc/virtual/2026/loc/sydney/poster/153057 - although the OpenReview page is not open, and the paper is not on arXiv, so I can’t verify the claims referenced in the related work section here, but it appears added to the citations solely based on abstract content. Feels like an LLM related work crawl that left the author list as Anonymous, but the claims should be verified against the original paper, and the author list updated to [ Zach Gazak ⋅ Ryan Swindle ⋅ Justin Fletcher ]:
Spatial failures of VLMs. Several benchmarks document failures on simple spatial questions (Rahmanzadehgervi et al., 2024; Tong et al., 2024; Fu et al., 2025; Kamath et al., 2023); the Sophon benchmark used in this work (Anonymous, 2026a) removes semantic content so that failures cannot be attributed to scene semantics. A companion paper (Anonymous, 2026b) studies what a fine-tune must change for the frozen last five decoder layers (the reader or motor regime of Gurnee et al. (2026)) to report a quantity that the layers below them (the workspace band) already encode. Here we study which computation the fine-tune installs and how that computation can be replaced or silenced.
The second Anonymous reference is referenced as “A companion paper (Anonymous, 2026b)”, which really doesn’t tell me much apart from possibly the guess that the second paper is written by the same authors as the Sophon benchmark, or a follow-up thereof. Either way, this reference reads hallucinated because it cannot be verified as it currently stands, and the authors should have omitted it. Moreover, the submission shows “under review”, so it begs the question of where exactly the authors came across this paper — in my scientific publishing practice, I am not familiar with submissions under double-blind review ever being cited in another active double-blind submission under review. This leads me to believe that reference 2026b has potentially leaked through LLMs crawling active submissions on OpenReview, and the authors simply did not verify their citation list to exclude it:
Anonymous. Gain and alignment at the readout layer of vision-language models, 2026b. under review.
Questions for authors
-
What is the paper's primary research question, and main claims?
-
Can you define the terminology used throughout the paper where it appears first, or refrain from introducing dense technical terms that are yet to be defined in the abstract? Can some of what this terminology represents be explained in plain language instead? In particular, please clarify the operational meaning of the "reader or motor regime", "workspace band", and the computation that fine-tuning "installs".
I would invite you to think about the minimal amount of obfuscation to what you are trying to communicate, and whether more technicality or terminology helps or hurts that intent.
-
What does this submission claim beyond the referenced Sophon benchmark and the "companion paper"? Please distinguish the methods, experiments, and conclusions contributed here from those inherited from either reference. Can your main claims be evaluated using the evidence provided in this submission and its appendix?
-
Can you provide reviewer-accessible links or anonymized copies of both citations 2026a and 2026b? For Sophon, please identify the relevant files in supplementary material, and clarify if these include the paper itself, or only the benchmark and generator.
3 comments
Reviewer feedback request [by AC]
Dear Reviewer,
Please kindly go through the authors' rebuttal and provide timely feedback. The discussion phase ends in 24 hours (11/18 23:59 AoE). If you have further concerns and need additional clarifications, please specify them and take the limited time the authors have into account. Thank you.
Best,
AC
I appreciate the authors taking the time to respond to my questions and offer clarifications. Unfortunately, I do not believe the paper in its current form is ready for acceptance. The main contribution as stated in the authors' response and throughout the paper seems very narrow in scope, and I still believe presentation, clarity and scientific rigor in how findings are written up is important, so my concerns about the quality of the writing remain without a paper revision. I maintain my score as stated.
Thank you for your review. We're sorry the submitted version was hard to follow, and we have rewritten the abstract and introduction.
Research questions and claims: The question is which computation a fine-tune installs on spatial tasks with known solutions and whether it can be removed. The paper makes three claims: 1) models tuned on rotation up to 45 degrees learn an orientation-matching shortcut which fails over the wider range of rotations. 2) a correct computation can be installed by tuning over 0-180 degrees, but the supervision cost is much higher, at 18,000 examples per task. Supplying correspondence cues in the training stimuli brings that down to 1,200. 3) Six removal losses return the rotation answer to chance, but they are not durable. The paper is a controlled study, not an interpretability method or fix.
Terminology and prior work: We were sloppy with a few terms, sorry. Terms are now defined at first use. "Installs" is defined in the first paragraph of the introduction, "workspace" in Section 3, and "reader or motor regime" is removed. The paper takes stimulus generator and choice of which layers to tune from the two anonymous citations. Every experiment and claim in the paper is novel and can be evaluated from this paper alone.
Anonymous citations: These were anonymous to enforce double blind review, as neither work had been released before submission of this paper. The benchmark codebase was also private in review at NeurIPS, and so the only way we could have it is if we were the team behind that work. It is in the supplement under
sophon_bench/sophon/(the benchmark and generator, not the paper). For you to say this is likely an "LLM related work crawl", that it "reads hallucinated", and then go so far as to dig for and publish names feels inappropriate and not in the spirit of double blind review. We ask that the names be removed from the review. The full paper that you mention, whose authors you made public, is now available on arXiv. The second paper is not yet public, but we are happy to provide it to the Area Chair on request.The rating. Your review gives a 1 at confidence 2 and does not identify an error in any experiment or result. We ask you to name the experimental claim you find unsupported, or to reconsider the rating.