- Rating
- 1
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This work proposed to use medical image alignment assessment as a test platform for generalist visual reasoning, and showed that recent frontier models have crossed an inflection point, while earlier generations perform near chance.
Critical strengths
This paper addresses a timely and practically relevant topic, and crisply isolates the generalization gap between frontier and locally fine-tuned models.
Critical weaknesses
Several methodological explanations, experimental analyses, and presentation details require further improvement.
Questions for authors
Comments
- In Figure 1, the 3D volumes are reduced to 2D slices for the alignment assessment. Since real-world registration operates in 3D, why evaluate only on 2D slices, which discard out-of-plane misalignment and may not reflect volumetric quality control?
- The authors should justify the 2D-slice choice and, ideally, also report performance on full 3D inputs.
- The motivation for using the two modality pairs to test image alignment assessment is unclear. The paper should state what each pair is meant to probe and why these two specifically support the "generalist visual reasoning" claim, rather than leaving the selection unexplained.
- In Table 1, all tested frontier models are closed-source (GPT, Gemini, Claude). To make the conclusions reproducible, more open-access models (e.g., LLaVA, Qwen-VL, InternVL) should be included.
- The MRI-PET test set contains only 29 subjects / 52 sessions, and the T1w-FLAIR set is similarly tiny. The central claim that frontier models "can now effectively perform" the task is therefore not statistically supported.
- In Tables 2 and 3, performance swings wildly between the two image-presentation methods with no control or explanation, e.g., GPT-5.2 rises from 57.7% (contours) to 88.5% (alpha-blend).
6 comments
Response to Comments 1 and 2
[Responding from UTC+0] There are a few reasons for this choice:
- For the rigid alignment task we’re focussing on three-planes are enough to detect misalignment, since the transform is global. We do not need to see more because the (mis)alignment will be present in at least one of the planes. In other words, the three-planes are orthogonal so there is no misalignment that can be out-of-plane in all three views.
- In practice, a human performing this task will only ever see 2D slices of the 3D data. A valid distinction being that humans will usually have the capacity to browse through the data, change slices etc. Enabling VLMs to do this will be an avenue for future work.
- The VLMs we test do not have the capacity to natively handle 3D data so we cannot test true volumetric quality control.
Response to Comment 3
[Responding from UTC+0] This is outlined in Section 2. We chose MRI-PET as “a challenging setting where although the two images come from the same subject, they convey different information. We use this as a test bed to probe the impact of various prompting/input choices on model performance.”
And T1w-FLAIR as “a theoretically simpler task where the two images both show aspects of brain structure. We test how well best performing settings from the MRI-PET experiment transfer to this dataset”.
In other words, MRI-PET is a difficult task to see whether VLMs can perform at all and T1w-FLAIR test whether models that perform well on MRI-PET generalise well to a different task.
As of 17:55, Nov 17, 2026 (AoE), I have not received a response, and my concerns remain unresolved. I am therefore keeping my score.