acceptodds
Under review as a conference paper at ICLR 2027

Medical Image Alignment Assessment as a Test of Generalist Visual Reasoning in Frontier Multimodal Models

Abstract

Frontier multimodal large language models (MLLMs) are increasingly positioned as general purpose visual reasoners as part of the quest for artificial general intelligence. A key test of this generality is whether they can perform novel visual judgments that humans can make reliably from visual evidence and task instructions, without task-specific parameter optimisation. We investigate this question through the task of medical image alignment assessment, where the goal is to establish whether there is anatomical correspondence between two images. Assessing spatial alignment is a fundamental component of medical imaging pipelines across within-subject, between-subject, and cross-scanner settings. Human visual assessment of image alignment is still the gold standard and most common approach; however, it requires trained operators and is impractical to scale for large datasets. Existing automated methods using conventional image quality metrics or convolutional neural networks (CNNs) are generally designed for particular modalities, anatomical regions, or alignment tasks, not generalisable readily to new settings. We evaluate recent generations of MLLMs on two exemplar medical image alignment tasks, varying both prompting strategies and image-presentation methods. We compare against a smaller, locally fine-tuned MLLM and a task-specific CNN, allowing us to examine the trade-off between frontier general purpose models that cannot be used locally and smaller models that require specific task optimisation but can be used locally. Our results show that we are reaching an inflection point, where frontier MLLMs can now effectively perform visual assessment of medical image alignment. Models released only a few months ago generalise poorly and, in some settings, perform barely above chance, whereas GPT-6 achieves over 85% across almost all scenarios tested, including under zero-shot evaluation. Fine-tuned local models can match or exceed frontier-model performance on the tasks on which they are trained, but transfer substantially less effectively to unseen settings. These findings identify medical image alignment as a useful test bed for generalist visual reasoning. They also suggest that frontier multimodal models are beginning to acquire transferable visual assessment capabilities that could support a common quality-control mechanism across heterogeneous medical-imaging pipelines.

Primary area:
applications to neuroscience & cognitive science
Keywords:
Frontier Models, Multimodal, Large Language Models, Image Understanding, Medical Imaging, Image Alignment

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_VJX2OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This work proposed to use medical image alignment assessment as a test platform for generalist visual reasoning, and showed that recent frontier models have crossed an inflection point, while earlier generations perform near chance.

Critical strengths

This paper addresses a timely and practically relevant topic, and crisply isolates the generalization gap between frontier and locally fine-tuned models.

Critical weaknesses

Several methodological explanations, experimental analyses, and presentation details require further improvement.

Questions for authors

Comments

  • In Figure 1, the 3D volumes are reduced to 2D slices for the alignment assessment. Since real-world registration operates in 3D, why evaluate only on 2D slices, which discard out-of-plane misalignment and may not reflect volumetric quality control?
  • The authors should justify the 2D-slice choice and, ideally, also report performance on full 3D inputs.
  • The motivation for using the two modality pairs to test image alignment assessment is unclear. The paper should state what each pair is meant to probe and why these two specifically support the "generalist visual reasoning" claim, rather than leaving the selection unexplained.
  • In Table 1, all tested frontier models are closed-source (GPT, Gemini, Claude). To make the conclusions reproducible, more open-access models (e.g., LLaVA, Qwen-VL, InternVL) should be included.
  • The MRI-PET test set contains only 29 subjects / 52 sessions, and the T1w-FLAIR set is similarly tiny. The central claim that frontier models "can now effectively perform" the task is therefore not statistically supported.
  • In Tables 2 and 3, performance swings wildly between the two image-presentation methods with no control or explanation, e.g., GPT-5.2 rises from 57.7% (contours) to 88.5% (alpha-blend).
6 comments

As of 17:55, Nov 17, 2026 (AoE), I have not received a response, and my concerns remain unresolved. I am therefore keeping my score.

AuthorsOpenReview ↗

Response to Comments 1 and 2

[Responding from UTC+0] There are a few reasons for this choice:

  • For the rigid alignment task we’re focussing on three-planes are enough to detect misalignment, since the transform is global. We do not need to see more because the (mis)alignment will be present in at least one of the planes. In other words, the three-planes are orthogonal so there is no misalignment that can be out-of-plane in all three views.
  • In practice, a human performing this task will only ever see 2D slices of the 3D data. A valid distinction being that humans will usually have the capacity to browse through the data, change slices etc. Enabling VLMs to do this will be an avenue for future work.
  • The VLMs we test do not have the capacity to natively handle 3D data so we cannot test true volumetric quality control.
AuthorsOpenReview ↗

Response to Comment 3

[Responding from UTC+0] This is outlined in Section 2. We chose MRI-PET as “a challenging setting where although the two images come from the same subject, they convey different information. We use this as a test bed to probe the impact of various prompting/input choices on model performance.”

And T1w-FLAIR as “a theoretically simpler task where the two images both show aspects of brain structure. We test how well best performing settings from the MRI-PET experiment transfer to this dataset”.

In other words, MRI-PET is a difficult task to see whether VLMs can perform at all and T1w-FLAIR test whether models that perform well on MRI-PET generalise well to a different task.

Reviewer_rXtGOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper evaluates frontier multimodal models for detecting medical image misalignment using MRI-PET and T1w-FLAIR data. The authors compare different prompting strategies and image presentation methods with locally fine-tuned MedGemma and ResNet-18 models. The results are interesting, particularly regarding the generalization ability of frontier models compared to task-specific models.

Critical strengths
  1. The paper addresses a relevant and practical problem in medical image quality control. The comparison across different MLLMs and prompting strategies is useful.

  2. The comparison with locally fine-tuned models provides interesting insights into the trade-off between task-specific performance and generalization.

Critical weaknesses
  1. The evaluation relies entirely on synthetic misalignments, which may not reflect real registration failures. The small test sets (52 MRI–PET and 40 T1w–FLAIR sessions) and lack of statistical analysis, confidence intervals, and error analysis further limit the reliability and generalizability of the findings.

  2. The paper does not clearly establish whether MLLMs perform genuine anatomical reasoning or simply detect visual mismatches in 2D overlays. The absence of conventional registration quality metrics and stronger vision-based baselines, along with unmatched training and testing input formats for local models, weakens the comparison and central claims.

Questions for authors
  1. Did you evaluate the models on real registration failures? How does performance on actual misalignment compare with synthetically generated cases?

  2. How do you ensure that models detect anatomical misalignment rather than interpolation artifacts, image boundaries, or other visual cues introduced by synthetic transformations? Are aligned images also resampled as controls?

  3. How does performance change with different misalignment magnitudes, particularly near the minimum thresholds of 2.5 mm translation and 3.5° rotation?

3 comments
AuthorsOpenReview ↗

Response to Question 1

[Responding from UTC+0] Thank you for this suggestion, owing to the difficulty of defining a genuine ground truth in registration tasks, we utilise the synthetic case where we create a ground truth misalignment similar to seminal works developing linear alignment methods (Jenkinson et al., https://doi.org/10.1006/nimg.2002.1132, 2002).

To investigate this properly, we identified 30 subjects where one of the alignment methods described in Section 2.1.2 failed human visual assessment and tested a subset of MLLMs on those plus 10 control cases from the same subjects using one of the other methods that achieved successful alignment. This now makes up Experiment 1c.

AuthorsOpenReview ↗

Response to Question 2

[Responding from UTC+0] Thank you for the suggestion, we do not believe this to be a concern since we are attempting to replicate a human-like visual assessment process where interpolation artefacts would be present too. All images undergo interpolation as part of the alignment process to generate ground-truth alignment, however it is true that the misaligned images undergo a second resampling which could introduce artefacts. We will investigate this further as future work but we expect the effect to be small, particularly for the MRI-PET experiment where the PET image is the moving image while the T1w is held fixed. This means that the PET image is resampled, however it is already inherently blurry so less susceptible to interpolation artefacts.

AuthorsOpenReview ↗

Response to Question 3

[Responding from UTC+0] Thank you for this suggestion, it is something we briefly looked into but did not fit into the original submission.

We now include an investigation of this in the best performing model, GPT-6 Astra which highlights an important factor that was not explored previously. Overall accuracy can hide sensitivity/specificity imbalances, as highlighted by GPT-6 Astra: Contours – Zero Shot, where overall 75% accuracy belies poor sensitivity explained by only detecting the most severe misalignment.

To reflect this we have updated Table 2 (and new Table in response to Question 1) so also include sensitivity and specificity and also added a discussion of this and new figure in the appendix to explore misalignment magnitude vs. accuracy.

Reviewer_SaVYOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies VLM performance on a medical image alignment assessment task. The authors use MRI–PET pairs, align them using three off-the-shelf methods, and select the best alignment as their reference after manual verification. They then introduce random misalignments and prompt the models to determine whether the images are aligned. They further test the best settings found in this experiment on T1w–FLAIR alignment assessment to evaluate generalizability. They evaluate multiple proprietary models and one open-source VLM, MedGemma 4B, and compare their performance with fine-tuned MedGemma and ResNet-18 models. Their results show that newer models generally perform better and that in-context examples often improve performance.

Critical strengths

None

Critical weaknesses
  1. The paper does not sufficiently justify why a single misalignment classification task in the medical domain is useful for evaluating generalist visual reasoning. While the authors explain its relevance to medical image quality control, they do not establish why performance on this task is a meaningful measure of broader visual generalization. In particular, the experiments do not distinguish anatomical correspondence assessment from detecting simpler visual mismatches.

  2. The paper’s main contribution is an empirical study of model performance over time on a narrowly defined task, but the evaluation is insufficiently thorough to support its broader conclusions. The paper does not include open-source VLMs except for a single small 4B model, limiting the insights it provides about these models. Although the experiments examine in-context examples, image presentation, training-set size, and comparisons with fine-tuned local models, they provide limited analysis of failure modes or controlled investigation of what drives the performance differences. Consequently, the study offers limited insight beyond reporting which models perform better under the tested settings.

Questions for authors
  1. Could the authors explain why medical image misalignment classification is a meaningful test of generalist visual reasoning? What properties of the task or findings distinguish this interpretation from detecting task-specific visual mismatches?

  2. Could the authors clarify the main insights they would like readers to take away from this study? In particular, how do the findings contribute to our understanding of VLM visual capabilities, and which conclusions might extend beyond the specific models and alignment settings evaluated?

3 comments
AuthorsOpenReview ↗

Response to Question 1

[Responding from UTC+0] The idea is not to test full generalist reasoning, but rather a specific aspect of it. We identify that this is a task that human readers can do easily but traditional automated methods struggle with. Indeed, we have been relying on image alignment since the advent of digital medical imaging with CT in the 1960s, however no reliable, generalisable methods exist to automatically assess alignment quality.

We argue that if a model can be considered to visually reason, it should be able to do this task, not that if a model can do this task it can be considered to be a generalist reasoner. In other words, we feel that performing this task is a necessary test of generalist visual reasoning but not a sufficient demonstration thereof.

The difference from task-specific visual mismatches is explored in Experiment 2. Task-specific models can indeed perform this task well, but only on modality/image-presentation that they have be trained on while recent VLMs perform strongly across the board (i.e. generalise well).

AuthorsOpenReview ↗

Response to Question 2

[Responding from UTC+0] The main insight is to highlight that we are reaching an inflection point where models are showing VLM capabilities that generalise across modalities and image presentations for this task and showing promise that image alignment metric or task-specific CNN based methods have not been able to do for over half a century.

It points towards the potential for frontier MLLMs to perform wider visual inspection tasks, particularly in the medical domain, where although the tasks may differ, the fundamental capacity to understand what you’re looking (anatomy, modality etc.) and where anomalies may be is shared.

I thank the authors for their answers. However, my main concern remains. In the first answer, the authors clarify that performing well on this task is not sufficient to establish general visual reasoning, which I agree with. In the second answer, they suggest that this performance “points towards the potential for frontier MLLMs to perform wider visual inspection tasks.” While this is presented as a possibility, it has not been supported by the experiments or the discussions.

I acknowledge that the results show better generalization across the tested modalities and image representations than the task-specific models. However, this is still generalization within alignment assessment. The broader implications remain speculative, and the response has not sufficiently clarified what we learn about VLM capabilities beyond this specific task.

As a result, I will retain my score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.