acceptodds
Withdrawn from ICLR 2027

From Image Space to Patient Space: Learning Patient-Centered Coordinates in Medical MLLMs

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Medical MLLMs are increasingly moving beyond visual recognition toward more complex medical reasoning, yet they still struggle with spatial-relation reasoning in medical images. Our analyses reveal that medical MLLMs already encode image-space spatial information in the intermediate representations. However, this image-space representation alone is insufficient for clinical spatial reasoning. We therefore propose *Patient-Space Coordinate Learning (PS-Coord)*, a dual-frame training framework that jointly supervises image-space and patient-space coordinates within intermediate hidden states. Across eight backbones spanning 7B–34B parameters, PS-Coord consistently improves CT spatial reasoning, with average gains of 19.67% and 16.17% on MIRP and MSD, respectively. The improvements further generalize to unseen imaging modalities, held-out rotations, and medical visual question answering. Representation analyses show substantially increased accessibility of patient coordinates after training, while counterfactual controls indicate that PS-Coord-trained models rely on the anatomical context of the current image for medical spatial decisions. These results suggest that explicitly connecting image and patient coordinate provides an effective way for enabling spatial reasoning capabilities already latent within medical MLLMs.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Multimodal Large Language Models, Medical Imaging, Spatial Reasoning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_rBfTOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies spatial reasoning in medical multimodal large language models (MLLMs), focusing on the distinction between image-space coordinates, which describe where an object appears on the displayed image, and patient-space coordinates, which describe its location relative to the patient’s anatomical axes. The authors first show, using layer-wise linear probes and hidden-state interventions, that existing medical MLLMs contain linearly accessible image-coordinate representations, but that these representations are not reliably connected to patient-centered spatial reasoning. To address this gap, the paper proposes Patient-Space Coordinate Learning (PS-Coord), a training framework that jointly supervises image and patient coordinates through two auxiliary prediction heads attached to an intermediate hidden state, together with standard answer supervision. Experiments on eight backbones ranging from 7B to 34B parameters report consistent improvements on CT spatial-reasoning benchmarks, transfer to radiographic tasks and unseen rotations, and gains on medical VQA benchmarks. Representation probing further shows that patient coordinates become substantially more linearly accessible after training, while counterfactual experiments suggest that PS-Coord-trained models use anatomical context from the current image when making patient-relative spatial judgments.

Critical strengths
  1. Clear and clinically meaningful formulation of spatial reasoning

The paper makes an important conceptual distinction between image-space and patient-space coordinates. This formulation captures a genuine challenge in medical imaging: the screen position of an anatomical structure changes under rotation or reflection, whereas its location relative to the patient should remain invariant. By decomposing patient-centered spatial reasoning into visual grounding, orientation inference, and coordinate-frame conversion, the paper provides a precise explanation of why a model may correctly identify where an object appears in an image yet still fail to reason about its anatomical location. This dual-frame perspective is intuitive, mathematically well specified, and potentially useful beyond the particular training method proposed here.

  1. Strong and complementary empirical evidence

The evaluation goes beyond reporting end-task accuracy. The authors combine layer-wise linear probing, target-binding analysis, prompt-transfer experiments, hidden-state interventions, held-out rotations, cross-modal transfer, and counterfactual removal or replacement of anatomical context. In particular, patient-coordinate accuracy falls to approximately chance when the anatomical background is removed or replaced, while image-coordinate accuracy remains high. This provides meaningful evidence that the trained models use image-specific anatomical context rather than relying only on marker position, question wording, or memorized anatomical priors. The substantial improvement on rotations excluded from training also supports the claim that the model learns a transferable frame-conversion rule rather than simply memorizing the training orientations.

Critical weaknesses
  1. The main results do not isolate the contribution of coordinate supervision from spatial instruction tuning

The primary comparison is between the original checkpoint and a model trained with both spatial question-answer supervision and the proposed coordinate losses. Consequently, the large improvements in the main table cannot be attributed specifically to the dual-coordinate objective. The appendix includes an answer-only condition with (\lambda_c=0), but this ablation is limited to two 7B backbones and shows relatively modest additional gains from coordinate supervision. For example, the MIRP improvement is 2.9 percentage points for Qwen2.5-VL-7B and only 0.05 points for Lingshu-7B. The paper would be substantially stronger if the main experiments included matched answer-only baselines for all eight backbones and all six benchmarks, together with statistical variation across multiple seeds. This would clarify how much of the reported improvement comes from the proposed representation-level supervision rather than targeted spatial instruction tuning more generally.

  1. The experimental setting is highly controlled

The training setup relies primarily on selected two-dimensional axial CT slices, artificial rotation and reflection transformations, explicit circular target markers, and slices filtered to contain recognizable orientation cues. This makes the coordinate labels and counterfactual tests clean, but it also leaves a considerable gap between the benchmark setting and realistic clinical spatial reasoning. The current experiments do not evaluate volumetric localization, oblique or non-axial views, unmarked abnormalities, truncated fields of view, acquisition artifacts, postoperative anatomy, or uncommon anatomical configurations such as situs inversus. These cases are precisely where memorized anatomical priors and standard orientation assumptions may fail.

Questions for authors

See weaknesses.

Reviewer_Yk2ZOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

In this paper, the authors argue that image coordinates (already encoded in MLLMs) alone are not enough for clinical spatial reasoning, which requires locating targets in the patient's anatomical frame. PS-Coord is proposed which finetunes the MLLMs on spaital QA built from roated and reflected axial CT slices. Two affine heads on an intermediate hidden state are trained to predict both image-space and patient-space coordinates. Across eight backbones, PS-Coord improves results on MIRP and MSD and transfers to other tasks. Probes show higher patient-coordinate readability after training, and counterfactual inputs indicates that the trained models rely on anatomical context.

Critical strengths
  1. A clinically relevant problem. Laterality and orientation errors matter in radiology, and existing benchmarks show that MLLMs often answer spatial questions from anatomical priors rather than from the image.
  2. Clean engineering and documentation. The data construction, coordinate geometry (App. E), training settings, checkpoint identifiers, and scoring protocols are described in detail, which aids reproducibility.
  3. Useful analysis tools. The probing, binding, and intervention analyses, together with the counterfactual background-replacement controls, are a reasonable toolkit for examining spatial representations in medical MLLMs.
Critical weaknesses
  1. Positioning relative to prior work; two of the four contributions overlap with existing findings : Linear coordinate subspace (contribution 1). Linearly decodable, causally relevant spatial representations in VLM hidden states have already been reported, e.g., Kang et al. (ICLR 2026, “Linear Mechanisms for Spatiotemporal Reasoning in Vision Language Models”) and work on linear spatial world models in LLMs (2025). The medical-domain replication is useful but should be presented as such, with these works cited. Two reference frames (contribution 2). The image-frame versus patient-frame distinction under rotation and reflection is already central to MIRP (Wolf et al., 2025), the paper’s main benchmark, which scores image-view correctness and anatomical correctness separately on transformed CT slices. In the general domain, COMFORT (Zhang et al., ICLR 2025) studies frame-of-reference ambiguity in VLMs, including reflected versus rotated axis conventions. Neither COMFORT nor the linear-representation literature is cited. The paper should state its novelty more narrowly and compare against these works.

  2. The gains appear to come mainly from in-domain fine-tuning, not from dual-frame supervision. The paper’s own answer-only ablation (λ_c = 0, Table 8, two models) gives: Qwen2.5-VL-7B: MIRP 83.10 vs 86.00, MSD 60.00 vs 63.67; Lingshu-7B: MIRP 83.24 vs 83.29, MSD 66.67 vs 70.00. So the coordinate heads add roughly 0–4 pp, while the headline average gains are about 16–20 pp. Two baselines are needed to support the central claim: (i) answer-only fine-tuning on the same data for all eight backbones, and (ii) an image-coordinate-only head, to show that the patient frame specifically drives any improvement. Results also use a single training seed without confidence intervals on 300–900-item benchmarks, so the small differences that remain cannot be judged.

  3. The patient frame is inferred from anatomy, which is a safety risk. PS-Coord teaches the model to infer display orientation from anatomical cues, and Figure 6 confirms that patient-frame answers depend on the anatomy in the image. In patients with situs inversus, dextrocardia, post-surgical anatomy, or transplants, this would produce confident laterality errors, the failure mode the paper aims to prevent. This is mentioned only briefly in the limitations (App. L). An evaluation on mirrored anatomy without a reflected display, or on real atypical-anatomy cases, would make the risk concrete.

  4. Unexplained probe asymmetry. In Figure 3a, the readout for the queried target is accurate on both axes (r ≈ 0.96–0.98). However, the correlation with the other target’s position is positive for x (+0.48/+0.46) and negative for y (−0.40/−0.42). This suggests incomplete binding with opposite signs on the two axes, possibly due to how marker pairs were sampled. It deserves explanation.

Questions for authors
  1. Can you report answer-only fine-tuning (λ_c = 0) and image-coordinate-only supervision for all eight backbones, ideally with multiple seeds and confidence intervals?

  2. How does the centroid-based patient coordinate compare with the true patient frame derived from DICOM orientation metadata? In which practical settings is the anatomy-inferred frame needed instead of the header?

  3. How would the framework extend to oblique or non-axial acquisitions (e.g., cardiac MR short- and long-axis views) and to 3D volumes?

  4. What explains the opposite-sign cross-target correlations for x and y in Figure 3a?

Reviewer_EfaDOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies how medical multimodal large language models (MLLMs) represent image and patient coordinates. The authors propose Patient-Space Coordinate Learning (PS-Coord), which combines answer supervision with coordinate losses. The method is evaluated on eight model checkpoints and six benchmarks. The paper also includes representation probes, rotation tests, and counterfactual experiments. The problem is important, and the results are promising. But some experimental choices and reporting issues make the main claims difficult to assess.

Critical strengths
  1. The paper addresses spatial reasoning in medical MLLMs and clearly distinguishes image coordinates from patient coordinates.
  2. The method is tested across eight model checkpoints, four model families, and six benchmarks.
  3. Linear probes, layer-wise analysis, and rotation tests provide insights into how spatial information is represented.
  4. These experiments provide evidence that predictions depend on anatomical context.
  5. The coordinate heads are removed after training, so they add no overhead during inference.
Critical weaknesses
  1. Answer-only fine-tuning already produces good gains. In Table 8, adding coordinate supervision improves MIRP by 2.90% points for Qwen2.5-VL-7B but only 0.05 points for Lingshu-7B. The paper does not include separate experiments for image-coordinate-only and patient-coordinate-only supervision. Matched-budget comparisons are important to determine whether both losses are necessary and how much they contribute beyond answer-only training.

  2. Tables 1 and 8 report different MIRP scores for Lingshu-7B: 76.67% and 83.29%, respectively. The authors should explain whether these differences are due to training budgets or else. And Appendix C.2 reports 32,000 inputs and 5,000 optimizer updates, while Table 5 specifies gradient accumulation over eight inputs, which would imply 40,000 inputs for 5,000 updates. The actual training procedure should be clarified. It is also not clear whether general medical VQA examples were included during training. If so, the authors should provide their sources and quantities, as this affects the interpretation of the reported VQA gains.

  3. The hidden-state interventions use gradient-derived directions rather than the ridge-regression directions used for coordinate probing. Therefore, the experiments do not directly establish that the identified coordinate directions cause the spatial predictions. Direct interventions along the probe directions, with suitable controls, would provide stronger evidence.

  4. The coordinate heads use 18th hidden states for all models, although larger models show their strongest image-coordinate correlations at deeper layers. A layer-selection ablation would help determine whether using deeper layers improves performance, especially for the larger models.

  5. As in Appendix B.3, the training prompts refer to a red circle rather than explicitly naming organs. Table 11 also reports relatively low strict grounding scores, with G-all8 ranging from 16.88% to 38.12%. These results do not establish consistently correct organ grounding across transformed views.

Similarly, the radiograph experiments examine image orientation and displayed-image laterality but do not directly demonstrate patient-coordinate localization across imaging modalities. The authors should narrow these claims to match the evidence presented.

6.Some benchmark inputs share underlying questions or cases. Reporting confidence intervals grouped by case or question would give stronger evidence for the reported improvements.

Questions for authors

Q1. Can the authors provide matched-budget comparisons of answer-only, image-coordinate-only, patient-coordinate-only, and dual-coordinate supervision?

Q2. Can they clarify the differences between Tables 1 and 8 and reconcile the reported input counts with the optimizer updates?

Q3. Were general medical VQA examples included in training? If so, what were their sources and quantities?

Q4. Can they test interventions directly along the coordinate directions identified by the linear probes?

Q5. Does using a deeper supervision layer improve performance for the larger models, and can the authors report confidence intervals grouped by case or question?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.