acceptodds
Withdrawn from ICLR 2027

Beyond Time: A Physically-Grounded Two-Stage VLM for 3D Medical Image Segmentation via Spatial Prompting

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Bringing temporal modeling from vision-language models into 3D medical image understanding offers a practical way to process full-length CT and MRI volumes and improve training stability. Existing methods still have two main limits. They often treat the spatial Z axis as a uniformly sampled time axis and ignore the true physical slice thickness, which weakens spatial perception. They also rely on external foundation models for one-way slice-by-slice mask propagation, which can accumulate errors in long sequences. We propose PhysiVLM, a physically constrained two-stage framework that explicitly separates 3D spatial reasoning from 2D mask generation. In the first stage, we use an anatomical 2.5D grid puzzle and spatial coordinate prompts to inject the prior of physical slice thickness, which allows the model to regress the global 3D depth range of a lesion at low cost and with high accuracy. In the second stage, we introduce a Spatial Prompting mechanism to build anchor-guided reasoning. The model takes the predicted depth range from the first stage, dynamically samples key slices, and then guides the VLM to perform slice-wise spatial semantic reasoning, parse lesion shape, and output structured 2D bounding boxes. We then feed these boxes into MedSAM2 to produce spatially consistent bidirectional mask propagation along the Z axis. Extensive experiments show that PhysiVLM reduces cumulative errors in long-sequence propagation, cuts redundant computation, and improves segmentation accuracy for lesions with complex cross-slice morphology. The code is available at: https://anonymous.4open.science/r/PhysiVLM-1E33

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
3D Medical Image Segmentation, Vision-Language Models, Spatial Prompting

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_t6uzOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper presents PhysiVLM, a text-conditioned VLM for 3D medical image segmentation, which aims to improve segmentation performance by introducing physical priors into a VLM. However, some experimental results, metric calculations, and statistical analysis in the current manuscript need further clarification.

Critical strengths

PhysiVLM offers a relative clear and practical coarse-to-fine framework that connects text-guided spatial localization with medical mask generation. The VLM interprets text descriptions and expresses target locations as depth intervals and 2D bounding boxes, while MedSAM2 generates pixel-level masks. This provides a approach to using VLMs for medical image segmentation.

Critical weaknesses

Several details about the datasets, metric calculations, consistency of results, and reproducibility remain unclear. These problems affect the main conclusions and confidence in the experiments (see Questions For Authors for details).

Questions for authors

Please address the questions below, particularly the dataset and evaluation protocols, and check the inconsistencies in the reported metrics and training settings. Explaining these differences is important for assessing the main conclusions and the reliability of the experiments:

  1. The MSD setup is unclear. The paper mentions three MSD subsets in the training data but does not name them. It is also unclear which tasks and evaluation settings the MSD results in Table 1 cover. Please specify these details.

  2. Different models have identical results across multiple metrics. In Table 1, U-KAN and U-RWKV report exactly the same values for all four MSD metrics. This seems unusual. Please check these results and explain why they are identical.

  3. The full list of 21 training datasets is missing. Line 323 states that PhysiVLM is trained on 21 public CT/MRI segmentation datasets, but these datasets are not fully listed. Please provide the complete list and explain how each dataset is used.

  4. R1Seg’s metric calculations need clarification. For large targets, Table 2 reports a Dice score of 76.5%, Precision of 70.0%, and Recall of 67.0%. If these metrics use the same samples, weights, and standard definitions, the mean Dice cannot exceed the harmonic mean of the mean Precision and Recall, which gives an upper bound of approximately 68.47%. Please check these values and explain how each metric is calculated and aggregated.

  5. VILA’s subgroup results do not match its overall results. If VILA in Table 2 is VILA-M3 in Table 1, and both tables use the same test set and aggregation method, the overall Dice should lie between the large-target score of 57.8% and the small-target score of 48.7%. It therefore cannot be the 33.3% reported in Table 1. HD95 shows a similar issue: both subgroup values are higher than the overall value. Please clarify the model, evaluation cohort, and aggregation method used in each table.

  6. PhysiVLM’s results imply inconsistent subgroup weights. If Tables 1 and 2 use the same set of valid cases, the same grouping, and case-wise averaging, let ww be the proportion of large targets. Then: Overall mean=w×Mean for large targets+(1−w)×Mean for small targets.\text{Overall mean} = w \times \text{Mean for large targets} + (1-w) \times \text{Mean for small targets}. From Dice, we obtain: w=73.2−68.585.8−68.5≈27.2%.w=\frac{73.2-68.5}{85.8-68.5}\approx27.2\%. From HD95, we obtain: w=165.2−143.1165.2−31.6≈16.5%.w=\frac{165.2-143.1}{165.2-31.6}\approx16.5\%.

    For the same set of cases, the proportion of large targets cannot be both 27.2% and 16.5%. This difference may come from how empty masks are handled, how metrics are aggregated, or the use of different MSD subsets in the two tables. Please explain the calculation steps. If the tables use different subsets, please also explain why.

  7. The Stage 2 training duration is inconsistent. Lines 808–809 in Appendix A.3 on page 15 state that Stage 2 is trained for 15,000 epochs, while Table 9 lists 1,500 epochs. Please confirm the actual number of epochs and the training schedule used.

The manuscript also has the following editing and reproducibility issues:

  1. The title does not match the content. The PDF is titled “AudioRouter: Audio-Guided Visual Compression for Streaming Video Language Models,” which is unrelated to the study presented. Please check the manuscript version.

  2. Unexplained citation markers remain in the text. Markers such as :contentReference[oaicite:…]index=… appear at lines 715, 787, 809, and 823, among others. Their meaning and sources are unclear. Please clarify what they refer to and clean up these markers.

  3. The anonymous code link could not be accessed. I could not access the anonymous code link during the review, so I was unable to check the implementation and evaluation pipeline. Please provide a working link or the relevant code materials.

Reviewer_ZqwGOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper introduces PhysiVLM, a physically constrained framework that separates global 3D spatial reasoning from local 2D mask generation for segmenting lesions in CT and MRI scans based on text descriptions.

Critical strengths

The author witnesses two significant weaknesses in the current temporal modeling of 3D medical segmentation and proposes a physically constrained framework that separates global 3D spatial reasoning from local 2D mask generation.

Critical weaknesses
  1. The topics on the webpage are quite different from the ones in the PDF.
  2. The authors should provide more information to verify the effectiveness of their method. In Table 1, although the proposed method slightly better than U-KAN, the improvement may come from the use of external datasets.
Questions for authors
  1. What does "slice-wise spatial semantic reasoning" mean? Is this a reasoning process such as CoT? Is there any specific reasoning mechanism proposed in this paper?
  2. In Table 1, please use bold text to label the best results.
  3. The intermediate visualization is missing. Please provide the visualization results for Depth Regression and Dynamic Sampling.
Reviewer_goQrOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PhysiVLM, a two-stage framework for 3D medical image segmentation. It addresses the limitations of existing 3D-as-Video approaches, which ignore physical Z-axis spacing and are prone to error accumulation during unidirectional mask propagation. In the first stage, a 2.5D grid and physical coordinate tokens are used to predict the depth interval of the lesion. In the second stage, key slices are sampled within this interval, and a VLM generates bounding boxes that serve as multi-anchor prompts for a frozen MedSAM2 to perform bidirectional mask propagation. Experiments demonstrate that the proposed method achieves superior segmentation performance and effectively mitigates long-range mask drift.

Critical strengths

1.Pointing out that simply treating the Z-axis of medical images as the time axis of a video discards physical thickness information is a valuable observation. The design of introducing absolute physical coordinates aligns well with the physical nature of medical imaging.

  1. Decoupling VLM from low-level pixel segmentation, and using multi-anchor initialization to shorten the propagation distance per run in MedSAM2, is sound and effectively mitigates error accumulation. The mathematical derivation and computation graph explanation are very clear, showing how the mask loss can still backpropagate to the VLM despite a frozen MedSAM2.
Critical weaknesses
  1. The title presented in the PDF header ("AUDIOROUTER: AUDIO-GUIDED VISUAL COMPRESSION FOR STREAMING VIDEO LANGUAGE MODELS") completely mismatches the actual content of the paper (PhysiVLM for 3D medical image segmentation). While this may be a careless formatting mistake, it is unacceptable for a top-tier conference submission.
  2. Although the experiments are generally solid, they require further improvement. The paper lacks comparisons with recent SOTA models equipped with self-supervised pre-trained weights, such as SuPreM and VoCo. Evaluating against these strong baselines is essential to justify the necessity of introducing a heavy VLM pipeline. In Table 1, the HD95 values for the proposed method are notably large, but they are also highlighted in bold. This is misleading. Furthermore, the authors fail to discuss the reasons behind the poor HD95 results. The same issue persists in the ablation study. The full version does not achieve the best HD95. And the authors say the model is trained on 21 public datasets, but the manuscript fails to provide a specific list of these 21 datasets.
  3. The core pipeline, using a VLM to generate bounding boxes that serve as prompts for SAM-based mask generation, has already been extensively explored in the natural image domain. PhysiVLM does not fundamentally alter this information flow, but merely extends the VLM-generated boxes from a single slice to multiple slices and delegates the subsequent propagation to MedSAM2. Furthermore, while the authors present the incorporation of true physical thickness as a major contribution, leveraging voxel spacing and origin coordinates for resampling has long been standard practice in traditional 3D medical image processing. Repackaging this foundational concept as a special token within an LLM involves certain implementation details and may offer practical value, but it lacks fundamental methodological novelty.
Questions for authors

no

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.