acceptodds
Under review as a conference paper at ICLR 2027

Think Before You Locate: Clinical Chain-of-Thought Supervision for Sequential Medical Image Grounding

Abstract

Clinicians rarely localize a finding from one image alone. They compare current and prior scans, follow structures across slices, and match regions across views or modalities. Yet vision-language models for sequential medical grounding usually learn only from bounding boxes. Boxes show where the target is, but they do not explain why that region in that image answers the question. We study whether explicit, image-grounded reasoning can improve localization, and which tasks benefit from it. We introduce \method, whose core is MedChain, a corpus of 182,573 filtered rationales spanning eight sequential grounding tasks and ten imaging modalities. A medical VLM (FlemingVL-38B) first drafts a caption-like rationale, and a multimodal refiner (Qwen3.5-397B-A17B) then re-examines the images, question and annotated box to write a box-conditioned explanation with explicit cross-image reasoning. We fine-tune Qwen3.5 on these rationales with deliberative chain-of-thought supervision (dCoT-SFT). Across 4B, 9B, and 27B models, dCoT-SFT sets a new state of the art on MedSG-Bench. The 27B model reaches 77.24 mIoU and 85.20 [email protected], improving over the strongest prior model by 4.69 and 5.49 points. In a paired comparison using fixed 4B checkpoints, dCoT-SFT improves over answer-only SFT by 3.56 mIoU (95% CI 3.13–4.00). The gains mainly come from change detection and object tracking, while performance drops on three appearance-matching tasks. Replacing a model's own rationale with a rationale from another task reduces mIoU by 12.2 points, showing that localization depends strongly on the reasoning context. Finally, adding GRPO with an IoU reward after dCoT-SFT gives only small and task-dependent gains, which we analyze through controlled multi-seed development experiments.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Sequential medical image grounding, Chain-of-thought supervision, Multimodal large language models, Visual grounding, Medical vision-language models, Reasoning distillation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_o7aROpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MedChain-R1, a two-stage multimodal framework in which teacher models generate medical explanations conditioned on ground-truth (GT) bounding boxes, and these explanations are then used to supervise vision-language models on sequential medical image localization. The authors construct an explanation dataset spanning eight task types and ten imaging modalities. Experiments across multiple model scales show promising results on MedSG-Bench. The paper also analyzes the context dependence of the explanations and the additional benefits of GRPO.

Critical strengths
  1. The motivation is reasonable. Generating explanations conditioned on GT bounding boxes and using them to supervise a model could help the model learn interpretable visual grounding.
  2. The method is evaluated across multiple tasks and imaging modalities, providing evidence of its applicability to a range of settings.
Critical weaknesses

The paper does not provide a sufficiently detailed analysis of the data distribution, bounding-box distribution, or lesion distribution in the test set. Without these analyses, it is difficult to determine whether the reported gains reflect improved disease understanding or are driven by other factors.

The source and construction of the paired images, as well as the origin of the GT annotations, are not clearly described. For example, it is unclear whether two images come from different slices in the same examination of one patient, or from examinations acquired at different time points. The paper also does not explain how the authors ensure that the bounding boxes in paired images correspond to the same lesion. Without these details, the reliability and clinical relevance of the sequential localization task are difficult to assess; in the worst case, the task may not represent a well-defined clinical problem.

Related work, such as TumorChain (ICLR 2026), also constructs chain-of-thought data and uses it for model training. The paper does not sufficiently clarify its technical novelty relative to these approaches. At present, the contribution appears to rely primarily on constructing chain-of-thought data.

The experiments do not disentangle the benefits of explanation-based supervision from those of direct bounding-box optimization. It is therefore unclear whether the gains arise from interpretable supervision or simply from additional localization training. If the explanations do improve performance, further analysis is needed to determine whether the improvement comes from better lesion or object classification, or from more accurate box prediction for structures the model could already detect.

Questions for authors

Is FlemingVL-38B necessary for the proposed pipeline? How would the results change if another model were used? In the second stage, the model already has access to the original image and the GT box, while the first-stage description is only an optional input. Why not have the second-stage model generate the explanation directly?

Please provide the image sources and construction rules for each task. In particular, distinguish among true longitudinal pairs, cross-slice, cross-view, and cross-modality pairs, as well as cropped or synthetically constructed samples. Please also clarify how patient identity and correspondence are established, whether image registration is performed, and how target consistency across paired images is verified. These details are essential for assessing the level of clinical reasoning required by each task and the scope of the method’s generalization.

Please report the category distribution of the target regions, distinguishing pathological lesions, normal anatomical structures, image-change regions, and local correspondence regions. The paper should also report the main disease or lesion categories and their sample counts. This would help determine whether the reported gains reflect improved clinical abnormality recognition or more general visual correspondence and localization

7 comments

We thank the reviewer for detailed and constructive comments. We agree that interpreting the gains requires clearer data provenance, target distributions, and controls separating explanation supervision from additional localization training. We address raised points through an additional test-set analysis and matched training comparisons.

W1 / Q3. Data, bounding-box, and target-category distributions

We evaluate on MedSG-Bench, which contains 9,630 questions covering 76 public datasets, ten modalities, and 114 clinical tasks. Additional test-set analysis finds that CT and MRI account for **51.5

The median box width and height are 50 and 49 pixels, and the median box-area fraction is **2.4

Selected annotated lesion categories include brain tumour (1,279 questions), breast findings (269), skin lesions (216), polyps (113), and stroke lesions (59). These are question counts, not counts of unique patients or lesions.

The target-group analysis below reports dCoT-SFT minus box-only SFT for the new matched 4B comparison, in mIoU percentage points:

Target groupQuestionsDifference [95% CI]
Image-change regions2,000+2.06 [1.48, 2.64]
Local correspondence regions—patch1,000−0.49 [−0.90, −0.13]
Pathological lesions2,032−0.01 [−0.74, 0.70]
Normal anatomical structures4,370+0.52 [−0.22, 1.25]
Other228−0.21 [−3.01, 2.49]

The clearest improvement is therefore in image-change localization, while the pathological-lesion group shows no supported average improvement. We will narrow our interpretation accordingly: these results support task-specific visual comparison and localization benefits, not a demonstrated improvement in disease recognition. They also do not isolate target size from task type.

W2 / Q2. Image sources, construction, and ground truth

We inherit the images, image sets, and localization annotations from the MedSG resources; MedChain adds rationales rather than constructing new image pairs or ground-truth boxes. The benchmark construction distinguishes the following tasks:

TaskConstruction and correspondence
Registered differenceOne image and a locally modified counterpart; changes are simulated.
Non-registered differenceThe same construction, with translations of up to 20 pixels per axis.
Multi-viewOrthogonal slices from a volume or existing mammographic views, including VinDr-Mammo.
Object trackingOrdered volumetric slices or annotated surgical-video frames.
ConceptA mask-extracted reference region matched to its source image.
PatchCropped regions matched to their original image locations.
Cross-modalRegions paired using modality metadata and semantic annotations.
ReferringA textual target selected from semantically unrelated candidate images.

These are the benchmark's construction rules, rather than a claim that every task tracks the same lesion within one patient.

Patient identity and correspondence. Our test-set audit identifies cross-slice separations of 1–21 slices and distinguishes same-case MRI contrasts from cross-case organ correspondence. Cross-case correspondence concerns the same anatomical concept, not the same lesion instance. Filename consistency checks support the released case/slice associations but do not independently verify patient identity or lesion-instance identity.

Ground truth and registration. Ground truth is inherited from source masks or boxes, or from construction coordinates for edited and cropped regions. We perform no additional registration. We will document source-specific alignment and pairing metadata rather than assume universal registration or lesion correspondence.

Clinical scope. Simulated difference pairs are not longitudinal examinations, so their results do not establish disease-progression assessment. We will add a task-to-source manifest specifying the construction, annotation source, and correspondence type, with unavailable metadata explicitly marked.

W3. Relationship to TumorChain

We agree that TumorChain is relevant and will discuss it explicitly. Both works construct CoT supervision, and TumorChain also includes localization; our distinction is therefore not simply “diagnosis versus grounding.”

TumorChain develops 3D CT tumour analysis across five digestive organs, with interleaved organ-level visual reasoning and report-derived reasoning annotations. Its CoT-construction process uses radiology and pathology text.

In contrast, MedChain generates image-conditioned, box-conditioned explanations of sequential grounding decisions across eight tasks and ten modalities. Its multimodal refiner directly inspects the image sequence rather than constructing explanations from clinical reports.

Our contribution is this task-specific supervision resource and its evaluation across model scales, reasoning interventions, and GRPO, not the general idea of training with CoT or a new optimization algorithm. We will revise the related-work discussion to make these distinctions explicit.

W4. Explanation supervision versus direct box optimization

To address this concern, we conducted an additional matched comparison, training box-only models on the same rows with the rationale removed while retaining the images, questions, boxes, and training settings. This comparison is distinct from the original fixed-checkpoint analysis.

The resulting effects are smaller and task-dependent. At 4B, explanation supervision improves registered difference, non-registered difference, object tracking, and cross-modal grounding by approximately +2.7, +1.4, +1.4, and +1.6 mIoU, respectively. These tasks also improve at 9B, whereas box-only supervision is competitive or better on multi-view, concept, and patch grounding. The overall differences are +0.61 at 4B and −0.48 at 9B.

Thus, the matched results support selective benefits, not a universal advantage of explanation supervision. The full task-wise comparison is provided in our response to Reviewer sfWq.

Target identification versus box precision. Our localization-error analysis distinguishes complete misses from box precision. On difference grounding, zero-overlap predictions remain below **3.5.

These patterns are consistent with changes in localization precision rather than uniformly improved target detection. However, nonzero overlap does not establish correct lesion classification, and conditional means can involve different subsets of examples. We therefore do not interpret this analysis as proof of improved disease understanding or faithful clinical reasoning.

Q1. Is FlemingVL-38B necessary?

Stage I supplies supplementary medical descriptions of modality, anatomy, and visible findings. Stage II re-examines the original images and uses these descriptions, where available, to construct task-specific comparisons. The intended division of labour is therefore medical description followed by multimodal reasoning refinement, rather than compensating for missing visual access.

FlemingVL-38B was selected through a qualitative comparison of six candidates on the same 100 examples under six prompt variants (Appendix A.1). This motivates its selection but does not establish that it is uniquely necessary.

We have not trained downstream models with alternative Stage-I teachers or performed a matched Stage-II-only comparison, so we cannot quantify those differences. The single-expert acceptance rate of **98.25.

We hope these responses address the reviewer's concerns. Please let us know if any questions or concerns remain; we would be happy to address them and provide further clarification during the remaining rebuttal period.

Could you elaborate on how this technically differs from distilling on sampled data, and highlight the key technical innovations?

Reviewer_sfWqOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies visual grounding in medical image sequences, including cross-slice tracking, difference grounding, multi-view matching, and cross-modal correspondence. Its central question is: Can explicit cross-image reasoning supervision improve localization over bounding-box-only supervision, and which tasks benefit? The authors propose MedChain-R1 and construct MedChain, containing 182,573 unique sample IDs, through a two-stage multimodal pipeline that generates clinical rationales conditioned on annotated boxes. They train 4B, 9B, and 27B models and further investigate the benefits of GRPO. The method is evaluated on eight tasks and 9,630 test examples from MedSG-Bench. Results show strong overall grounding performance, task-dependent benefits from reasoning supervision, and limited, unstable additional gains under the tested GRPO configurations.

The methodological novelty is limited, as the overall framework closely resembles existing reasoning-based multi-image grounding methods such as UniVG-R1. The main contribution lies in designing and constructing supervision data for cross-image medical reasoning and providing empirical analysis, rather than introducing new optimization algorithms or theoretical results.

Critical strengths

A useful data resource with strong benchmark results. MedChain covers eight tasks and ten imaging modalities. All three model sizes achieve strong overall results, with the 27B model reaching 77.24 mIoU and 85.20 [email protected]. The same-backbone 4B comparison and continued training of MedSeq-Grounder provide additional positive evidence. Although these comparisons do not fully isolate the contribution of reasoning supervision, they support the practical value of the training approach. The resource and empirical results are my main reasons for considering acceptance.

Careful analysis of gains, failures, and evaluation artifacts. The authors report tasks where reasoning supervision hurts and show that training gains do not always coincide with reliance on reasoning at inference time. In the 4B comparison, dCoT-SFT retains its advantage over answer-only SFT after excluding test examples with identified image or source-series overlap. A zero-learning-rate control also traces an apparent GRPO performance drop to preprocessing, providing a useful lesson for model evaluation.

Critical weaknesses

1. valuation is limited to a single benchmark.

Although MedSG-Bench covers eight tasks and ten imaging modalities, all reported benchmark results come from this evaluation framework. The paper therefore does not establish whether the gains from dCoT-SFT transfer to independent medical grounding datasets. This limits the evidence for the broader applicability of the proposed reasoning supervision, a limitation the authors also acknowledge in Appendix E.

2. The central training comparison does not fully isolate the benefit of reasoning supervision. Section 5.2 and Appendix E acknowledge differences between answer-only SFT and dCoT-SFT in effective batch size, optimizer implementation, and checkpoint selection. Neither run was repeated across training seeds. Equal optimization steps do not ensure equal data exposure or comparable optimization conditions. The reported +3.56 mIoU and paired confidence interval therefore establish a difference between two fixed checkpoints, but do not fully attribute it to reasoning supervision. The MedSeq-Grounder continuation experiment also lacks an equally trained answer-only continuation. The evidence is positive, but this attribution gap limits confidence in the paper’s central claim. A matched comparison on one backbone would address this concern more directly than adding further model sizes or benchmarks.

Questions for authors

Q1. Could you evaluate the existing answer-only SFT and dCoT-SFT checkpoints on at least one independent medical grounding benchmark with compatible inputs and tasks, without additional fine-tuning?

Comparing them under the same evaluation settings would help determine whether the benefit of reasoning supervision extends beyond MedSG-Bench.

Q2. Does the central result hold under matched training conditions?

Please specify all differences between answer-only SFT and dCoT-SFT, including effective batch size, actual sample exposure, optimizer implementation, and checkpoint selection. Do the overall improvement and task-level gains and losses persist when these conditions are matched? Please include repeated-run results if available.

Q3. Can you reconcile the discrepancies in Figures 7 and 9?

Some qualitative examples contain inconsistencies between the images, reasoning text, and captions.

  • Figure 9, page 36: The reference crop appears to show a relatively large lung region containing vascular markings, making its description as a pulmonary nodule unclear. The caption also claims that the method tightens the baseline box. However, the baseline predicts (44,99,158,250), while the proposed method predicts (44,99,158,254). The latter has the same width and a greater height, which does not support that description. These discrepancies weaken the use of this example to explain GRPO-related semantic degradation.
  • Figure 7, page 34: The example appears to involve different CT imaging planes but is labeled “Crossmodal Grounding”; the basis for this classification needs clarification. The caption describes the target image as axial, whereas the reasoning text calls it sagittal and interprets the trachea accordingly. These descriptions need to be reconciled.

Please verify the sample IDs, target annotations, model outputs, and captions. For Figure 9, explain the “pulmonary nodule” description and the claim that the predicted box is tightened. Please clarify whether the displayed rationales were independently checked for clinical correctness, separately from the MedChain training-data audit. For Figure 7, clarify the task category and the target image’s anatomical plane.

If the authors provide the experiments and results requested above, I would consider raising my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer sfWq

We thank the reviewer for the careful and fair review, and for recognising MedChain as a useful data resource with strong benchmark results, and our careful analysis of gains, failures and evaluation artifacts. To support the community, we will open-source everything: the MedChain dataset, all MedChain-R1 models (4B, 9B and 27B variants), the training code, and the Medical Modality Bridging algorithm code. We answer points below.

W1 / Q1. Evaluation on an independent benchmark

We searched for an independent benchmark with compatible inputs and outputs, that is, Sequential Medical Image Grounding and a question as input, and one bounding box as output. To the best of our knowledge, MedSG-Bench is the first and currently the only public benchmark of this kind.

For this reason our evaluation uses MedSG-Bench, which itself covers eight tasks and ten imaging modalities over 9,630 test questions. If the reviewer knows of a benchmark with inputs and outputs similar to MedSG-Bench, we would be grateful for the pointer.

W2 / Q2. Matched training conditions

The central result holds, and becomes stronger, when training conditions are matched. In the submission, the answer-only baseline used the same training samples as dCoT-SFT and differed only in settings forced by compute limits. None of these differences was tuned to favour our method:

SettingAnswer-only SFT (submitted)dCoT-SFT
Effective batch size32, because it was trained on fewer nodes64
Optimizer implementationFused AdamW kernel; the algorithm and hyperparameters are the sameStandard AdamW
Checkpoint selectionValidation every 250 stepsValidation every 500 steps

Validating the baseline every 250 steps gave it a finer choice of checkpoints. So these differences, if anything, favoured the baseline.

Matched re-run. We retrained the answer-only baseline with every one of these settings matched to dCoT-SFT: effective batch size 64, standard AdamW, the same training samples, and validation every 500 steps. Results are on all 9,630 MedSG-Bench test questions (Qwen3.5-4B; mIoU / [email protected], %):

Qwen3.5-4BRDGNRDGMVOTVCGVPGCMGRGOverall
Answer-only SFT (matched)67.56 / 80.9066.90 / 80.2055.11 / 62.2258.30 / 68.4068.03 / 77.6091.20 / 99.5079.99 / 85.1072.95 / 82.5070.96 / 80.01
dCoT-SFT82.86 / 93.3081.30 / 91.2054.01 / 58.9764.77 / 71.0070.64 / 78.8090.21 / 99.7083.39 / 87.0069.67 / 79.2874.96 / 82.37
Δ mIoU+15.30+14.40−1.10+6.47+2.61−0.99+3.40−3.28+4.00

Overall improvement. Under matched conditions the baseline falls slightly, from 71.39 / 80.31 to 70.96 / 80.01. The dCoT-SFT gain therefore grows from +3.56 to +4.00 mIoU and from +2.06 to +2.36 [email protected]. The original comparison did not overstate our improvement.

Task-level gains and losses. Every task keeps the sign of its Δ from the submission:

  • Difference grounding still gains most: RDG +15.30, NRDG +14.40. OT, CMG and VCG also still improve.
  • The three small losses remain: MV −1.10, VPG −0.99, RG −3.28.
    • The MV and VPG losses shrink, from −2.20 and −2.45.
    • VPG is at ceiling for both models ([email protected] ≥ 99.5).

We will replace the baseline in the paper with these matched results and list all training settings for both models in the appendix.

Q3. Figures 7 and 9

We thank the reviewer for examining these figures so carefully. We verified the sample IDs, target annotations, model outputs and captions of both examples:

Figure 7Figure 9
Task (verified)Multi-viewConcept
Sample IDmulti_view/0000095concept/0000013
Annotated targetbronchus (LUNA16)right lung (LUNA16)
Ground-truth box(140,16),(195,55)(44,99),(158,253)
Our box(140,16),(195,55), IoU 1.00(44,99),(158,254), IoU 0.99
MedSeq-Grounder boxIoU 0.94(44,99),(158,250), IoU 0.98

For Figure 7, the sample belongs to the multi-view task, and the reference and target images are coronal and sagittal views respectively, as the model's rationale correctly states; we thank the reviewer for noting this and have corrected the caption accordingly. For Figure 9, we checked both boxes carefully and have removed the word "tightened" from the caption. We have also added another qualitative example that directly shows the GRPO-related semantic drift. The rationales shown in the qualitative figures are model outputs and were not separately verified by the experts who audited the MedChain training rationales.


We hope these answers address the reviewer's concerns. We are grateful to the reviewer for reading our paper so carefully and for the valuable suggestions. If any concerns, remaing We would be happy to answer any further questions.

Your responses address my other concerns, but external generalization (W1/Q1) remains untested.

I understand that an exact counterpart to MedSG-Bench may not exist. However, my concern is whether the gains from reasoning supervision transfer beyond MedSG, rather than whether another benchmark covers all eight tasks or requires multiple images. A single-image grounding benchmark can still test whether the improvements extend beyond MedSG’s task construction and data distribution. This evaluation does not require annotated reasoning chains; images, queries, and target boxes are sufficient. UniVG-R1 provides a relevant precedent through its zero-shot evaluations on LISA, LLMSeg, ReVOS, and ReasonVOS, using boxes derived from segmentation annotations.

Could you compare the matched answer-only and dCoT models on MeCoVQA-G+, after checking for overlap with the training data? The target masks could be converted into bounding boxes for evaluation using mIoU and [email protected]. This would provide useful evidence of transfer in medical grounding.

Thank you for this suggestion. We compared the matched answer-only SFT and dCoT-SFT models, Qwen3.5-4B, on MeCoVQA-G+, without any training on it.

Overlap check. The MeCoVQA-G+ test set has 3,157 questions, and their IDs name 36 source datasets. We compared these with the source datasets of our training data (MedSG). 30 of the 36 are also used in the training data, either directly or through a dataset built from them (e.g., KiTS and MSD Liver are part of AbdomenCT-1K). To avoid any overlap, we left out all questions from these 30 datasets. We kept only the questions from the 6 datasets that our models have never seen: autoPET, TotalSegmentator, Ultrasound Nerve Segmentation, FUSC2021, MSD Hepatic Vessel, and BTCV Cervix. This leaves 1,092 questions in 4 modalities, including PET, a modality that does not appear in MedSG at all.

Setup. Each target mask was converted into the tightest bounding box around it. We used the same evaluation script, prompt and greedy decoding as on MedSG-Bench

Results (1,092 questions from unseen source datasets; mIoU / [email protected], %; number of questions in brackets):

Qwen3.5-4BPET (755)CT (117)Dermoscopy (51)Ultrasound (169)Overall (1,092)
Answer-only SFT (matched)18.77 / 16.9116.77 / 17.6649.46 / 50.334.66 / 0.0017.80 / 15.93
dCoT-SFT (matched)26.86 / 28.1221.59 / 22.7956.86 / 59.485.72 / 0.3924.43 / 24.73
Δ mIoU+8.09+4.82+7.40+1.05+6.62
  • dCoT-SFT transfers better. Overall, it improves by +6.62 mIoU and +8.79 [email protected].
  • It is ahead in modality. The gain is good on PET (+8.09 mIoU, +11.21 [email protected]), a modality never seen in training.

This was a very useful suggestion. It shows that the gain from reasoning supervision is not limited to MedSG: it transfers to an independent medical grounding benchmark on datasets and a modality the models have never seen. We will add this experiment and the overlap check to the appendix.

Reviewer_5YWSOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes MedChain-R1 for sequential medical image grounding. The authors construct MedChain, a dataset of 182K clinical rationales generated through a two-stage multimodal pipeline, and fine-tune Qwen3.5 models using CoT supervision and optional GRPO. Experiments on MedSG-Bench demonstrate state-of-the-art performance, with additional analyses of task-dependent effects and the benefits of reasoning supervision and GRPO.

Critical strengths
  1. Strong empirical performance. The proposed method achieves state-of-the-art results on MedSG-Bench, improving over the strongest prior model by 4.69 mIoU and 5.49 [email protected] points.

  2. Extensive experimental evaluation. The paper evaluates multiple model scales and provides comparisons between answer-only and CoT supervision, along with task-wise analyses and investigations of GRPO.

Critical weaknesses

W1. Insufficient motivation and validation of the proposed two-stage rationale generation pipeline.

Although the paper is generally understandable, the motivation for several key design choices is not sufficiently established. In particular, the authors use FlemingVL-38B to generate preliminary medical descriptions and Qwen3.5-397B-A17B to refine them into clinical reasoning. However, since the stage-II model already has access to the original medical images, questions, and ground-truth bounding boxes, it is confusing why the first stage is necessary.

Furthermore, the model selection for both stages is insufficiently justified. Although Appendix A.1 mentions evaluating six candidate models for Stage I, Table 4 only lists the candidates without reporting their evaluation results. Similarly, no comparison or justification is provided for selecting Qwen3.5-397B-A17B as the Stage II refiner. These missing analyses make it difficult to assess the necessity and effectiveness of the proposed design.

W2. Extensive experiments but limited explanatory analysis.

The paper presents extensive experimental results, but much of the analysis remains descriptive rather than providing deeper insights into why the proposed method works. Making it more like an experimental report instead of a research paper.

For example, the authors observe that CoT supervision benefits difference grounding and object tracking but hurts multi-view, patch, and referring grounding. They suggest that reasoning is particularly useful for tasks requiring cross-image comparison. However, this explanation does not account for the performance degradation on multi-view grounding, which also requires cross-image comparison. Although the paper acknowledges this inconsistency, there are no further experiments conducted to investigate its cause.

Similarly, the authors observe that GRPO provides only marginal improvements after dCoT-SFT and discuss several possible explanations, including insufficient reward variance, task imbalance, and semantic drift. However, these explanations are not convincingly validated through controlled experiments.

Consequently, despite the extensive experimental evaluation, the paper provides limited understanding of the mechanisms underlying the observed improvements and performance degradation. In fact, from my perspective, systematically investigating why CoT supervision degrades performance on certain tasks and why GRPO provides limited additional benefits could constitute a more interesting and informative research contribution than simply reporting these observations. As it stands, the findings remain largely empirical observations rather than generalizable insights into reasoning supervision for sequential medical grounding.

W3. Limited contribution and novelty.

I acknowledge that the main contributions of this paper are the proposed MedChain dataset and the empirical analysis of CoT supervision for sequential medical grounding. Beyond these two aspects, I do not identify any substantial methodological or theoretical contribution. However, as discussed in W1 and W2, neither contribution is sufficiently substantiated. The dataset is constructed using existing multimodal models, while the necessity and effectiveness of its key design choices remain unclear. Meanwhile, the extensive experimental analysis mainly reports empirical observations without providing deeper explanations or generalizable insights.

Consequently, although the reported performance improvements are encouraging, the paper reads more like an experimental report than a research work with a clearly motivated methodology and substantial scientific insights. This limits my assessment of its novelty and overall contribution to a top AI conference.

Questions for authors

Why is Figure 1 never explicitly referenced or discussed in the main text?

13 comments
AuthorsOpenReview ↗

1/3 Response to Reviewer 5YWS

We sincerely thank the reviewer for their thoughtful assessment, constructive feedback, and recognition of the strong empirical results and comprehensive experimental evaluation. We appreciate the concerns regarding the rationale behind our two-stage pipeline, and the novelty and scientific significance of our contributions. In the following response, we provide further methodological justification, clarify the underlying reasons for the observed performance differences, and strengthen the discussion of our key contributions and insights. We address each concern in detail below.

W1. Why two stages, and how the models were chosen

Why Stage I is needed. We agree that Stage II also sees the images, the question and the box. The difference is what each model is good at (§3.1; App. D.1).

  • Stage I is a medical model (Clinical understanding and semantic grounding). Stage I leverages a medically specialized VLM to identify imaging modalities, anatomical structures, and clinically relevant findings using appropriate medical terminology. It describes each image on its own and rarely says how the images relate (§3.1).
  • Stage II is a general corss image reasoning model. It reasons well step by step, but it is not trained for medicine. In our comparison of candidate models, the general models (Qwen3-VL and GPT-4o) were less reliable on medical content than FlemingVL-38B.

Stage II therefore receives the Stage I medical description, where available, together with the images, looks at the images again, and writes the comparison across images that ends at the annotated box. Stage I gives the medical content; Stage II turns it into step by step reasoning. The quality of the final rationales supports this design: a medical expert accepted 98.25% of 2,000 sampled rationales (95% CI 97.58–98.74; App. A.4).

How the Stage I model was chosen. Finding a model with real medical expertise for this problem is hard, because not every model uses correct terminology across ten imaging modalities. We tested six strong and relevant models that accept multi-image input and can write the kind of output we need (Table 4): two general models, three medical models and the task-specific MedSeq-Grounder. Each model was run on the same 100 examples with six different prompts. We manually evaluated and analyzed the outputs and found that FlemingVL-38B gave more accurate and more clinically faithful answers than the other models, across all prompts. The revised App. A.1 will state clearly that this was a qualitative comparison made by manually comparing the outputs, which is why Table 4 lists the candidates without scores.

How the Stage II model was chosen. Stage II needs a model that can read several images and reason step by step. We selected Qwen3.5-397B-A17B because it was the strongest general multimodal reasoning model available, which can do step-by-step cognitive reasoning like us for this specific type of sequential grounding.

AuthorsOpenReview ↗

2/3 Response to Reviewer 5YWS

W2. Depth of the analysis

We agree that sequential medical grounding is a new setting. Before this work it was not known whether reasoning supervision helps here, on which tasks, or whether reinforcement learning adds anything on top. We believe a careful map of where it helps and where it does not, with confidence intervals and with the negative results reported, is valuable to the community and saves others from repeating these runs. We agree that such a map is stronger with explanations, and we ran targeted analyses for the two cases the reviewer raises.

(a) Why reasoning hurts multi-view grounding. We split every multi-view prediction of the matched 4B models (see our reply to Reviewer sfWq) into miss (no overlap), loose box (overlap, IoU < 0.5) and hit.

  • Misses do not increase: 12.6% → 10.2%.
  • Loose boxes increase: 26.1% → 29.6%; the mean IoU of overlapping boxes drops 64.5 → 60.3.

So reasoning still finds the right structure in the new view; it draws its extent less exactly. The same structure changes shape and size across views, and a rationale that identifies which structure does not say how far it extends. Reasoning helps to find the structure, not to draw its extent. This is also why GRPO, whose IoU reward directly rewards tighter boxes, gives its clearest gain on multi-view (+1.11 at 9B; Table 19).

(b) Why GRPO adds little. We tested three explanations.

  • Low reward contrast: the best-supported cause. In the logged training rollouts (three seeds, 2,048 groups of eight answers each), all eight answers are already correct in about 72% of groups and all eight are wrong in about 7%. Only about 21% of groups contain both correct and incorrect answers, so in most groups the rewards are close (mean IoU spread 0.07) and the learning signal is weak. Better answers do exist (on 128 hard prompts, pass@1 is 67.7% but pass@8 is 84.4%), but too few groups contain them. Multi-view has the most mixed groups (35%), and it is the task GRPO helps most.
  • Task imbalance: not the cause. A task-balanced, threshold-aware reward gives +0.36 [−0.16, 0.89], which is not significant (Table 21).
  • Semantic drift: a hypothesis. At 4B, GRPO lowers concept grounding by 4.55 mIoU and 6.70 [email protected], while most other tasks improve (Fig. 3). An IoU reward credits overlap whether or not the model identifies the correct finding, which matters most for concept grounding, where the same lesion or structure must be recognised across very different images (§5.3). At 9B, concept grounding is unchanged (+0.17 mIoU, p = 0.74; Table 19). As stated in §5.3, we do not isolate this mechanism.

The gain is small but stable: +0.37 [0.05, 0.69] at 4B, +0.30 at 9B, and SD 0.08 across three seeds (Tables 3, 19 and 20).

AuthorsOpenReview ↗

3/3 Response to Reviewer 5YWS

W3. Contribution and novelty

Our contribution is fourfold, and aims to make reasoning supervision possible in a domain where it did not exist, and to measure carefully what it gives.

  1. A method to generate reasoning for a hard domain (Algorithm 1). Writing a correct chain of thought over several CT, MRI or ultrasound images is difficult: medical models describe images but do not reason across them, and general models reason but are less reliable on medical content. Our pipeline, medical modality bridging, combines the two, keeps only rationales that end at the annotated box, and is audited by a medical expert on 2,000 samples. It is reproducible and can be applied to other multi-image medical tasks.
  2. A new dataset. MedChain contains 182,573 rationales across eight tasks and ten modalities, with an expert audit and a train–test overlap audit (App. A.4–A.5). To our knowledge it is the first reasoning corpus for sequential medical grounding.
  3. Strong models at three sizes. We train 4B, 9B and 27B models. All three outperform the strongest prior model, and the 27B model reaches 77.24 mIoU and 85.20 [email protected], an improvement of 4.69 and 5.49 points. The same supervision also improves the prior model's own backbone (72.55 → 75.71 mIoU; App. C.2).
  4. Experimental insights. In our 4B comparison, reasoning supervision improves overall performance over answer-only SFT (+3.56 mIoU; Table 2). GRPO adds a small further improvement (+0.37 mIoU; Table 3), though not on every task. Reasoning supervision is not uniformly helpful: it helps small, comparison-based targets and costs box precision on multi-view. The model really uses its reasoning: replacing it with a rationale from another task costs 12.18 mIoU (Table 2). For future work, we show that better answers are already among the model's samples (a single sample is correct 67.7% of the time, but one of eight is correct 84.4% of the time), so the natural next step is a reward that prefers them, for example one that also scores semantic correctness (§5.3; App. D.5).

Together, the method, the dataset, the models and the analysis give later work a starting point that did not exist before. We will release the data, code, data generation pipeline, and checkpoints open source.

Q1. Figure 1 is not referenced

Thank you for catching this; we apologise for the oversight. The revised §1 will cite panel (a) where the eight tasks are introduced, panel (b) where we explain why box-only supervision leaves the reasoning implicit, and panel (c) where the main results are summarised.

We hope these answers address the reviewer's concerns, and we would be happy to answer any further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of the discussion

We thank the Area Chair, the Senior Area Chair and all three reviewers for their time and constructive engagement. We summarise the strengths the reviewers identified, how we addressed each concern, and the changes in the revised PDF (all changes highlighted in yellow).

Strengths noted by the reviewers

ReviewerStrengths noted
sfWq"A useful data resource with strong benchmark results"; "the resource and empirical results are my main reasons for considering acceptance"; "careful analysis of gains, failures, and evaluation artifacts"; presentation rated "clear and easy to follow"
5YWS"Strong empirical performance" (state of the art on MedSG-Bench, +4.69 mIoU and +5.49 [email protected] over the strongest prior model); "extensive experimental evaluation" across model scales, answer-only versus CoT supervision, task-wise analyses and GRPO; the new transfer experiment "provides useful evidence of improved generalization"
o7aR"The motivation is reasonable": box-conditioned explanations "could help the model learn interpretable visual grounding"; evaluation "across multiple tasks and imaging modalities, providing evidence of its applicability"

Concerns and how we addressed them

ReviewerConcernResponseStatus
sfWqBenefit of reasoning not isolated from training differencesMatched answer-only baseline: +4.00 mIoU, +2.36 [email protected]; every task keeps its sign (Table 2, App. C.8)Addressed; score raised from 2 to 3
sfWqEvaluation on a single benchmarkZero-shot transfer to MeCoVQA-G+ on 1,092 questions from unseen sources: +6.62 mIoU, +8.79 [email protected], largest gain on PET (App. C.9)Addressed
sfWqFigures 7 and 9Captions corrected; new concept-grounding example after GRPO (Figure 18)Addressed
5YWSW1: necessity of the two-stage pipelinePaired Stage I + Stage II versus Stage II-only examples (App. A.11, Figures 4–11)Resolved (confirmed by reviewer)
5YWSW3: contribution and Algorithm 1Contributions merged; Algorithm 1 fully specified with fixed filter rules (App. A.3)Resolved (confirmed by reviewer)
5YWSW2: mechanisms behind task-dependent effectsReasoning controls show that writing the reasoning helps on every task, including multi-view; the multi-view loss at 4B disappears with scale; rationale and box correctness compared (App. D, Figure 18, Case Study 5)Reviewer acknowledged the strengthened evaluation; systematic failure-category analysis will be added
5YWSQ1: Figure 1 not referencedAll three panels now referenced in §1Addressed
o7aRTest-set, box and target distributionsTest-set composition and lesion categories (App. A.10)Addressed
o7aRImage sources, construction and ground truthTask construction and correspondence table; difference pairs described as synthetic, not longitudinal (App. A.10)Addressed
o7aRRelation to TumorChainNew discussion (Extended Related Work)Addressed
o7aRExplanation versus box-only supervisionMatched box-only baseline: +4.00 mIoUAddressed
o7aRNecessity of FlemingVL-38B; difference from distillationStage I versus Stage II-only examples (App. A.11); relation to distillation (App. A.9)Addressed

Contribution to the community To our knowledge, MedChain is the first reasoning corpus for sequential medical grounding: 182,573 rationales across eight tasks and ten imaging modalities, with an expert audit and a train–test overlap audit. With it, dCoT-SFT sets a new state of the art at 4B, 9B and 27B, improves over a matched answer-only model, and transfers to an unseen benchmark. The controlled analysis maps where reasoning supervision helps and why its effect differs across tasks. We will release the MedChain dataset, the MedChain-R1 checkpoints at 4B, 9B and 27B, the training code and the rationale-generation code, so that others can build on this resource.

Commitments for the final version

  • A systematic failure analysis that labels failures as anatomical identification, cross-image correspondence or box estimation, linked to rationale correctness, as suggested by Reviewer 5YWS.
  • A task-to-source manifest for MedSG-Bench, as promised to Reviewer o7aR.

We are grateful to the reviewers, whose suggestions have substantially strengthened the paper.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.