acceptodds
Under review as a conference paper at ICLR 2027

MedInSight: Active Cross-Image Evidence Acquisition for Clinical Diagnosis

26 views

Abstract

Clinical diagnosis from heterogeneous medical images requires finding and integrating visual evidence across multiple images. Existing methods lack the ability to actively acquire and align distributed visual evidence, leading to missed subtle visual findings or incorrect cross-image relationships, thereby degrading overall performance. To address these challenges, we propose MedInSight, an end-to-end agentic framework for active cross-image clinical reasoning. Central to MedInSight is Cross-image Evidence Routing (CER), which leverages predictive uncertainty in per-image grounding and spatial uncertainty across images for targeted cross-image inspection. To optimize these visual interactions, we further propose Cross-horizon Credit Assignment (CCA), which combines short-horizon action-level credit magnitude with long-horizon stage-level direction for fine-grained credit assignment beyond uniform outcome-based advantages. Together, CER and CCA couple targeted cross-image evidence acquisition with fine-grained policy optimization, enabling progressive integration of distributed visual evidence for clinical diagnosis. We instill these capabilities through a two-stage training pipeline consisting of supervised fine-tuning (SFT) followed by reinforcement learning (RL). Extensive experiments demonstrate that MedInSight achieves state-of-the-art performance across medical diagnosis and reasoning benchmarks, improving its base model, Qwen3-VL-8B, by 22.96 points on average with consistent gains on out-of-domain datasets. Codes and checkpoints will be released.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Clinical Diagnosis, Multi-image Reasoning, Multimodal Medical Agents

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_xo1zOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies diagnostic reasoning across multiple medical images. The proposed framework, MedInSight, uses Cross-image Evidence Routing (CER) to select images for further inspection. It also uses Cross-horizon Credit Assignment (CCA) to refine the training signal for individual inspection actions. The authors evaluate the model on six medical benchmarks, with ablations, human assessments, and case studies. The results suggest that targeted evidence selection and finer-grained training feedback can improve diagnostic performance.

Critical strengths

1.The method addresses two practical questions in multi-image diagnosis: what to inspect next, and how to learn useful inspection behavior. CER and CCA address these questions in a coherent way. The appendix also includes a control that gives the model access to all images. That comparison supports the value of targeted evidence selection. 2.The ablation results are consistent across benchmarks. With the same backbone, the full method outperforms both supervised fine-tuning and vanilla GRPO on all three ablation benchmarks. The gains over supervised fine-tuning are about 11 percentage points. Comparisons against random selection and access to all images provide further support. These results are the main reason I lean toward acceptance.

Critical weaknesses

1.The performance gains from CCA are clearer than the evidence for why it works. The long-horizon ablation also changes the strength of the correction, making it hard to isolate the role of action weighting. Shuffled CCA is a useful additional control. However, it is evaluated on only 57 cases, where the full method answers three more cases correctly. Figure 9 also needs a clear account of how the analyzed trajectories were selected. Without that detail, it is hard to assess possible selection effects. 2.Some implementation and evaluation details need to be reconciled. Table 2 and Figure 3 describe the short-horizon variant differently. The loss formula in the main text also differs from the DAPO aggregation described in the appendix. In addition, MMRareBench uses open-ended questions, while the main prompt relies on candidate diagnoses. The authors should provide the prompt used for this evaluation. These details matter for reproducibility and interpretation.

Questions for authors

I would appreciate clarification on the following points. 1.For Figure 9, how was each case’s analyzed trajectory selected from its rollout group? Did it have to produce a correct answer or have a positive advantage? 2.Do Table 2 and Figure 3 use the same short-horizon variant? Please provide the exact definition. Also, was the total correction magnitude matched in the long-horizon ablation? 3.Does the implementation average the loss within each trajectory first, or across all valid tokens? 4.What inference prompt was used for MMRareBench T1? How was the candidate-support readout handled when no candidate diagnoses were provided?

10 comments
AuthorsOpenReview ↗

Response to Reviewer xo1z (1/2)

We sincerely thank the reviewer for the detailed and insightful feedback. The comments have been invaluable in helping us clarify our contributions and strengthen our paper. We have conducted several new experiments and revised the manuscript to address all the points raised. Below, we provide a point-by-point response.


Response to W1.1, Q2.2: Correction magnitude in the long-horizon ablation

We thank the reviewer for raising the question about the long-horizon variant. Yes, the total correction magnitude is matched. We acknowledge that the equation in line 426 omits the normalization factor used in our experiments. Actually, in the long-horizon-only ablation, we also normalize the action-level magnitude over the trajectory. We correct its definition to κt=dk(t),g(t)L∣T∣\kappa_t=\frac{d_{k(t),g(t)}^{\mathcal{L}}}{|\mathcal{T}|}, where ∣T∣|\mathcal{T}| is the number of visual inspection actions in the trajectory defined in Section 3.3. Compared with full Cross-horizon Credit Assignment (CCA), the long-horizon-only ablation replaces the normalized action-specific weights wtSw_t^{\mathcal{S}} with uniform weights 1/∣T∣1/|\mathcal{T}|, while retaining the same stage-level direction. Since both sets of weights sum to one over the trajectory, the two variants distribute the same total correction budget across actions, so their total correction magnitudes are matched. We correct Line 426 to "The long-horizon-only variant uses κt=dk(t),g(t)L/∣T∣\kappa_t=d_{k(t),g(t)}^{\mathcal{L}}/|\mathcal{T}|, retaining stage-level direction with a uniform action-level magnitude." The revision is highlighted in blue in the revised manuscript in Section4.3 [Line 427–428].

Response to W1.2: Sample size of the Shuffled CCA

We thank the reviewer for this constructive suggestion. We agree that a larger evaluation set is necessary for a reliable comparison, and we expand the evaluation accordingly. Specifically, we collect all test cases containing exactly four images from four of our evaluation benchmarks, yielding 407 cases in total (72 from MedThinkVQA, 256 from MedFrameQA, 22 from MedXQA-Multi, and 57 from our held-out test set Dtest\mathcal{D}_{\mathrm{test}}), denoted as Dcollect\mathcal{D}_{\mathrm{collect}}. The diagnostic accuracy (%) of the three variants is reported below.

MethodDtest\mathcal{D}_{\mathrm{test}} subset (57 cases)Dcollect\mathcal{D}_{\mathrm{collect}} (407 cases)
Vanilla GRPO29.8242.75
Shuffled CCA36.8447.42
Full CCA (MedInSight)42.1155.53

On the 407-case Dcollect\mathcal{D}_{\mathrm{collect}}, full CCA achieves 55.53% accuracy, outperforming Shuffled CCA (47.42%) and vanilla GRPO (42.75%) by 8.11 and 12.78 percentage points, respectively. Compared with Shuffled CCA, this corresponds to 33 additional correctly answered cases, rather than the 3-case difference on the original 57-case subset. Since Shuffled CCA preserves the total correction magnitude and stage-level direction while permuting its assignment across actions, this larger-scale comparison further supports that the gains of CCA come from assigning credit to the appropriate inspection actions, rather than from the additional shaping signal itself.

We have added these results to Appendix B.4 in the revised manuscript.

Response to W1.3, Q1: Trajectory selection for Figure 9

We thank the reviewer for this thoughtful question. For Figure 9, we analyze trajectories that reach the correct diagnosis and examine how often their visual inspection actions provide evidence that helps distinguish candidate diagnoses. The rationale for restricting this evaluation to correctly answered cases is to directly verify whether MedInSight genuinely localizes diagnostically decisive regions when arriving at a correct diagnosis. We have clarified this selection strategy and its rationale in Appendix B.4 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer xo1z (2/2)

Response to W2.1, Q2.1: Definition of the short-horizon variant

Thank you for pointing out this inconsistency. Table 2 and Figure 3 use the same short-horizon variant. In both cases, it uses the local transition sign as its credit direction, replacing the stage-level direction while retaining the same action-level weights. Its exact definition is κt=sign(δtS)wtA\kappa_t = \mathrm{sign}(\delta_t^{\mathcal{S}}) w_t^{\mathcal{A}}. Full CCA and this variant therefore share the same action-level magnitude weights wtSw_t^{\mathcal{S}} and differ only in whether the credit direction is given by the stage-level direction dk(t),g(t)Ld_{k(t),g(t)}^{\mathcal{L}} or the local transition sign sign(δtS)\mathrm{sign}(\delta_t^{\mathcal{S}}). For clarification, we have made the following revisions in Section 4.3 of the revised manuscript:

  • Lines 428-429 (ablation study, Table 2): We correct the definition to "The short-horizon-only variant uses κt=sign(δtS)wtA\kappa_t=\mathrm{sign}(\delta_t^{\mathcal{S}}) w_t^{\mathcal{A}}, retaining the action-level magnitude but replacing the stage-level direction with the local transition direction."

  • Line 468 (training dynamics, Figure 3): We clarify that this variant is the same as in Table 2: "Here, the short-horizon variant is the same as in Table 2, using each local transition sign as its credit direction."

All revisions are highlighted in blue in the revised manuscript in Section4.3 [Line 428–429].

Response to W2.2, Q3: Loss aggregation

We thank the reviewer for this important question. We double-check our training configuration and confirm that our implementation retains GRPO-style trajectory-level loss aggregation. Specifically, we first compute each token’s policy loss using its assigned advantage, including the CCA credit adjustment where applicable. We then average these token-level losses over all unmasked tokens within each trajectory, followed by averaging the resulting values across all trajectories in the generated group. The phrase “DAPO’s token-level aggregation” in Appendix D.1 is inaccurate. We have revised it to: “The policy loss follows the trajectory-level loss aggregation of GRPO alongside the Clip-Higher mechanism from DAPO …"

This revision is highlighted in blue in Appendix D.1, the main-text formula and appendix description are now consistent with the implementation.

Response to W2.3, Q4: MMRareBench T1 inference prompt

Thank you very much for pointing out this omission. The complete MMRareBench T1 inference prompts are now provided in Figures 16 and 17 in Appendix E.2. Specifically, given the input query and heterogeneous images, MedInSight first proposes four or five differential diagnoses and assigns each a fixed label (A–D or A–E). These labels stay fixed across the subsequent reasoning stages. The candidate-support readout is therefore reported over these self-generated candidates: after each inspection and stage reflection, the model reports an updated confidence score for each differential diagnosis option, and the final answer is the best-supported diagnosis.

To quantify the limitation of a fixed candidate set, we further measure initial candidate coverage on MMRareBench T1. The ground-truth diagnosis appears among the initial candidates in 92.62% of cases. Manual analysis of the remaining 7.38% of cases suggests that these candidate omissions primarily reflect gaps in the model’s knowledge of extremely rare diseases.

Our motivation for including MMRareBench is to evaluate whether our method can generalize beyond multiple-choice VQA to free-text diagnosis without externally provided answer options. We also add this motivation and the corresponding analysis to Section 4.2 [Line 374–377]. All revisions are highlighted in blue in the revised manuscript.


Thank you for your valuable comment again, we hope our responses can address your concern and we look forward to your further reply!

Thanks for the clarifications and the extra experiments. The new results have nudged me toward acceptance. I’d now put my assessment at around 2.5/4, though I’m still on the fence. Two points still give me pause: whether the correction magnitudes are matched, and how trajectories were selected for Figure 9. Clearing those up would help me settle on a final recommendation.

Reviewer_fxb9OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes an end-to-end agent framework called MedInSight for clinical diagnosis and reasoning using heterogeneous medical images. The aim is to address the challenges faced by existing medical visual language models in actively acquiring, associating, and integrating diagnostic evidence scattered across different medical images. The authors propose a cross-image evidence routing mechanism that combines image-level prediction uncertainty with cross-image spatial uncertainty to guide the model in targeted exploration and acquisition of visual evidence. It demonstrates good performance and generalization ability in six benchmark tests covering case-level diagnosis and multi-image medical reasoning.

Critical strengths

This paper models cross-image diagnosis as a reasoning process that actively acquires and progressively integrates distributed visual evidence. This research perspective aligns closely with the practical needs of clinical diagnosis, which requires repeated examination of different images, localization of key signs, and comprehensive judgment. It attempts to integrate cross-image evidence acquisition and fine-grained policy learning into a unified framework, providing a valuable technical path for heterogeneous image reasoning in complex medical scenarios.

This paper conducts extensive experiments on six medical diagnostic and reasoning benchmark datasets and compares them with general closed-source models, open-source visual language models, and medical-specific models. The results show that it has good task adaptability and achieves significant performance improvements on multiple cross-domain benchmark datasets.

Critical weaknesses

CER utilizes prediction and spatial uncertainty to guide image exploration, but regions with higher uncertainty do not necessarily have higher clinical diagnostic value. Could the authors further explain how this mechanism distinguishes truly diagnostically significant abnormal signals from image noise, artifacts, or irrelevant regions? The authors primarily use final diagnostic accuracy to demonstrate the usefulness of CER, but fail to provide evidence-level evaluations, such as whether the selected images and regions of interest truly correspond to key lesions, or present typical failure cases.

Questions for authors

Figure 4 shows that increasing the inference budget does not always consistently improve diagnostic accuracy, indicating that the active evidence integration process may introduce redundant interactions, accumulation of erroneous evidence, or excessively long inference chains. The authors do not explain the mechanism of this performance degradation, nor do they explain whether MedInSight still outperforms Qwen3-VL-8B and other multi-image inference methods under the same inference token budget or computational cost. Under the same inference token budget or computational cost constraints, does MedInSight still have a significant advantage compared to the base model Qwen3-VL-8B and other methods with multi-image inference capabilities?

2 comments
AuthorsOpenReview ↗

Response to Reviewer fxb9 (1/2)

We sincerely thank the reviewer for the insightful feedback. Your comments raise important considerations from a real-world clinical perspective that have prompted us to further strengthen the clinical grounding of our work. We have conducted several additional experiments and revised the manuscript accordingly to address all the points raised. Below, we provide a point-by-point response.


Response to Q1: Clinical validity of localized visual evidence

We sincerely thank the reviewer for your insightful questions regarding the role of Cross-image Evidence Routing (CER) in our framework. First, we would like to clarify a potential misunderstanding regarding the role of CER. The predictive uncertainty in CER is not designed to estimate the diagnostic value of a region. Rather, diagnostic valuable inspection behavior is learned autonomously by the agent through on-policy RL with Cross-horizon Credit Assignment (CCA). The primary motivation for designing CER is to identify uncertain grounding decisions that warrant further examination and guide the agent to cross-check its visual findings by jointly inspecting heterogeneous images from different modalities, views, or sequences. In summary, this mechanism uses the predictive uncertainty to guide targeted cross-image inspection. Table 4 in Appendix B.2 supports the effectiveness of this design. Replacing CER with all-image inspection, random selection, or low-uncertainty anchor selection consistently degrades performance. This indicates that CER serves as an effective starting point for targeted cross-image inspection.

CCA shapes the inspection policy through fine-grained advantage adjustments based on changes in the confidence margin with respect to the ground-truth diagnosis. Inspecting noise, artifacts, or irrelevant regions does not increase the margin toward the correct diagnosis and therefore receives little or negative credit, whereas inspections that reveal diagnostically relevant evidence receive positive credit. Over training, this pushes the policy toward regions that actually support the diagnosis rather than regions that are merely uncertain.

To further evaluate whether on-policy RL with CCA enables the model to localize diagnostically significant regions beyond improving final-answer accuracy, we conduct two evaluations comparing MedInSight with its base model, Qwen3-VL-8B-Instruct. (i) Grounding accuracy: on the Cross-modal Grounding (CMG) and Multi-view Grounding (MV) tasks of MedSG-Bench [1], we report the mIoU and [email protected] between the predicted boxes and the ground-truth annotations. (ii) Clinical validity: we recruit three licensed physicians from two tertiary hospitals to assess the inspection trajectories on 856 cases randomly sampled from the six evaluation benchmarks. To isolate the quality of the inspection behavior from answer correctness, we only include cases where both models produce the correct diagnosis. A trajectory is judged clinically valid if its inspected regions cover the key abnormal findings required for the diagnosis. Clinical validity is reported as mean ± std across the three physicians, and grounding metrics as mean ± std over five runs.

MetricQwen3-VL-8BMedInSight (Ours)
Clinical case-level validity (%) ↑64.06 ± 0.6692.56 ± 0.47
CMG mIoU ↑19.89 ± 2.4362.38 ± 1.47
CMG [email protected] ↑7.15 ± 1.5764.75 ± 1.82
MV mIoU ↑10.01 ± 1.0845.70 ± 1.63
MV [email protected] ↑2.87 ± 0.9650.35 ± 1.39

As shown, MedInSight significantly improves grounding accuracy in both tasks (e.g., CMG [email protected] from 7.15 to 64.75 and MV [email protected] from 2.87 to 50.35), demonstrating its superior ability to precisely locate target regions across different modalities and views. More importantly, even in cases where both models arrive at the correct diagnosis, physicians assess 92.56% of MedInSight’s trajectories as clinically valid, compared to only 64.06% for the baseline model. This suggests that MedInSight’s examination process is grounded in regions of diagnostic significance. We have included these results in Appendix B.5 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer fxb9 (2/2)

Response to Q2.1: Performance degradation under larger inference budgets

We thank the reviewer for raising this insightful question. As discussed in Section 4.3 (Lines 480–483), the performance degradation beyond 12 actions in Figure 4 mainly stems from redundant visual interactions. Our evaluation also shows that errors associated with redundant visual evidence always occur in excessively long reasoning trajectories. However, erroneous evidence accumulation is not specific to high action budgets, it can also arise during ordinary inference, as illustrated in Figure 22.

In clinical practice, abnormal findings rarely occur in isolation within a patient: multiple lesions often coexist, yet not all of them are related to the primary condition. Moreover, each lesion may present non-specific features that can be plausibly linked to the patient's symptoms. As a result, excessive exploration tends to introduce such findings and give rise to more competing diagnostic hypotheses during reasoning. As these findings accumulate, the most specific evidence chain supporting the correct diagnosis may be diluted, making it harder for the model to commit to the best-supported diagnosis.

To better illustrate and support this explanation, we additionally provide a case study in Appendix H (Figure 23). In this case, the model initially identifies the key inflammatory findings and is able to reach the correct diagnosis. With a larger inspection budget, however, it repeatedly revisits already examined regions and focuses on additional, nonspecific findings that can be plausibly linked to the patient’s symptoms. As this redundant visual evidence accumulates, competing diagnostic hypotheses emerge and gradually distract the model from the most specific evidence chain, causing its reasoning to drift away from the correct diagnosis.

Response to Q2.2: Efficiency comparison with more models

We thank the reviewer for this important practical consideration. To examine whether MedInSight's advantage persists under matched inference budgets, we evaluate MedInSight, the base model Qwen3-VL-8B-Instruct, and the multi-image agentic method MedAgent-Pro [2] under the same action budgets at∈{8,12,16}a_t \in \lbrace 8, 12, 16 \rbrace. For each setting, we report accuracy, output textual tokens, end-to-end latency, and the average number of turns per case. The results are presented in the table below:

Methodata_tAccuracy (%) ↑Output Textual Tokens / CaseLatency / Case (s)Avg. Turns / CaseΔAcc / 100 Tokens ↑
Qwen3-VL-8B-Instruct815.79626.9522.194.35baseline
1221.05781.4729.065.39baseline
1622.81853.8930.125.86baseline
MedAgent-Pro826.321,863.0836.796.390.85
1231.582,242.1945.567.670.72
1635.092,432.4248.988.330.78
MedInSight (Ours)838.601,345.9537.686.793.17
1242.111,478.3243.517.443.02
1642.111,531.4645.198.072.85

Across all budget settings (at={8,12,16}a_t = \lbrace 8, 12, 16 \rbrace), MedInSight achieves the highest accuracy (38.60%, 42.11%, and 42.11%, respectively), outperforming both the base model (15.79%-22.81%) and MedAgent-Pro (26.32%-35.09%). Notably, this advantage does not simply stem from consuming additional inference compute. Compared to MedAgent-Pro, MedInSight generates 28%-37% fewer output tokens while maintaining comparable or even lower latency; indeed, even under a strict budget of at=8a_t = 8, it already surpasses the performance of MedAgent-Pro at at=16a_t = 16 (38.60% vs. 35.09%). While MedInSight introduces modest additional overhead compared to the base model, the latency per interaction remains virtually unchanged. Overall, relative to the base model, MedInSight yields an accuracy gain of 2.85-3.17 percentage points for every 100 additional tokens, which is approximately 3.7-4.2 times greater than that of MedAgent-Pro. This indicates that the additional computational resources are utilized more effectively to acquire diagnostically valuable evidence. We have incorporated this analysis into Appendix B.9 of the revised manuscript.


Once again, thank you for your insightful comment, which helps us significantly strengthened the clinical grounding of our evaluation. We look forward to your further reply!

Reference

[1] MedSG-bench: A benchmark for medical image sequences grounding. NeurIPS 2026.

[2] MedAgent-Pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow. ICLR 2026.

Reviewer_9rA2OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes MedInSight, a multi-image diagnostic agent trained with cold-start SFT and GRPO, based on the Qwen3-VL-8B platform. It first locates lesions image-by-image and then performs joint examinations across images. The method has two core modules. CER selects anchor images using the entropy increase of coordinate tokens, and then selects target images using the 1-IoU between the relocated bounding box and the original bounding box. CCA determines action-level weights based on the change in the margin of correct options in the model's self-reported confidence, and determines the direction based on the sign of the stage-level margin change, adding these weights to the GRPO advantage. It achieves an average score of 50.32 on six benchmarks, 22.96 points higher than the base model.

Critical strengths
  • Most existing thinking-with-images methods only reason within a single image. This paper presents a traceable zoom-in + select workflow for cross-modal, cross-perspective, and cross-sequence evidence association, which is closer to clinical image interpretation. The cases in Figures 18–19 specifically demonstrate how cross-image examination can correct or strengthen the initial judgment.
  • Table 2 breaks down the two types of uncertainty in CER and the two time scales of CCA. Shuffled CCA, while keeping the total amount of correction constant, shuffles the allocation objects, directly testing whether ''which action to assign credit to'' is important. Table 4 and blinded manual assessment (Figure 9) further rule out the explanation of ''just looking at more graphs''.
  • The SFT trajectory was generated by a teacher who could not see the answers and was verified by a verifier. Plain text leak screening eliminated 5,593 questions, and duplicates were removed by PMCID and evaluation set. The three out-of-domain benchmarks improved by +24.0, +10.2, and +32.2 respectively compared to the base model.
Critical weaknesses
  • In Table 1, the average scores of Gemini 3 Pro (53.64), Gemini 2.5 Pro (53.51), GPT-5 (52.38), o3 (52.05), and Claude 4.6 Opus (50.93) are all higher than the paper's 50.32. The ''state-of-the-art'' in the abstract and introduction could be changed to ''achieved average best among open-source models''. This change does not affect the paper's main contributions.
  • The credit signal for CCA comes from the candidate confidence levels reported by the model itself. The authors also note in the captions of Figures 18–19 that these values ​​are not calibration probabilities. It is recommended to report the calibration status of the confidence levels (such as ECE or Brier) to confirm that the gain of CCA comes from effective credit assignment, rather than a shift in the confidence level representation.
  • Since Qwen3.5-9B's performance significantly surpasses that of Qwen3-VL-8B, why not use Qwen3.5-9B or a more powerful open-source model in the same series as the base model?
Questions for authors
  • How is the confidence level calibrated? If κ is applied only to the action token and not to the observation token, will the results change?
  • Could you please provide a comparison with at least one published medical vision agent method (such as MedVR or Ophiuchus) and give the RL training set size and number of training steps?
3 comments
AuthorsOpenReview ↗

Response to Reviewer 9rA2 (1/3)

We sincerely thank the reviewer for the constructive feedback. Your thoughtful comments have helped us comprehensively analyze our contributions and further strengthen our paper. We have conducted several additional experiments and revised the manuscript accordingly to address all the points raised. Below, we provide a point-by-point response.


Response to W1: Wording of "state-of-the-art performance"

We thank the reviewer for this valuable feedback. We acknowledge that the "state-of-the-art" claim is imprecise. Therefore, we revise the wording accordingly:

  • Abstract: "Extensive experiments demonstrate that MedInSight achieves the best average performance among open-source models across medical diagnosis and reasoning benchmarks, …"
  • Introduction (Line 100): "Extensive experiments demonstrate that MedInSight achieves the best average performance among evaluated open-source models consistently on both case-level diagnosis and multi-image medical reasoning, …"
  • Introduction (Line 110): "We achieve the best average performance among the evaluated open-source models on multiple public medical VQA benchmarks, showcasing MedInSight's robust generalization and enhanced reasoning capabilities."

These revisions have been highlighted in blue in the revised manuscript.

Response to W2, Q1.1: Calibration of self-reported confidence

We sincerely thank the reviewer for raising this practical concern. We acknowledge that our initial evaluation focuses primarily on diagnostic accuracy. Following the reviewer’s suggestion, we conduct a comprehensive calibration analysis to assess the reliability of the model’s confidence readouts.

Calibration metrics

To evaluate how reliable MedInSight's confidence estimates are throughout the reasoning process, we measure calibration at two complementary levels: action-level and final stage-level. At both levels, we compute Expected Calibration Error (ECE) and Brier score from the model's confidence readouts and the corresponding correctness labels on MedThinkVQA (MT), MedXQA-Multi (MX), and Dtest\mathcal{D}_{\mathrm{test}}.

Evaluation results

  • Action-level calibration analysis: Our method demonstrates superior calibration across all datasets (Table below). We achieve substantially lower ECE: 0.0961 on MT (58.7% better than the second-best, Vanilla GRPO, at 0.2326), 0.0886 on MX (57.9% better than 0.2103), and 0.1103 on Dtest\mathcal{D}_{\mathrm{test}} (49.9% better than 0.2200). We also achieve the lowest Brier scores, improving over Vanilla GRPO by 14.9%, 6.2%, and 9.9%, respectively.

  • Final stage-level calibration analysis: The same trend holds for the final diagnostic confidence. Our method achieves the lowest ECE on all datasets: 0.0732 on MT (61.2% better than Vanilla GRPO at 0.1889), 0.1055 on MX (38.4% better than 0.1714), and 0.1218 on Dtest\mathcal{D}_{\mathrm{test}} (38.3% better than 0.1975). It also attains the lowest Brier scores (0.2392, 0.2746, and 0.2194).

Action-level calibration analysis

MethodMT ECE ↓MT Brier ↓MX ECE ↓MX Brier ↓Dtest\mathcal{D}_{\mathrm{test}} ECE ↓Dtest\mathcal{D}_{\mathrm{test}} Brier ↓
SFT0.25800.30060.27040.30750.28980.2723
Vanilla GRPO0.23260.27840.21030.26970.22000.2473
MedInSight0.09610.23690.08860.25300.11030.2227

Final stage-level calibration analysis

MethodMT ECE ↓MT Brier ↓MX ECE ↓MX Brier ↓Dtest\mathcal{D}_{\mathrm{test}} ECE ↓Dtest\mathcal{D}_{\mathrm{test}} Brier ↓
SFT0.24340.31160.23080.31860.27390.2882
Vanilla GRPO0.18890.28740.17140.28860.19750.2520
MedInSight0.07320.23920.10550.27460.12180.2194

We have incorporated these results in Appendix B.6 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9rA2 (2/3)

Response to W3: Choice of base model

We thank the reviewer for this insightful question. We choose Qwen3-VL-8B-Instruct as our base model to demonstrate that the performance gains stem from our proposed framework rather than from the inherent capacity of the base model.

To verify that MedInSight is not tied to a specific base model, we additionally apply the exact same training pipeline and hyperparameters to Qwen3.5-9B to examine whether the gains persist on a stronger base model. The results on MedThinkVQA (MT), MedXQA-Multi (MX), and Dtest\mathcal{D}_{\mathrm{test}} are summarized below:

ModelMTMXDtest\mathcal{D}_{\mathrm{test}}
Qwen3.5-9B44.1738.9029.26
MedInSight (Qwen3.5-9B)60.1467.7848.87
Δ+15.97+28.88+19.61
Qwen3-VL-8B-Instruct29.1722.6718.65
MedInSight (Qwen3-VL-8B-Instruct)53.1954.8941.48
Δ+24.02+32.22+22.83

As shown, MedInSight consistently brings substantial improvements on both base models. These results show that the effectiveness of our framework generalizes across base models of different strength, and that a stronger base model further raises the final performance, e.g., reaching 60.14% on MT and 48.87% on Dtest\mathcal{D}_{\mathrm{test}}.

We have incorporated these results into Table 8 in Appendix B.7 of the revised manuscript. We hope these results address your concerns and provide a more comprehensive comparison for your reference.

Response to Q1.2: Ablation on the CCA coefficient κ\kappa

We thank the reviewer for this valuable feedback. We conduct an ablation in which the Cross-horizon Credit Assignment (CCA) correction κ\kappa is applied only to action tokens, while observation tokens receive only the outcome advantage.

Critical observations

The table below show that applying κ\kappa only to action tokens leads to a consistent yet modest drop across all three benchmarks (−0.41-0.41, −1.67-1.67, and −0.64-0.64 points), indicating that CCA remains substantially effective even when credit is assigned solely to action tokens. Importantly, observation tokens represent the model's interpretation of newly acquired visual evidence and directly influence the subsequent reasoning process. Consequently, assigning fine-grained credit to both the action tokens and their corresponding observation tokens provides a learning signal that spans both evidence localization and downstream diagnostic reasoning, enabling the model to achieve additional performance gains.

κ\kappa applied toMTMXDtest\mathcal{D}_{\mathrm{test}}
Action tokens only52.7853.2240.84
Action + Observation tokens (MedInSight)53.1954.8941.48
Δ−0.41−1.67−0.64

We have incorporated these results into Table 9 in Appendix B.8 of the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 9rA2 (3/3)

Response to Q2: Comparison with published medical vision agents

We thank the reviewer for this valuable suggestion. MedVR [1] does not release its model weights, and it is trained purely with RL on a 36K-sample subset selected from OmniMedVQA. Since this selected subset is not released, we also sample a training set of the same size (36K samples) from OmniMedVQA and train MedVR from scratch, strictly following the hyperparameters in its official codebase. Due to the time limit, we evaluate the reproduced model on three benchmarks.

Results

The results show that MedInSight is indeed more effective for multi-image clinical diagnosis than agentic RL methods designed for single-image reasoning. As shown in the table below, MedInSight outperforms the reproduced MedVR on all three benchmarks, with gains of +15.41 on MT, +26.97 on MX, and +5.47 on Dtest\mathcal{D}_{\mathrm{test}}. Since Ophiuchus [2] does not publicly release its model weights and training data, we are currently unable to reproduce it faithfully.

ModelMTMXDtest\mathcal{D}_{\mathrm{test}}
MedVR*37.7827.9236.01
MedInSight (Ours)53.1954.8941.48

We additionally compare with four recent medical vision models, including the published medical agent methods MedAgent-Pro [3], MedSIGHT [4], and VILA-M3 [5], as well as the concurrent evidence-seeking agent ClinSeekAgent [6]. All methods are evaluated under the same protocol as in Table 1.

ModelVenueMedThinkVQAMedFrameQAMedXQA-MultiLiveClinMMRareBenchDtest\mathcal{D}_{\mathrm{test}}-VQAAVG
VILA-M3-8BCVPR 202526.034.220.54.022.227.322.37
MedSIGHT-8BICML 202638.150.224.81.612.835.427.15
ClinSeek-35B-A3BarXiv 202631.446.136.812.745.138.135.03
MedAgent-ProICLR 202622.437.527.014.150.230.930.35
MedInSight-8B (Ours)–53.254.254.932.765.441.550.32

As shown, MedInSight consistently achieves the best performance on all six benchmarks, with an average of 50.32%, outperforming the second-best counterpart, ClinSeek-35B-A3B (35.03%), by 15.29 points. Compared with other published agentic methods, MedInSight still outperforms all baseline approaches by significant margins.

We have incorporated these results into Table 1 of the revised manuscript. The results of MedVR will also be added once its evaluation on all benchmarks is complete. We hope these results address your concerns and provide a more comprehensive comparison for your reference.

Response to Q3: RL training data and training steps

We thank the reviewer for this constructive point. The RL stage of MedInSight uses 2,971 training samples, and the model is trained for 160 steps. We have added these training details to Appendix D.1 of the revised manuscript.


Thank you for your insightful comment again, which helps us provide a more comprehensive analysis of our method. We look forward to your further reply!

References

[1] MedVR: Annotation-free medical visual reasoning via agentic reinforcement learning. ICLR 2026.

[2] Ophiuchus: Incentivizing Tool-augmented "Think with Images" for Joint Medical Segmentation, Understanding and Reasoning. ICML 2026.

[3] MedAgent-Pro: Towards evidence-based multi-modal medical diagnosis via reasoning agentic workflow. ICLR 2026.

[4] MedSIGHT: Towards Grounded Visual Comprehension in Medical Large Vision-Language Models. ICML 2026.

[5] VILA-M3: Enhancing vision-language models with medical expert knowledge. CVPR 2025.

[6] ClinSeekAgent: Automating Multimodal Evidence Seeking for Agentic Clinical Reasoning. arXiv 2026.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response for MedInSight (1/2)

We sincerely thank all reviewers for their time and constructive feedback. We are encouraged that the reviewers recognize the clinical relevance and sound motivation of modeling cross-image diagnosis as active, progressive evidence acquisition (9rA2, fxb9, xo1z), which goes beyond single-image thinking-with-images through a traceable workflow for cross-modal, cross-view, and cross-sequence evidence association (9rA2), and find that CER and CCA address what to inspect next and how to learn useful inspection behavior in a coherent, unified framework (xo1z, fxb9). The thorough and convincing experimental validation of our paper is recognized by all reviewers (9rA2, fxb9, xo1z), spanning from strong out-of-domain generalization (9rA2, fxb9) and comprehensive ablation study (9rA2, xo1z). Furthermore, The paper's clarity and strong organization are commended as well (9rA2, fxb9).

In response to the valuable concerns raised, we conduct extensive new experiments and incorporate detailed clarifications. We summarize these key revisions below.

Reviewer ConcernRelevant Reviewer(s)Our Actions and Revisions
1. Scope of Performance Claims9rA2We revise the "state-of-the-art" claim to "the best average performance among the evaluated open-source models" in the Abstract and Introduction.
2. Reliability of Our self-reported Confidence9rA2We conduct a new calibration analysis (ECE and Brier score) at both the action level and the final stage level. MedInSight achieves the lowest ECE and Brier scores on all three different medical multi-image benchmarks compared with SFT and vanilla GRPO, demonstrating the reliability of our self-reported confidence (Appendix B.6).
3. Generalization across Base Models9rA2We conduct a new quantitative experiment by applying our full training pipeline to the stronger Qwen3.5-9B backbone across three medical multi-image benchmarks, where substantial accuracy gains demonstrate that our framework generalizes effectively across model generations (Appendix B.7).
4. Comparison with Medical Vision Agents9rA2We expand our evaluation to published medical vision agents (MedAgent-Pro, MedSIGHT, VILA-M3, and ClinSeekAgent) across all six benchmarks under the exact same evaluation protocol as Table 1. Furthermore, we reproduce MedVR from its official codebase. These results demonstrate that MedInSight consistently outperforms all these specialized agent baselines.
5. Ablation of CCA9rA2We expand the ablation of CCA by applying the coefficient κ\kappa only to action tokens and comparing it with applying κ\kappa to both action and observation tokens. The results confirm that extending fine-grained credit to observation tokens yields additional gains by jointly reinforcing visual localization and downstream reasoning.
6. Clinical Validity of Localized Visual Evidencefxb9We clarify the role of CER and expand our evaluation to the multi-image grounding benchmark MedSG-Bench alongside a clinician validity assessment by three licensed physicians across 856 cases (Appendix B.5). These results demonstrate that MedInSight achieves accurate cross-image visual localization and generates clinically valid inspection trajectories.
7. Inference Budget and Efficiencyfxb9We add an additional case study (Figure 23 of Appendix H) to explain the performance drop at large budgets and analyze typical failure mode caused by redundant and erroneous evidence accumulation (Figures 22–23, Appendix H), and add a budget comparison with the base model and MedAgent-Pro, demonstrating that MedInSight achieves the highest accuracy with fewer tokens than MedAgent-Pro (Appendix B.9).
AuthorsOpenReview ↗

General Response for MedInSight (2/2)

(Continuing from above)

Reviewer ConcernRelevant Reviewer(s)Our Actions and Revisions
8. Trajectory Selection and Consistency in Figure 9xo1zWe provide additional clarification on the experimental details of Figure 9 in Appendix, including the full trajectory selection protocol, the handling of all-correct groups, and the groups used to compute advantages after resampling. To verify that the conclusions of Figure 9 remain consistent across multi-image settings, we conduct an additional clinician evaluation on 3-image cases and, given the time limit, we expand the experiment with quantitative δt\delta_t analyses across 3-image to 6-image cases as well as all sampled trajectories beyond all successful trajectory.
9. Correction magnitude of Table 2xo1zWe provide additional statistical analyses, formal mathematical proof across short-horizon and long-horizon variants, proving that the correction magnitude of MedInSight closely matches that of both short-horizon and long-horizon baselines and that the observed gains stem from stage-aligned credit assignment rather than reward scaling. In addition, we also include additional rollout evaluations from both the SFT checkpoint and the 80-step RL checkpoint, empirically demonstrating that the special edge-case condition raised by the reviewer does not occur in our experimental setting.
10. Implementation Details9rA2, xo1zWe report the RL training data size and steps (2,971 samples, 160 steps), correct the description of loss aggregation to match the implementation (Appendix D.1), and provide the complete MMRareBench T1 prompts together with analysis (Appendix E.2, Section 4.2).

We believe these substantial revisions, including extensive new experiments and clarifications, thoroughly address the reviewers' concerns and significantly strengthen the manuscript. We commit to releasing our code and checkpoints to ensure full reproducibility. We thank the reviewers once again for their valuable guidance in improving our work.

Best regards,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.