acceptodds
Under review as a conference paper at ICLR 2027

Diagno-SV: Decoupling Perception, Reasoning, and Knowledge for Trustworthy Medical VLMs

Abstract

While Large Vision-Language Models (VLMs) show immense promise in emulating the complex cognitive processes of clinical diagnostics, ensuring the reliability of their multimodal reasoning remains a critical and unsolved challenge. Current Chain-of-Thought approaches in VLMs typically conflate visual perception, logical deduction, and medical knowledge, creating an opaque "black box." This ambiguity fosters a critical failure mode we identify as retrospective forgery, a phenomenon where models fabricate visual findings or distort facts post-hoc to justify flawed diagnostic hypotheses. To address this, we introduce Diagno-SV (Diagnosis-Separation for medical VLM), a novel framework for diagnostic-oriented optimization via trilevel decoupling. Diagnosis-SV forces the model to decompose its reasoning into independently verifiable steps of visual perception, logical deduction, and medical knowledge. Building on this transparency, we design a curriculum-based diagnostic refinement workflow. A dedicated multimodal diagnostic module pinpoints errors, distinguishing between "looking wrong" and "thinking wrong", and constructs preference pairs for Direct Preference Optimization. The resulting Diagno-SV model achieves state-of-the-art performance across six medical VQA benchmarks, including MMMU-Med and PMC-VQA. More importantly, our method significantly mitigates retrospective forgery, reducing the rate of visual and factual fabrication from 42.74% to 26.32%, paving a new path toward trustworthy visual-based medical AI.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Medical Vision-Language Models, Trustworthy Medical AI, Retrospective Forgery, Reasoning Decoupling, Direct Preference Optimization, Medical Visual Question Answering

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_8CCNOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the fabrication of visual evidence or medical knowledge in explanations generated by medical vision-language models. It proposes Diagno-SV, which organizes the reasoning process into visual perception, logical deduction, and medical knowledge. The method combines supervision on structured reasoning trajectories, teacher-guided error-specific correction, and curriculum-based preference optimization to post-train an existing model. The authors evaluate the approach on multiple medical visual question answering benchmarks and through analysis of errors in explanations, reporting improved question answering performance and fewer fabricated statements identified by the evaluator model.

Critical strengths
  1. The paper addresses an important reliability problem in medical VLMs: correct answers do not necessarily imply faithful visual descriptions or well-supported medical justifications. Separating observations, inferences, and medical knowledge provides a useful framework for identifying error sources and generating more targeted training feedback.
  2. The proposed training workflow combines teacher-guided correction with curriculum-based preference optimization. Reported gains across six benchmarks and fewer fabricated statements in explanations support its practical value.
Critical weaknesses
  1. The added value of the central design is not established. Table 2 mainly compares SFT with successive DPO iterations. This supports the effectiveness of the overall workflow, but does not distinguish the effects of tri-level decomposition, teacher correction, preference-data quality, and additional training. A standard CoT plus teacher correction and DPO baseline using the same backbone, source data, and a comparable budget is needed to establish the method's distinctive contribution.
  2. GPT-4o participates in training-data generation, correction, and forgery-rate assessment, with no independent validation reported and an insufficiently specified evaluation protocol (Section 4.3.3 and Table 4). Moreover, structured textual explanations are easier to inspect but do not establish causal decoupling of perception, reasoning, and knowledge inside the model. The evidence supports improved output quality, but remains insufficient for the stronger claims about reasoning faithfulness and internal transparency.
Questions for authors
  1. Does tri-level decomposition retain a clear advantage over standard CoT plus teacher correction and DPO when source data, teacher supervision, and training budgets are matched?
  2. What are the sample size and adjudication criteria for forgery-rate assessment? Is there independent evaluation or a clinical-expert audit, together with an assessment of evaluator agreement?
  3. Section 3.4 and Algorithm 2 primarily describe correcting trajectories with incorrect final answers. Are trajectories with correct answers but incorrect justifications also audited and used for training? How are they selected?
  4. In practice, how do you distinguish retrospective forgery from results caused by falses in detecting evidences?
    
1 comment
AuthorsOpenReview ↗

Additional results addressing Q1-Q4

Thank you for the review. We provide the matched-budget comparison and independent clinical results addressing Q1-Q4.

Q1. Matched correction-and-DPO comparison

The full controlled method (D) achieves 68.65±0.2368.65\pm0.23\\% mean accuracy across six benchmarks, compared with 67.17±0.2567.17\pm0.25\\% for ordinary CoT with error-specific correction and DPO (C). Main-comparison values report the mean and sample standard deviation over training seeds 17, 42, and 123. The evaluation includes OmniMedVQA-open-8.

Both conditions share the backbone, source questions, frozen candidate bank, rejected trajectories, retrieval evidence, and difficulty labels. Both receive visual, logical, and knowledge feedback, with matched pair counts and update budgets. C's auditor identifies these error types while the student retains ordinary CoT output. At a common effective-token budget, D achieves 68.55% accuracy against 67.10% for C. The 1.45-point advantage agrees with the 1.48-point gain in the main comparison.

Q2. Independent audit and agreement

The audit uses 600 fixed questions, 100 per benchmark, sampled with seed 20261108. Both clinical checkpoints use training seed 42. Two modality-qualified physicians independently annotate anonymized outputs, and a third adjudicates disagreements. Study-wide pooled inter-physician agreement is κ=0.74\kappa=0.74 for visual errors, 0.66 for logical errors, and 0.78 for knowledge errors.

Clinical adjudication identifies fabrication in 198/600 C outputs (33.00%) and 162/600 D outputs (27.00%). The reduction is 6.00 percentage points, with a paired image-clustered 95% confidence interval of 3.09 to 8.99 points. This interval uses 10,000 bootstrap resamples of 360 image clusters, stratified by benchmark, with shared resamples across models. Final technical missingness is zero. InternVL3.5-8B gives the same ordering, at 34.00% versus 28.00%.

The denominator is all audited outputs. Fabrication is the union of explicitly unsupported visual and medical assertions used in diagnostic justification, with each trajectory counted once. Indeterminate support without confirmed fabrication occurs in 20 C outputs and 18 D outputs. Assigning these cases outside or inside the category gives label-uncertainty bounds of 33.00% to 36.33% and 27.00% to 30.00%, respectively. These bounds concern label uncertainty, separately from the statistical confidence interval.

Q3. Correct answers with invalid justifications

The valid preference pool includes 9,840 such rejected trajectories out of 41,000 pairs (24.00%). The remaining 31,160 have incorrect final answers. Rejection is triggered by an incorrect answer or a failed visual, logical, or knowledge check. Chosen trajectories must answer correctly and pass all three checks.

The same clinical audit separately measures justification validity among correct answers. C produces 402 correct answers, including 72 with invalid justifications. D produces 408 correct answers, including 54 with invalid justifications. The conditional rates are 17.91% and 13.24%. Justification validity is assessed using the visual, logical, and knowledge error criteria.

Q4. Evidence-detection errors and interpretation

Physicians assess each asserted finding against the image and each medical proposition against supporting evidence. Missing graph edges receive a noncoverage label. Insufficient or conflicted retrieval evidence and indeterminate visual findings are tracked separately. Only explicitly unsupported assertions used in the diagnostic justification contribute to fabrication.

The diagnostic components expose observations, deductions, and medical propositions for separate checks. The checks determine the error-specific correction and preference target. The matched training comparison and independent audit establish improved accuracy and validity of generated justifications, which are the measured benefits of this diagnostic training procedure.

Reviewer_buajOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes a diagnosis-separation framework for medical VLM. The authors identify retrospective forgery as a critical failure mode for medical VLM. To address this, the authors decouple the medical reasoning into visual perception, logical deduction, and medical knowledge. Based on this workflow, the authors conduct SFT and DPO based on data generated from GPT-4o.

Critical strengths

Strengths

  1. The experimental results across six medical VQA benchmarks verify the effectiveness of this method.
  2. The authors conduct error analysis of reasoning workflow, providing an intuitive understanding of how the framework improves the performance.
Critical weaknesses
  1. The authors use GPT-4o to detect retrospective forgery. However, GPT-4o itself could suffer from hallucination. Using the same model family as both teacher and evaluator may introduce shared biases, making the forgery rate analysis less convincing.
  2. The proposed approach is incremental, assembling existing techniques (structured CoT, SFT, DPO) in a straightforward pipeline.
  3. There are missing details regarding how to detect Visual Perception Error, Reasoning Error (logic gap), and Knowledge Error.
  4. The paper does not compare with the models released in 2026. The latest compared baselines are released in in 2025.
  5. Typos: Figure 2, Curriculum-Based Reforcement-> Reinforcement
Questions for authors

Please see weaknesses.

Reviewer_cJUnOpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Diagno-SV, a framework for improving medical VLMs by decomposing diagnostic trajectories into visual perception, logical reasoning, and medical knowledge. The approach comprises two stages: (1) SFT on structured reasoning trajectories synthesized by a multi-agent framework and (2) DPO using a diagnostic module that corrects visual, logical, and factual errors.

The experiments use Qwen2.5-VL-7B as the backbone and report improvements across six medical VQA datasets. The method reportedly improves average accuracy from 56.9% to 68.8% and reduces the proposed retrospective forgery rate from 42.74% to 26.32%.

Strengths

The paper addresses the important problem of faithful and reliable medical VLM reasoning through structured decomposition of perception, reasoning, and knowledge. It also reports substantial performance improvements across multiple medical VQA benchmarks.

Weaknesses

The paper raises concerns about potential data contamination, insufficient implementation details for reproducibility, and overstated SOTA claims due to incomplete baseline comparisons. Furthermore, its methodological novelty is unclear given the substantial overlap with prior work such as S-Chain (Le-Duc et al., 2025), which is neither discussed nor adequately differentiated.

Critical strengths
  1. The paper addresses an important problem of faithful CoT: medical VLMs may produce plausible diagnostic explanations that are not grounded in the image or supported by medical knowledge.

  2. Useful structured CoT for failure diagnosis. Separating perception, reasoning, and knowledge provides a potentially useful structure for analyzing diagnostic failures.

  3. Impressive result improvements. The reported improvements over the Qwen2.5-VL backbone are substantial, and the evaluation covers multiple established medical VQA benchmarks.

Critical weaknesses
  1. Potential data contamination undermines the reliability of the reported results. The authors use PMC-VQA to construct 12k SFT trajectories and 41k DPO preference pairs (Sections 3.1 and 4.1), while also evaluating on PMC-VQA (Table 1). Although official test splits are mentioned, the paper does not clarify whether training samples were drawn exclusively from the training split or provide image-level and source-article-level deduplication checks. This raises a serious concern about potential data leakage, particularly when multiple questions may share the same image or source publication. Without explicit verification of train–test separation, the reported performance gains cannot be confidently attributed to generalization rather than possible data contamination.

My suggestions: I would like to see more details on dataset construction in the appendix, particularly the filtering criteria, split provenance, deduplication procedures, and preference-pair generation process, to better assess the validity and reproducibility of the reported results.

  1. Insufficient methodological specification and lack of reproducibility. The proposed framework is severely underspecified, making it difficult to assess or reproduce the central methodological contributions.
LocationWhat the paper providesWhat I think is missing for reproducibility
§3.3, lines 203–225Four agents represented by mathematical functionsAgent models, prompts, decoding configurations, and implementation details. I do not see any of them in Appendix.
§3.3, lines 220–222Knowledge verification via an external database SSSpecific databases/knowledge graphs, retrieval queries, evidence selection, and conflict resolution. Again, I do not see any of them in Appendix.
§3.4, lines 270–286Teacher-generated correctionsExact prompts, correction criteria, and data quality-control procedures
§4.1, lines 357–36612k SFT trajectories, 41k DPO pairs, GPT-4o teacher, learning rates, batch sizes, epochsData-generation protocol, DPO β\beta, reference-model setup, and DPO training schedule
§4.1, lines 363–366Qwen2.5-VL-7B-Instruct backbone, learning rates, batch sizes, epochs, context length, and GPU hardwareDetailed model and training configs, including full fine-tuning vs. LoRA, optimizer, learning-rate scheduler, reference-model configuration, and so on. No info in Appendix.
  1. Overstated state-of-the-art claims and outdated baseline comparisons. The paper repeatedly claims state-of-the-art performance across six medical VQA benchmarks (Section 4.2, lines 370–373). However, Table 1 does not support this broad characterization. Both the proprietary and open-source baseline selections are outdated for an ICLR 2027 submission. The proprietary comparisons are restricted to GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Flash, all released in 2025. Moreover, its performance remains substantially below the reported proprietary baselines on MMMU-Med (61.8% versus 75.2% for GPT-4.1 and 76.9% for Gemini 2.5 Flash). By September 2026, significantly newer generations of frontier models, including GPT-6 Astra, had already been introduced. Comparing against these older models is insufficient to establish competitiveness with contemporary frontier multimodal systems. The open-source comparisons are similarly incomplete. For example, Fleming-VL-8B, released in November 2025, reports a five-benchmark average of 73.3% on OmniMedVQA, PMC-VQA, PathVQA, SLAKE, and VQA-RAD, compared with 70.3% calculated from Diagno-SV's reported results on the same benchmarks.

Small corrections for authors. Table 1 cites GPT-4.1 as “OpenAI (2024),” but GPT-4.1 was introduced on April 14, 2025. The paper's 2024 OpenAI reference actually concerns GPT-4o.

  1. Limited methodological novelty and insufficient discussion of closely related work. The novelty of the proposed tri-level decomposition is unclear in light of existing work on structured, visually grounded medical reasoning. In particular, S-Chain Le-Duc et al., 2025; arXiv:2510.22728 already introduced structured visual chain-of-thought supervision that explicitly connects image regions, visual descriptions, intermediate reasoning, and diagnostic predictions. It also investigated the integration of external medical knowledge and the faithfulness of intermediate reasoning to visual evidence.

More importantly, S-Chain goes beyond textual decomposition by providing expert-annotated ROI bounding boxes, conducting component-wise interventions on intermediate reasoning steps, and introducing explicit objectives for aligning reasoning representations with visual tokens. These contributions directly overlap with Diagno-SV's stated motivation and its claims regarding visual grounding and independently verifiable reasoning.

I also could not find a discussion or citation of S-Chain in the manuscript, despite its clear relevance and public availability since October 2025.

Direct Novelty Comparison

DimensionS-Chain (2025)Diagno-SV (Under Review)
Core motivationFaithful, interpretable medical visual reasoningFaithful, interpretable medical visual reasoning
Structured reasoningLocalization → Lesion description → Grading → DiagnosisVisual perception → Reasoning → Medical knowledge
Visual groundingExplicit ROI bounding boxes linked to reasoningNatural-language ROI focus and visual description
Medical knowledgeStructured reasoning combined with medical RAGExternal knowledge verification as a separate step
TrainingStructured CoT supervised fine-tuning with grounding regularizersSynthetic trajectory SFT followed by curriculum-based DPO
Supervision sourceExpert-annotated reasoning and ROIsGPT-4o-generated and GPT-4o-corrected trajectories
Faithfulness evaluationComponent-level interventions, localization metrics, and grounding experimentsGPT-4o-judged visual and factual fabrication rate
ScopePrimarily brain MRI and dementia reasoningBroad medical VQA benchmarks

My suggestions. Overall, the core idea of separating visual observations from intermediate reasoning is not sufficiently novel on its own. The paper should substantially revise its novelty claims, properly situate the proposed approach relative to S-Chain, and demonstrate empirically what additional benefit is attributable to its diagnostic refinement procedure.

Reference.

S-Chain: Structured Visual Chain-of-Thought For Medicine: https://arxiv.org/abs/2510.22728

Questions for authors

I appreciate the authors' efforts to improve the faithfulness and reliability of medical VLMs through structured reasoning and preference optimization. However, I believe the paper's contributions primarily lie in combining existing techniques rather than introducing substantial methodological novelty. I encourage the authors to address the concerns regarding data contamination, reproducibility, baseline comparisons, and differentiation from related work to strengthen their claims.

My intention is to provide constructive feedback rather than simply advocate rejection. I sincerely hope the authors continue improving this work and succeed in publishing it.

My criticisms are based on my own assessment. I used AI to help investigate potential technical issues, cross-check results, and polish my review. I remain responsible for the accuracy of my observations and final assessment.

I wish the authors all the best!

3 comments
AuthorsOpenReview ↗

Additional results addressing W3 and W4

Thank you for the review. We provide the completed comparisons addressing W3 and W4.

W3. Contemporary baselines

We evaluate the released Fleming-VL-8B and MedMO-8B-Next checkpoints with greedy decoding. All entries use the same six evaluation manifests, answer extraction, normalization, and scoring. The benchmarks are MMMU-Med, VQA-RAD, SLAKE, PathVQA, PMC-VQA, and OmniMedVQA-open-8, the public-image subset used in this evaluation.

ModelSix-benchmark mean accuracy (%)
Qwen2.5-VL-7B backbone56.80
Fleming-VL-8B71.00
MedMO-8B-Next73.53
Cleaned online Diagno-SV (O)68.70±0.2568.70\pm0.25
Controlled full method (D)68.65±0.2368.65\pm0.23

Diagno-SV entries report the mean and sample standard deviation across training seeds 17, 42, and 123. These common-protocol scores establish the comparison with contemporary medical VLMs. The cleaned online model improves its backbone by 11.90 percentage points. The revision will state the performance claim in terms of backbone gains and matched training comparisons.

W4. Contribution relative to S-Chain

S-Chain trains ROI localization, lesion description, grading, and diagnosis with expert-annotated sequences, and reports ROI-alignment and inter-disease contrastive objectives. Its RAG variant supplies retrieved medical context.

Diagno-SV introduces a procedure for converting diagnosed failures in student trajectories into preference supervision. Visual observations are checked against the image, deductions against the corrected premises, and medical propositions against external evidence. These checks determine which trajectory is rejected, how visual, logical, then knowledge correction proceeds, and whether the correction qualifies as chosen. A chosen trajectory must answer correctly and pass all three checks before curriculum DPO learns its preference over the faulty trajectory. Retrieved evidence participates in this verification and pair-acceptance process. Of the 41,000 valid pairs, 9,840 reject correct-answer trajectories with invalid justifications, making diagnostic support part of the training target.

We test the added value of the explicit diagnostic interfaces against ordinary CoT with error-specific correction and DPO. Both conditions share the backbone, source questions, frozen candidate bank, rejected trajectories, retrieval evidence, and difficulty labels. Pair counts and update budgets are matched. Across the same six benchmarks, accuracy is 67.17±0.2567.17\pm0.25\\% for the CoT comparator (C) and 68.65±0.2368.65\pm0.23\\% for D, a 1.48-point gain over three seeds.

On a fixed panel of 600 questions, 100 per benchmark, independent clinical assessment finds fabrication in 198 C outputs (33.00%) and 162 D outputs (27.00%). Two physicians assess each output independently, with a third adjudicating disagreements. The paired image-clustered 95% confidence interval for the 6.00-point reduction is 3.09 to 8.99 points. This comparison establishes an accuracy and justification-quality benefit from the diagnostic interfaces under matched error-specific feedback and preference training.

Dear Reviewer, could you please follow up on the authors' response and indicate whether your concerns have been adequately addressed? Thank you.

Reviewer_cJUnOpenReview ↗

Thanks authors for your efforts!

Thank you to the authors for their clarifications and thoughtful responses.

However, I believe that W1, W2, and W4 should have been addressed BEFORE submission, as they concern fundamental expectations for a research paper. In its current form, the 13-page submission lacks essential methodological details to the extent that the proposed method cannot be reproduced. While this does not necessarily mean the reported results are incorrect, the missing information prevents independent verification and substantially limits my confidence in their reliability.

Furthermore, the novelty of the proposed method should have been clearly established relative to existing literature, particularly works such as S-Chain and GEMeX-ThinkVG. The authors should have explained the key methodological differences, especially regarding training strategies, to substantiate their novelty claims.

I appreciate the authors' efforts during the rebuttal. However, addressing these fundamental issues BEFORE submission would have saved reviewers considerable time, allowing us to focus on deeper technical evaluation rather than clarifying essential omissions. The rebuttal should primarily address reviewers' concerns, rather than provide fundamental information missing from the original submission.

==> These omissions also leave me uncertain whether other issues remain unaddressed.

For these reasons, I respectfully maintain my original score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.