- Potential data contamination undermines the reliability of the reported results. The authors use PMC-VQA to construct 12k SFT trajectories and 41k DPO preference pairs (Sections 3.1 and 4.1), while also evaluating on PMC-VQA (Table 1). Although official test splits are mentioned, the paper does not clarify whether training samples were drawn exclusively from the training split or provide image-level and source-article-level deduplication checks. This raises a serious concern about potential data leakage, particularly when multiple questions may share the same image or source publication. Without explicit verification of train–test separation, the reported performance gains cannot be confidently attributed to generalization rather than possible data contamination.
My suggestions: I would like to see more details on dataset construction in the appendix, particularly the filtering criteria, split provenance, deduplication procedures, and preference-pair generation process, to better assess the validity and reproducibility of the reported results.
- Insufficient methodological specification and lack of reproducibility. The proposed framework is severely underspecified, making it difficult to assess or reproduce the central methodological contributions.
| Location | What the paper provides | What I think is missing for reproducibility |
|---|
| §3.3, lines 203–225 | Four agents represented by mathematical functions | Agent models, prompts, decoding configurations, and implementation details. I do not see any of them in Appendix. |
| §3.3, lines 220–222 | Knowledge verification via an external database S | Specific databases/knowledge graphs, retrieval queries, evidence selection, and conflict resolution. Again, I do not see any of them in Appendix. |
| §3.4, lines 270–286 | Teacher-generated corrections | Exact prompts, correction criteria, and data quality-control procedures |
| §4.1, lines 357–366 | 12k SFT trajectories, 41k DPO pairs, GPT-4o teacher, learning rates, batch sizes, epochs | Data-generation protocol, DPO β, reference-model setup, and DPO training schedule |
| §4.1, lines 363–366 | Qwen2.5-VL-7B-Instruct backbone, learning rates, batch sizes, epochs, context length, and GPU hardware | Detailed model and training configs, including full fine-tuning vs. LoRA, optimizer, learning-rate scheduler, reference-model configuration, and so on. No info in Appendix. |
- Overstated state-of-the-art claims and outdated baseline comparisons. The paper repeatedly claims state-of-the-art performance across six medical VQA benchmarks (Section 4.2, lines 370–373). However, Table 1 does not support this broad characterization. Both the proprietary and open-source baseline selections are outdated for an ICLR 2027 submission. The proprietary comparisons are restricted to GPT-4.1, Claude Sonnet 4, and Gemini 2.5 Flash, all released in 2025. Moreover, its performance remains substantially below the reported proprietary baselines on MMMU-Med (61.8% versus 75.2% for GPT-4.1 and 76.9% for Gemini 2.5 Flash). By September 2026, significantly newer generations of frontier models, including GPT-6 Astra, had already been introduced. Comparing against these older models is insufficient to establish competitiveness with contemporary frontier multimodal systems. The open-source comparisons are similarly incomplete. For example, Fleming-VL-8B, released in November 2025, reports a five-benchmark average of 73.3% on OmniMedVQA, PMC-VQA, PathVQA, SLAKE, and VQA-RAD, compared with 70.3% calculated from Diagno-SV's reported results on the same benchmarks.
Small corrections for authors. Table 1 cites GPT-4.1 as “OpenAI (2024),” but GPT-4.1 was introduced on April 14, 2025. The paper's 2024 OpenAI reference actually concerns GPT-4o.
- Limited methodological novelty and insufficient discussion of closely related work. The novelty of the proposed tri-level decomposition is unclear in light of existing work on structured, visually grounded medical reasoning. In particular, S-Chain Le-Duc et al., 2025; arXiv:2510.22728 already introduced structured visual chain-of-thought supervision that explicitly connects image regions, visual descriptions, intermediate reasoning, and diagnostic predictions. It also investigated the integration of external medical knowledge and the faithfulness of intermediate reasoning to visual evidence.
More importantly, S-Chain goes beyond textual decomposition by providing expert-annotated ROI bounding boxes, conducting component-wise interventions on intermediate reasoning steps, and introducing explicit objectives for aligning reasoning representations with visual tokens. These contributions directly overlap with Diagno-SV's stated motivation and its claims regarding visual grounding and independently verifiable reasoning.
I also could not find a discussion or citation of S-Chain in the manuscript, despite its clear relevance and public availability since October 2025.
Direct Novelty Comparison
| Dimension | S-Chain (2025) | Diagno-SV (Under Review) |
|---|
| Core motivation | Faithful, interpretable medical visual reasoning | Faithful, interpretable medical visual reasoning |
| Structured reasoning | Localization → Lesion description → Grading → Diagnosis | Visual perception → Reasoning → Medical knowledge |
| Visual grounding | Explicit ROI bounding boxes linked to reasoning | Natural-language ROI focus and visual description |
| Medical knowledge | Structured reasoning combined with medical RAG | External knowledge verification as a separate step |
| Training | Structured CoT supervised fine-tuning with grounding regularizers | Synthetic trajectory SFT followed by curriculum-based DPO |
| Supervision source | Expert-annotated reasoning and ROIs | GPT-4o-generated and GPT-4o-corrected trajectories |
| Faithfulness evaluation | Component-level interventions, localization metrics, and grounding experiments | GPT-4o-judged visual and factual fabrication rate |
| Scope | Primarily brain MRI and dementia reasoning | Broad medical VQA benchmarks |
My suggestions. Overall, the core idea of separating visual observations from intermediate reasoning is not sufficiently novel on its own. The paper should substantially revise its novelty claims, properly situate the proposed approach relative to S-Chain, and demonstrate empirically what additional benefit is attributable to its diagnostic refinement procedure.
Reference.
S-Chain: Structured Visual Chain-of-Thought For Medicine: https://arxiv.org/abs/2510.22728
Additional results addressing Q1-Q4
Thank you for the review. We provide the matched-budget comparison and independent clinical results addressing Q1-Q4.
Q1. Matched correction-and-DPO comparison
The full controlled method (D) achieves 68.65±0.23 mean accuracy across six benchmarks, compared with 67.17±0.25 for ordinary CoT with error-specific correction and DPO (C). Main-comparison values report the mean and sample standard deviation over training seeds 17, 42, and 123. The evaluation includes OmniMedVQA-open-8.
Both conditions share the backbone, source questions, frozen candidate bank, rejected trajectories, retrieval evidence, and difficulty labels. Both receive visual, logical, and knowledge feedback, with matched pair counts and update budgets. C's auditor identifies these error types while the student retains ordinary CoT output. At a common effective-token budget, D achieves 68.55% accuracy against 67.10% for C. The 1.45-point advantage agrees with the 1.48-point gain in the main comparison.
Q2. Independent audit and agreement
The audit uses 600 fixed questions, 100 per benchmark, sampled with seed 20261108. Both clinical checkpoints use training seed 42. Two modality-qualified physicians independently annotate anonymized outputs, and a third adjudicates disagreements. Study-wide pooled inter-physician agreement is κ=0.74 for visual errors, 0.66 for logical errors, and 0.78 for knowledge errors.
Clinical adjudication identifies fabrication in 198/600 C outputs (33.00%) and 162/600 D outputs (27.00%). The reduction is 6.00 percentage points, with a paired image-clustered 95% confidence interval of 3.09 to 8.99 points. This interval uses 10,000 bootstrap resamples of 360 image clusters, stratified by benchmark, with shared resamples across models. Final technical missingness is zero. InternVL3.5-8B gives the same ordering, at 34.00% versus 28.00%.
The denominator is all audited outputs. Fabrication is the union of explicitly unsupported visual and medical assertions used in diagnostic justification, with each trajectory counted once. Indeterminate support without confirmed fabrication occurs in 20 C outputs and 18 D outputs. Assigning these cases outside or inside the category gives label-uncertainty bounds of 33.00% to 36.33% and 27.00% to 30.00%, respectively. These bounds concern label uncertainty, separately from the statistical confidence interval.
Q3. Correct answers with invalid justifications
The valid preference pool includes 9,840 such rejected trajectories out of 41,000 pairs (24.00%). The remaining 31,160 have incorrect final answers. Rejection is triggered by an incorrect answer or a failed visual, logical, or knowledge check. Chosen trajectories must answer correctly and pass all three checks.
The same clinical audit separately measures justification validity among correct answers. C produces 402 correct answers, including 72 with invalid justifications. D produces 408 correct answers, including 54 with invalid justifications. The conditional rates are 17.91% and 13.24%. Justification validity is assessed using the visual, logical, and knowledge error criteria.
Q4. Evidence-detection errors and interpretation
Physicians assess each asserted finding against the image and each medical proposition against supporting evidence. Missing graph edges receive a noncoverage label. Insufficient or conflicted retrieval evidence and indeterminate visual findings are tracked separately. Only explicitly unsupported assertions used in the diagnostic justification contribute to fabrication.
The diagnostic components expose observations, deductions, and medical propositions for separate checks. The checks determine the error-specific correction and preference target. The matched training comparison and independent audit establish improved accuracy and validity of generated justifications, which are the measured benefits of this diagnostic training procedure.