acceptodds
Under review as a conference paper at ICLR 2027

RadThinking: Teaching Vision-Language Models to Think the Way Radiologists Are Taught

Abstract

Radiology education teaches a way of thinking, not a set of answers. A trainee learns to screen each region of an organ, characterize each finding, compare it with earlier scans, and apply a clinical guideline, a rule over the finding's representation: its size, enhancement, change, and a few more. Vision-language models are trained on images paired with answers and judged by the answer alone; the representation in between goes unchecked. We present RadThinking, which turns this way of thinking into supervision: the representation a radiologist is taught to build becomes the representation the model learns to write. Each step of reading a scan becomes a question with a reference answer, grounded in a 3D bounding box; questions link into chains; five clinical guidelines become decision tables, explicit rules from a finding's representation to its category. The representation is grounded, so each answer in it is scored at its box on the scan, and explicit, so a radiologist can check it as they check a trainee. RadThinking holds 122,127 questions over 5,365 abdominal CT scans of 2,679 patients from two countries. Across eight models, the best gets every answer right in only 23% of chains, numerical errors aside, and turning thinking on brings no consistent gain. Looking inside open-weight models shows why. Once its intermediate answers are written, Qwen3.8-27B follows that text, not the images, right or wrong. Before it writes, a linear probe on its hidden state predicts the final answer about 9 points more often than the model does. The model holds more than it writes and follows what it writes. So the representation it writes must be correct, and RadThinking supervises every answer in it. Trained on it, a 4B model outperforms zero-shot GPT-6, Claude Opus 5, and Gemini 3.8 Flash: 64% versus 52% on individual questions and 30% versus 23% on chains. Radiology education already wrote this representation down. A small model that learns it outperforms much larger models that have not.

Primary area:
datasets and benchmarks
Keywords:
Longitudinal analysis, Cancer screening, VQA Dataset, Benchmark

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_bN9DOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The study introduces a dataset of 122,127 questions spanning 5,365 CT scans, with visually grounded evidence, reference answers, dependency links, and explicit clinical guideline rules. By connecting intermediate observations to clinical conclusions, the dataset enables evaluation of both model predictions and reasoning steps. By connecting intermediate observations to clinical conclusions, the dataset enables evaluation of both model predictions and their supporting evidence. It provides a useful resource for assessing and improving the reasoning capabilities of medical vision-language models.

Critical strengths
  1. A substantial dataset for training CoT LLM RadThinking connects 122,127 questions across 5,365 CT scans with visual grounded evidence, reference answers, dependency links, and explicit clinical guideline rules. This dataset is useful to address the explainable challenge of reasoning LLM in medical applications. It also provides a tool for evaluating and improving the reasoning capacity of LLM in medical applications.

  2. Extensive informative benchmark Evaluating intermediate answers exposes failures that conclusion-only assessment can conceal. No zero-shot model exceeds 23.5% categorical chain accuracy. The dataset also supports substantial improvements through supervised fine-tuning. Although these comparisons do not isolate the benefit of intermediate supervision, they demonstrate that the resource provides useful training LLM models in medical applications. In other words, the dataset is useful for identifying model weaknesses and improve performance.

Critical weaknesses
  1. Although the dataset is extracted from CT images and reports, LLMs participate in extracting findings and constructing reference rationales. Appendix B.5 describes physician review of 5% of rationales per round, but the reported error rate below 2% is not accompanied by exact sample sizes, error counts, inter-rater agreement, or a clearly documented independent final audit.

  2. Section 4.2 primarily compares fine-tuned models with zero-shot models, leaving improvements attributable to ordinary domain adaptation, familiarity with question templates, or output-format learning. Appendix F describes a matched comparison between conclusion-only supervision and supervision of intermediate answers plus conclusions, but the corresponding quantitative comparison is not presented in Tables 2 or 12. Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.

Questions for authors
  1. I strongly recommend the authors provide an overview figure to describe the collecting dataset pipeline. It can help readers understand the paper easier.

  2. In the final physician-review round described in Appendix B.5, how many rationales were reviewed, how many were judged incorrect, and how were reviewer disagreements resolved? Did this review independently verify intermediate answers and conclusions against the original CT images and reports, or assess consistency with the pipeline-generated fact sheets?

  3. Appendix F describes conclusion-only supervision and supervision of intermediate answers plus conclusions. Were both conditions evaluated? If so, could you provide their comparative results and specify which factors were matched, particularly the backbone, training cases, image inputs, training budget, and output format?

  4. The study's title uses the term "radiology", but the proposed dataset focus only in CT imaging modality. Therefore, I would like to suggest to provide more specific title for the study.

6 comments
AuthorsOpenReview ↗

Response to Reviewer bN9D (1/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!

Thank you for the great questions and suggestions; they have really made the paper better, and we appreciate the time you spent on it! We are glad to hear the words "Extensive informative benchmark" and "the dataset is useful for identifying model weaknesses and improve performance". Building a 3D medical imaging dataset of this scale, with physician-reviewed reference answers for every intermediate task, five clinical guidelines as decision tables, and longitudinal links across 5,365 scans of 2,679 patients, took our team more than two years, eight physicians, and five review rounds. Datasets like this are rare in the field because they are so hard to build, and your kind words truly mean a lot to us!

We have addressed all five feedbacks as follows.


Q1. Although the dataset is extracted from CT images and reports, LLMs participate in extracting findings and constructing reference rationales. Appendix B.5 describes physician review of 5% of rationales per round, but the reported error rate below 2% is not accompanied by exact sample sizes, error counts, inter-rater agreement, or a clearly documented independent final audit.

In the final physician-review round described in Appendix B.5, how many rationales were reviewed, how many were judged incorrect, and how were reviewer disagreements resolved? Did this review independently verify intermediate answers and conclusions against the original CT images and reports, or assess consistency with the pipeline-generated fact sheets?

This is an important point. The physician review is the most labor-intensive part of this dataset: eight physicians reviewed rationales over five rounds across more than a year. They are now in Appendix B.5 of the revised PDF, and we provide them here for your reference. Every answer comes from the radiology report and the radiologist-reviewed segmentations, not generated by the language model (Section 2.1, Appendix B.2, B.4, and B.6). The physicians verified each sampled rationale against the original CT images and reports, not against the pipeline-generated fact sheets. In the final round, 600 rationales were reviewed and 11 were judged incorrect (1.8%). Table R5 below gives every round. We explain below.

  1. 600 rationales reviewed in the final round, 11 judged incorrect (1.8%). Table R5 lists every round: the rationales in that build, the number sampled (~5%), the readers per rationale, the errors found, and the error rate. Each error was traced to its cause and fixed at the source. The pipeline was rerun before the next round.
  2. Disagreements: each sampled rationale was read by one physician, due to the enormous time and effort, so there is no inter-rater agreement. We now say so in Appendix B.5.
  3. The final round is an independent audit, verified against the original CT images and reports, not the pipeline-generated fact sheets. The fact sheet is a pipeline output; checking against it would only test one stage. An error was any rationale that missed or invented a finding, gave a wrong temporal-change label, misapplied a guideline, or reached a conclusion that the report and the scan do not support. The sample was drawn from the output of the final pipeline run, and no rebuild followed, so the rationales it checked are the ones in the dataset.

Table R5. Physician review by round. A round samples ~5% of the reference rationales of that build at random; errors are fixed at the source and the pipeline is rerun before the next round.

roundrationales in the buildsampledreaders per rationaleerrors founderror rate
111,79558016711.6%
211,7955801315.3%
311,7955801223.8%
411,7956001162.7%
5 (final)11,7956001111.8%

The sample sizes, error counts, readers per rationale, the disagreement procedure, and what the review verified against are now in Appendix B.5 of the revised PDF. We appreciate your careful reading of this part of the paper; these details make the paper stronger.

AuthorsOpenReview ↗

Response to Reviewer bN9D (2/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!

Q2. Section 4.2 primarily compares fine-tuned models with zero-shot models, leaving improvements attributable to ordinary domain adaptation, familiarity with question templates, or output-format learning. Appendix F describes a matched comparison between conclusion-only supervision and supervision of intermediate answers plus conclusions, but the corresponding quantitative comparison is not presented in Tables 2 or 12. Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.

Appendix F describes conclusion-only supervision and supervision of intermediate answers plus conclusions. Were both conditions evaluated? If so, could you provide their comparative results and specify which factors were matched, particularly the backbone, training cases, image inputs, training budget, and output format?

Thank you for the careful reading of Appendix F. Yes, both conditions were evaluated, with the backbone, training cases, image inputs, training budget, and output format all matched. Supervision of intermediate answers plus conclusions raises the share of sequences with every task right from 5.4% to 28.3% and the conclusion from 16.7% to 50.1% (Table R1; Table 13 in Appendix F of the revised PDF). The reinforcement-learning rewards are answered in part (3/3). We explain below.

  1. The backbone, training cases, image inputs, training budget, and output format are all matched. Backbone: the same released Primus (NV-Reason-CT, 4B) weights. Training cases: the same training split, the same individual VQA examples, and the same linked sequences. Image inputs: the same CT inputs and preprocessing. Training budget: the same LoRA and optimizer settings, seed, and 22,104 optimizer updates (one epoch). Output format: identical for the individual VQA examples; for the linked sequences the training text differs by design, the conclusion alone or the intermediate answers followed by the conclusion. At evaluation, both checkpoints answered the same test sequences and were scored on their explicit answers, not on the layout of the response. Domain adaptation, familiarity with question templates, and output-format learning are therefore the same in both conditions.
  2. The gains come from supervising the intermediate answers. Conclusion-only supervision reaches a Phase 1-4 average of 25.0% and 5.4% categorical chain accuracy. Supervision of intermediate answers plus conclusions reaches 63.9% and 28.3%, and raises patient-level accuracy from 10.4% to 98.0%. The improvement is largest on the intermediate tasks (Phases 1 to 3): 28.1%, 3.9%, and 51.4% against 69.7%, 71.9%, and 63.9% (Table R1).

Table R1. Primus (NV-Reason-CT, 4B) fine-tuned with conclusion-only supervision vs. intermediate-plus-conclusion supervision, under the same training settings. The zero-shot row is without fine-tuning, from Table 2.

supervisionPhase 1Phase 2Phase 3Phase 4avg. (1-4)patient-levelmeasurements in rangecategorical chain accuracy
none (zero-shot, Table 2)2.81.022.24.27.50.00.00.0
conclusion only28.13.951.416.725.010.40.05.4
intermediate answers + conclusions (SFT, Table 2)69.771.963.950.163.998.010.428.3

With all five factors matched, intermediate supervision is what improves the performance of intermediate answers, the conclusions, and the categorical chain accuracy. Section 4.2 of the revised PDF now states this.

AuthorsOpenReview ↗

Response to Reviewer bN9D (3/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!

Q2 (continued). Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.

Thank you for raising this. We ran both rewards defined in Appendix F.1, and both are now in Table 14 in Appendix F of the revised PDF; Table R6 below reproduces them.

The intermediate-step-plus-conclusion reward improves over the conclusion-only reward on the linked-question metrics we report. Both rewards start from the same SFT adapter and share the training sequences, prompts, group size, generation cap, and seed. Table R6 gives both on the linked-question evaluation. Relative to conclusion-only, intermediate-step-plus-conclusion raises Phases 1 to 3 and Avg. 1–4 from 65.5% to 69.2%, measurements in range from 5.2% to 11.9%, and categorical-chain accuracy from 30.4% to 33.2%. Phase 4 is 58.5% versus 56.8%. Patient-level accuracy is similar (98.7% versus 98.5%). We now report both in Table 14 so that readers can see the comparison.

Table R6. Reinforcement learning with the two rewards (percent).

rewardPhase 1Phase 2Phase 3Phase 4Avg. 1–4measurements in rangecategorical-chain accuracy
conclusion-only69.470.765.356.865.55.230.4
intermediate-step + conclusion71.674.472.258.569.211.933.2

Q3. I strongly recommend the authors provide an overview figure to describe the collecting dataset pipeline. It can help readers understand the paper easier.

Very good suggestion, thank you! We have added Figure 4 in Appendix B, which draws the eight stages of Table 8 as one figure, from the three sources (the CT with its segmentations, the report, and the metadata) through to the final automated audit.


Q4. The study's title uses the term "radiology", but the proposed dataset focus only in CT imaging modality. Therefore, I would like to suggest to provide more specific title for the study.

Thank you for the suggestion! We have revised the title to “RadThinking: Teaching Vision-Language Models to Read Abdominal CT the Way Radiologists Are Taught” to make the imaging modality clear.


Q5. Could the authors provide the institutional review board (IRB) or ethics committee approval number for each institution that contributed CT images to the dataset?

Thank you for asking. Yes, the IRB approvals were obtained before data curation. Because the review is double-blind, we cannot reveal the approval numbers here. We will add them to the Ethics Statement in the camera-ready version.

Reviewer_oYHbOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

This study creates RadThinking, comprising 122,127 questions over 5,365 abdominal CT scan images of 2,679 patients from two countries

Critical strengths

The dataset is reasonably large: 122,127 questions over 5,365 abdominal CT scan images of 2,679 patients from two countries

Critical weaknesses

Innovations are lacking. Importantly, this study did not cite relevant work, including MedThinkVQA, ChestX-Reasoner, M3CoTBench, GEMeX-RMCoT, 3DReasonKnee, and MedReason-R1

Questions for authors

Should MedGemma 1.5 4B be a baseline model?

5 comments
AuthorsOpenReview ↗

Response to Reviewer oYHb (1/3): Request for Reconsideration Based on Clarifications and Additional Evidence

Thank you for reviewing our paper and for recognizing that "the dataset is reasonably large". Beyond its size, the dataset does two things. (1) It provides a sequence of VQA tasks tests that show where frontier vision-language models (VLMs) such as GPT-6 fail. (2) It also enables small open-source VLMs (e.g., Primus (NV-Reason-CT)) to outperform large proprietary VLMs (e.g., GPT-6, Claude Opus 5). We explain this in the point-by-point response below.


Q1. Innovations are lacking.

We introduce a new benchmark methodology: a sequence of VQA tasks that train and test VLMs for complex decision making in radiology. We develop these tasks based on the clinical guidelines that are widely used to train human radiologists. Every radiologist must perform this sequence of VQA tasks when reading medical images. We use these same tasks to analyze where frontier VLMs such as GPT-6 fail, and further develop a small open-source VLM (4B) to perform much better than enormously large proprietary VLMs. We hope this new benchmark methodology can be extended to other domains and inspire applications there.

We explain below.

  1. A reference answer for every intermediate task. In the related benchmarks listed in Q2, the intermediate tasks are written reasoning judged as text; only the final answer has a reference. In RadThinking every task in the sequence is a question with a reference answer, linked to the tasks it depends on, and the final decision task (the conclusion of a sequence) follows a clinical guideline written as a decision table (Section 2.3, Appendix B.4). A task is therefore checked exactly, and a sequence fails at a known task.
  2. It shows where frontier VLMs fail. The best zero-shot accuracy on the final decision is 56.8%, yet no VLM gets every task of a sequence right in more than 23.5% of sequences (categorical chain accuracy in Table 1), and 63% of Hulu-Med-32B’s correct decisions come with at least one wrong intermediate answer (Section 1); the weakest task is the comparison with the earlier scan (Phase 3). The decision follows the intermediate tasks: swapping two written sizes, with the images unchanged, lowers Qwen3.8-27B’s agreement with the reference decision from 48.1% to 11.1% (Section 4.1).
  3. Fixing those tasks fixes the VLM. Training a 4B Primus on the intermediate answers together with the conclusion raises the share of sequences with every task right from 5.4% (conclusion-only training) to 28.3%, and decision accuracy from 16.7% to 50.1% (Table 13 in Appendix F of the revised PDF), and the fine-tuned 4B VLM exceeds zero-shot GPT-6, Claude Opus 5, and Gemini 3.8 Flash on individual VQA tasks and on full sequences (Section 4.2).

RadThinking is therefore, to our knowledge, the first dataset in which every task of a radiology read has a reference answer, grounded in a 3D box, linked across the scans of a patient, and checked against five clinical guidelines as decision tables (Section 1, Appendix C). The reference answers come from the radiologists’ reports and radiologist-reviewed segmentations, and were checked by eight physicians over five rounds (Section 2.5). Where a VLM fails is visible, and the reference answer it must learn is given for every task.

The other two reviewers describe the contribution the same way. Reviewer 2T59: "This is the paper's clearest contribution in dataset design" and "For a dataset-track submission, this contribution matters more than model novelty." Reviewer bN9D: "Evaluating intermediate answers exposes failures that conclusion-only assessment can conceal." Our submission is in the datasets and benchmarks area. It shows where frontier VLMs fail and gives a reference answer for every intermediate task, so the failure can be fixed: trained on these answers, a small open-source VLM (4B) outperforms GPT-6, Claude Opus 5, and Gemini 3.8 Flash on this task.

AuthorsOpenReview ↗

Response to Reviewer oYHb (2/3): Request for Reconsideration Based on Clarifications and Additional Evidence

Q2. Importantly, this study did not cite relevant work, including MedThinkVQA, ChestX-Reasoner, M3CoTBench, GEMeX-RMCoT, 3DReasonKnee, and MedReason-R1.

Very helpful - thanks! Following your suggestions, we have now added and discussed all six papers in the revised related work (Section 1 and Appendix C, Table 10). Although these papers are related to our work, none of them presents a benchmark methodology that addresses the three critical challenges identified in Q1. Here, we summarize key differences between these papers and our RadThinking benchmark:

  1. Scans linked over time: 2 of the 6. ChestX-Reasoner [2] has one task with a current and a prior radiograph of the same patient; about a third of MedThinkVQA [1] cases include follow-up images, with one diagnosis question per case. RadThinking links all the scans of a patient: 1,411 of the 2,679 patients have two or more scans, 4,214 findings carry a temporal-change label (Phase 3), and patient-level questions ask about one organ across all of the patient's scans (Section 2.2).
  2. Clinical guidelines as decision tables: 0 of the 6. 3DReasonKnee [5] records MOAKS grades, a scoring system; none implements a published clinical guideline as a decision table that maps imaging features to a category and a recommended follow-up. RadThinking implements five (LI-RADS, Bosniak, Fleischner, the ACR white papers on adrenal masses and pancreatic cysts), so the benchmark checks the features and the rule, not only the label (Appendix B.4).
  3. Sixteen organs on 3D CT: 0 of the 6. Five of the six use 2D images; the one 3D dataset, 3DReasonKnee [5], covers one body part, the knee. RadThinking covers 16 organs on 3D abdominal CT of 2,679 patients, 1,153 of them with a recorded cancer diagnosis (Section 2.4).

Table R3. RadThinking and the six related papers, compared on the three differences above and on how each task is checked.

resourceimagesscans linked over timeclinical guidelines as rulestask checked on the 3D scan
MedThinkVQA [1]2D case images, many specialties (8,067 Eurorad cases)a third of the cases have follow-up images; one diagnosis question per casenono (tasks checked as text)
ChestX-Reasoner [2]chest radiographs (59k questions, 301k tasks)one task: current versus prior radiographnono (tasks checked as text)
M3CoTBench [3]2D images, 24 examination types, 13 tasksnonono (tasks checked as text)
GEMeX-RMCoT [4]chest radiographsnono2D regions
3DReasonKnee [5]3D knee MRI, one body part (7,970 volumes, 494k items)no (one scan per question; volumes from a longitudinal study)MOAKS grades (a scoring system)one 3D box per question
MedReason-R1 [6]chest CT slices, 2D (CT-RATE-VQA, 84k questions)nonono (model-generated tasks)
RadThinking3D abdominal CT, 16 organs (5,365 scans, 2,679 patients, 122,127 questions)yes (1,411 patients; findings matched across scans, Phase 3)five decision tablesevery question

The six papers are now cited and compared with RadThinking in the revised PDF. None of them does what RadThinking does, so the contribution stated in Q1 stands. We appreciate you bringing these six papers to our attention.


Reference

  • [1] Yao, Zonghai, Benlu Wang, Yifan Zhang, Junda Wang, Iris Xia, Zhipeng Tang, Shuo Han, Feiyun Ouyang, Zhichao Yang, Arman Cohan, and Hong Yu. "Medical thinking with multiple images." ICLR (2026).
  • [2] Fan, Ziqing, Cheng Liang, Chaoyi Wu, Ya Zhang, Yanfeng Wang, and Weidi Xie. "ChestX-Reasoner: Advancing radiology foundation models with reasoning through step-by-step verification." arXiv:2504.20930 (2025).
  • [3] Jiang, Juntao, Jiangning Zhang, Yali Bi, Jinsheng Bai, Weixuan Liu, Weiwei Jin, Zhucun Xue, Yong Liu, Xiaobin Hu, and Shuicheng Yan. "M3CoTBench: Benchmark chain-of-thought of MLLMs in medical image understanding." ICLR (2026).
  • [4] Liu, Bo, Xiangyu Zhao, Along He, Yidi Chen, Huazhu Fu, and Xiao-Ming Wu. "GEMeX-RMCoT: An enhanced Med-VQA dataset for region-aware multimodal chain-of-thought reasoning." ACM Multimedia (2025).
  • [5] Sambara, Sraavya, Sung Eun Kim, Xiaoman Zhang, Luyang Luo, Shreya Johri, Mohammed Baharoon, Du Hyun Ro, and Pranav Rajpurkar. "3DReasonKnee: Advancing grounded reasoning in medical vision language models." arXiv:2510.20967 (2025).
  • [6] Li, Yifan, Fenghe Tang, Yingtai Li, and S. Kevin Zhou. "MedReason-R1: Learning to reason for CT diagnosis with reinforcement learning and local zoom." arXiv:2510.19626 (2025).
AuthorsOpenReview ↗

Response to Reviewer oYHb (3/3): Request for Reconsideration Based on Clarifications and Additional Evidence

Q3. Should MedGemma 1.5 4B be a baseline model?

Very good suggestion! We have now added MedGemma 1.5-4B [1] to Table 1, together with MedReason-R1 [2] from your suggested related work and MediX-R1-8B [3]. All three are zero-shot and evaluated in the same way as the other models in Table 1. Table R4 gives the results of these newly added baselines. None of the three changes the paper’s findings.

  1. MedGemma 1.5-4B does not change the ranking. It reaches 38.1% on individual VQA (Phase 1-4 average) and gets every task of a sequence right in 2.3% of sequences, against 50.2% and 23.5% for the best zero-shot rows of Table 1, and 60.8% and 28.3% for the 4B Primus fine-tuned on RadThinking (Table 2), a VLM of the same size.
  2. Reasoning VLMs do not pass the sequence of VQA tasks either. MedReason-R1, trained with reinforcement learning to reason on chest CT, gets every task right in 2.9% of sequences; MediX-R1-8B, the best of the three, in 12.0%, still below the 23.5% of Table 1. Reasoning toward the answer does not make the intermediate answers right, which is the point of Q1.
  3. The sequence of VQA tasks shows where each VLM fails. MedGemma 1.5-4B tells normal from abnormal regions well (64.7% on Phase 1) but fails on the comparison with the earlier scan (23.3% on Phase 3) and on the decision (27.5% on Phase 4); MediX-R1-8B is the opposite, 43.8% on Phase 3 but 16.1% on describing the finding (Phase 2). One overall accuracy would show none of this.

Table R4. The three new zero-shot baselines on RadThinking (percent), now added to Table 1. Individual VQA: each task answered on its own; linked questions: the whole sequence answered in one response. Avg. is the Phase 1-4 average; chain is the share of sequences with every task right (categorical chain accuracy in Table 1); patient-level accuracy and measurements in range are the same as in Table 1.

evaluationmodelPhase 0Phase 1Phase 2Phase 3Phase 4avg. (1-4)patient-levelmeasurements in rangechain
individual VQAMedGemma 1.5-4B [1]23.264.736.623.327.538.1n/a1.1n/a
individual VQAMedReason-R1 [2]13.465.131.524.723.436.2n/a1.0n/a
individual VQAMediX-R1-8B [3]35.461.916.143.840.640.6n/a0.5n/a
linked questionsMedGemma 1.5-4B [1]24.257.317.66.96.122.08.62.52.3
linked questionsMedReason-R1 [2]26.063.29.08.34.521.24.70.22.9
linked questionsMediX-R1-8B [3]27.448.528.515.328.130.146.60.012.0

No zero-shot VLM, general or medical, gets every task right in more than 23.5% of sequences, and the 4B VLM trained on RadThinking exceeds all of them, including the three added here, at the same size as MedGemma 1.5-4B. The three new baselines change none of the paper’s findings.


Reference

  • [1] Sellergren, Andrew, Chufan Gao, Fereshteh Mahvar, Timo Kohlberger, Fayaz Jamil, et al. “MedGemma 1.5 technical report.” arXiv:2604.05081 (2026).
  • [2] Li, Yifan, Fenghe Tang, Yingtai Li, and S. Kevin Zhou. “MedReason-R1: Learning to reason for CT diagnosis with reinforcement learning and local zoom.” arXiv:2510.19626 (2025).
  • [3] Mullappilly, Sahal Shaji, Mohammed Irfan Kurpath, Omair Mohamed, Mohamed Zidan, Fahad Khan, Salman Khan, Rao Anwer, and Hisham Cholakkal. “MediX-R1: Open ended medical reinforcement learning.” arXiv:2602.23363 (2026).
Reviewer_2T59OpenReview ↗
Rating
4(+2)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

RadThinking organizes abdominal CT interpretation into linked questions with reference answers, 3D evidence localization, and explicit dependencies. It covers regional screening, finding characterization, temporal changes, and clinical decisions. The dataset contains 122,127 questions over 5,365 CT volumes from 2,679 patients and incorporates five clinical guidelines. The authors evaluate eight models zero-shot and use hidden-state probes and interventions on intermediate answers to examine the relationship between intermediate evidence and final conclusions. Supervised fine-tuning and reinforcement learning experiments assess the dataset’s training utility. Fine-tuning substantially improves categorical chain accuracy, although this metric excludes numerical errors. The contribution of intermediate supervision beyond task-specific adaptation remains to be established through a matched comparison.

The paper addresses a meaningful problem. Its novelty rests on the combination of longitudinal 3D CT question chains, intermediate reference answers, and explicit dependencies. The related-work discussion and dataset comparisons explain these structural differences, but matched training experiments have not established the additional benefit of intermediate supervision. The paper also does not report whether the training gains transfer to independently constructed medical imaging benchmarks.

Critical strengths
  1. A question dependency graph that supports evaluation and supervision of individual interpretation steps. Questions record reference answers and evidence sources and link to relevant 3D image regions. Questions requiring combined evidence additionally record their dependencies. This connects finding detection, characterization, temporal changes, and clinical decisions. Researchers can identify where a model fails and whether a correct conclusion accompanies incorrect intermediate evidence. The same annotations also support process supervision. This is the paper’s clearest contribution in dataset design.

  2. A longitudinal abdominal CT resource with demonstrated training utility. The dataset contains 2,679 patients and 5,365 CT volumes from two countries, including 1,411 patients with multiple scans. It supports research on changes in findings and patient trajectories beyond single-scan VQA. Patient-level splitting and the held-out Turkish cohort provide a reasonable evaluation foundation. Supervised fine-tuning increases categorical chain accuracy from 0.0% to 28.3% for Primus and from 10.9% to 30.6% for Hulu-Med, supporting the resource’s training utility. For a dataset-track submission, this contribution matters more than model novelty.

Critical weaknesses
  1. The key comparison isolating the contribution of intermediate supervision is not reported. Appendix F on page 31 explicitly describes two matched conditions: supervising only final conclusions and supervising both intermediate answers and final conclusions. However, the manuscript does not report numerical results comparing these conditions. This comparison is central to the paper’s argument. Do the gains come from intermediate supervision, or simply from fine-tuning on the target task?
  2. The transferability of the training gains across datasets has not been evaluated. All reported fine-tuning gains are measured within RadThinking’s own evaluation framework. Although the test set includes held-out Turkish patients, it uses the same question construction, annotation, and scoring procedures. The current evidence therefore does not establish whether these gains transfer to independent medical imaging tasks. The paper does not evaluate its fine-tuned models on a task-matched external benchmark.
Questions for authors
  1. Has the matched comparison between conclusion-only supervision and intermediate-answer-plus-conclusion supervision described in Appendix F been completed? If so, please provide the results and their location. Otherwise, please run this comparison using the same base model, training split, image inputs, and comparable training budgets. Please report performance on both intermediate answers and final conclusions.

  2. Have the RadThinking-fine-tuned models been evaluated on an independent 3D medical imaging benchmark? If not, please compare the same model before and after RadThinking fine-tuning on a task-matched external benchmark, such as DeepTumorVQA or relevant M3D-Bench VQA tasks. Keep preprocessing and scoring fixed, with no additional training on the external benchmark. I consider this important evidence for assessing the transferability of the training gains.

If the authors provide the experiments and results requested above, I would consider raising my score.

4 comments
AuthorsOpenReview ↗

Response to Reviewer 2T59 (1/2): Thank You for Your Two Suggestions That Made Our Paper Stronger; Both Experiments Completed, Both Support Our Claim

Thank you for the constructive review; your two suggested experiments have made our paper stronger! We also appreciate your recognizing "the paper's clearest contribution in dataset design" and that "for a dataset-track submission, this contribution matters more than model novelty".

We ran both experiments you suggested. (1) Supervising the intermediate answers improves the categorical chain accuracy from 5.4% to 28.3% and the conclusion accuracy from 16.7% to 50.1% (see Table R1 below) compared with fine-tuning on the target task alone. (2) The gain carries over to two external benchmarks: from 34.5% to 42.5% on DeepTumorVQA, and from 46.0% to 52.5% on AMOS-MM (see Table R2 below).

These results reiterate the value of a sequence of VQA tasks for training and testing vision-language models (VLMs) on complex decision-making in radiology. We describe both experiments in detail in the point-by-point response below.


Q1. The key comparison isolating the contribution of intermediate supervision is not reported. Appendix F on page 31 explicitly describes two matched conditions: supervising only final conclusions and supervising both intermediate answers and final conclusions. However, the manuscript does not report numerical results comparing these conditions. This comparison is central to the paper's argument. Do the gains come from intermediate supervision, or simply from fine-tuning on the target task?

Has the matched comparison between conclusion-only supervision and intermediate-answer-plus-conclusion supervision described in Appendix F been completed? If so, please provide the results and their location. Otherwise, please run this comparison using the same base model, training split, image inputs, and comparable training budgets. Please report performance on both intermediate answers and final conclusions.

Thank you for catching this! Yes, the comparison was completed before submission. We apologize for accidentally commenting out the comparison table during submission. Following your suggestion, it is now Table 13 in Appendix F of the revised PDF; Table R1 below reproduces it. Phases 1 to 3 are the intermediate answers and Phase 4 is the conclusion, so the table reports both. The gains come from intermediate supervision, not from fine-tuning on the target task alone. We explain below.

  1. Intermediate answers: +40.7 points on average from intermediate supervision. Conclusion-only supervision never trains the intermediate answers, so on the three intermediate tasks (Phases 1 to 3) the VLM reaches only 28.1%, 3.9%, and 51.4% (average 27.8%). Supervising the intermediate answers together with the conclusion, under the same training settings, raises them to 69.7%, 71.9%, and 63.9% (average 68.5%).
  2. Conclusions: +33.4 points from intermediate supervision. Conclusion-only supervision trains the conclusion directly, yet it reaches only 16.7% on the conclusion (Phase 4) and 10.4% patient-level accuracy. Supervising the intermediate answers together with the conclusion, under the same training settings, raises them to 50.1% and 98.0%.
  3. Every task right: +22.9 points from intermediate supervision. The share of sequences with every task right (categorical chain accuracy in Table 1) is 5.4% with conclusion-only supervision and 28.3% with intermediate supervision (compared with zero-shot: 0.0%). So the gain comes from supervising the intermediate answers.

Table R1. Primus (NV-Reason-CT, 4B) fine-tuned with conclusion-only supervision vs. intermediate-plus-conclusion supervision, under the same training settings. The zero-shot row is without fine-tuning, from Table 2.

supervisionPhase 1Phase 2Phase 3Phase 4avg. (1-4)patient-levelmeasurements in rangecategorical chain accuracy
none (zero-shot, Table 2)2.81.022.24.27.50.00.00.0
conclusion only28.13.951.416.725.010.40.05.4
intermediate answers + conclusions (SFT, Table 2)69.771.963.950.163.998.010.428.3

In short, with everything else matched, supervising the intermediate answers raises the intermediate answers (27.8% to 68.5% on average), the conclusions (16.7% to 50.1%), and the share of sequences with every task right (5.4% to 28.3%); fine-tuning on the target task alone, which is what conclusion-only supervision is, raises them far less. So the gains come from intermediate supervision, not simply from fine-tuning on the target task. Table 13 in Appendix F of the revised PDF reports this, and Section 4.2 now states it.

AuthorsOpenReview ↗

Response to Reviewer 2T59 (2/2): Thank You for Your Two Suggestions That Made Our Paper Stronger; Both Experiments Completed, Both Support Our Claim

Q2. The transferability of the training gains across datasets has not been evaluated. All reported fine-tuning gains are measured within RadThinking's own evaluation framework. Although the test set includes held-out Turkish patients, it uses the same question construction, annotation, and scoring procedures. The current evidence therefore does not establish whether these gains transfer to independent medical imaging tasks. The paper does not evaluate its fine-tuned models on a task-matched external benchmark.

Have the RadThinking-fine-tuned models been evaluated on an independent 3D medical imaging benchmark? If not, please compare the same model before and after RadThinking fine-tuning on a task-matched external benchmark, such as DeepTumorVQA or relevant M3D-Bench VQA tasks. Keep preprocessing and scoring fixed, with no additional training on the external benchmark. I consider this important evidence for assessing the transferability of the training gains.

Thank you for this suggestion, which we agree is important evidence. Following it, we have now compared the same VLM before and after RadThinking fine-tuning on two task-matched external benchmarks. The gains transfer: on DeepTumorVQA [1] accuracy rises from 34.5% to 42.5%, and on AMOS-MM [2] from 46.0% to 52.5% (Table R2; Table 15 in Appendix F of the revised PDF). We explain below.

  1. M3D-Bench cannot be evaluated, so we followed your suggestion of DeepTumorVQA and also added AMOS-MM. The M3D-Bench VQA questions are available, but the CT volumes they refer to (M3D-Cap) were removed from Hugging Face after a copyright notice from Radiopaedia (https://huggingface.co/datasets/GoodBaiBai88/M3D-Cap/discussions/4). DeepTumorVQA is abdominal CT, as RadThinking is; we used its two reasoning categories, visual reasoning and medical reasoning. AMOS-MM is 3D CT of the chest, abdomen, and pelvis; we used its official validation split, disease-analysis and clinical-decision questions (risk evaluation and therapeutic suggestion). Both are multiple-choice benchmarks, and neither shares RadThinking's question construction, annotation, or scoring.
  2. The gains transfer: +8.0 points on DeepTumorVQA and +6.5 points on AMOS-MM. On both benchmarks the gain is largest on the questions answered by reading the scan, which is what RadThinking trains (visual reasoning, 25.0% to 38.0%; disease analysis, 44.0% to 55.0%). The questions that need medical knowledge beyond the scan also improve (medical reasoning, +3.0; clinical decision, +2.0). So the gain from fine-tuning a VLM on RadThinking directly transfers to other benchmarks.

Table R2. Accuracy (percent) of the same VLM "before and after RadThinking fine-tuning", tested directly on two external benchmarks. Both VLMs received the same CT inputs, each benchmark’s original prompts, and its own scorer; no DeepTumorVQA or AMOS-MM example was used for fine-tuning. Because of the rebuttal time, we evaluated a random subset of each benchmark (200 questions, 100 per category). AMOS-MM clinical decision combines 50 risk-evaluation and 50 therapeutic-suggestion questions. We will post results on the full test sets as soon as those runs finish.

benchmarkquestionsnbefore RadThinking fine-tuningafter RadThinking fine-tuningdifference
DeepTumorVQAvisual reasoning10025.038.0+13.0
DeepTumorVQAmedical reasoning10044.047.0+3.0
DeepTumorVQAall20034.542.5+8.0
AMOS-MMdisease analysis10044.055.0+11.0
AMOS-MMclinical decision10048.050.0+2.0
AMOS-MMall20046.052.5+6.5

In short, the gain from fine-tuning on RadThinking transfers: +8.0 points on DeepTumorVQA and +6.5 points on AMOS-MM, with no training on either benchmark (Table 15 in Appendix F of the revised PDF).

Both experiments you asked for are now in the paper: the gain comes from intermediate supervision (Q1, Table 13) and transfers to other benchmarks (Q2, Table 15). Thank you again for the constructive review. We hope this resolves the two weaknesses and, as you kindly indicated, we would be grateful if you would consider raising your score!


Reference

  • [1] Chen, Yixiong, Wenjie Xiao, Pedro R. A. S. Bassi, Xinze Zhou, Sezgin Er, Ibrahim Ethem Hamamci, Zongwei Zhou, and Alan Yuille. "Are vision language models ready for clinical diagnosis? A 3D medical benchmark for tumor-centric visual question answering." arXiv:2505.18915 (2025).
  • [2] Ji, Yuanfeng, Chongjian Ge, Ruijiang Li, and Ping Luo. "AMOS-MM: Abdominal Multimodal Analysis Challenge." MICCAI 2024 challenge, Zenodo (2024). https://doi.org/10.5281/zenodo.10992155

Thank you for the additional experiments. The matched supervision comparison and external benchmark results address my main concerns and strengthen the evidence for the dataset’s training value, so I will raise my score accordingly.

AuthorsOpenReview ↗

We appreciate your strong support!

Dear Reviewer 2T59,

Thank you for raising your score, and for the two suggestions that made our paper stronger.

We would like to ask what we could do to earn a 4 from you. We are happy to provide more before the discussion period ends.

Unfortunately, we received a surprisingly negative score, as you can see, and the paper is now on the borderline. We certainly do not want this to delay the public release of the RadThinking dataset. A rejection and resubmission would cost at least another six months. We have already given more than 20 research teams around the world early access to RadThinking to support their research, and we would like it to benefit many more.

Our team has a strong track record of publishing high-quality, unique datasets for the medical community. Longitudinal reasoning is a very important but understudied area, largely because public datasets are lacking. So we are eager to release this dataset as soon as possible, and to see it receive the recognition it deserves.

We would greatly appreciate your endorsement.

Yours, sincerely,

RadThinking Team

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response (1/2): We Studied a Debate in Radiology AI: Should Complex Decision Making Be Taught as a Sequence of Tasks, the Way Radiologists Are Taught, or Learned End to End from Enormously Big Data and Compute?

Dear reviewers and ACs,

Thank you all for your time and care. Your reviews recognize two practical contributions of our RadThinking dataset. A radiologist reads a CT scan as a sequence of tasks, from finding each lesion to applying a clinical guideline. First, RadThinking gives a reference answer for every task, not only for the final conclusion. Second, it scores vision-language models (VLMs) on every task. Together, these two make a third contribution: they turn a debate about reading CT into experiments. Must complex decision making be taught as a sequence of tasks, or can it be learned end to end at scale?

Two of you asked whether our gains come from intermediate supervision or simply from fine-tuning on the target task. Your question is at the center of the debate. Our point-by-point replies, and the experiments you asked for, are under your reviews and in the revised PDF. Here we share our current view of the debate with you.

01 | The debate

Interpreting an abdominal CT scan is complex decision making. A radiologist screens each region, describes each finding, compares it with the earlier scan, and applies a clinical guideline to reach the conclusion. How should a VLM learn this?

  • Strategy A: teach the tasks. People define the intermediate tasks and label their answers. The VLM is supervised on every task.
  • Strategy B: learn at scale. People label only the conclusion. With enough data, the VLM finds its own way to it.

Strategy B has a great record: self-play in Go, and language models trained on text at scale. It is the strategy the "bitter lesson" [1] favors, and we admire it.

02 | Why RadThinking takes Strategy A

  1. Medical data are scarce. Strategy B needs huge amounts of data. A Go program can make its own data by playing against itself. Language models learn from text that is already on the internet. Medicine has neither. CT scans are private and stay in hospitals. Patient outcomes take years to know. Scans of the same patient over time are rarer still. We assume this will not change soon. So each scan must teach the model more.

  2. The sequence of tasks are already defined. Radiologists learn them in training. Clinical guidelines state the rules. With Strategy B, a VLM must find these rules from data alone. That would cost a fortune in scans and compute. RadThinking gives every task a ground-truth answer, so every task can be trained: 122,127 answers across 5,365 CT scans. This brings the guidelines into VLM training, so the VLM can learn them faster. The VLM and its training method stay the same.

  3. A radiologist can check the answers. Each scan-level task is grounded in a 3D box on the scan. So the VLM's behavior can be tracked task by task. When it makes a mistake, you can see at which task, and study why.

AuthorsOpenReview ↗

General Response (2/2): We Studied a Debate in Radiology AI: Should Complex Decision Making Be Taught as a Sequence of Tasks, the Way Radiologists Are Taught, or Learned End to End from Enormously Big Data and Compute?

03 | Turning the debate to experiments

With RadThinking, you can finally test this debate: the same scans can supervise the conclusion alone, or every task.

  1. In one matched comparison, supervising the whole sequence tripled conclusion accuracy. Same VLM, same training scans, same training budget. Both runs train on the same individual tasks and the same sequences. They differ in which tasks of each sequence are supervised. Zero-shot: 4.2% conclusion accuracy. The conclusion alone: 16.7%. The intermediate answers as well: 50.1% (Table 13 in Appendix F). This is one 4B VLM, one data size, and one seed.

  2. A correct conclusion does not guarantee a correct read. Across 11 zero-shot VLMs, the best reaches 56.8% on the conclusion, above the 4B VLM supervised on RadThinking (50.1%). Yet none gets every task right in more than 23.5% of sequences. The supervised VLM reaches 28.3% (categorical chain accuracy in Tables 1 and 2; numerical answers are scored separately). For Hulu-Med-32B, 63% of the correct conclusions come with at least one wrong intermediate answer (Section 1). Strategy B may call this the VLM's own way. Strategy A may call it luck. Either way, an answer in the read is wrong, and only a reference answer for every task lets you see it.

Two results outside medicine agree. In theory, some problems become learnable only when their intermediate tasks are supervised [2]. A driving model plans better with its supervised intermediate tasks than without them [3].

04 | Scale matters too

We do not claim that Strategy B cannot work in medicine. If hospitals one day share clean data at scale, Strategy B may catch up, and we would be glad to see it.

With our the RadThinking dataset, you can now measure the debate. Train one VLM twice on the same scans: on the conclusion alone, and on every task. Repeat with more scans. While the gap holds, teaching the tasks pays. If it closes on every task, scale is enough. Table 13 is a first point on this curve. Its conclusion-only run also trains on the individual tasks, so it is not yet pure Strategy B.

Whichever strategy is right, with RadThinking, you can now score any VLM, however it was trained, on every task of the read, and a radiologist can check each answer. We respectfully ask the reviewers and the Area Chair to consider this contribution.

Reference

  • [1] Sutton, Rich. "The bitter lesson." (2019).
  • [2] Wies, Noam, Yoav Levine, and Amnon Shashua. "Sub-task decomposition enables learning in sequence to sequence tasks." ICLR (2023).
  • [3] Hu, Yihan, Jiazhi Yang, Li Chen et al. "Planning-oriented autonomous driving." CVPR (2023).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.