- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The study introduces a dataset of 122,127 questions spanning 5,365 CT scans, with visually grounded evidence, reference answers, dependency links, and explicit clinical guideline rules. By connecting intermediate observations to clinical conclusions, the dataset enables evaluation of both model predictions and reasoning steps. By connecting intermediate observations to clinical conclusions, the dataset enables evaluation of both model predictions and their supporting evidence. It provides a useful resource for assessing and improving the reasoning capabilities of medical vision-language models.
Critical strengths
-
A substantial dataset for training CoT LLM RadThinking connects 122,127 questions across 5,365 CT scans with visual grounded evidence, reference answers, dependency links, and explicit clinical guideline rules. This dataset is useful to address the explainable challenge of reasoning LLM in medical applications. It also provides a tool for evaluating and improving the reasoning capacity of LLM in medical applications.
-
Extensive informative benchmark Evaluating intermediate answers exposes failures that conclusion-only assessment can conceal. No zero-shot model exceeds 23.5% categorical chain accuracy. The dataset also supports substantial improvements through supervised fine-tuning. Although these comparisons do not isolate the benefit of intermediate supervision, they demonstrate that the resource provides useful training LLM models in medical applications. In other words, the dataset is useful for identifying model weaknesses and improve performance.
Critical weaknesses
-
Although the dataset is extracted from CT images and reports, LLMs participate in extracting findings and constructing reference rationales. Appendix B.5 describes physician review of 5% of rationales per round, but the reported error rate below 2% is not accompanied by exact sample sizes, error counts, inter-rater agreement, or a clearly documented independent final audit.
-
Section 4.2 primarily compares fine-tuned models with zero-shot models, leaving improvements attributable to ordinary domain adaptation, familiarity with question templates, or output-format learning. Appendix F describes a matched comparison between conclusion-only supervision and supervision of intermediate answers plus conclusions, but the corresponding quantitative comparison is not presented in Tables 2 or 12. Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.
Questions for authors
-
I strongly recommend the authors provide an overview figure to describe the collecting dataset pipeline. It can help readers understand the paper easier.
-
In the final physician-review round described in Appendix B.5, how many rationales were reviewed, how many were judged incorrect, and how were reviewer disagreements resolved? Did this review independently verify intermediate answers and conclusions against the original CT images and reports, or assess consistency with the pipeline-generated fact sheets?
-
Appendix F describes conclusion-only supervision and supervision of intermediate answers plus conclusions. Were both conditions evaluated? If so, could you provide their comparative results and specify which factors were matched, particularly the backbone, training cases, image inputs, training budget, and output format?
-
The study's title uses the term "radiology", but the proposed dataset focus only in CT imaging modality. Therefore, I would like to suggest to provide more specific title for the study.
6 comments
Response to Reviewer bN9D (2/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!
Q2. Section 4.2 primarily compares fine-tuned models with zero-shot models, leaving improvements attributable to ordinary domain adaptation, familiarity with question templates, or output-format learning. Appendix F describes a matched comparison between conclusion-only supervision and supervision of intermediate answers plus conclusions, but the corresponding quantitative comparison is not presented in Tables 2 or 12. Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.
Appendix F describes conclusion-only supervision and supervision of intermediate answers plus conclusions. Were both conditions evaluated? If so, could you provide their comparative results and specify which factors were matched, particularly the backbone, training cases, image inputs, training budget, and output format?
Thank you for the careful reading of Appendix F. Yes, both conditions were evaluated, with the backbone, training cases, image inputs, training budget, and output format all matched. Supervision of intermediate answers plus conclusions raises the share of sequences with every task right from 5.4% to 28.3% and the conclusion from 16.7% to 50.1% (Table R1; Table 13 in Appendix F of the revised PDF). The reinforcement-learning rewards are answered in part (3/3). We explain below.
- The backbone, training cases, image inputs, training budget, and output format are all matched. Backbone: the same released Primus (NV-Reason-CT, 4B) weights. Training cases: the same training split, the same individual VQA examples, and the same linked sequences. Image inputs: the same CT inputs and preprocessing. Training budget: the same LoRA and optimizer settings, seed, and 22,104 optimizer updates (one epoch). Output format: identical for the individual VQA examples; for the linked sequences the training text differs by design, the conclusion alone or the intermediate answers followed by the conclusion. At evaluation, both checkpoints answered the same test sequences and were scored on their explicit answers, not on the layout of the response. Domain adaptation, familiarity with question templates, and output-format learning are therefore the same in both conditions.
- The gains come from supervising the intermediate answers. Conclusion-only supervision reaches a Phase 1-4 average of 25.0% and 5.4% categorical chain accuracy. Supervision of intermediate answers plus conclusions reaches 63.9% and 28.3%, and raises patient-level accuracy from 10.4% to 98.0%. The improvement is largest on the intermediate tasks (Phases 1 to 3): 28.1%, 3.9%, and 51.4% against 69.7%, 71.9%, and 63.9% (Table R1).
Table R1. Primus (NV-Reason-CT, 4B) fine-tuned with conclusion-only supervision vs. intermediate-plus-conclusion supervision, under the same training settings. The zero-shot row is without fine-tuning, from Table 2.
| supervision | Phase 1 | Phase 2 | Phase 3 | Phase 4 | avg. (1-4) | patient-level | measurements in range | categorical chain accuracy |
|---|---|---|---|---|---|---|---|---|
| none (zero-shot, Table 2) | 2.8 | 1.0 | 22.2 | 4.2 | 7.5 | 0.0 | 0.0 | 0.0 |
| conclusion only | 28.1 | 3.9 | 51.4 | 16.7 | 25.0 | 10.4 | 0.0 | 5.4 |
| intermediate answers + conclusions (SFT, Table 2) | 69.7 | 71.9 | 63.9 | 50.1 | 63.9 | 98.0 | 10.4 | 28.3 |
With all five factors matched, intermediate supervision is what improves the performance of intermediate answers, the conclusions, and the categorical chain accuracy. Section 4.2 of the revised PDF now states this.
Response to Reviewer bN9D (3/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!
Q2 (continued). Moreover, the reported reinforcement-learning result uses an answer-only reward, rather than demonstrating the benefit of intermediate-step rewards.
Thank you for raising this. We ran both rewards defined in Appendix F.1, and both are now in Table 14 in Appendix F of the revised PDF; Table R6 below reproduces them.
The intermediate-step-plus-conclusion reward improves over the conclusion-only reward on the linked-question metrics we report. Both rewards start from the same SFT adapter and share the training sequences, prompts, group size, generation cap, and seed. Table R6 gives both on the linked-question evaluation. Relative to conclusion-only, intermediate-step-plus-conclusion raises Phases 1 to 3 and Avg. 1–4 from 65.5% to 69.2%, measurements in range from 5.2% to 11.9%, and categorical-chain accuracy from 30.4% to 33.2%. Phase 4 is 58.5% versus 56.8%. Patient-level accuracy is similar (98.7% versus 98.5%). We now report both in Table 14 so that readers can see the comparison.
Table R6. Reinforcement learning with the two rewards (percent).
| reward | Phase 1 | Phase 2 | Phase 3 | Phase 4 | Avg. 1–4 | measurements in range | categorical-chain accuracy |
|---|---|---|---|---|---|---|---|
| conclusion-only | 69.4 | 70.7 | 65.3 | 56.8 | 65.5 | 5.2 | 30.4 |
| intermediate-step + conclusion | 71.6 | 74.4 | 72.2 | 58.5 | 69.2 | 11.9 | 33.2 |
Q3. I strongly recommend the authors provide an overview figure to describe the collecting dataset pipeline. It can help readers understand the paper easier.
Very good suggestion, thank you! We have added Figure 4 in Appendix B, which draws the eight stages of Table 8 as one figure, from the three sources (the CT with its segmentations, the report, and the metadata) through to the final automated audit.
Q4. The study's title uses the term "radiology", but the proposed dataset focus only in CT imaging modality. Therefore, I would like to suggest to provide more specific title for the study.
Thank you for the suggestion! We have revised the title to “RadThinking: Teaching Vision-Language Models to Read Abdominal CT the Way Radiologists Are Taught” to make the imaging modality clear.
Q5. Could the authors provide the institutional review board (IRB) or ethics committee approval number for each institution that contributed CT images to the dataset?
Thank you for asking. Yes, the IRB approvals were obtained before data curation. Because the review is double-blind, we cannot reveal the approval numbers here. We will add them to the Ethics Statement in the camera-ready version.
Response to Reviewer bN9D (1/3): Thank you for appreciating our contributions. This two-year, multi-institutional study involved 36 doctors. It pushes the frontier of vision-language models (VLMs) in complex radiology decision-making by teaching them to think like radiologists. We present RadThinking!
Thank you for the great questions and suggestions; they have really made the paper better, and we appreciate the time you spent on it! We are glad to hear the words "Extensive informative benchmark" and "the dataset is useful for identifying model weaknesses and improve performance". Building a 3D medical imaging dataset of this scale, with physician-reviewed reference answers for every intermediate task, five clinical guidelines as decision tables, and longitudinal links across 5,365 scans of 2,679 patients, took our team more than two years, eight physicians, and five review rounds. Datasets like this are rare in the field because they are so hard to build, and your kind words truly mean a lot to us!
We have addressed all five feedbacks as follows.
This is an important point. The physician review is the most labor-intensive part of this dataset: eight physicians reviewed rationales over five rounds across more than a year. They are now in Appendix B.5 of the revised PDF, and we provide them here for your reference. Every answer comes from the radiology report and the radiologist-reviewed segmentations, not generated by the language model (Section 2.1, Appendix B.2, B.4, and B.6). The physicians verified each sampled rationale against the original CT images and reports, not against the pipeline-generated fact sheets. In the final round, 600 rationales were reviewed and 11 were judged incorrect (1.8%). Table R5 below gives every round. We explain below.
Table R5. Physician review by round. A round samples ~5% of the reference rationales of that build at random; errors are fixed at the source and the pipeline is rerun before the next round.
The sample sizes, error counts, readers per rationale, the disagreement procedure, and what the review verified against are now in Appendix B.5 of the revised PDF. We appreciate your careful reading of this part of the paper; these details make the paper stronger.