- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes VEC-CT, a visual cue-guided error-corrective clinical reasoning framework for lung CT report generation. The authors construct LungCT-CoT, a dataset that decomposes the diagnostic process into six clinically ordered sub-tasks to support structured reasoning. VEC-CT employs an Error Tracer to identify the earliest erroneous sub-task and a Visual Cue Extractor to provide task-specific visual information for targeted reasoning refinement. During inference, only the optimized Reasoning Model is retained, without explicit error tracing or correction. Experimental results demonstrate improvements over representative baseline models on report generation metrics and clinical classification tasks.
Critical strengths
This paper proposes VEC-CT to address the problem of error propagation across intermediate clinical reasoning steps in lung CT report generation. The proposed framework integrates structured clinical reasoning, error tracing, and task-specific visual cues to improve the reliability of diagnostic report generation. The authors construct LungCT-CoT, which provides task-level supervision across six clinically relevant sub-tasks. Extensive experiments are conducted to compare the proposed method with various general-purpose and medical MLLMs, along with ablation studies to evaluate the contributions of different components. Additionally, qualitative comparisons and failure-case analyses provide further insights into the model's performance and limitations.
Critical weaknesses
- The paper claims that LungCT-CoT contains over 100,000 clinical QA pairs, whereas the task-level statistics in Appendix A.4 sum to only 92,796 pairs.
- The baseline models are evaluated without fine-tuning on LungCT-CoT, whereas VEC-CT is specifically fine-tuned on this dataset. Therefore, the reported performance improvements may primarily result from dataset-specific fine-tuning rather than the proposed method.
- The paper lacks sufficient details regarding the human verification of the generated QA pairs, including the number and qualifications of clinical reviewers, the proportion of manually reviewed samples, and inter-rater agreement.
- The Error Tracer and Visual Cue Extractor are used only during training, while inference relies solely on the Reasoning Model.
- The paper does not directly evaluate the reliability of the proposed error-correction mechanism, such as the proportion of incorrect predictions successfully corrected or correct predictions degraded after correction. Thus, it is unclear whether the improvements genuinely result from error correction.
- The framework uses a single 2D CT slice to generate patient-level diagnostic reports, although some reference findings require information unavailable from a single slice. This potential mismatch between input information and reference labels may affect the validity of training and evaluation.
- Although several failure cases are presented, the paper lacks a systematic analysis of different error types, their frequencies, and their potential clinical consequences.
- The private clinical dataset is insufficiently described, with limited information about the contributing institutions, patient population, data distribution, and annotation procedures. Its contribution to the reported performance also remains unclear.
- The clinical criteria underlying the five malignancy risk categories (A–E) are not clearly defined, raising concerns about the clinical validity and interpretability of the classification results.
Questions for authors
-
The authors should clarify why the reported task-level statistics sum to 92,796 QA pairs, rather than the claimed total of over 100,000.
-
The authors should provide comparisons with baseline models fine-tuned on LungCT-CoT under comparable training settings to isolate the contribution of the proposed method.
-
The authors should clarify how many clinical experts participated in the QA verification process, their qualifications, and the proportion of samples that underwent manual review. Inter-rater agreement should also be reported.
-
The authors should clarify how the proposed framework achieves error-corrective reasoning at inference time when the Error Tracer and Visual Cue Extractor are removed.
-
The authors should quantify the correction success rate, the rate at which correction introduces new errors, and the extent to which performance improvements can be attributed to successful error correction.
-
The authors should explain how reference findings that cannot be inferred from a single CT slice, such as those requiring volumetric imaging, prior examinations, or additional clinical information, are handled.
-
The authors should clarify the clinical guidelines or annotation criteria used to define the five malignancy risk categories.
-
The authors should provide a quantitative analysis of different error categories, their frequencies, and their potential clinical impact.
-
The authors should provide more details about the private clinical dataset, including the number of institutions and patients, data distribution across splits, and annotation procedures. An additional ablation study should be conducted to evaluate the contribution of the private dataset.
6 comments
Response to Reviewer hojp (2)
Q4. The authors should clarify how the proposed framework achieves error-corrective reasoning at inference time when the Error Tracer and Visual Cue Extractor are removed.
Thank you for raising this question. The Error Tracer and Visual Cue Extractor are used only during training to guide explicit correction. In the second reasoning round, the earliest-error index specifies the subtask from which the model regenerates the reasoning trace, while task-specific visual cues direct its attention to the corresponding lesion region. We supervise the model with corrected reasoning traces to help it learn to observe lesions accurately and use relevant imaging features for clinical reasoning.
At test time, the model performs single-pass inference without a separate, explicit correction stage. The lesion observation and reasoning capabilities learned through correction training are implicitly reflected in this process. Using the original CT slice as its sole visual input, the model follows the predefined clinical task sequence from the first task to the final report. No correction starting point is therefore required.
Tables 4 and 5 in Section 4.3 compare the full method with VEC-CT (w/o Correction), which uses only Clinical CoT supervision, and VEC-CT (Pre-Cue), which receives visual cues before reasoning and generates the report in a single pass. These comparisons examine the contributions of correction training and the timing of visual cue provision, respectively. The full method outperforms both variants on the text-generation and automatic clinical metrics, supporting the benefit of visual cue-guided correction during training for single-pass report generation at test time. We also present generated reports and corresponding attention maps in Appendix D.5.2 (Ablation Variant Analysis).
Combined ablation results from Tables 4 and 5. Values are mean ± standard deviation or mean (95% CI). Best mean scores are bold.
| Method | BLEU-4 | ROUGE-L | METEOR | BERT-F1 | GREEN | RaTEScore | RadGraph Partial |
|---|---|---|---|---|---|---|---|
| VEC-CT (w/o FT) | 0.05±0.06 | 0.20±0.08 | 0.24±0.09 | 0.69±0.04 | 0.347 (0.325, 0.369) | 0.507 (0.499, 0.514) | 0.187 (0.179, 0.196) |
| VEC-CT (w/o CoT) | 0.12±0.21 | 0.31±0.19 | 0.34±0.22 | 0.71±0.08 | 0.440 (0.417, 0.462) | 0.616 (0.606, 0.627) | 0.303 (0.287, 0.319) |
| VEC-CT (w/o Correction) | 0.20±0.19 | 0.40±0.19 | 0.42±0.20 | 0.78±0.08 | 0.404 (0.387, 0.421) | 0.628 (0.619, 0.638) | 0.368 (0.354, 0.383) |
| VEC-CT (Pre-Cue) | 0.23±0.25 | 0.42±0.23 | 0.41±0.26 | 0.78±0.09 | 0.439 (0.415, 0.462) | 0.620 (0.608, 0.632) | 0.381 (0.361, 0.401) |
| VEC-CT (Ours) | 0.28±0.24 | 0.47±0.23 | 0.47±0.24 | 0.81±0.09 | 0.504 (0.482, 0.525) | 0.665 (0.654, 0.676) | 0.446 (0.429, 0.463) |
Q5. The authors should quantify the correction success rate, the rate at which correction introduces new errors, and the extent to which performance improvements can be attributed to successful error correction.
Thank you for this suggestion. Table 19 in Appendix D.4 (Training-Stage Error Analysis and Correction Dynamics) presents a paired evaluation of first- and second-round outputs. CSR measures the fraction of initially unsuccessful responses that pass after correction; EIR measures the fraction of initially successful responses that fail after correction; and C/N measures successful corrections as a fraction of all evaluated responses.
We divide the more than 5,000 training steps into five stages and randomly sample 200 examples per stage. Overall accuracy is the proportion of responses judged correct across all applicable subtasks in these examples. Classification and judgment outputs are checked against reference labels, while generative outputs are evaluated by an LLM for agreement with references in anatomical location, key findings, medical details, and negation. Both rounds use the same criteria.
Second-round accuracy is higher at every stage, with the gain increasing from 2.0 to 12.7 percentage points. Early improvements are limited, suggesting that the model may not yet have learned to use visual cues effectively for correction. As training progresses, CSR and C/N increase while EIR decreases, suggesting more effective correction and better preservation of initially correct responses.
Table 19. Correction evaluation across five training stages. First- and second-round values denote overall accuracy across all evaluated subtasks in each stage’s sampled examples.
| Stage | First round | Second round | CSR | EIR | C/N |
|---|---|---|---|---|---|
| Stage 1 | 45.2% | 47.2% | 22.2% | 22.5% | 12.2% |
| Stage 2 | 48.2% | 54.3% | 28.4% | 17.9% | 14.7% |
| Stage 3 | 53.8% | 60.9% | 33.0% | 15.1% | 15.2% |
| Stage 4 | 56.3% | 66.0% | 38.4% | 12.6% | 16.8% |
| Stage 5 | 58.9% | 71.6% | 44.4% | 9.5% | 18.3% |
Response to Reviewer hojp (3)
Q6. The authors should explain how reference findings that cannot be inferred from a single CT slice, such as those requiring volumetric imaging, prior examinations, or additional clinical information, are handled.
Thank you for raising this point. All six VQA tasks in LungCT-CoT use a single CT slice as the imaging input. Processing complete three-dimensional CT volumes requires greater computational resources, GPU memory, and training time. Given the available computational resources and the time required for data construction and training, we adopted a single-slice setting, constructing QA pairs from each selected slice and its corresponding metadata rather than using the complete CT volume.
We organize the metadata at the slice level and link the reference annotations to the input slice and the target lesion shown on it. During verification, we check whether the findings in each reference answer are supported by that slice and its corresponding metadata. This criterion also applies to answers extracted directly from the original clinical reports. We exclude QA pairs whose reference answers require adjacent slices, the complete CT volume, prior examinations, or additional clinical information, so that the evaluation targets remain consistent with the information available to the model.
Q7. The authors should clarify the clinical guidelines or annotation criteria used to define the five malignancy risk categories.
Thank you for this suggestion. For LIDC-IDRI cases in LungCT-CoT, the five grades in the Malignancy task follow the original LIDC-IDRI annotation protocol described by Armato et al. (2011). Each CT scan was reviewed by four experienced thoracic radiologists in two phases. In the first phase, the radiologists independently marked lesions. In the second phase, each radiologist reviewed their own markings alongside the anonymized markings of the other readers before independently providing a final assessment. We use the resulting nodule-level malignancy ratings recorded in the preprocessed metadata. Ratings of 1 (highly unlikely), 2 (moderately unlikely), 3 (indeterminate), 4 (moderately suspicious), and 5 (highly suspicious) correspond to grades A–E, respectively.
The reference labels for Malignancy and CancerDiagnosis were already recorded in the preprocessed metadata before QA generation. For the same nodule, Malignancy provides a fine-grained, five-level assessment of malignancy suspicion, while CancerDiagnosis provides a coarse-grained, three-category assessment by grouping grades A–B as No, grade C as Indeterminate, and grades D–E as Yes.
During verification, we use these existing reference labels to check whether the generated textual answers are consistent with the CT findings and the answers to related tasks. We have clarified the annotation basis and verification procedure in Appendix A.3 (Data Verification).
Reference
Armato, S. G., III, McLennan, G., Bidaut, L., et al. (2011). The Lung Image Database Consortium (LIDC) and Image Database Resource Initiative (IDRI): A completed reference database of lung nodules on CT scans. Medical Physics, 38(2), 915–931. https://doi.org/10.1118/1.3528204
Response to Reviewer hojp (1)
Q1. The paper claims that LungCT-CoT contains over 100,000 clinical QA pairs, whereas the task-level statistics in Appendix A.4 sum to only 92,796 pairs.
Thank you for pointing out this discrepancy. The figure of “over 100,000” refers to the 105,315 QA pairs we initially constructed across seven tasks, including Cancer Subtype. During quality verification, we excluded 5,399 QA pairs, including 3,082 from Diagnostic Finding, leaving 99,916 QA pairs.
Of the 7,120 remaining QA pairs for Cancer Subtype, 5,964 (83.76%) were labeled as adenocarcinoma, and the other three subtypes accounted for 1,156 pairs (16.24%) combined. We sought additional cases for the underrepresented subtypes, but the limited number of suitable cases left the distribution highly imbalanced. We therefore removed the Cancer Subtype task from the dataset and experiments. After these two filtering steps, the final LungCT-CoT dataset contains 92,796 QA pairs across six tasks. We have clarified these counts in Appendix A.4 (Dataset Analysis) of the revised paper.
Q2. The authors should provide comparisons with baseline models fine-tuned on LungCT-CoT under comparable training settings to isolate the contribution of the proposed method.
Thank you for this suggestion. Our experiments include two baselines fine-tuned on LungCT-CoT using the same backbone: VEC-CT (w/o CoT), trained with final diagnostic reports as supervision, and VEC-CT (w/o Correction), trained with Clinical CoT supervision but without corrective training.
The full method builds on Clinical CoT fine-tuning by adding error localization and task-specific visual cues during corrective training. Its comparison with VEC-CT (w/o Correction) assesses the additional contribution of this training procedure. We also include the pretrained backbone without fine-tuning and the Pre-Cue variant, which provides visual cues before reasoning, to examine the effects of fine-tuning and cue timing.
Tables 4 and 5 in Section 4.3 show that the full method achieves higher mean scores than both fine-tuned baselines across the text-generation and automatic clinical metrics. Appendix D.5.2 (Ablation Variant Analysis) presents generated reports and corresponding attention maps for the same input side by side, illustrating differences in the attended lesion regions and report content.
Combined ablation results from Tables 4 and 5. Values are mean ± standard deviation or mean (95% CI). Best mean scores are bold.
Q3. The authors should clarify how many clinical experts participated in the QA verification process, their qualifications, and the proportion of samples that underwent manual review. Inter-rater agreement should also be reported.
Thank you for this suggestion. We clarified the annotator background and manual review procedure in Appendix A.3 (Data Verification). Six annotators with several years of experience in radiology reporting conducted two rounds of review.
Across the seven initially constructed tasks, 5,399 QA pairs (5.13% of the initial 105,315 pairs) remained ambiguous or difficult to assess after model-based verification and were referred for manual review, including 3,082 pairs from Diagnostic Finding.
In the first round, all six annotators independently scored the referred pairs from 1 to 5. They assessed whether each question could be answered from the selected CT slice alone, whether the slice supported the key findings in the reference answer, and whether answers to related subtasks were consistent. Available source annotations were also checked.
In the second round, the annotators reviewed one another’s anonymized scores and comments, revisited the supporting evidence, and rescored each pair. We retained a pair if at least three annotators scored it above 3 and all substantive disagreements about the key content of its reference answer had been resolved.