acceptodds
Under review as a conference paper at ICLR 2027

Decompose and Describe: Narrowing the Diagnosis-to-Report Gap in 3D CT Report Generation

Abstract

Automated 3D CT report generation could support radiology workflows, yet medical vision–language models (MedVLMs) often omit diagnostic findings from generated reports. We ask whether the diagnostic ability of these models carries over to free-form reporting. Across five MedVLMs, diagnostic VQA achieves higher macro-F1 than reports generated from the same CT studies and evaluated on the same disease categories. We call this task-dependent discrepancy the diagnosis-to-report gap. Conventional Report Adaptation improves reporting for most models but does not close the gap. To investigate and narrow it, we introduce DeCAD (Decompose CT Analysis and Describe), which progressively decomposes CT report generation at diagnostic, regional textual, and regional visual levels. The model identifies positive findings before writing a report, organizes diagnosis and description into regional tasks, and restricts each task to visual tokens from the corresponding region. Diagnostic and regional textual decomposition improve report-level diagnostic performance across the five MedVLMs, whereas regional visual decomposition provides limited additional gains. In a standard VLM without medical VQA pretraining, the first two levels remain effective; regional visual decomposition also provides further gains with a region-level contrastively pretrained encoder. Full DeCAD achieves state-of-the-art performance on CT-RATE and Merlin. Analyses show greater CT-token attention during diagnosis and higher recall of reference-positive findings in both the diagnostic output and final report than with Report Adaptation. These results show that explicitly organizing diagnosis improves diagnostic reporting and helps narrow the diagnosis-to-report gap. Code and models will be released upon acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
CT Report Generation, Medical Vision-Language Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_z2KxOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the “diagnosis-to-report gap” in medical vision–language models: diagnostic VQA achieves higher macro-F1 than free-form report generation on the same CT studies and disease categories. To narrow this gap, the authors introduce DeCAD, which progressively decomposes CT report generation at diagnostic, regional textual, and regional visual levels. The model generates an explicit diagnostic assessment before report composition, organizes diagnostic assessment and report description by anatomical region, and restricts each regional task to the corresponding visual tokens. Diagnostic and regional textual decomposition consistently improve report-level diagnostic performance across five medical vision–language models, whereas regional visual decomposition provides limited additional gains. In a standard VLM without medical VQA pretraining, the first two levels remain effective, and regional visual decomposition provides further gains with a region-level contrastively pretrained visual encoder. Full DeCAD achieves diagnostic macro-F1 scores of 52.1% on CT-RATE and 39.4% on Merlin, demonstrating that explicitly organizing diagnosis and description improves diagnostic reporting. However, diagnostic evaluation remains restricted to labeled disease categories and relies on automated report labeling, motivating broader clinical evaluation and expert assessment.

Critical strengths
  1. Consistent improvements across multiple models and datasets. Diagnostic and regional textual decomposition improve report-level diagnostic macro-F1 over Report Adaptation across all evaluated models on CT-RATE and Merlin. This consistency provides meaningful evidence that the approach has practical value across different model architectures and both chest and abdominal CT reporting.

  2. A useful empirical characterization of the diagnosis-to-report gap. The paper compares diagnostic VQA and free-form reporting on the same CT studies using aligned disease categories and reference labels. Although this comparison does not fully isolate the causes of the gap, it highlights an important limitation of evaluating diagnostic capability through VQA alone and motivates evaluating whether identified findings are expressed in generated reports.

Critical weaknesses
  1. The incremental methodological contribution over prior work is not sufficiently established. CT-AGRG: Automated Abnormality-Guided Report Generation from 3D Chest CT Volumes (ISBI 2025) already predicts abnormalities before generating corresponding descriptions. Dia-LLaMA: Towards Large Language Model-driven CT Report Generation (MICCAI 2025) uses diagnostic text prompts to guide report generation, while Large Language Model With Region-Guided Referring and Grounding for CT Report Generation (IEEE TMI 2025), referred to as Reg2RG, introduces regional visual features and region-report alignment. DeCAD differs by autoregressively generating the diagnostic assessment and report within the same model and systematically examining successive levels of decomposition. Nevertheless, its central methodological ideas substantially overlap with these established directions. Although the paper includes a comparison with Reg2RG, it does not provide direct quantitative comparisons with CT-AGRG or Dia-LLaMA, or controlled comparisons of alternative diagnostic guidance strategies using a shared backbone and equivalent supervision. Consequently, the experiments demonstrate effectiveness more clearly than they establish a substantive advance over prior approaches, which limits my assessment of the contribution's significance.

  2. The interpretation of the central empirical findings is limited by insufficient controls. Although the diagnosis-to-report comparison uses the same CT studies and disease labels, diagnostic VQA explicitly names each disease and queries it separately, whereas free-form reporting requires the model to identify and organize findings without these disease-specific cues. The observed gap therefore does not fully distinguish differences in task prompting from failures to express diagnostic information during report generation. Furthermore, the paper links the benefits of regional visual decomposition in its standard VLM to region-level contrastive pretraining, but all settings in Table 3 use a regionally aligned encoder. A comparison within the same architecture, varying both regional alignment and regional visual decomposition, is needed to isolate this explanation. The current results support the reported performance differences but do not establish their causes. Because systematic empirical insight is a central contribution beyond the existing methodological components, these unresolved explanations weaken the case for acceptance.

Questions for authors
  1. The incremental value of diagnostic decomposition over existing diagnostic guidance strategies. The paper positions its main distinction as a systematic analysis of different levels of decomposition. To clarify the contribution of diagnostic decomposition, do the authors have a comparison between autoregressively generated assessments and independently predicted diagnostic text prompts, using the same backbone and diagnostic supervision? Which existing results distinguish the benefit of assessment-before-report generation from that of adding explicit diagnostic supervision? This would help establish the substantive advance over strategies such as CT-AGRG and Dia-LLaMA.

  2. The sensitivity of the diagnosis-to-report gap to prompting. The paper defines this gap as a task-dependent discrepancy and examines the role of report-specific fine-tuning through Report Adaptation. Have the authors also evaluated a reporting prompt that provides the same disease checklist as diagnostic VQA while keeping the model weights fixed? This control would help clarify how much of the gap is attributable to explicit disease cues and how far a simple prompting change can narrow it without additional training.

  3. Evidence for the role of regional image–text alignment. Table 3 demonstrates that regional visual decomposition is effective with a fixed, regionally aligned encoder. Do the authors have further evidence attributing this benefit specifically to region-level image–text alignment, such as a comparison of the gains from visual decomposition under region-level versus global image–text contrastive pretraining within the same architecture? If no such control is available, should regional alignment be presented as a possible explanation rather than an established cause? This distinction would affect the assessment of the paper’s conclusions about when regional visual decomposition is beneficial.

2 comments
AuthorsOpenReview ↗

Response to Reviewer z2Kx (Part 1 of 2)

Thank you for recognizing the effectiveness and practical value of our work, as well as for your constructive suggestions to further strengthen the manuscript. We address your comments point by point below.

1. Concerns regarding comparison with existing diagnostic guidance strategies (W1, Q1)

Thank you for raising this point. First, for clarity in the following discussion, we distinguish the two diagnostic-guidance settings. The “independently predicted diagnostic text prompts” mentioned in the review correspond to external-classifier guidance, where a separate classifier predicts diagnostic labels that are then supplied to the report-generation model as text prompts. In contrast, our method uses autoregressive self-diagnosis guidance, in which DeCAD first generates a diagnostic assessment from the CT input and subsequently generates the report within the same model.

Second, we agree that comparisons with alternative diagnostic-guidance strategies are necessary to clarify our contribution. Since CT-AGRG does not provide public code, and Dia-LLaMA was originally developed on a dataset different from CT-RATE, we reimplemented Dia-LLaMA following its original training protocol on CT-RATE to ensure a matched data and evaluation setting. We further constructed an external-classifier-guided DeCAD variant, in which a linear classifier was trained by linear probing on DeCAD’s frozen visual encoder, and its predicted diagnostic labels were formatted as Dia-LLaMA-style prompts and provided to the LLM for report generation. The linear-probed classifier achieves an F1/precision/recall of 53.1/43.5/74.9. Both DeCAD variants use the same visual encoder, LLM, CT-report training data, and diagnostic supervision; they differ only in whether diagnostic information is supplied by an external classifier or generated autoregressively within the model. As shown in Table R9, our DeCAD achieves the highest F1 for report generation, indicating the advantage of its autoregressive self-diagnosis design.

Third, we acknowledge that external-classifier-guided methods may achieve strong report-generation performance when equipped with a strong classifier. However, their performance may depend on the externally supplied diagnostic information and the LLM’s instruction-following ability. Consequently, improved report-generation scores alone do not establish that the report-generating model itself grounds the generated findings in the visual input, rather than primarily following the diagnostic prompts. We therefore do not claim that self-diagnosis guidance universally outperforms a strong external classifier.

Importantly, we would like to emphasize that our contribution is not the general use of diagnostic guidance or regional representations, which have been explored by CT-AGRG, Dia-LLaMA, and Reg2RG. Rather, we rethink CT report generation by identifying the diagnosis-to-report gap in existing MedVLMs and systematically examining which forms of visual and textual decomposition help bridge it. Starting from our empirical analysis of existing MedVLMs, we identify practical principles for better translating visual diagnostic capability into free-form reports. DeCAD distills these insights into a lightweight design that can be implemented with standard SFT, without relying on large-scale medical pretraining or external diagnostic guidance at inference time. We believe this perspective is valuable because it shifts attention from adding increasingly powerful external components to understanding and better leveraging the diagnostic capability already present in MedVLMs.

We have added this discussion to the revised Appendix E.2.1 and Table 10 to further distinguish our approach from existing diagnostic guidance strategies. And we have also compared our approach with the reimplemented Dia-LLaMA in revised Table 4. Thank you again for your insightful suggestion.

MethodGuidanceF1RecallPrecisionMETBLEURaTERGGR
Dia-LLaMAClassifier30.426.245.824.024.366.225.741.7
DeCADClassifier50.767.944.821.619.765.822.437.6
Our DeCADAutoregressive52.154.255.626.327.166.523.642.0

Table R9. Comparison of external-classifier guidance and autoregressive self-diagnosis guidance on CT-RATE.

AuthorsOpenReview ↗

Response to Reviewer z2Kx (Part 2 of 2)

2. Concerns regarding the influence of disease-specific prompting on the diagnosis-to-report gap (W2, Q2)

We thank the reviewer for this insightful comment. We added a disease-specific prompting control that provides the same disease checklist as diagnostic VQA while keeping model weights and CT inputs fixed. As shown in Table R10, disease-list prompting improves F1 for all models, for example, from 17.8 to 27.2 for CT-CHAT and from 21.3 to 36.1 for HuluMed-7B, and outperforms Report Adaptation without additional training for three of five models. However, Diagnostic Decomposition still achieves higher F1 for every model, including 30.7 for CT-CHAT and 45.0 for HuluMed-7B. Therefore, disease-specific cues can contribute to the diagnosis-to-report gap, but explicit diagnostic supervision further improves the model’s ability to generate diagnostically complete reports beyond disease-specific prompting alone. We have updated the result of disease-specific prompting in Section E.2.2 and Table 11 in the revised Appendix.

SettingFine-tuning?CT-CHATMedGemma-1.5-4BLingshu-7BHuluMed-7BRadSight-8B
Direct✗17.8 / 37.4 / 18.113.7 / 39.4 / 12.47.1 / 28.6 / 5.421.3 / 52.5 / 15.434.6 / 43.9 / 30.9
Disease-specific prompting✗27.2 / 24.4 / 40.720.6 / 23.7 / 38.210.8 / 15.0 / 17.636.1 / 31.0 / 59.538.0 / 37.1 / 40.7
Report Adapt.✓17.7 / 39.7 / 15.119.8 / 46.0 / 19.521.5 / 56.2 / 17.837.7 / 53.8 / 33.237.2 / 43.8 / 33.7
Diagnostic Decomp.✓30.7 / 43.9 / 30.132.3 / 50.7 / 28.930.8 / 47.2 / 27.845.0 / 57.4 / 42.942.2 / 45.3 / 40.6

Table R10. Disease-list prompting and diagnostic decomposition on CT-RATE. F1/P/R are macro-averaged diagnostic metrics (%).

3. Concerns regarding ablation study of regional image–text alignment (W2, Q3)

Thank you for highlighting the need to isolate the role of regional image-text alignment. We therefore repeated the decomposition ablation using the same architecture initialized with a whole-image contrastively pretrained encoder, rather than the regionally aligned encoder used in Table 3. As shown in Table R11, under whole-image alignment, diagnostic decomposition improves F1 from 36.2 to 43.8 on CT-RATE (25.3 to 30.1 on Merlin), and adding text decomposition further improves it to 46.9 (33.6). However, adding visual decomposition provides no further gain, decreasing F1 from 46.9 to 46.0 on CT-RATE and from 33.6 to 30.9 on Merlin. In contrast, with region-level alignment, visual decomposition improves F1 from 41.6 to 52.1 on CT-RATE and from 34.1 to 39.4 on Merlin when diagnostic and text decomposition are enabled. This interaction shows that the benefit of visual decomposition depends on region-level image-text alignment, rather than arising merely from adding another decomposition module. We have updated ablation of regional image–text alignment in Table 3 in the revised manuscript.

Encoder pretrainingDiag. Decomp.Text Decomp.Visual Decomp.CT-RATE F1 / P / RMerlin F1 / P / R
Whole-image align.✗✗✗36.2 / 43.2 / 32.025.3 / 29.2 / 23.9
Whole-image align.✓✗✗43.8 / 48.5 / 42.530.1 / 29.2 / 33.6
Whole-image align.✓✓✗46.9 / 51.5 / 45.833.6 / 30.7 / 39.0
Whole-image align.✓✓✓46.0 / 54.0 / 44.130.9 / 29.4 / 38.9
Region-level align.✓✓✗41.6 / 47.2 / 42.834.1 / 37.1 / 38.3
Region-level align.✓✓✓52.1 / 55.6 / 54.239.4 / 43.1 / 42.3

Table R11. Decomposition ablation with whole-image contrastive pretraining on CT-RATE and Merlin. The architecture and decomposition settings are identical to Table 3; only encoder pretraining granularity differs.

Reviewer_V3a8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors tackle an interesting issue in MedVLMS, the "diagnosis-to-report gap". They illustrate how MedVLMs achieve higher diagnostic accuracy when answering VQA questions than they do when they are generating reports. The authors suggest this means the ability for the models to generate more diagnostically correct reports is already present within the model.

To address this issue, the author's propose 3 decompositions. The first is diagnostic decomposition where the Medical VLM is tasked with first describing the diagnosis and then generating the report. The second is textual decomposition where the model is tasked with describing specific regions. The third is visual decomposition where the model only receives visual tokens from the region is reporting on.

The models are evaluated on CT-RATE and Merlin diagnostic labels. The DeCAD method is shown to improve all tested MedVLM models and outperform all models on Diagnostic evaluation. Ablations show that the biggest boost comes from the diagnostic decomposition, but the text and visual decompositions are also important.

Critical strengths
  1. This model addresses a clear issue in the reporting paradigm, where a model can properly diagnose a scan (QA), but may not include that diagnosis in a generated report. This is shown to be a substantial problem across multiple different state-of-the-art MedVLMs. This paper identifies a very clear gap in the capabilities of these models and proposes a reasonable solution.

  2. The ablations for the three decomposition techniques are well done. These clearly show when and how the various decompositions are useful.

Critical weaknesses
  1. The paper mainly focuses on the diagnostic F1 metric. The paper does not justify that improving this necessarily results in a better report. This is most clearly illustrated in Table 4 where CT-CHAT greatly underperforms the author's model on the Diagnostic evaluation but CT-CHAT outperforms the author's model on two clinical metrics: RaTE and GREEN. This leads to the question: which model is actually generating a more useful report?

  2. Visual decomposition (one of the three main decomposition techniques) shows inconsistent or limited benefit over just Diagnostic and Textual decomposition (Table 1 and 2). Why is this the case?

Questions for authors
  1. How does this model compare to a reasoning model like NV-Reason-CT?
  2. Should all true diagnostic labels be present in the generated report? Or does it make more clinical sense to keep these tasks decoupled?
2 comments
AuthorsOpenReview ↗

Response to Reviewer V3a8 (Part 1 of 2)

Thank you for your insightful and constructive feedback. We appreciate your positive assessment of the importance and effectiveness of our work. Your thoughtful comments have helped us further improve the clarity and rigor of the manuscript. We address your concerns point by point below.

1. Concerns regarding diagnostic and clinical report evaluation metric (W1)

Thank you for raising this important point. First, we would like to claim that report quality is multidimensional: diagnostic macro-F1, RaTEScore, and GREEN quantify complementary aspects. Our macro-F1 evaluates diagnostic accuracy over the predefined CT-RATE categories, jointly measuring whether reference-positive findings are expressed in the generated report (recall) and whether reported positive findings are supported by the reference (precision). Diagnostic accuracy is foundational to clinically useful radiology reporting, and this metric provides a direct and reproducible measure of whether reports correctly communicate clinically relevant findings.

Second, we further perform a shuffled-report control for Reg2RG, CT-CHAT, ViSD-Boost, and AdaRAG-CT-8B (Table R5). We keep all generated reports unchanged but randomly reassign each report to a different patient, breaking case-specific report-reference correspondence while preserving report content and distribution. The diagnostic F1 decreases by 6.6, 5.9, 20.9, and 23.2 points, respectively, whereas RaTEScore decreases by only 0.9, 1.3, 3.1, and 3.2 points, and GREEN by 2.7, 3.6, 2.3, and 0.5 points. We additionally show illustrative CT-CHAT cases in Table R6, where high GREEN and RaTEScore coexist with multiple missed reference-positive findings. Together, these results suggest that, in our setting, GREEN and RaTEScore may be less sensitive than diagnostic F1 to case-specific diagnostic errors, particularly missed findings.

A rigorous assessment of how clinical-report metrics respond to diagnostic errors would require a dedicated study with carefully controlled reports and expert validation; it cannot be comprehensively established within this rebuttal. We have therefore revised the paper to clarify the scope of our claims: Full DeCAD achieves the strongest diagnostic macro-F1 while maintaining competitive text-generation quality and competitive clinical scores. At the same time, diagnostic accuracy is foundational to clinically useful radiology reporting. By evaluating predefined diagnostic categories with precision and recall, our metric provides a direct, transparent, and reproducible measure of whether reports correctly communicate clinically relevant findings.

MethodShuffled-reportF1-scoreRaTEScoreGREEN
Reg2RG17.166.937.9
Reg2RG✓10.5 (-6.6)66.0 (-0.9)35.2 (-2.7)
CT-CHAT17.866.942.2
CT-CHAT✓11.9 (-5.9)65.6 (-1.3)38.6 (-3.6)
ViSD-Boost39.465.834.9
ViSD-Boost✓18.5 (-20.9)62.7 (-3.1)32.6 (-2.3)
AdaRAG-CT-8B41.865.726.3
AdaRAG-CT-8B✓18.6 (-23.2)62.5 (-3.2)25.8 (-0.5)

Table R5. Shuffled-report control on CT-RATE.

CaseTP / FP / FN / TNGREEN (%)RaTEScore (%)
valid_466_a_20 / 0 / 3 / 1593.387.6
valid_410_a_21 / 0 / 4 / 1387.579.8
valid_805_a_10 / 0 / 4 / 1484.676.0
valid_90_b_10 / 0 / 3 / 1580.083.8
valid_682_a_20 / 0 / 4 / 1477.883.1
valid_644_a_10 / 0 / 5 / 1373.379.5
valid_1132_a_10 / 0 / 3 / 1572.779.1
valid_432_a_10 / 0 / 5 / 1370.085.4

Table R6. Illustrative CT-CHAT cases with high GREEN and RaTEScore despite missed reference-positive findings. TP, FP, FN, and TN are computed over the 18 CT-RATE diagnostic categories for each case.

2. Concerns regarding visual decomposition (W2)

Thank you for this important question. We have already analyzed this issue in Section 4.3, where visual decomposition is beneficial with a region-level aligned encoder but provides limited or inconsistent gains for existing MedVLMs. We further added a controlled experiment in Table R11 (also see our response to Reviewer z2Kx) and updated Table 3 in our revised manuscript: with whole-image contrastive pretraining, visual decomposition provides no further gain over diagnostic plus textual decomposition, whereas with region-level alignment it improves F1 from 41.6 to 52.1 on CT-RATE and from 34.1 to 39.4 on Merlin. These results support that the benefit of visual decomposition depends on region-level image-text alignment.

AuthorsOpenReview ↗

Response to Reviewer V3a8 (Part 2 of 2)

3. Comparison with NV-Reason-CT (Q1)

Thank you for bringing up NV-Reason-CT [R2], a relevant concurrent work posted on arXiv in September 2026, around the ICLR submission period. First, we evaluated its released checkpoint on CT-RATE under our unified inference and evaluation protocol. As shown in Table R7, NV-Reason-CT achieves strong report-generation performance, outperforming DeCAD by 6.8 macro-F1 points. This difference should be interpreted in light of the substantially different training settings (Table R8). NV-Reason-CT is trained on 70,111 CT scans from public datasets and an internal NIH dataset, with ~550K multimodal instruction examples spanning reports, QA, dialogue, reasoning, and refusals. It retains 13,824 visual tokens for each CT crop throughout language decoding and applies GRPO after SFT. In contrast, DeCAD uses only public CT-RATE and Merlin data, report-generation supervision, and 32 visual tokens for each target-region generation step, with standard SFT.

Second, we would like to clarify the motivation and scope of our work. Rather than aiming to match the scale of a large reasoning-oriented model, our DeCAD distills the insights from our empirical analysis into a lightweight report-generation design. It incorporates two simple but important principles: explicitly generating a diagnostic assessment before description, and organizing both assessment and description by anatomical region. With public data, compact region-level visual representations, and standard SFT, this formulation achieves strong diagnostic report generation. Thus, DeCAD provides a simple and practical design recipe for building diagnostically effective CT report-generation models without requiring large-scale multimodal instruction curation, high-resolution visual-token processing, or reinforcement learning.

[R2] NV-Reason-CT: 3D Visual Language Model for CT Analysis, arXiv 2026.9.25.

MethodF1RecallPrecisionMETBLEURaTERGGR
NV-Reason-CT58.957.161.420.313.164.719.236.0
Our DeCAD52.154.255.626.327.166.523.642.0

Table R7. Comparison between NV-Reason-CT and DeCAD on CT-RATE under a unified inference and evaluation protocol.

MethodTraining dataTraining instruction examplesInstruction typesVisual tokensTraining paradigm
NV-Reason-CT70,111 CT scans~550KReport, diverse QA, dialogue, reasoning etc.13,824 per CT cropSFT+GRPO
Our DeCAD46,337 (CT-RATE) / 14,715 (Merlin)108k (CT-RATE) / 167k (Merlin)Report only32 per regional callSFT

Table R8. Comparison of training data, instruction supervision, visual-token budgets, and training paradigms between NV-Reason-CT and DeCAD.

4. Concerns regarding inclusion of diagnostic labels in reports (Q2)

We agree that a clinically appropriate report need not enumerate every possible diagnostic label as a separate item. Our goal is not to require verbatim label listing, but to examine whether findings that are represented in the reference report are adequately expressed in the generated report. DeCAD keeps diagnostic assessment and report description structurally distinct: the assessment is generated as an intermediate step, while the final report remains free-form clinical prose and is evaluated independently. Here is an example:

<assessment>
["emphysema", "lung_nodule", "atelectasis", "coronary_artery_wall_calcification",
"arterial_wall_calcification"]
</assessment>
<report>
Trachea, both main bronchi are open. Mediastinal main vascular structures, heart contour, size are normal. Thoracic aorta diameter is normal. Pericardial effusion-thickening was not observed. Calcific atheroma plaques are observed in the aortic arch and coronary arteries. Thoracic esophagus calibration was normal and no significant tumoral wall thickening was detected. No enlarged lymph nodes in prevascular, pre-paratracheal, subcarinal or bilateral hilar-axillary pathological dimensions were detected. When examined in the lung parenchyma window; There are emphysematous changes in both lungs. Linear atelectasis areas are observed in the left lung upper lobe inferior lingula and right lung middle lobe medial. There are several millimetric nonspecific nodules in both lungs.
Upper abdominal organs included in the sections are normal. No space-occupying lesion was detected in the liver that entered the cross-sectional area. Bilateral adrenal glands were normal and no space-occupying lesion was detected. Bone structures in the study area are natural. Vertebral corpus heights are preserved.
</report>
Reviewer_8KP9OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the discrepancy between diagnostic VQA and free-form CT report generation. The authors propose DeCAD, which generates an explicit diagnostic assessment before the report and further decomposes the task into regional descriptions and regional visual inputs. Diagnostic and textual decomposition improve reporting across five medical VLMs on CT-RATE, with additional experiments on Merlin. A separate model with a regionally pretrained encoder benefits from all three stages and reaches diagnostic macro-F1 scores of 52.1% on CT-RATE and 39.4% on Merlin.

Critical strengths
  1. The paper identifies a useful limitation of current medical VLMs and tests a simple way to address it. Diagnostic decomposition improves report-level macro-F1 across all five CT-RATE models, and regional textual decomposition adds further gains. This consistency is more convincing than an improvement on a single backbone.

  2. The progressive experiments distinguish diagnostic, textual, and visual decomposition. The finding that regional visual restriction helps the regionally pretrained model but provides little benefit to existing MedVLMs is informative. The supplement also evaluates the Merlin report-label extractor and excludes intermediate diagnostic lists from report scoring.

Critical weaknesses
  1. The diagnosis-to-report gap is established as a performance difference, but its interpretation needs more care. Yes/no prompting can change the tendency to predict a disease as present: for CT-CHAT, diagnostic VQA has 85.0% recall but only 22.7% precision (Table 1). VQA answers and reports also use different label-extraction routes. These differences leave open how much of the gap reflects accessible visual knowledge versus prompting and evaluation effects. Attention measurements alone do not resolve this distinction.

  2. The contribution of decomposition is not fully separated from additional supervision and computation. Diagnostic targets explicitly repeat positive findings, and regional training changes the number of training sequences. At inference, the model is called separately for each region. Appendix A matches epoch counts, but does not establish equal optimizer updates or token budgets across these settings. This limits the interpretation of the gains as an effect of task organization alone.

Questions for authors
  1. Does the VQA advantage remain relative to an always-positive or shuffled-image control, and after checking report-label extraction on generated reports? This would help distinguish visual diagnostic ability from response and extraction effects.

  2. How many optimizer updates, training tokens, and inference calls does each decomposition setting use? Is there a comparison with Report Adaptation at a matched training budget, or with the same diagnostic supervision supplied without assessment-first generation?

5 comments
AuthorsOpenReview ↗

Response to Reviewer 8KP9 (Part 1 of 3)

Thank you for your insightful feedback. We appreciate your positive assessment of the value and effectiveness of our work, as well as your constructive suggestions. We address your concerns point by point below.

1. Concerns regarding comparability of VQA and report-based diagnostic evaluation (W1, Q1)

Thank you for raising this important point. We address three aspects of the VQA-report comparison:

(i) Whether VQA captures vision-conditioned disease discrimination?

To assess whether VQA reflects vision-conditioned disease discrimination rather than merely the effect of explicit disease-specific prompting, we evaluate both the model’s continuous confidence and its final yes/no decisions under original and shuffled CT inputs. In disease-specific VQA, the model is given a CT volume and a disease query and determines whether the queried disease is present.

First, to obtain a continuous VQA confidence score, we apply a softmax to the logits for the constrained “yes” and “no” responses and use the resulting “yes” probability. We then compute AUC for each disease category and macro-average across categories. AUC measures confidence-based discriminability, i.e., whether positive cases receive higher “yes” confidence than negative cases. In contrast, macro-F1 measures operational binary diagnostic quality, i.e., whether the model converts its evidence into correct final yes/no decisions, including the resulting precision-recall trade-off. As shown in Table R1, the original-image macro-AUCs are 69.7, 70.1, 64.5, 80.9, and 82.3 for CT-CHAT, MedGemma, Lingshu, HuluMed, and RadSight, respectively.

Second, as you suggested, we add a shuffled-image control (Table R1). We retain each disease query, ground-truth label, and evaluation protocol, while randomly replacing the paired CT volume with that of another patient, thereby breaking image-label correspondence. We report both the original- and shuffled-image macro-AUC, together with F1, precision, and recall. After shuffling, macro-AUC decreases by 18.3, 16.9, 13.1, 29.0, and 30.5 points for CT-CHAT, MedGemma, Lingshu, HuluMed, and RadSight, respectively. F1 also decreases for all models, with the largest drops for HuluMed (19.3 points) and RadSight (29.0 points). Thus, both the continuous confidence scores and final yes/no decisions depend on the visual input.

Together, these analyses show that VQA captures vision-conditioned disease discrimination. Accordingly, VQA macro-F1 evaluates models’ end-to-end vision-conditioned diagnostic decisions, jointly reflecting their discrimination of positive versus negative cases and their conversion of this evidence into final yes/no outputs.

ModelsOriginal-image: (AUC / F1 / P / R)Shuffled-image: (AUC / F1 / P / R)ΔAUC / ΔF1 / ΔP / ΔR
CT-CHAT69.7 / 31.4 / 22.7 / 85.051.4 / 26.7 / 17.8 / 80.018.3 / 4.7 / 4.9 / 5.0
MedGemma70.1 / 20.6 / 33.7 / 27.953.2 / 12.1 / 16.2 / 18.516.9 / 8.5 / 17.5 / 9.4
Lingshu64.5 / 24.6 / 27.5 / 59.351.4 / 20.8 / 19.3 / 50.613.1 / 3.8 / 8.2 / 8.7
HuluMed80.9 / 46.3 / 33.4 / 87.251.9 / 27.0 / 19.3 / 52.529.0 / 19.3 / 14.1 / 34.7
RadSight82.3 / 49.7 / 45.1 / 66.851.8 / 20.7 / 19.3 / 27.330.5 / 29.0 / 25.8 / 39.5

Table R1. AUC denotes the macro-average of per-disease AUROC values computed from the softmax-normalized yes/no logits. ΔAUC, ΔF1, ΔP, and ΔR denote the performance drop from the original-image to shuffled-image setting.

AuthorsOpenReview ↗

Response to Reviewer 8KP9 (Part 2 of 3)

[continued] 1. Concerns regarding comparability of VQA and report-based diagnostic evaluation (W1, Q1)

(ii) How the disease-specific yes/no formulation affects the observed VQA-report difference?

We add an always-positive VQA baseline (Table R2) to assess whether an affirmative response tendency alone can reproduce the observed VQA-report difference. Although this image-free strategy attains a non-trivial macro-F1 of 31.0, it does not consistently reproduce the observed gaps across models. For CT-CHAT, the always-positive and observed gaps are similar (13.2 vs. 13.6), consistent with its high-recall/low-precision VQA behavior. This tendency is itself part of the model’s VQA diagnostic behavior: it reflects how the model converts visual evidence into a final yes/no decision, with high sensitivity but limited specificity indicating less selective disease discrimination. However, for HuluMed and RadSight, the observed gaps substantially exceed the always-positive gaps (25.0 vs. 9.7 and 15.1 vs. -3.6, respectively). Thus, although model-specific yes/no response behavior is a meaningful component of VQA performance and can contribute to the VQA-report difference, an indiscriminate affirmative strategy cannot universally account for the observed gaps, particularly for HuluMed and RadSight.

ModelsVQA F1Always-Positive VQA F1Report F1VQA–Report GapAlways-Positive VQA-Report Gap
CT-CHAT31.431.017.813.613.2
MedGemma20.631.013.76.917.3
Lingshu24.631.07.117.523.9
HuluMed46.331.021.3259.7
RadSight49.731.034.615.1-3.6

Table R2. Comparison of VQA and report-based diagnostic performance, with an always-positive VQA baseline. The VQA–report gap is computed as the difference between VQA macro-F1 and report F1.

(iii) Whether different label-extraction procedures confound the comparison?

We clarify the comparability of the label-extraction procedures. The VQA and report-based evaluations use the same CT studies, disease categories, and reference binary labels; they differ only in how model predictions are converted into disease labels. For report-based evaluation, we extract diagnostic labels from generated reports using RadBERT, following [R1]. On CT-RATE, we validated RadBERT against the official diagnostic labels, achieving an F1 score of 98.1, precision of 99.4, and recall of 96.9. On Merlin, our independently trained ModernBERT extractor achieves a macro-AUC of 98.8 and macro-F1 of 95.3 on the held-out test set across 30 disease categories (Appendix Table 7). Thus, report-side label extraction introduces limited measurement error and yields labels closely aligned with the official labels used for VQA evaluation. The observed VQA-report differences are therefore unlikely to be primarily explained by the use of different label-extraction procedures.

Taken together, these analyses show that VQA is a visually grounded, disease-specific probe of diagnostic discrimination. The VQA-report difference should nevertheless be interpreted as task-dependent and in light of model-specific yes/no decision behavior. Thanks again for your insightful feedback. We have incorporated this analysis into Section E.3 and Table 12, 13 in the revised Appendix.

[R1] Generalist foundation models from a multimodal dataset for 3d computed tomography, Nature Biomedical Engineering 2026

AuthorsOpenReview ↗

Response to Reviewer 8KP9 (Part 3 of 3)

2. Concerns regarding additional supervision and computational cost (W2, Q2)

Thank you for pointing this out. First, we added Table R3 to report the training and inference cost for every setting on CT-RATE. Under the original epoch-matched protocol, Report Adaptation and Diagnostic Decomposition both use 1,450 optimizer updates, whereas the regional decomposition settings use 3,386 updates over two epochs. For visual-token exposure, let NN denote the number of visual tokens for a whole CT study, which depends on the underlying model, and let K=4K=4 denote the number of regional tasks. Report Adaptation and Diagnostic Decomposition each use NN visual tokens. Diag.+Text Decomp. supplies the full visual-token sequence to each region, resulting in KNKN cumulative visual-token exposures. Diag.+Text+Visual supplies only region-specific visual tokens, with its cumulative visual-token exposure kept within the whole-CT visual-token budget NN.

Second, we therefore additionally control for optimizer updates by retraining Diag.+Text Decomp. and Diag.+Text+Visual for HuluMed-7B and RadSight with the same 1,450 updates as Report Adaptation and Diagnostic Decomposition in Table R4. As expected, reducing the regional models’ optimizer updates lowers their absolute performance relative to the original epoch-matched results. Nevertheless, the main conclusion is unchanged under this control: Diag.+Text Decomp. still improves F1 over Diagnostic Decomposition from 45.0 to 50.6 for HuluMed-7B and from 42.2 to 44.7 for RadSight, while the additional effect of regional visual decomposition remains small and model-dependent (48.6 and 45.2 F1, respectively).

Third, we would like to clarify the inference-cost trade-off of regional decomposition. While it incurs additional decoding cost, the best regional variant improves F1 over Report Adaptation by 12.9 points for HuluMed-7B (37.7→50.6) and 8.0 points for RadSight (37.2→45.2) under the update-matched setting. We consider this trade-off practically reasonable for medical reporting, where improved diagnostic accuracy and fewer omitted positive findings can justify additional inference cost. Notably, the CT volume is encoded only once, and region-specific tokens are selected from the shared visual representation. Regional requests can also be batched for parallel decoding to reduce practical inference overhead.

SettingEpochs (original)Steps/epochTotal updates (original)Inference calls/CT
Report Adapt.27251,4501
Diagnostic Decomp.27251,4501
Diag.+Text Decomp.21,6933,3864
Diag.+Text+Visual21,6933,3864

Table R3. Training and inference cost across decomposition settings on CT-RATE.

SettingStepsHuluMed-7B (F1 / P / R)RadSight (F1 / P / R)
Report Adaptation1,45037.7 / 53.8 / 33.237.2 / 43.8 / 33.7
Diagnostic Decomposition1,45045.0 / 57.4 / 42.942.2 / 45.3 / 40.6
Diag.+Text Decomp.1,45050.6 / 58.2 / 49.544.7 / 49.7 / 45.7
Diag.+Text+Visual1,45048.6 / 56.9 / 47.445.2 / 50.6 / 46.2
Diag.+Text Decomp.3,38651.4 / 61.2 / 48.645.8 / 46.1 / 49.2
Diag.+Text+Visual3,38649.9 / 60.4 / 48.146.4 / 46.6 / 50.6

Table R4. Update-matched decomposition ablation on CT-RATE. All settings are trained for 1,450 optimizer updates. F1, P, and R denote macro-averaged diagnostic F1, precision, and recall (%), respectively.

Comments on the paper

2 comments
Area_Chair_88KAOpenReview ↗

Author - Reviewer Discussion

Dear Reviewers:

Thank you for your thoughtful reviews. The authors have responded to your reviews. Please check their response and reach out to them if you have any further questions. Please act as soon as possible, as the discussion period with the authors closes on Nov 18.

Best regards,

  • AC.
AuthorsOpenReview ↗

Summary of reviewer feedback and our revisions

We thank the reviewers for their insightful and constructive feedback, which has greatly helped us improve the quality of our manuscript.

We are pleased that reviewers found our work:

  • Clear Motivation and Significance: Reviewers recognized that our work “identifies a useful limitation of current medical VLMs” and “tests a simple way to address it” (Reviewer 8KP9), addresses “a clear issue in the reporting paradigm” (Reviewer V3a8), and provides “a useful empirical characterization of the diagnosis-to-report gap” (Reviewer z2Kx).
  • Technical Merit and Design: Reviewers considered our approach a “reasonable solution” (Reviewer V3a8), and found that “the progressive experiments distinguish diagnostic, textual, and visual decomposition” in an informative manner (Reviewer 8KP9). The ablations were also viewed as “well done,” clearly showing “when and how the various decompositions are useful” Reviewer (V3a8).
  • Consistent Empirical Performance: Reviewers noted “consistent improvements across multiple models and datasets” (Reviewer z2Kx) and that the diagnosis-to-report gap is substantial “across multiple different state-of-the-art MedVLMs” (Reviewer V3a8). They further found that diagnostic and regional textual decomposition improve report-level diagnostic macro-F1 across all five CT-RATE models, emphasizing that “this consistency is more convincing than an improvement on a single backbone” (Reviewer 8KP9).

Meanwhile, we are trying to address the following primary concerns:

  • Comparability of VQA and Report-based Evaluation (Reviewer 8KP9): We added shuffled-image and always-positive controls (Tables R1–R2), showing that VQA captures vision-conditioned disease discrimination and that the VQA-report gap cannot be explained solely by response bias; we also validated the report-label extraction procedures. In the follow-up discussion, Reviewer 8KP9 found our model-specific interpretation of the VQA–report gap more convincing after the additional controls.
  • Additional Supervision and Computational Cost (Reviewer 8KP9): We report training and inference costs in Table R3 and add update-matched experiments in Table R4; diagnostic plus textual decomposition still improves performance under the same training budget. In the follow-up discussion, Reviewer 8KP9 acknowledged that the matched-update results addressed the concern about extra training steps.
  • Assessment-first Generation versus Additional Diagnostic Supervision (Reviewer 8KP9, follow-up): Additional control experiments (Table R12) show that diagnostic supervision alone yields modest improvements, while combining it with assessment-first generation achieves substantially larger gains, highlighting the benefits of assessment-first generation beyond additional supervision.
  • Diagnostic and Clinical Report Evaluation Metrics (Reviewer V3a8): Shuffled-report controls and case examples (Tables R5–R6) show that RaTEScore and GREEN may miss case-specific diagnostic omissions; we therefore clarify their complementary role to diagnostic F1.
  • Contribution over Existing Diagnostic-Guidance Strategies (Reviewer z2Kx): Our contribution lies not in the general use of diagnostic guidance or regional representations, but in identifying the diagnosis-to-report gap in MedVLMs and systematically studying which forms of diagnostic, textual, and visual decomposition help bridge it. Matched comparisons with Dia-LLaMA and an external-classifier-guided variant (Table R9) further show that DeCAD’s autoregressive self-diagnosis achieves the strongest report-level diagnostic F1.
  • Visual Decomposition and Regional Image-Text Alignment (Reviewers V3a8, z2Kx): The controlled alignment ablation (Table R11) shows that visual decomposition provides clear gains specifically with region-level image-text alignment.

We have revised the manuscript accordingly, and all changes are highlighted in blue. If you have any further questions or concerns, we would be glad to discuss them. We would also be grateful if you could kindly consider updating your score in light of our responses and the additional results. Updates reflecting the follow-up discussion are marked in italics.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.