- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper explores the verbalization gap between the information contained in visual representations and what is stated in text reports, for 3D computed tomography (CT) scans. The proposed framework then addressed said gap with explicitly graded verbalization as learnt from resampled tokens, to include a measure of certainty (or confidence) into the linguistic representation reflected in derived radiology reports. The effect is represented by significant gains in clinical efficacy (Table 2) on CT-RATE, and does address a pertinent shortcoming with CT radiology reports.
Critical strengths
- Articulates the existence of a verbalization gap and attempts to mitigate it through the inclusion of graded verbalization conditioning
Critical weaknesses
- Further attribution of verbalization gap (to model or conventional limitations) not made
- Lack of some natural comparisons, such as against raw classifier predictions
Questions for authors
- The central issue identified in this study is the presence of a verbalization gap between clinically-relevant information derivable from input CT images (as measured by abnormality label classification), and that which is derivable from the CT (text) report generated from the same CT image(s).
While this is indeed a relevant shortcoming of typical (automatically generated) CT reports, it is less clear as to whether the verbalization gap arises more from an actual inability of existing CT report generation models to determine (and include) such clinically-relevant information, or more because typical CT report formats and practices do not encourage the inclusion of such information (perhaps to aid conciseness for human readers), despite the model(s) being able to provide it.
For example, the Introduction performs linear probing on 18 abnormality labels to quantify the retrieval of relevant clinical information, from both modalities (CT image and generated text report). In principle, it is likely possible for the (multimodal) LLM that generated the text report to have included estimated probability scores for all 18 labels (as indeed shown under Graded Verbalization in Figure 2), if so prompted. Such a structured report would certainly provide more fine-grained information for each abnormality, than a typical human-style CT report (e.g. Figures 2, 5 & 6) that tend to produce binary (absent/present) judgments, or even the proposed method (which tends to use more fine-grained descriptors such as "minimal" and "more clearly", from Figure 5).
The distinction between the ability of existing models to derive clinically-relevant information, and the choice as to whether and how to express such information in the report, might therefore be emphasized and discussed if relevant.
-
Related to the above, an involved analysis for a structured report generator that (simply) directly expresses/translates raw probabilities for all abnormalities in terms of CRG for clinical efficacy - and also computes , precision, recall and accuracy based on the explicitly-stated probabilities - may be appropriate. This would further quantify the verbalization gap in terms of model ability (upper limit) and loss due to stylistic conventions for text report formatting.
-
In Sections 3.2 and 3.3, it is stated that graded verbalization includes evidence/information from a separate image-based classifier, by converting the raw classifier probability predictions into an additional prompt for further conditioning the final LLM decoder on the visual tokens (Figure 2).
This does imply that the quality of the separate classifier may significantly impact the accuracy and utility of the generated prompt. Given this, details on the training methodology of the classifier should be fully presented in the Appendix (Section B), especially for the training data used and (internal) validation metrics.
-
Related to the above, the probabilities from the separate classifier can in principle be used to generate a structured text report using the various decision thresholds, without requiring an additional LLM decoder (e.g. by asserting "high probability for <abnormality>" for meeting the high precision threshold, and so on). If possible, a direct assessment of the binarized classifier output might be also presented in Table 2.
-
In Section 3.2, the minimal LOW assertion level contains all , and corresponds to "cannot be excluded". While some ablation was attempted (Table 9), it might be discussed as to whether an abnormality might in practice be excluded, and whether additional threshold(s) on that possibility would have been appropriate.
-
In Section 4, it is stated that for CE, generated reported are converted into 18 binary abnormality labels using a RadBERT-based classifier. The application and validation of this conversion might be described in supplementary material, and it might further be clarified as to whether the lack of mention of an abnormality in report text would be translated as the absence of that abnormality.
2 comments
We thank the authors for the comprehensive clarifications.
Thank you for the thorough review and the concrete suggestions. They let us better quantify where the verbalization gap comes from.
Q1. Model ability vs. reporting convention. Three observations separate the two.
Stating the information as text costs 0.014 to 0.039, while deriving it from the visual tokens costs 0.215. Most of the gap is the model's inability to turn visual evidence into stated findings, and style accounts for a small part. A probe on the 128-token prefix the decoder receives still recovers the labels at macro-F1 0.607 (Appendix H.4), so the information reaches the decoder and is lost in decoding.
Q2. Structured probability report as upper limit. Writing all 18 probabilities into a template leaves the predictions unchanged, so the classifier itself (F1 0.636, CRG 0.565) is this upper limit, and the template row above shows that fixed-form text loses 0.014. Both rows are now in Tables 2 and 3.
Q3. Classifier details. Same patient-level training partition as the generator, FlexiCT frozen. Gated attention-MIL pooling (Ilse et al., ICML 2018; hidden 256, dropout 0.25), LayerNorm, 512-d GELU MLP, 18 outputs. AdamW (lr 1e-3, weight decay 1e-4, 600 warm-up steps, cosine decay, batch 128) with the class-weighted BCE of Eq. 2. Model selection and per-class thresholds (grid search maximizing F1) use the early-stopping split, with no further calibration.
Appendix B now contains these (Table 10) and a per-class table with threshold, precision, tier, AUROC, F1 and support (Table 11).
Q4. Classifier output in the main table.
Our binary variant matches the template in F1 and exceeds it in CRG and precision, while writing a full report with localization, measurements and findings beyond the 18 labels (Figs. 5 and 6). The template carries only the 18 decisions, which shows in its NLG scores. All rows are now in Table 3.
Q5. Exclusion threshold. LOW applies only to findings above their class threshold, so it means "selected, from a class with lower precision". Below-threshold findings are left unspecified in the graded prompt. The all-18 binary variant tests explicit exclusion, since it states absent for every class below threshold, and it gives the highest CE (F1 0.622, recall 0.658). A separate high-NPV threshold for reliable absence is a natural refinement, which we discuss in Sec. 5.
Q6. Evaluator. We use the RadBERT-based evaluator of prior CT report generation work (Hamamci et al., 2026b). Negated and unmentioned findings are both scored as absent. It reproduces the official labels from the reference reports with macro-F1 0.998 (lowest class 0.991, Table 16) and loses 0.014 F1 on the template text above. Hedged statements are partly scored as absent, e.g., writing the same decisions with graded phrases lowers recall from 0.681 to 0.499 (Appendix H.1).
We hope this addresses your questions and would be grateful if you considered raising your score.