- Rating
- 2
- Confidence
- 4
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
TELOS is designed to address errors that occur during the generation of multi-word clinical terms in radiology reports. Such expressions are common in medical language, but virtual language models (VLMs) may complete these expressions with frequently occurring substitutes, thereby changing their clinical meaning. The framework combines linguistic context and visual information and exploits explicit memory for medical terms while promoting correct completion through targeted suffix supervision. The evaluation covers three VLM backbone networks and four radiology datasets, employing quantitative metrics and qualitative examples to examine term generation and clinical accuracy for different subword length groups. The results show that the improvement is particularly significant for longer clinical expressions and requires far fewer training parameters than the corresponding LoRA configuration.
Critical strengths
This work focuses on a specific limitation of medical virtual language models (VLMs) and proposes a coherent mechanism to address this limitation. Error rate estimation (BER) leverages textual context and image retrieval terms, while causal rewriting supports continuity between individual lexical units in clinical expressions. Suffix-level contrastive supervision further addresses erroneous term completion. Although some unresolved issues remain in experimental evidence, these components are closely related to the proposed problem.
Due to the limited number of trainable parameters, TELOS is an ideal choice for modifying existing VLMs with limited parameter overhead.
Critical weaknesses
- The primary assessment did not directly demonstrate that TELOS generates more reliable multi-word medical terms. Figure 3 measures the consistency between the generated sub-word length distribution and the reference distribution, as its caption states. Matching these distributions does not prove that the generated expressions convey the correct clinical meaning, which is precisely the shortcoming shown in Figure 1. Therefore, I believe this analysis is supplementary evidence rather than a core validation of the method. The report-level clinical scores in Tables 1 and 2 provide useful background information, but they do not assess correctness at the multi-word terminology level independently. Figures 4 and 5 in the appendix seem to better align with this objective and would be more helpful in the main text.
A proper assessment should also consider the limitations of word matching. Expressions such as “vascular lesion” and “avascular lesion” may have similar word length patterns despite significant differences in meaning. Conversely, “avascular lesion” and “lesion without internal vessels” may use different phrasing to describe similar findings. These examples suggest that consistency in length distribution or exact matching alone cannot adequately measure clinical correctness. The appendix contains some useful qualitative comparisons, but broader evidence should assess whether the generated terms retain the expected concepts, negations, and uncertainties. Concept-aware assessments or radiologist assessments can compensate for this deficiency, and representative examples should be provided in the text to support this. For now, the results in the text are insufficient to fully support the core reliability claims.
- Length-based analyses contain significantly fewer terms in five- and six-word groups, so their improvements in estimation may not be as stable as those of shorter expressions. Figures 4 and 5 demonstrate how confidence intervals or error bars can be obtained through term-level bootstrapping resampling, explicitly specifying the resampling unit. Reporting the number of terms in each group also helps readers assess the strength of evidence for longer terms.
Questions for authors
Please address my major concerns.
4 comments
Official comment of Reviewer 1ns9
The reviewer appreciates the author's feedback. However, I decided to maintain my original score for the following reasons:
-
The statement that confidence intervals "do not overlap" does not match the graph.
-
Clinical semantics and uncertainty validation are insufficient.
-
The completion test uses the training context, and long-phrase samples are scarce.
Response to reviewer 1ns9
If there is anything further that you feel we should clarify, revise, or supplement, please do not hesitate to let us know. We would be more than happy to address any additional concerns or make further improvements to the manuscript. Please feel free to point out even the smallest details. Thank you for your patience and generous support.
Supplementary to question 2
We agree that surface exact matching cannot capture clinical polarity ("vascular" vs. "avascular") nor reward valid clinical paraphrases ("avascular lesion" vs. "lesion without internal vessels"). We validate that TELOS resolves both challenges across four hierarchical levels:
-
Backbone embedding discrimination on antonym pairs: To test whether frozen subword embeddings conflate similarly spelled words with contrasting meanings, we examined six word pairs from training reports, averaging subword vectors from the frozen LLaVA-Rad input embedding layer. Six pairs exhibited cosine similarities below 0.70:
Isolated word pair Input embedding cosine hyperinflated / hypoinflated 0.623 calcified / noncalcified 0.602 symmetrical / asymmetrical 0.592 vascular / avascular 0.514 distended / nondistended 0.427 dilated / nondilated 0.380 Even among words sharing the same negating prefix, vector similarity varied substantially (0.602 for calcified/noncalcified down to 0.380 for dilated/nondilated, where only the final subword is shared). This demonstrates that the frozen backbone inherently represents contrasting concepts in distinct vector spaces rather than collapsing them.
-
Dictionary retrieval attribute discrimination: In the 1,265-entry LLaVA-Rad clinical dictionary, we evaluated attribute discrimination across candidate entries differing only in the queried attribute. By querying both directions and ranking entries by cosine similarity:
- Similar-looking antonyms (displacement): 4 matched pairs (displaced vs. nondisplaced) 8/8 (100%) matching terms ranked higher;
- Same contrast, including hyphenated spellings: 5 matched pairs 9/10 (90%) matching terms ranked higher;
- Directional opposites (laterality): 64 matched pairs (left vs. right) 122/128 (95.3%) matching terms ranked higher. These high retrieval discrimination rates confirm that the Bilateral Evidence Router reliably selects matching attributes and polarities rather than false-positive antonyms.
-
Semantic equivalence across varied phrasings via BERTScore: To evaluate whether reports preserve true clinical meaning under flexible phrasing (e.g., "avascular lesion" vs. "lesion without internal vessels"), we employ BERTScore, which computes greedy maximal cosine similarity over deep contextual token embeddings rather than exact string matches. Given candidate tokens and reference tokens with normalized contextual representations :
- Recall:
- Precision:
- F1 (BERTScore):
Because contextual embeddings project synonymous clinical expressions ("avascular" and "without internal vessels") into proximate latent neighborhoods, BERTScore rewards semantic equivalence despite divergent surface vocabulary. Conversely, contradictory tokens ("vascular" vs. "avascular") receive low cosine similarity, heavily penalizing polarity errors. In Table 1, TELOS achieves the highest BERTScore across all backbones: 0.860 on LLaVA-1.5 (vs. 0.818 for LoRA), 0.853 on Qwen2.5-VL (vs. 0.821 for LoRA), and 0.874 on LLaVA-Rad (vs. 0.863 for base), directly confirming that TELOS maintains genuine semantic equivalence across diverse phrasing.
-
Assertion-aware clinical graphs and expert radiologist audit:
- RadGraph-F1: Parses reports into clinical entities and relations, explicitly scoring observation assertion labels (definitely present, definitely absent / negated, uncertain). Mismatching opposing assertions ("vascular" vs. "avascular") penalizes both assertion and relation matching. In Table 1, TELOS achieves 0.240 vs. 0.158 for LoRA on LLaVA-1.5 (+51.9% relative gain) and 0.222 vs. 0.206 on Qwen2.5-VL.
- CheXbert Diagnostic F1: Evaluates condition-level multi-label classification (grouping positive and uncertain findings vs. negative), confirming categorical diagnostic correctness.
- Double-blind radiologist evaluation: In our 50-case evaluation by board-certified radiologists (detailed in response to Reviewer WrLo, Question 2), TELOS reduced clinically meaningful diagnostic and negation errors by 18.9%--24.3% relative to LoRA.
Response to Reviewer 1ns9
We thank Reviewer 1ns9 for the valuable recommendations regarding evidence placement, assertion-aware metrics, and statistical validation.
1. Core evidence and figure allocation: We agree that matching subword length distributions represents supplementary consistency evidence rather than direct proof of semantic correctness. In contrast, evaluating exact controlled completion rates, term-level precision, recall, and F1 across token lengths 2 to 6+ directly benchmarks terminology realization fidelity. In the revised manuscript (highlighted in blue), we have promoted both core terminology realization evaluations to the main text as Figure 3 (MIMIC-CXR) and Figure 4 (CheXpert Plus under institutional shift) in Section 4.4, fully equipped with 95% bootstrap confidence intervals, and relocated the subword length distribution analysis to Appendix A.6.
2. Clinical concept assessment: Polarity, negation, and uncertainty
Surface word matching cannot capture clinical polarity. Our evaluation explicitly addresses this through assertion-aware clinical metrics:
3. Bootstrap confidence intervals and sample sizes for length-stratified groups: We added 95% confidence intervals to all five metrics using 1,000 paired word-level bootstrap resamples within each length group. For controlled completion, metrics average the 30 contexts per word, then average across sampled words; for free generation, metrics are recomputed from word-level occurrences across held-out test reports using identical resampled word identities across methods. The term counts across token-length groups 2 to 6+ are detailed below (and now incorporated into Table 4 in Appendix A.5 of the revised manuscript, highlighted in blue):
As visualized in the updated plots with 95% bootstrap confidence intervals (revised main-text Figure 3 and Figure 4), TELOS exhibits non-overlapping error bars with LoRA on completion rate and F1 across the 5- and 6+ groups, confirming that suffix drift mitigation on longer expressions is statistically reliable.