- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper proposes LASER, a label-free framework for learning radiology sentence embeddings that distinguish clinical concepts while preserving severity ordering. It uses LLM-generated synthetic sentences across concept–severity pairs to fine-tune multilingual E5-large with five complementary losses. At inference, concept and severity labels are assigned through k-NN classification using a synthetic codebook, while report retrieval uses cosine similarity between mean-pooled sentence embeddings. The method is evaluated on English chest X-ray and German cardiac CT reports against pretrained encoders, with experiments covering classification, retrieval, severity ordering, and loss-function ablations.
Critical strengths
-
Strong experimental evaluation: The authors compare LASER against pretrained encoders and fine-tuned baselines using the same synthetic data, including cross-entropy classification and Rank-N-Contrast. On CheXpert, LASER800 achieves a QWK of 0.632, outperforming Classifier800 (0.528), RNC800 (0.412), and RadBERT (0.350). The evaluation also includes five random seeds, ablation studies, and datasets spanning two languages and imaging modalities.
-
Well-motivated problem and practical approach: Learning severity-aware clinical embeddings without manual annotations is an important problem. The use of LLM-generated synthetic sentences and a k-NN codebook provides a simple, reusable approach that could generalize to other clinical findings.
Critical weaknesses
-
Inconsistent performance across datasets: LASER shows clear improvements on CheXpert but not on the cardiac dataset, where Classifier200 achieves higher accuracy, F1, and QWK than LASER200 (Table 8), and RNC outperforms LASER in accuracy at several training sizes (Table 10).
-
Overstated claims: The abstract and Discussion claim consistent superiority, but E5-large outperforms LASER on several cardiac classification and retrieval metrics (Tables 1–3, 12). Additionally, removing the concept loss yields comparable or better severity performance (Tables 6–7).
-
Potential test-set bias: The authors select the number of synthetic training sentences based on test-set performance without describing a separate validation split, potentially biasing the reported results.
-
The evaluation lacks supervised models trained on the available labeled data, zero-shot LLM grading, and simple keyword-based baselines, making it difficult to assess LASER's practical advantages.
-
Performance varies with the number of synthetic sentences, without sufficient explanation. Moreover, the correlation values reported in Section 5.3 do not match Table 5, raising concerns about reporting accuracy.
Questions for authors
-
Was any real labeled data used for hyperparameter selection, including the choice of synthetic training size (200/800)? If not, what validation procedure was used?
-
Why do the severity correlation values reported in Section 5.3 differ in sign and magnitude from those in Table 5? Which values are correct?
-
Were the Classifier and RNC baselines independently tuned, or did they use LASER's hyperparameters? How does LASER compare against supervised and zero-shot LLM grading baselines?
-
Why does performance decline as the number of synthetic training sentences increases? Could limited diversity or duplication in the generated data explain this trend?