- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper investigates a "text-guided image selection" method for radiology report generation, addressing a practical scenario where radiology reports are readily available, while the corresponding medical images are either difficult to obtain or costly to acquire. Unlike traditional active learning, the proposed framework leverages existing text reports to determine which corresponding images should be acquired, thereby expanding the training dataset.
The authors employ four data acquisition strategies—LESS, Ask-LLM, FeatEnt, and CLIP—to evaluate the information content of candidate reports during the selection process, without requiring direct access to the corresponding images. Furthermore, the paper provides a theoretical foundation for text-guided selection from an information-theoretic perspective and introduces a retrieval-based proxy method to facilitate gradient-based influence estimation when the candidate images are unavailable.
The framework is evaluated on the MIMIC-CXR and CheXpert-Plus datasets across various data acquisition budgets. Additionally, cross-institutional experiments are conducted. The study demonstrates that data acquisition based solely on text can yield downstream task performance comparable to methods that utilize image-text pairs. Under certain experimental conditions, specific acquisition strategies also outperform random selection.
Critical strengths
S1. The problem holds profound practical significance.
This paper addresses a critical challenge in vision-language learning within the medical domain: how to select valuable "image-report" pairs when only the radiology reports are available during the data-selection phase. By shifting the data-acquisition decision-making process from the image space to the text space, this framework offers a viable alternative to traditional active learning and holds the potential to facilitate the development of data-efficient medical vision-language models (VLMs).
S2. Comprehensive experimental evaluation across diverse scenarios.
The proposed methodological framework is clear and well-defined. The experiments encompass multiple datasets, varying data-acquisition budgets, and a range of evaluation metrics. In particular, the comparisons between "text-only" and "image-text" based data acquisition (Figure 3), intra-institutional versus cross-institutional experiments (Tables 3 and 4), and additional ablation studies provide robust evidence for the feasibility of the framework.
Critical weaknesses
W1. Performance improvements over random selection are inconsistent, and the applicability of different acquisition strategies remains unclear.
The authors emphasize—both in the abstract and the introduction—that a key contribution of LESS is its performance gain over random selection when the data budget is limited. However, the experimental results do not consistently support these claims across various datasets, acquisition functions, and budget constraints. For instance, Table 3 shows that with a 5% budget, LESS performs significantly worse than random selection on the CheXpert-Plus dataset (CheXpert-F1 Micro: 0.280 vs. 0.342). Furthermore, Table 4 indicates that with a 10% budget, random selection outperforms all other evaluated acquisition methods across all reported metrics on the IU-Xray/MIMIC-CXR datasets. While certain acquisition strategies may yield performance gains under specific conditions, the paper offers limited explanation for these inconsistencies. Moreover, it fails to provide sufficient guidance on which acquisition function should be prioritized under different data scenarios.
W2. The gap between theoretical motivation and downstream acquisition utility.
The theoretical analysis in Section 3.1 (Proposition 1) suggests that, under certain assumptions, more informative text descriptions can reduce the uncertainty associated with their corresponding images. However, there is no direct link between reducing conditional uncertainty and the utility that such image-report pairs might provide for downstream VLM training. Similarly, the effective ranking analysis in Table 2 reveals differences in the concentration of retrieved image representations, yet it fails to directly establish a relationship between the informativeness of the text, the acquisition score, and actual improvements in downstream performance. Consequently, while the theoretical motivation offers an intuitive rationale for text-guided selection, its connection to the effectiveness of the acquisition strategy remains insufficiently validated.
W3. Insufficient validation and reproducibility for retrieval-based proxy models.
In Section 4.1, the retrieval-based proxy model uses the Jaccard similarity between sets of pathology labels to select proxy images. However, similarity in pathology labels does not necessarily imply similarity in visual features or training gradients. Although Appendix C.1 reports an average Jaccard similarity of 0.875 and an exact match rate of 66% for pathology labels, these statistics do not directly demonstrate whether the gradients computed using retrieved proxy images can reliably approximate the gradients obtained using the actual corresponding images. Furthermore, the paper does not specify how ties are resolved when multiple images achieve the same maximum Jaccard similarity. Different images with the same similarity score could potentially lead to different gradient estimates and acquisition rankings.
W4. The incremental benefit of complex acquisition functions over simpler sampling strategies remains unclear.
Appendix F.2 demonstrates that pathology-stratified sampling alone can significantly improve downstream performance. Specifically, Figure 17 shows that with an initial training set size of 10,000 and an acquisition budget of 25%, stratified random sampling yields an overall improvement of 0.087 in CheXbert-F1, while stratified LESS yields an improvement of 0.083. Although these results were obtained using an initial training set size that differs from the one used in the main experiments, they suggest that, under certain conditions, relatively simple sampling strategies can achieve performance comparable to—or even better than—more complex acquisition functions. While the authors have included stratified random sampling as a baseline, further comparison with lightweight, text-based selection methods—such as prioritizing rare pathologies or clustering text embeddings—would help clarify the additional contribution of the acquisition functions under study.
W5. The paper’s claims regarding its own innovation—specifically in relation to existing "text-guided data filtering" methods—are not sufficiently clear.
While the problem definition addressed in this paper is practically significant, the underlying concepts have already been explored in prior research. For instance, CiT (ICCV 2023; "Curation in Training for Effective Vision-Language Data") also leverages text representations to guide the selection of image-text training data. Both approaches share the same core philosophy: using textual information to direct the filtering of multimodal data. The authors are advised to provide a more explicit discussion on the conceptual similarities and differences, and to clearly articulate the specific aspects in which their proposed framework offers a significant innovation over existing "text-guided data filtering" methods.
Questions for authors
see Weaknesses
2 comments
W4: The incremental benefit of complex acquisition functions over simpler sampling strategies remains unclear.
Thank you for your suggestion. We note that the suggested "clustering text embeddings" baseline is closely related to our existing CLIP(txt) acquisition function, which measures the cosine distance between each candidate and the pre-training set reports in the BiomedCLIP embedding space.
Moreover, we ran an additional experiment using a rare pathology acquisition baseline that assigns higher scores to candidates whose pathologies are under-represented in the pre-training set, based on the inverse frequency of their least common positive label. The results are presented in Table 3.1 below. The rare pathology baseline does not outperform random selection at any budget or dataset (cf. Table 3, pg. 9 of the paper), suggesting that the model-based acquisition functions capture richer signals than simple label frequency heuristics. For completeness, we will include this baseline in the camera-ready version.
Table 3.1: Downstream performance of the rare pathology acquisition baseline.
| Dataset | Budget | CheXbert-F1 Macro | CheXbert-F1 Micro |
|---|---|---|---|
| CheXpert-Plus | 5% | 0.155 | 0.261 |
| CheXpert-Plus | 10% | 0.205 | 0.341 |
| MIMIC-CXR | 5% | 0.201 | 0.278 |
| MIMIC-CXR | 10% | 0.253 | 0.357 |
W5: The paper's claims regarding its own innovation—specifically in relation to existing "text-guided data filtering" methods—are not sufficiently clear.
We appreciate your suggestion. We agree that text-guided data filtering methods [1-3] are conceptually related to our work in that both use text representations to guide the selection of VLM training data. However, there are two fundamental differences in terms of the problem setting and the selection objective.
First, filtering methods assume that image-text pairs have already been collected and aim to remove noisy or irrelevant pairs from an existing large dataset. In our setting, candidate images do not exist at selection time and can only be acquired after selection. Second, and more importantly, filtering methods assess whether a sample meets a fixed quality standard or matches a pre-defined task description, i.e., fixed criteria that are defined independently of the model being trained. In contrast, following the data selection paradigm, the acquisition functions considered in our work estimate the expected utility of each candidate for the model currently being trained. As a result, the value of a candidate in our setting depends on the model's current state, not on fixed quality criteria applied uniformly across all samples. To our knowledge, this problem has not been previously addressed in the context of radiology report generation VLMs.
We acknowledge that this connection to text-guided filtering methods is not discussed in the submitted paper, and we will include a dedicated paragraph in the Related Work section of the camera-ready version.
[1] Xu, Hu, et al. "Cit: Curation in training for effective vision-language data." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.
[2] Xu, Hu, et al. "Demystifying clip data." International Conference on Learning Representations. Vol. 2024. 2024.
[3] Fang, Alex, et al. "Data filtering networks." International Conference on Learning Representations. Vol. 2024. 2024.
We thank the reviewer for their detailed and constructive comments. Our responses are provided below.
Thank you for this comment. We acknowledge the reviewer's observations, which are closely related to points addressed in our responses above. The performance drop of LESS on CheXpert-Plus at the 5% budget is explained by a label distribution skew in the selected subset, as detailed in our response to point Q1 raised by reviewer fmMG. In that same response, we also provide practical guidance on which acquisition function to prioritize under different data scenarios. In short, we recommend LESS with stratified acquisition for pools with high redundancy, Ask-LLM for iterative multi-turn acquisition settings, and FeatEnt at single-turn acquisition settings with a small budget when label information is unavailable. We will include this analysis in the revised manuscript.
Regarding the IU-Xray/MIMIC-CXR result at 10% budget, we note that this cross-institution setting involves significant domain shift (e.g., different reporting styles) between the pre-training set (IU-Xray) and the pool (MIMIC-CXR). This distribution mismatch may reduce the reliability of the acquisition scores, since a model pre-trained on IU-Xray is applied to score candidates from MIMIC-CXR, which might explain the relative strength of random selection in this scenario.
Finally, we note that the abstract does not make claims specific to LESS. It refers to acquisition functions broadly and qualifies the claim to a specific metric (CheXbert-F1) and budget range (small data budgets). We will revise the manuscript to clarify the settings in which each method performs well.
Thank you for your comment. We would like to clarify that Proposition 1 is not intended to establish a direct link between reducing conditional uncertainty and the utility of a candidate sample for downstream VLM training. Instead, its purpose is to motivate the feasibility of text-guided selection by establishing that radiology reports, which contain richer semantic information than coarse labels, provide a more reliable proxy for the underlying visual content. We do not claim that reducing conditional entropy predicts downstream utility.
Table 2 (pg. 5) validates this proposition empirically, showing that radiology reports reduce uncertainty about the corresponding visual content more effectively than coarse pathology labels, thus supporting their use as the basis for text-guided data selection. Overall, Section 3.1 is not intended to establish a direct connection between acquisition scores and downstream performance; that connection is evaluated empirically through the downstream results presented in Tables 3 and 4 (pg. 9).
We acknowledge that the paper could be clearer about the scope and purpose of the theoretical analysis, and we will revise Section 3.1 (pg. 4) in the camera-ready version to make this distinction explicit.
We appreciate your comment. To address the concern regarding gradient approximation quality, we compute gradients on the MIMIC-CXR pool using (i) ground truth image-text pairs and (ii) approximate image-text pairs based on our proposed retrieval-based surrogate (Formulation 1, pg. 6), projected onto an 8,192-dimensional space (using random projections) following the LESS approach. The mean cosine similarity between the two sets of gradients is 0.97, which confirms that the approximate image-text pairs produce reliable gradient approximations and supports the validity of our approach. We will include this analysis in the camera-ready version.
Regarding tie breaking, when multiple samples from the pre-training set achieve the same maximum Jaccard similarity, we randomly sample among them using a fixed seed for reproducibility.