acceptodds
Under review as a conference paper at ICLR 2027

Text-guided Active Image Selection for Radiology Report Generation

Abstract

Training vision-language models (VLMs) typically relies on large-scale paired datasets. However, na\"ive data scaling is often suboptimal or even impractical when one modality is significantly more expensive to obtain than the other. In medical imaging, for example, text reports are considerably cheaper to transfer and store compared to their corresponding imaging scans. In this work, we introduce the task of text-guided active image selection to efficiently construct a dataset for training report generation VLMs. Unlike standard active learning, where the pool consists of unlabeled inputs, we instead formulate the selection task in the text space. We instantiate this formulation with a suite of acquisition functions that identify the most informative reports for the downstream image-to-text task; their corresponding images are then queried to expand the training set. Experiments on two large-scale chest X-ray datasets show that text-guided selection achieves performance comparable to image-text selection, where both modalities are available. Moreover, we show that acquisition functions improve over random selection in terms of CheXbert-F1 at small data budgets. Finally, we demonstrate the feasibility of text-guided selection for cross-institution acquisition. These findings motivate rethinking data selection for medical VLMs by shifting the acquisition process from the expensive image space to the more accessible text space.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
data selection, radiology report generation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_2MGZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates a "text-guided image selection" method for radiology report generation, addressing a practical scenario where radiology reports are readily available, while the corresponding medical images are either difficult to obtain or costly to acquire. Unlike traditional active learning, the proposed framework leverages existing text reports to determine which corresponding images should be acquired, thereby expanding the training dataset.

The authors employ four data acquisition strategies—LESS, Ask-LLM, FeatEnt, and CLIP—to evaluate the information content of candidate reports during the selection process, without requiring direct access to the corresponding images. Furthermore, the paper provides a theoretical foundation for text-guided selection from an information-theoretic perspective and introduces a retrieval-based proxy method to facilitate gradient-based influence estimation when the candidate images are unavailable.

The framework is evaluated on the MIMIC-CXR and CheXpert-Plus datasets across various data acquisition budgets. Additionally, cross-institutional experiments are conducted. The study demonstrates that data acquisition based solely on text can yield downstream task performance comparable to methods that utilize image-text pairs. Under certain experimental conditions, specific acquisition strategies also outperform random selection.

Critical strengths

S1. The problem holds profound practical significance.

This paper addresses a critical challenge in vision-language learning within the medical domain: how to select valuable "image-report" pairs when only the radiology reports are available during the data-selection phase. By shifting the data-acquisition decision-making process from the image space to the text space, this framework offers a viable alternative to traditional active learning and holds the potential to facilitate the development of data-efficient medical vision-language models (VLMs).

S2. Comprehensive experimental evaluation across diverse scenarios.

The proposed methodological framework is clear and well-defined. The experiments encompass multiple datasets, varying data-acquisition budgets, and a range of evaluation metrics. In particular, the comparisons between "text-only" and "image-text" based data acquisition (Figure 3), intra-institutional versus cross-institutional experiments (Tables 3 and 4), and additional ablation studies provide robust evidence for the feasibility of the framework.

Critical weaknesses

W1. Performance improvements over random selection are inconsistent, and the applicability of different acquisition strategies remains unclear.

The authors emphasize—both in the abstract and the introduction—that a key contribution of LESS is its performance gain over random selection when the data budget is limited. However, the experimental results do not consistently support these claims across various datasets, acquisition functions, and budget constraints. For instance, Table 3 shows that with a 5% budget, LESS performs significantly worse than random selection on the CheXpert-Plus dataset (CheXpert-F1 Micro: 0.280 vs. 0.342). Furthermore, Table 4 indicates that with a 10% budget, random selection outperforms all other evaluated acquisition methods across all reported metrics on the IU-Xray/MIMIC-CXR datasets. While certain acquisition strategies may yield performance gains under specific conditions, the paper offers limited explanation for these inconsistencies. Moreover, it fails to provide sufficient guidance on which acquisition function should be prioritized under different data scenarios.

W2. The gap between theoretical motivation and downstream acquisition utility.

The theoretical analysis in Section 3.1 (Proposition 1) suggests that, under certain assumptions, more informative text descriptions can reduce the uncertainty associated with their corresponding images. However, there is no direct link between reducing conditional uncertainty and the utility that such image-report pairs might provide for downstream VLM training. Similarly, the effective ranking analysis in Table 2 reveals differences in the concentration of retrieved image representations, yet it fails to directly establish a relationship between the informativeness of the text, the acquisition score, and actual improvements in downstream performance. Consequently, while the theoretical motivation offers an intuitive rationale for text-guided selection, its connection to the effectiveness of the acquisition strategy remains insufficiently validated.

W3. Insufficient validation and reproducibility for retrieval-based proxy models.

In Section 4.1, the retrieval-based proxy model uses the Jaccard similarity between sets of pathology labels to select proxy images. However, similarity in pathology labels does not necessarily imply similarity in visual features or training gradients. Although Appendix C.1 reports an average Jaccard similarity of 0.875 and an exact match rate of 66% for pathology labels, these statistics do not directly demonstrate whether the gradients computed using retrieved proxy images can reliably approximate the gradients obtained using the actual corresponding images. Furthermore, the paper does not specify how ties are resolved when multiple images achieve the same maximum Jaccard similarity. Different images with the same similarity score could potentially lead to different gradient estimates and acquisition rankings.

W4. The incremental benefit of complex acquisition functions over simpler sampling strategies remains unclear.

Appendix F.2 demonstrates that pathology-stratified sampling alone can significantly improve downstream performance. Specifically, Figure 17 shows that with an initial training set size of 10,000 and an acquisition budget of 25%, stratified random sampling yields an overall improvement of 0.087 in CheXbert-F1, while stratified LESS yields an improvement of 0.083. Although these results were obtained using an initial training set size that differs from the one used in the main experiments, they suggest that, under certain conditions, relatively simple sampling strategies can achieve performance comparable to—or even better than—more complex acquisition functions. While the authors have included stratified random sampling as a baseline, further comparison with lightweight, text-based selection methods—such as prioritizing rare pathologies or clustering text embeddings—would help clarify the additional contribution of the acquisition functions under study.

W5. The paper’s claims regarding its own innovation—specifically in relation to existing "text-guided data filtering" methods—are not sufficiently clear.

While the problem definition addressed in this paper is practically significant, the underlying concepts have already been explored in prior research. For instance, CiT (ICCV 2023; "Curation in Training for Effective Vision-Language Data") also leverages text representations to guide the selection of image-text training data. Both approaches share the same core philosophy: using textual information to direct the filtering of multimodal data. The authors are advised to provide a more explicit discussion on the conceptual similarities and differences, and to clearly articulate the specific aspects in which their proposed framework offers a significant innovation over existing "text-guided data filtering" methods.

Questions for authors

see Weaknesses

2 comments

We thank the reviewer for their detailed and constructive comments. Our responses are provided below.

W1: Performance improvements over random selection are inconsistent, and the applicability of different acquisition strategies remains unclear.

Thank you for this comment. We acknowledge the reviewer's observations, which are closely related to points addressed in our responses above. The performance drop of LESS on CheXpert-Plus at the 5% budget is explained by a label distribution skew in the selected subset, as detailed in our response to point Q1 raised by reviewer fmMG. In that same response, we also provide practical guidance on which acquisition function to prioritize under different data scenarios. In short, we recommend LESS with stratified acquisition for pools with high redundancy, Ask-LLM for iterative multi-turn acquisition settings, and FeatEnt at single-turn acquisition settings with a small budget when label information is unavailable. We will include this analysis in the revised manuscript.

Regarding the IU-Xray/MIMIC-CXR result at 10% budget, we note that this cross-institution setting involves significant domain shift (e.g., different reporting styles) between the pre-training set (IU-Xray) and the pool (MIMIC-CXR). This distribution mismatch may reduce the reliability of the acquisition scores, since a model pre-trained on IU-Xray is applied to score candidates from MIMIC-CXR, which might explain the relative strength of random selection in this scenario.

Finally, we note that the abstract does not make claims specific to LESS. It refers to acquisition functions broadly and qualifies the claim to a specific metric (CheXbert-F1) and budget range (small data budgets). We will revise the manuscript to clarify the settings in which each method performs well.

 

W2: The gap between theoretical motivation and downstream acquisition utility.

Thank you for your comment. We would like to clarify that Proposition 1 is not intended to establish a direct link between reducing conditional uncertainty and the utility of a candidate sample for downstream VLM training. Instead, its purpose is to motivate the feasibility of text-guided selection by establishing that radiology reports, which contain richer semantic information than coarse labels, provide a more reliable proxy for the underlying visual content. We do not claim that reducing conditional entropy predicts downstream utility.

Table 2 (pg. 5) validates this proposition empirically, showing that radiology reports reduce uncertainty about the corresponding visual content more effectively than coarse pathology labels, thus supporting their use as the basis for text-guided data selection. Overall, Section 3.1 is not intended to establish a direct connection between acquisition scores and downstream performance; that connection is evaluated empirically through the downstream results presented in Tables 3 and 4 (pg. 9).

We acknowledge that the paper could be clearer about the scope and purpose of the theoretical analysis, and we will revise Section 3.1 (pg. 4) in the camera-ready version to make this distinction explicit.

 

W3: Insufficient validation and reproducibility for retrieval-based proxy models.

We appreciate your comment. To address the concern regarding gradient approximation quality, we compute gradients on the MIMIC-CXR pool using (i) ground truth image-text pairs and (ii) approximate image-text pairs based on our proposed retrieval-based surrogate (Formulation 1, pg. 6), projected onto an 8,192-dimensional space (using random projections) following the LESS approach. The mean cosine similarity between the two sets of gradients is 0.97, which confirms that the approximate image-text pairs produce reliable gradient approximations and supports the validity of our approach. We will include this analysis in the camera-ready version.

Regarding tie breaking, when multiple samples from the pre-training set achieve the same maximum Jaccard similarity, we randomly sample among them using a fixed seed for reproducibility.

W4: The incremental benefit of complex acquisition functions over simpler sampling strategies remains unclear.

Thank you for your suggestion. We note that the suggested "clustering text embeddings" baseline is closely related to our existing CLIP(txt) acquisition function, which measures the cosine distance between each candidate and the pre-training set reports in the BiomedCLIP embedding space.

Moreover, we ran an additional experiment using a rare pathology acquisition baseline that assigns higher scores to candidates whose pathologies are under-represented in the pre-training set, based on the inverse frequency of their least common positive label. The results are presented in Table 3.1 below. The rare pathology baseline does not outperform random selection at any budget or dataset (cf. Table 3, pg. 9 of the paper), suggesting that the model-based acquisition functions capture richer signals than simple label frequency heuristics. For completeness, we will include this baseline in the camera-ready version.

Table 3.1: Downstream performance of the rare pathology acquisition baseline.

DatasetBudgetCheXbert-F1 MacroCheXbert-F1 Micro
CheXpert-Plus5%0.1550.261
CheXpert-Plus10%0.2050.341
MIMIC-CXR5%0.2010.278
MIMIC-CXR10%0.2530.357

 

W5: The paper's claims regarding its own innovation—specifically in relation to existing "text-guided data filtering" methods—are not sufficiently clear.

We appreciate your suggestion. We agree that text-guided data filtering methods [1-3] are conceptually related to our work in that both use text representations to guide the selection of VLM training data. However, there are two fundamental differences in terms of the problem setting and the selection objective.

First, filtering methods assume that image-text pairs have already been collected and aim to remove noisy or irrelevant pairs from an existing large dataset. In our setting, candidate images do not exist at selection time and can only be acquired after selection. Second, and more importantly, filtering methods assess whether a sample meets a fixed quality standard or matches a pre-defined task description, i.e., fixed criteria that are defined independently of the model being trained. In contrast, following the data selection paradigm, the acquisition functions considered in our work estimate the expected utility of each candidate for the model currently being trained. As a result, the value of a candidate in our setting depends on the model's current state, not on fixed quality criteria applied uniformly across all samples. To our knowledge, this problem has not been previously addressed in the context of radiology report generation VLMs.

We acknowledge that this connection to text-guided filtering methods is not discussed in the submitted paper, and we will include a dedicated paragraph in the Related Work section of the camera-ready version.

[1] Xu, Hu, et al. "Cit: Curation in training for effective vision-language data." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

[2] Xu, Hu, et al. "Demystifying clip data." International Conference on Learning Representations. Vol. 2024. 2024.

[3] Fang, Alex, et al. "Data filtering networks." International Conference on Learning Representations. Vol. 2024. 2024.

Reviewer_ygYDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies text-guided active image selection for radiology report generation. The authors consider a setting where radiology reports are available, but accessing their corresponding medical images is expensive. They propose selecting informative reports first and then acquiring the corresponding images for VLM training.

The paper adapts several existing data selection methods, including LESS, Ask-LLM, FeatEnt, and CLIP, to this setting. Experiments on MIMIC-CXR and CheXpert-Plus show that text-only selection can achieve performance comparable to image-based selection, with some improvements over random selection at small data budgets. Cross-institution acquisition is also evaluated.

Critical strengths
  1. Selecting training images based on available radiology reports is a useful direction, especially when medical image access and storage are expensive. The proposed setting is different from conventional active learning and has potential applications in medical VLM development.

  2. The paper evaluates several acquisition strategies on two large chest X-ray datasets and includes comparisons with image-based acquisition and cross-institution experiments. These results help demonstrate the feasibility of text-guided selection.

Critical weaknesses
  1. The paper mainly adapts existing acquisition methods to the text-only setting. Although the problem formulation is interesting, there is no substantially new acquisition algorithm. The information-theoretic proposition is also a direct consequence of the data processing inequality and provides limited theoretical insight into downstream selection quality.

  2. According to Table 3, the improvements in CheXbert-F1 are relatively small and sometimes disappear at larger acquisition budgets. Some methods even perform worse than random selection. Furthermore, the experiments suggest that pathology-stratified sampling plays an important role, making it difficult to determine the additional benefit of the proposed selection strategies.

Questions for authors

Can the authors compare all acquisition methods against a pathology-stratified random baseline under exactly the same acquisition budgets? How much improvement comes from the acquisition score itself?

2 comments

Thank you for your review. Our responses to each comment are provided below.

W1: The paper mainly adapts existing acquisition methods to the text-only setting. Although the problem formulation is interesting, there is no substantially new acquisition algorithm. The information-theoretic proposition is also a direct consequence of the data processing inequality and provides limited theoretical insight into downstream selection quality.

We appreciate the reviewer's comment.

The main contribution of our work is a new problem formulation: we ask whether text-only data selection, where candidate images cannot be observed at selection time, can yield high-utility training subsets for radiology report generation VLMs. To study this question, we adapt state-of-the-art data selection methods to operate under this constraint, which requires principled design choices specific to this setting (e.g., the retrieval-based surrogate for LESS, Formulation 1 on pg. 6). The empirical finding that text-only selection yields downstream performance comparable to that of image-text selection is a non-trivial result with practical implications for settings where candidate images are unavailable or costly to obtain at selection time. We agree that developing novel acquisition algorithms tailored to text-space selection is a natural and important next step, which we leave for future work.

Furthermore, the purpose of Proposition 1 (pg. 4) is to motivate the feasibility of the task, and not to provide theoretical insight into downstream selection quality. Specifically, the proposition formalizes why text-space selection can serve as a reliable proxy for multi-modal selection when images are unavailable, and why this proxy becomes more reliable as reports become more semantically rich. In the radiology domain, reports provide detailed clinical descriptions of image findings, making them a semantically rich proxy for the corresponding images. We will clarify the role of the proposition more explicitly in the revised manuscript.

 

W2: According to Table 3, the improvements in CheXbert-F1 are relatively small and sometimes disappear at larger acquisition budgets. Some methods even perform worse than random selection. Furthermore, the experiments suggest that pathology-stratified sampling plays an important role, making it difficult to determine the additional benefit of the proposed selection strategies._
Q1: Can the authors compare all acquisition methods against a pathology-stratified random baseline under exactly the same acquisition budgets? How much improvement comes from the acquisition score itself?_

Thank you for your comment. We address both points together, as they are closely related.

Regarding the comparison against a stratified random baseline: for the main results on MIMIC-CXR shown in Table 3 (pg. 9), we already apply the same stratified acquisition strategy to all methods, including random selection (cf. lines [419-421], Section 6.2, pg. 8). The CheXpert-Plus results presented in Table 3 are obtained without stratification, as we found that it does not improve downstream performance on that dataset (cf. Lines [421-422], Section 6.2, pg. 8). For a direct comparison between naive and stratified acquisition, we refer the reviewer to Appendix F.2 (pg. 23), which shows that both the acquisition function and the stratification strategy contribute independently to downstream performance.

Moreover, we acknowledge that gains over random selection are modest. This is consistent with findings reported in the broader data selection literature. For example, the papers that introduced LESS [3] and Ask-LLM [4] both note that random selection is a strong baseline, with improvements on individual tasks often being small. Beyond acknowledging this general trend, we also provide an analysis of cases where methods perform worse than random in our setting. As discussed in our response to point Q1 raised by reviewer fmMG, LESS on CheXpert-Plus tends to over-select samples from two dominant pathologies at the 5% budget, thus leaving rare pathologies severely under-represented, which explains its performance drop at that budget.

[3] Xia, Mengzhou, et al. "Less: Selecting influential data for targeted instruction tuning." arXiv preprint arXiv:2402.04333 (2024).

[4] Sachdeva, Noveen, et al. "How to train data-efficient llms." International Conference on Learning Representations. Vol. 2026. 2026.

Thank you for your detailed response and clarifications. While these clarifications help address some of my concerns, my main reservations regarding the limited methodological novelty and the modest, inconsistent improvements over random selection remain. Therefore, I will maintain my original rating.

Reviewer_fmMGOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper introduces the novel task of text-guided active image selection for building training datasets for radiology report generation VLMs. Considering that text reports are far cheaper to access than medical images, the authors propose to perform data selection entirely in the text space, thereby avoiding the cost of processing candidate images at selection time. The authors considered four acquisition functions: a gradient-based influence estimation method (LESS), an LLM self-scoring method (Ask-LLM), a model-based feature space diversity method (FeatEnt), and a CLIP-based coverage method (CLIP). Experiments on MIMIC-CXR and CheXpert-Plus demonstrate that text-only selection can approach the performance of image-text selection and outperform random baselines at small budgets. The paper also explores cross-institution acquisition settings. Overall, the problem is well-motivated and the proposed framework is conceptually interesting.

Critical strengths

1, Using text to guide image selection is a practically motivated problem formulation. The idea of shifting data selection from the expensive image space to the accessible text space addresses a genuine bottleneck in medical VLM development, especially in privacy-sensitive and resource-constrained environments.

2, The authors explore multiple strategies (gradient-based, LLM-based, diversity-based, coverage-based) that capture different notions of informativeness, giving a broad view of the design space.

3, Demonstrating the method’s applicability when the pool comes from a different institution adds realism and practical value.

4, The analysis of pre-training set size, stratified vs. naive selection, and report length bias shows thoughtful investigation of design choices.

Critical weaknesses

1, Comparison with existing methods is limited. The only baseline is random selection. The paper must compare against standard active learning methods that operate on images (e.g., uncertainty sampling, core-set selection, BADGE) to convincingly argue that text-only selection is a viable alternative. Without such comparisons, the claimed advantage over “image-text selection” is not rigorously supported.

2, The generalization ability of the method is not demonstrated. Only one VLM architecture (vicuna-7b + BiomedCLIP) is tested. It is unclear whether the conclusions hold for other backbones the paper should at least validate on a second, distinct VLM to demonstrate robustness.

3, Standard active learning is iterative; the model’s knowledge evolves after each acquisition round. The current static, single-shot selection ignores this dynamic aspect and fails to show whether the methods remain effective over multiple rounds. Multi-round experiments are essential for a paper claiming to advance active learning.

4, LESS method relies on a potentially flawed surrogate. The retrieval-based proxy for candidate images uses Jaccard similarity on pathology labels, assuming label similarity implies visual similarity. This assumption is not validated, and errors in the surrogate could propagate to poor selection. The paper lacks an analysis of how surrogate quality affects downstream performance.

Questions for authors

1, Though the authors proposed four mechanisms for sample selection, their performance varies on different datasets. The current work did not provide any suggestion on how to select samples based texts in VLM in practice. The authors tried some potentially useful methods but did not construct a well designed and robust sample selection method.

2, For random selection, the performance will have a large variance if different seeds are used. It is necessary to show the variance of the random selection method, indicating if the proposed methods outperform random selection just in average or under all the tested randoms seeds.

3, Testing the performance of the methods under different VLM backbones is valuable for understanding if these methods are model-specific or general.

4 comments

We would like to thank the reviewer for their constructive feedback. Please find our point-by-point responses below.

W1: Comparison with existing methods is limited. The only baseline is random selection. The paper must compare against standard active learning methods that operate on images (e.g., uncertainty sampling, core-set selection, BADGE) to convincingly argue that text-only selection is a viable alternative. Without such comparisons, the claimed advantage over "image-text selection" is not rigorously supported.

Thank you for your suggestion. First, we note that standard image-based active learning methods assume access to candidate images at selection time. In our setting, however, only a pool of radiology reports is available during data selection, which makes these methods inapplicable as baselines.

Furthermore, the results presented in Section 6.1 (pg. 8) already provide empirical evidence that, in our experimental setup, text-only selection methods yield downstream performance comparable to their image-text variants. To further complement this analysis, we ran an additional experiment using an established active learning method [1] that operates on candidate images and accounts for both feature space entropy and epistemic uncertainty. We report the downstream results in Table 1.1 below and show that this image-based method performs comparably to image-text selection methods already considered in the paper, such as LESS (Upper Bound) and Ask-VLM. This further strengthens our finding that text-only selection is a viable alternative even relative to methods with full image access at selection time, and we will include this result in the camera-ready version.

Table 1.1: Downstream performance (CheXbert-F1 macro) on MIMIC-CXR.

Method5% budget10% budget25% budget
LESS (UB)0.2160.2720.314
Ask-VLM0.2140.2760.317
[1]0.2150.2480.319

[1] Melo, Luckeciano C., et al. "Deep bayesian active learning for preference modeling in large language models." Advances in Neural Information Processing Systems 37 (2024): 118052-118085.

 

W2: The generalization ability of the method is not demonstrated. Only one VLM architecture (vicuna-7b + BiomedCLIP) is tested. It is unclear whether the conclusions hold for other backbones the paper should at least validate on a second, distinct VLM to demonstrate robustness.
Q3: Testing the performance of the methods under different VLM backbones is valuable for understanding if these methods are model-specific or general.

We appreciate the reviewer's suggestion. To assess whether our findings generalize across model architectures, we run experiments using the LLM backbone of Gemma-3-4B-IT [2] paired with BiomedCLIP as the vision encoder. Note that we choose BiomedCLIP as it is an open-source model that was pre-trained on a large-scale biomedical dataset (PMC-15M) and it has not been exposed to any of the chest X-ray datasets considered in our work (MIMIC-CXR, CheXpert-Plus, IU-Xray), making it a suitable choice that mitigates the risk of data leakage. We report results on CheXpert-Plus for a single seed in Table 1.2 below. We observe that LESS outperforms other selection methods, including the random baseline, across budgets. We will provide full results on both CheXpert-Plus and MIMIC-CXR in the camera-ready version.

Table 1.2: Downstream performance (CheXbert-F1) on CheXpert-Plus with Gemma-3-4B-IT as the language backbone and BiomedCLIP as the vision encoder, across data budgets of 5%, 10%, and 25%.

MethodCheXbert-F1 macroCheXbert-F1 micro
5%10%25%5%10%25%
Random0.2210.2680.2870.3810.4520.484
LESS0.2590.2840.3100.4430.4790.503
FeatEnt0.1870.2310.3010.3600.4100.490
Ask-LLM0.2270.2540.3020.4030.4480.473

[2] Team, Gemma, et al. "Gemma 3 technical report." arXiv preprint arXiv:2503.19786 (2025).

W3: Standard active learning is iterative; the model's knowledge evolves after each acquisition round. The current static, single-shot selection ignores this dynamic aspect and fails to show whether the methods remain effective over multiple rounds. Multi-round experiments are essential for a paper claiming to advance active learning.

Thank you for raising this point. We agree that multi-turn acquisition experiments are a valuable addition to our evaluation. To this end, we repeat the acquisition experiments on CheXpert-Plus for four turns and on MIMIC-CXR for two turns, acquiring 5% of the remaining pool at each turn. Note that, at each turn, the VLM is re-trained from scratch on the pre-training data and on all subsets acquired up to and including the current turn, and the updated model is used for the next round of data selection. Single-seed results on CheXpert-Plus are presented in Tables 1.3 and 1.4, and preliminary single-seed results on MIMIC-CXR are shown in Tables 1.5 and 1.6 below (turns 3 and 4 are still in progress). We observe that Ask-LLM outperforms other methods from turn 2 onwards. We will provide complete multi-turn acquisition results across seeds for both datasets in the camera-ready version.

Table 1.3: Multi-turn acquisition results on CheXpert-Plus (metric: CheXbert-F1 macro).

MethodTurn 1Turn 2Turn 3Turn 4
Random0.1930.2300.2570.262
LESS0.1720.2440.2490.245
FeatEnt0.2280.2130.2400.251
Ask-LLM0.1800.2660.2630.258

Table 1.4: Multi-turn acquisition results on CheXpert-Plus (metric: CheXbert-F1 micro).

MethodTurn 1Turn 2Turn 3Turn 4
Random0.3370.3960.4300.438
LESS0.3200.3910.4020.403
FeatEnt0.3810.3750.4280.426
Ask-LLM0.3590.4460.4540.447

Table 1.5: Multi-turn acquisition results on MIMIC-CXR (metric: CheXbert-F1 macro). Note that the experiments for turn 3 and 4 are still in progress and will be included in the camera-ready version.

MethodTurn 1Turn 2
Random0.2020.260
LESS0.2630.266
FeatEnt0.2060.221
Ask-LLM0.2130.285

Table 1.6: Multi-turn acquisition results on MIMIC-CXR (metric: CheXbert-F1 micro). Note that the experiments for turn 3 and 4 are still in progress and will be included in the camera-ready version.

MethodTurn 1Turn 2
Random0.2820.363
LESS0.3770.386
FeatEnt0.2830.321
Ask-LLM0.2960.415

 

W4: LESS method relies on a potentially flawed surrogate. The retrieval-based proxy for candidate images uses Jaccard similarity on pathology labels, assuming label similarity implies visual similarity. This assumption is not validated, and errors in the surrogate could propagate to poor selection. The paper lacks an analysis of how surrogate quality affects downstream performance.

Thank you for your comment. We provide an analysis of the retrieval-based surrogate in Appendix C.1 (pg. 17). Furthermore, as shown in Section 6.1 (cf. Fig. 3, pg. 8), the overlap between subsets selected by LESS using our retrieval-based surrogate and LESS using ground truth candidate image-text pairs is consistently above 95%, while the difference in downstream performance between the two is small (within ~2% CheXbert-F1). These results suggest that the proposed surrogate is effective in our experimental setup. In the revised manuscript, we will make the connection between Appendix C.1 and Section 6.1 more explicit to highlight this empirical validation.

Q1: Though the authors proposed four mechanisms for sample selection, their performance varies on different datasets. The current work did not provide any suggestion on how to select samples based texts in VLM in practice. The authors tried some potentially useful methods but did not construct a well designed and robust sample selection method.

Thank you for raising this point. Performance variability across datasets reflects differences in pool composition and in what each method selects, both of which can be characterized to guide practitioners in selecting the most appropriate acquisition function for their setting.

Pool characterization. Table 1.7 below summarizes the key statistics of each dataset's pool. MIMIC-CXR has substantially higher pool redundancy than CheXpert-Plus (29.5% vs. 9.9% of reports have at least one near-identical report in the pool, where near-identical is defined as a cosine similarity of at least 0.99 in BiomedCLIP feature space). MIMIC-CXR also has a larger proportion of normal cases and shorter reports on average. In contrast, CheXpert-Plus contains a more pathology-diverse pool with clinically richer reports, as reflected by its higher median number of RadGraph entities per report (28 vs. 21 for MIMIC-CXR).

Table 1.7: Statistics of each pool across datasets (MIMIC-CXR and CheXpert-Plus).

DatasetPool sizeMean report length (words)Median of RadGraph entities per reportRedundancy (near-duplicates)% of normal (No Finding) cases
MIMIC-CXR99,51045.32129.5%37.5%
CheXpert-Plus45,73463.7289.9%9.6%

Selection behavior analysis. To understand why methods perform differently across datasets, we analyzed the subsets selected at the 5% acquisition budget for LESS and Ask-LLM, where the most pronounced differences are observed. On CheXpert-Plus, where we do not apply a stratified acquisition strategy (as it did not improve downstream performance on this dataset), LESS heavily over-selects samples from two dominant pathologies (Edema and Pleural Effusion), thus leaving rare pathologies significantly under-represented. On the other hand, Ask-LLM consistently selects longer and more clinically detailed reports at the same budget across datasets (on CheXpert-Plus, Ask-LLM selected a subset with a mean report length of 136 words and a median of 45 RadGraph entities per report, compared to 64 words and 28 entities per report for random selection), which explains its strong BLEU-1 performance but may also contribute to its instability at very low budgets where learning from complex reports with only a few samples could be more challenging.

Guidance for practitioners. Based on these findings, we offer the following recommendations:

  • LESS paired with a stratified acquisition strategy is the strongest method on redundant pools such as MIMIC-CXR, where it performs better than other methods on CheXbert-F1 across low budgets (5% and 10%). Without stratification, LESS risks over-selecting samples from dominant labels at low budgets, as observed on CheXpert-Plus.
  • Based on the finding that emerged during the rebuttal (cf. our response to point W3 above), Ask-LLM is well-suited to iterative multi-turn acquisition settings, where the model is updated after each turn. As shown in Tables 1.3-1.6 in our response to point W3 above, Ask-LLM outperforms other methods from turn 2 onwards on both datasets. This finding suggests that Ask-LLM becomes more reliable as the model improves.
  • In cases where label information is not available during data selection, FeatEnt is the most stable choice at low budgets (5%) for single-turn acquisition, as it achieves the highest CheXbert-F1 macro on CheXpert-Plus at that budget and remains competitive on MIMIC-CXR on the same metric and budget.

We will incorporate this analysis into a dedicated section in the camera-ready version.

Q2: For random selection, the performance will have a large variance if different seeds are used. It is necessary to show the variance of the random selection method, indicating if the proposed methods outperform random selection just in average or under all the tested randoms seeds.

We appreciate your comment. We would like to clarify that standard deviations across three independent seeds are already reported in Table 3 (pg. 9) for all methods including random selection. Each seed defines a fully independent experimental setup (i.e., a different pre-training set, validation set, and pool) and, within each setup, random selection is run once using the same seed for reproducibility.

Futhermore, we verified the per-seed results for the within-institution experiments (Table 3, pg. 9). On CheXpert-Plus, all methods that outperform random on average do so consistently across all three seeds. On MIMIC-CXR, LESS clearly outperforms random selection on 2 out of 3 seeds in terms of CheXbert-F1 and performs comparably (with a difference of up to 0.01 in CheXbert-F1) on the third seed at both the 5% and 10% data budgets. These results confirm that the reported improvements over random selection are reliable across different experimental setups.

For the cross-institution experiments, variance reporting differs by setting. For IU-Xray/MIMIC-CXR, variance is not applicable as the pre-training set is the full IU-Xray training set and the pool is fixed. For CheXpert/MIMIC-CXR, where the pre-training set is sampled, results are currently reported for a single seed. We will run all methods across three seeds and report standard deviations in the camera-ready version.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.