acceptodds
Under review as a conference paper at ICLR 2027

SentZero: An Enhanced Sentence-Centric Feature Learning Framework for Multi-Task Zero-Shot Chest X-Ray Analysis

Abstract

Vision–language (VL) pretraining using paired chest X-ray (CXR) images and radiology reports has shown strong potential for medical image understanding. However, existing methods often remain dependent on task-specific fine-tuning because radiology reports are lengthy, clinically dense, and difficult to align with simple zero-shot prompts. Recent sentence-level approaches partially address this limitation using clinical phrases extracted by large language models (LLMs), but they largely overlook the intrinsic characteristics of radiology discourse. In particular, limited positive-pair diversity constrains further gains, while clinically equivalent sentences frequently recur across patients, creating false negatives in contrastive learning. To address these issues, we propose **SentZero**, an enhanced sentence-centric VL pretraining framework for zero-shot, multi-task CXR analysis. SentZero introduces LLM-based abstract-level sentence structuring and mapping to expand positive-pair diversity, together with an additional loss term to mitigate false negatives. We further introduce sentence-conditioned residual modulation of visual embeddings, enabling visual features to adapt to the semantic characteristics of each input sentence. Across diverse downstream tasks and datasets, SentZero improves zero-shot generalization and outperforms prior multi-task zero-shot methods.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Vision-language pretraining, Chest X-ray, Medical Pretraining

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Xa7COpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes SentZero for zero-shot chest X-ray classification and grounding through vision–language pretraining. It combines LLM-based abstract-level sentence augmentation, a patch-level false-negative mitigation objective based on shared cross-report statements, sentence-conditioned residual modulation, and multi-layer visual feature aggregation, achieving strong results across multiple public benchmarks.

Critical strengths

1.Clear and compelling motivation: Starting from the contrastive learning objective, the paper identifies the risk of pushing apart compatible image–text pairs when identical descriptions appear in other samples. This is an interesting and important problem, and the sentence-level formulation elegantly supports both zero-shot classification and grounding within a unified pretraining framework.

2.Comprehensive experimental evaluation: The paper evaluates the method on multiple chest X-ray benchmarks, including CheXpert. Detailed ablation studies further support the effectiveness of the proposed components.

Critical weaknesses

1.Attribution of the false-negative mitigation gains remains incomplete: Table 3 compares masking and positive relabeling, but these alternatives modify contrastive pair assignments, whereas the proposed approach introduces additional local image–text supervision. Without a control applying the same local alignment objective to original positive pairs, it remains unclear how much of the improvement comes from targeted false-negative handling rather than more general local supervision.

2.Limited validation of LLM choices: Although reusing RadZero’s LLaMA3-extracted phrases helps ensure comparability, the authors do not justify using Qwen for the subsequent structured extraction. Without comparisons across LLM choices, it remains unclear how sensitive the final performance is to the specific extraction models.

Questions for authors

1.Could the authors apply the same local image–text alignment loss to original positive pairs, keeping patch selection and loss settings unchanged, to distinguish the benefits of additional local supervision from targeted false-negative handling?

2.Could the authors evaluate alternative LLMs to assess the method’s robustness to the choice of extraction model? For example, could LLaMA also be used for structured extraction to maintain consistency across the two stages?

3.Although Table 3 evaluates false-negative masking, could the authors explain why excluding other positive sentences from the same sample is appropriate, whereas additionally excluding shared sentences across samples generally reduces performance? What mechanism explains the advantage of retaining these false-negative pairs while adding local attraction?

4.The ViT [CLS] output, which typically serves as a global image representation, does not appear to be directly used in the subsequent alignment or prediction stages. Could the authors explain this choice and whether combining the attention-weighted patch representation with the global [CLS] embedding could be beneficial?

5 comments

We thank the reviewer for the thoughtful and constructive comments. We particularly appreciate the recognition of our model’s strong performance over the baselines across multiple benchmarks, a strength highlighted by all reviewers. We are also grateful for the positive assessment of the motivation behind our proposed components and of our evaluation design. Universal encoders that achieve strong zero-shot performance across multiple chest X-ray tasks still remain rare, and we would be grateful if the reviewer could take these results into account in the final assessment.

We address each point of weakness and question you argued below. We use W for weaknesses and Q for questions. The numbers (1, 2, 3) follow the order of the main headings in the review. For a more natural flow, we answer Q3 directly after Q1, since both concern the false-negative loss, and then Q2 and Q4.


[W1 & Q1] Separating targeted false-negative handling from general local supervision

As suggested, we applied the false negative patch loss Lfn\mathcal{L}_{fn} to the original positive pairs instead of the false negative pairs. However, this did not achieve higher performance, indicating that the gain from our proposed loss does not simply come from additional local supervision. Combined with the motivation and working hypothesis behind our false negative handling, which we explain in our response to [Q3] below, this result supports the validity of the proposed loss.

OpenICXR14PadChestCXD10CheXpertCXD10 (grounding)MS-CXR
Lfn\mathcal{L}_{fn} on original positive pairs0.88290.83540.86040.83700.89730.70310.8982
Lfn\mathcal{L}_{fn} on false negative pairs (Ours)0.88910.83890.86090.84330.90370.73150.9222

[Q3] Mechanism of the proposed false negative loss

The design of our proposed false negative patch loss follows from the focal nature of radiographic findings, which we discussed with a medical expert during study design. For instance, in a sentence such as "There is opacity", only a few patches contain the evidence; the remaining patches do not show opacity. If a false negative pair is directly converted into a positive (false negative transition) or excluded from the loss computation (false negative masking), the following issues arise and can in fact lead to detrimental results.

  • Transition (relabel as positive). Every patch of the matched image is pulled toward the sentence, including the majority that are unrelated to the finding. This sends an erroneous signal to those patches and explains the large drop in Table 3.
  • Masking. The matched image–sentence pair is removed from the loss entirely, so it receives neither attraction nor repulsion. The patches unrelated to the finding should still be separated from the sentence, and this repulsion is lost.

Since the image representation is a similarity-weighted sum over all patches, any change to the contrastive logit of a pair acts on all patches of that image in one direction. As a result, the two methods above can misguide the global pull–push geometry of all patches from the same image sample.

Our loss mechanism. Our proposed loss leaves the contrastive pair assignments unchanged, and therefore preserves the global pull–push geometry. Additional attraction is applied only to the top-M% patches that are most similar to the shared sentence, which are the patches that should not be pushed away from it. Therefore, unlike the two methods above, our loss largely preserves the global contrastive objective while selectively mitigating the effect on only those patches whose similarity could be reduced by potential false negatives. In addition, following the original formulation of the MP-NCE loss, we exclude the other positives from the denominator when computing it.

We have incorporated the above content into Section 4.4 of the revised manuscript.

[W2 & Q2] Choice of LLM for abstract-level mapping

We reused RadZero's LLaMA3-extracted sentences for comparability, but did not use LLaMA3 for the second stage because it is now outdated. We instead chose Qwen3-Next-80B: its parameter count is close to that of Llama-3.3-70B-Instruct, and it is neither too old nor too recent, without being a frontier-scale model. Note that recent baselines such as GLINT have similarly moved to a newer model (gpt-oss-120b).

To test sensitivity to the choice of LLM for structured tuple generation, we repeated the abstract-level mapping with two other LLMs, 1) Llama-3.3-70B-Instruct (older than Qwen3-Next) and 2) gpt-oss-120b (similar release period but more parameters), and retrained the full model for each. As shown in the following table, our method is not highly sensitive to the mapping LLM.

Mapping LLMOpenICXR14PadChestCXD10CheXpertCXD10 (grounding)MS-CXR (grounding)
Qwen3-Next-80B (main)0.88910.83890.86090.84330.90370.73150.9222
Llama-3.3-70B-Instruct0.88490.83540.86410.84100.90700.70780.9281
gpt-oss-120b0.88420.84450.86150.85010.90320.70460.8982

[Q4] Use of the [CLS] token

We did not use [CLS] because we wanted a single representation that serves both classification and grounding. Grounding is read directly from the patch attention weights, and adding a global token to the image representation would instead disperse the attention weights rather than concentrate them on the relevant patches. In addition, with a ViT backbone followed by bidirectional transformer blocks, global context is already mixed into every patch embedding.

We tested the reviewer's suggestion by combining the [CLS] embedding with the attention-weighted patch representation, which are as follows. Adding [CLS] did not help: classification is unchanged or slightly lower, consistent with the reasoning above.

OpenICXR14PadChestCXD10CheXpertCXD10 (grounding)MS-CXR (grounding)
With [CLS] token0.88300.83360.86140.84180.90250.71280.9162
Ours (no [CLS] token)0.88910.83890.86090.84330.90370.73150.9222
AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer Xa7C,

Thank you again for your thoughtful review and concrete suggestions. We are writing to politely remind you that our responses to all reviews have been posted, and we would like to check whether our response has addressed your concerns.

Following your suggestions, we:

  • W1 & Q1: ran the control you proposed. Applying our proposed patch-level loss for false negative handling to the original positive pairs, with patch selection and loss settings unchanged, lowers all seven benchmarks.
  • Q3: explained why retaining false-negative pairs with local attraction works better than masking or relabeling. Findings are focal, so relabeling pulls unrelated patches toward the sentence, and masking removes the repulsion those patches still need. Our loss keeps the pair assignments and attracts only the top-M% patches. This is now in Sec. 4.4.
  • W2 & Q2: repeated the abstract-level mapping with Llama-3.3-70B-Instruct and gpt-oss-120b and retrained the full model for each.
  • Q4: tested combining the [CLS] embedding with the attention-weighted patch representation. It gives no gain in classification and lowers grounding.

If any of these points remains unresolved, please let us know as early as possible. Given the limited time in the discussion period, the sooner we receive your feedback, the better equipped we will be to run any additional experiments you might request. We are ready to engage in further discussion immediately.

Thank you again for your time and effort in evaluating our work.

 

Best regards,

The Authors

Reviewer_gpDhOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SentZero learns from chest X-ray images and report sentences for disease classification and lesion localization. It keeps the original report phrases and adds more general sentences about findings and their presence. It brings image features closer to sentences shared across studies and uses text to adjust pooled image features. The experiments test the full method, its components, and alternative training losses across several datasets.

Critical strengths
  • The model performs well across several benchmarks. It leads most of the reported comparisons and improves lesion localization over the listed baselines. Tests within the authors' own setup also show gains on localization and most classification datasets when the proposed components are added. These internal comparisons are useful because the external baselines do not all use the same architecture.
  • The paper includes useful checks of its design. It tests separate components, alternative ways to handle false negatives, and text conditioning against an image-only version with matched model size. Text conditioning helps on most reported measures. Keeping the original detailed phrases alongside general sentences is also a sensible way to add shared concepts without discarding the original report information.
Critical weaknesses
  • The explanation of the false-negative component is not fully supported by the component tests. It helps localization when added to the other proposed modules, but can hurt when added to the basic configuration. The basic configuration also matches the full model on MS-CXR. The results therefore suggest that the components work together, while the benefit of false-negative handling on its own remains unclear. Information about the quality of shared-sentence matches and a control using ordinary positive pairs would help explain the gains.
  • Some details needed to reproduce the results are missing. The appendix says that evaluation images were excluded from SentZero and RAD-DINO pretraining, which addresses the basic overlap concern. The authors should still identify the checkpoint and explain how they checked overlap at the image, study, and patient levels, especially for MS-CXR. They should also state which baselines were rerun and which data were used to choose hyperparameters. These are open reporting questions; the available evidence does not establish data leakage or improper test-set tuning.
Questions for authors
  • How do shared-sentence matches differ across positive, negative, and uncertain findings? Is the benefit of false-negative handling mainly due to its interaction with the other modules? Please explain the mixed component-test results.
  • Which RAD-DINO checkpoint was used, and how were evaluation cases excluded from pretraining? Which data were used to select hyperparameters, and which baseline results were rerun?
4 comments

We thank the reviewer for the thoughtful comments. We especially appreciate the recognition of our model’s strong performance over the baselines across multiple benchmarks, a strength noted by all reviewers. We are also grateful for the positive assessment of our framework design, including the false-negative handling, text conditioning, and two-level granularity with abstract-level mapping. Universal encoders with strong zero-shot performance across multiple chest X-ray tasks are still far from common, and we would be grateful if the reviewer could take these results into account in the final assessment.

We address each point of weakness and question you argued below. We use W for weaknesses and Q for questions. The numbers (1, 2, ...) follow the order of the main headings in the review.


[W1 & Q1] Effect of the false-negative loss and statistics on shared-sentence matches

Effect of the false-negative loss. Adding the false-negative loss to the basic configuration (Table 2, row 2 → row 5) slightly lowers grounding performance but improves all five classification benchmarks, so the overall effect is positive. In the full-component setting (row 6 → row 7), it improves both grounding and classification. As the reviewer noted, the benefit of the false-negative loss is more pronounced when all proposed components are used. We attribute this to abstract-level mapping, which improves performance but also introduces additional false-negative pairs, as detailed below.

Statistics of false-negative pairs. We analyzed how often false-negative pairs occur, since this is the motivation for our problem setting. As discussed in the Introduction, recent phrase-extraction-based methods have improved zero-shot vision-language pretraining, but they also cause sentences to overlap across studies. This overlap is unavoidable, because radiology reports describe the presence or absence of a limited set of findings. To quantify it, we measured false-negative statistics over one training epoch on the entire dataset with a global batch size of 256 (see the table below and Figure 4 in the Appendix C of the revised manuscript). Even without abstract-level mapping, more than 50% of the original sentences have at least one false-negative sentence, and roughly 10% of all pairs within each minibatch are false negatives.

Sentence sourceFN pairs / all pairs (%)Sentences with ≥ 1 FN (%)Images with ≥ 1 FN (%)
Original sentences9.80 ± 0.8153.7 ± 1.794.3 ± 1.4
Original sentences + abstract-level mapping10.90 ± 0.7262.4 ± 1.399.4 ± 0.5

The table below breaks down the percentage of sentences with at least one false negative by finding status (positive, negative, or uncertain).

Sentence sourcePositiveNegativeUncertainAll
Original sentences28.4 ± 2.081.8 ± 1.624.4 ± 9.453.7 ± 1.7
Original sentences + abstract-level mapping45.0 ± 1.785.0 ± 1.345.3 ± 7.362.4 ± 1.3

The false-negative problem is not unique to RadZero; it affects all prior methods that rely on phrase extraction. The table below reports the same statistics for CARZero, RadZero, and GLINT, which support our argument that false negatives need to be handled explicitly. We have also added these results to Figure 4 in Appendix C.

MethodFN pairs / all pairs (%)Sentences with ≥ 1 FN (%)Images with ≥ 1 FN (%)
CARZero12.63 ± 1.3559.2 ± 2.476.9 ± 2.7
RadZero9.80 ± 0.8153.7 ± 1.794.3 ± 1.4
GLINT8.80 ± 0.7553.6 ± 1.796.2 ± 1.2

Interaction between the false-negative loss and abstract-level mapping. Abstract-level mapping provides supervision at two granularities, but it also increases cross-study overlap. The increase is large for positive findings (28.4% → 45.0%), which are exactly the findings that have a localized region to ground. The false-negative loss therefore acts as a regularizer that offsets the additional false negatives introduced by the mapping. We have revised Sec. 4.3 to describe the false-negative loss as complementary to abstract-level mapping.

[W2 & Q2] Reproducibility details and dataset leakage control

RAD-DINO checkpoint and dataset leakage. We use the RAD-DINO-MAIRA-2 checkpoint, which is the image encoder of the vision-language model MAIRA-2. Its training datasets and splits are documented in the RAD-DINO and MAIRA-2 papers [1, 2], and we checked that the datasets for our zero-shot evaluation were not included. For instance, Open-I is used only for evaluation, the official test sets of CXR14 and CheXpert are excluded, and the MS-CXR validation and test splits are not used. We have added the checkpoint name and citation to the revised manuscript.

We note two further points. First, our model outperforms CoNNs, which uses the same RAD-DINO-MAIRA-2 checkpoint. Second, RAD-DINO is an image-only self-supervised encoder with no zero-shot capability of its own. Any data leakage would therefore matter most at the vision–language stage, and we controlled that stage more strictly, as described below.

Overlap control of MS-CXR. As you pointed out, MS-CXR is derived from MIMIC-CXR, our pretraining corpus. To prevent data leakage, we identified every MS-CXR test image in the MIMIC-CXR metadata by its patient and study IDs and excluded it from our training data. The other test benchmarks come from sources entirely separate from MIMIC-CXR, so they do not overlap with our training data.

Baselines. All baseline numbers in Table 1 are taken from published papers (as reported in CARZero [3], RadZero [4], CoNNs [5], and GLINT [6]); no baseline was retrained. Since CoNNs did not report PadChest AUROC, we obtained this single value by running inference with the authors’ released checkpoint, without any retraining or modification.

[W2 & Q2] Hyperparameter setting

For false negative loss hyperparameters, we report zero-shot test results for the loss coefficient λ_fn and the patch selection ratio M in Tables 5–6. We do not claim that any single hyperparameter setting is superior to the others; these tables are provided to illustrate the overall trends and to keep our experiments transparent. Since some value must be chosen, we report the one that gives the most stable results across datasets.

As you pointed out, in real-world deployment, hyperparameter selection on a held-out set can make the process more refined and robust. We therefore re-evaluated the results across hyperparameters on a held-out set separate from training. For this purpose, we used the MIMIC-CXR test split, which provides physician-verified labels for 13 major findings (5,159 cases × 13 findings = 67,067 labels) [7, 8]. This split is not used for training or for any of our zero-shot benchmarks. Held-out selection picks the same configuration (λ_fn = 0.01, M = 20%), so the main results are unchanged. We have added the tables below to Table 7 in Appendix B of the revised manuscript.

λ_fn (M = 20%)AUROCAUPRC
0.01 (final)0.79390.4093
0.030.78910.4055
0.050.78800.4054
M (λ_fn = 0.01)AUROCAUPRC
10%0.78920.4022
20% (final)0.79390.4093
30%0.79080.4063

For contrastive loss temperature, we chose 0.1, which is within the range commonly used in general-domain and medical vision–language contrastive learning (0.07–0.1) [9-11].


References

[1] Bannur, Shruthi, et al. "Maira-2: Grounded radiology report generation, 2024." URL https://arxiv. org/abs/2406.04449.

[2] Pérez-García, Fernando, et al. "Exploring scalable medical image encoders beyond text supervision." Nature Machine Intelligence 7.1 (2025): 119-130.

[3] Lai, Haoran, et al. "Carzero: Cross-attention alignment for radiology zero-shot classification." 2024 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR). IEEE, 2024.

[4] Park, Jonggwon, et al. "RadZero: Similarity-Based Cross-Attention for Explainable Vision-Language Alignment in Chest X-ray with Zero-Shot Multi-Task Capability." Advances in Neural Information Processing Systems 38 (2026): 56008-56034.

[5] Lian, Chenyu, et al. "Concept-Guided Noisy Negative Suppression for Zero-Shot Classification and Grounding of Chest X-Ray Findings." International Conference on Medical Image Computing and Computer-Assisted Intervention. Cham: Springer Nature Switzerland, 2026.

[6] Park, Jonggwon, et al. “GLINT: Sparsely Gated Vision-Language Alignment for Fine-Grained Radiology Representations.” Advances in Neural Information Processing Systems 40 (2026).

[7] Yang, Lin, et al. "Advancing multimodal medical capabilities of Gemini." arXiv preprint arXiv:2405.03162 (2024).

[8] Park, K., Sayres, R., Sellergren, A., Pollard, T., Jamil, F., Kohlberger, T., Lau, C., & Kiraly, A. (2025). Application of Med-PaLM 2 in the refinement of MIMIC-CXR labels (version 1.0.0). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/7wmp-jx90

[9] Radford, Alec, et al. "Learning transferable visual models from natural language supervision." International conference on machine learning. PmLR, 2021.

[10] Chen, Zixiang, et al. "Understanding transferable representation learning and zero-shot transfer in clip." International Conference on Learning Representations. Vol. 2024. 2024.

[11] Zhang, Yuhao, et al. "Contrastive learning of medical visual representations from paired images and text." Machine learning for healthcare conference. PMLR, 2022.

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer gpDh,

Thank you again for your thoughtful review and concrete suggestions. We are writing to politely remind you that our responses to all reviews have been posted, and we would like to check whether our response has addressed your concerns.

Following your suggestions, we:

  • W1 & Q1 (role of the false-negative loss): showed that it improves all five classification benchmarks on the basic configuration, and both classification and grounding in the full model. Sec. 4.3 now describes it as complementary to abstract-level mapping.
  • W1 & Q1 (shared-sentence statistics): reported the matches by finding status. More than 50% of the original sentences have at least one false negative. We also found that this is a common problem for all phrase-extraction-based methods, not only our setup: the rate is 59.2% for CARZero, 53.7% for RadZero and 53.6% for GLINT. Abstract-level mapping raises the rate for positive findings from 28.4% to 45.0%.
  • W2 & Q2 (reproducibility and data leakage control): named the checkpoint (RAD-DINO-MAIRA-2) and verified again that data leakage was controlled. We checked the training data documented in the RAD-DINO and MAIRA-2 papers, and matched the MS-CXR test set against our training data by patient and study ID.
  • W2 & Q2 (hyperparameters): measured the performance across λ_fn and M on a held-out set, the MIMIC-CXR test split. Our final configuration (λ_fn = 0.01, M = 20%) still gives the highest AUROC and AUPRC.

If any of these points remains unresolved, please let us know as early as possible. The sooner we receive your feedback, the better equipped we will be to run any additional experiments you might request.

Thank you again for your time and effort in evaluating our work.

 

Best regards,

The Authors

Reviewer_84noOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

SentZero is a CLIP-style image-sentence pretraining framework for chest X-rays, built on a LoRA-tuned RAD-DINO ViT plus four trainable transformer blocks, and an MPNet text encoder. It starts from RadZero's recipe (LLaMA3-extracted finding sentences, similarity-weighted patch aggregation, multi-positive contrastive loss) and adds four components: (1) multi-layer feature aggregation (2) abstract-level mapping, where a second LLM (Qwen3-Next-80B) reduces each finding sentence to a (Topic, Presence) tuple and added as an extra positive (3) a FiLM-style residual modulation of the attention-pooled visual feature conditioned on the sentence (4) a false-negative loss that, for sentences shared verbatim across studies in a batch, pulls the top-20% most similar patches of the other image toward that sentence. Zero-shot AUROC on five classification sets and pointing-game accuracy on two grounding sets are reported against eight baselines.

Critical strengths

1. The best reported zero-shot results on the standard multi-task CXR benchmark, including a large grounding gain. SentZero is first on six of seven benchmarks against eight baselines that include the two most recent methods (CoNNs, GLINT), and its accuracy on ChestXDet10 is 7.6 points above the strongest prior result.

2. Abstract-level sentence mapping is a simple, transferable idea. Reducing each finding sentence to a (Topic, Presence) tuple and adding the coarse sentence as an extra positive costs one LLM pass over the text and no architectural change. In Table 2 it is the largest single contributor to classification (rows 2 to 3), and it is also what makes cross-study matching possible, since detailed sentences rarely coincide verbatim. The underlying observation, that radiology reports draw from a narrow set of clinical statements and that this redundancy can be used as supervision at two granularities rather than treated as noise, is the paper's genuine insight.

Critical weaknesses

1. The "no component" baseline in Table 2 (row 1) uses RadZero's training sentences and, by the paper's own description of RadZero (Sec. 2), the same core mechanism (similarity-weighted patch aggregation with a multi-positive contrastive loss), yet it already scores 0.875 / 0.814 / 0.859 / 0.809 / 0.908 on classification and 0.651 / 0.892 on grounding, above RadZero by 1-5 points on every dataset and already competitive with the strongest baselines (CoNNs, GLINT), beating them on 3 of 7 benchmarks, before any proposed component is added. The paper does not say what accounts for this gap (backbone, LoRA, extra transformer blocks, resolution, text encoder, training budget). Thus, If I understand it correctly, the headline claim, "SentZero outperforms prior multi-task zero-shot methods," is confound, and the components should be credited only with the row 1 to row 7 deltas. This is significant because it is the paper's primary claim, and in its current form the comparison in Table 1 cannot support it.

2. The ablations does not distinguish the proposed components from noise, and hyperparameters are selected on the test sets. Every ablation uses a single seed and no interval is reported. Many component deltas are 0.001-0.005 AUROC, the false-negative loss, presented as a core contribution, changes classification by small margin and shows an effect only on grounding, which is measured on 542 and 167 images. Tables 5-6 then choose λ_fn and the patch ratio M by comparing results on these same test benchmarks, where the selected value wins on only 3 of 7 benchmarks in each table.

Questions for authors

**Clarification requests **

  1. Table 2 row 1 uses RadZero's sentences and, per your Sec. 2 description, RadZero's core mechanism, yet it is 1-5 points above RadZero on every dataset. What differs (backbone, LoRA, the four extra transformer blocks, input resolution, text encoder, batch size, epochs)? Please add one of the following: (a) RadZero's released checkpoint re-evaluated under your exact protocol, plus RadZero's objective trained on your RAD-DINO/LoRA/518px stack (which is what row 1 claims to be, so state explicitly whether that is what row 1 is), or (b) the four components added on top of RadZero's original backbone. Either isolates setup from components.

  2. Please report mean ± std over at least three seeds for SentZero in Table 1, and bootstrap intervals for the two grounding sets (542 images and 167 paris).

  3. Were λ_fn and the patch ratio M (Tables 5-6) selected by comparing results on the test benchmarks? If so, what would the main results be under a selection made on MIMIC validation data or any held-out split?

**Suggestions **

  • State the row 1 to row 7 deltas as the method's contribution in the abstract and Sec. 4.2, separately from the Table 1 comparison.
  • Consider reporting results on MS-CXR split by phrase length, since Appendix C claims the model handles long phrases.
  • Fix the duplicated phrase in Sec. 3.2 ("the pair is the pair is treated") and add a citation for Qwen3-Next-80B.
4 comments

We thank the reviewer for their careful and constructive feedback. We particularly appreciate the reviewer’s recognition of our model’s strong performance relative to the baselines across multiple benchmarks, a strength noted by all reviewers. We also appreciate the reviewer’s positive assessment of our use of sentence-centric framework based on structured tuples. As universal encoders with strong zero-shot performance across multiple chest X-ray tasks are still far from common, we would be grateful if the reviewer could take these results into account in the final assessment.

We address each point of weakness and question you argued below. We use W for weaknesses and Q for questions. The numbers (1, 2, ...) follow the order of the main headings in the review.


[W1 & Q1] Basic setting description / Isolation of setup from components

As you noted, our method shares some basic elements with RadZero; however, we did not adopt its original configuration as is. The main reason is its pretrained vision encoder, XrayDINOv2, which was pretrained at a relatively low input resolution of 224 px. Although the authors of RadZero paper report that it can be applied to 518 px inputs, we were concerned about this resolution mismatch and instead used RAD-DINO-MAIRA-2, which is pretrained at 518 × 518, together with a LoRA adapter. With this change, we also increased the number of additional transformer blocks from 2 to 4.

The first row of Table 2 therefore corresponds to a modified reimplementation of RadZero, as described in your question Q1(a): RadZero trained within our setup, with the encoder and architectural changes noted above and none of our proposed components. To keep the comparison fair, we matched RadZero’s number of training epochs, global batch size, input resolution, and text encoder across all main and ablation experiments.

As we will discuss in our response to the second weakness below, comparing the base model with the full model (row 1 vs. row 7 in Table 2) shows that using all of our proposed components improves classification and grounding by up to +3.4 and +8.0 percentage points (pp), respectively. These gains are comparable to the improvements that recent SOTA models have achieved over their predecessors (please refer to Table 1 and the table in our response to [W2] below). Following the reviewer’s suggestion, the revised abstract and Sec. 4.3 now report the row 1 → row 7 deltas as the contribution of the method, separately from the Table 1 comparison.


[W2] Size of the component gains

We acknowledge that the gain from each individual component is modest on some datasets. However, in terms of the overall trend, each component improves performance fairly consistently, and we do not claim that every individual increment is significant on every dataset. Our claim concerns the combined effect: with all components enabled, classification improves by up to +3.4 pp and grounding by up to +8.0 pp over the setting with no components (row 1 vs. row 7 in Table 2).

To put this gain in context, we compare it with the maximum gain each recent baseline achieved over the preceding SOTA. As shown below, those gains range from +2.5 to +3.8 pp for classification and from +3.4 to +9.5 pp for grounding. The gain from our full set of components falls within both ranges, which indicates that it is meaningful.

Max. classification AUROC gainMax. pointing accuracy gain
RadZero (vs. CARZero)+3.1 pp+9.5 pp
CoNNs (vs. RadZero)+3.8 pp+3.4 pp
GLINT (vs. RadZero)+2.5 pp+4.2 pp
Ours (base vs. full component)+3.4 pp+8.0 pp

The table also shows that as more zero-shot encoders have been proposed recently, gains have begun to level off and large improvements have become increasingly hard to achieve. Moreover, no recent method in Table 1 improves uniformly over its predecessor: RadZero falls below CARZero on CheXpert, and CoNNs falls below RadZero on PadChest. This reflects how difficult it is for a single multi-task zero-shot encoder to improve across the entire benchmark. We therefore respectfully ask the reviewer to assess our improvements in light of this trend.

[W2 & Q2] Multi-seed results and bootstrap confidence intervals

Multi-seed runs. Following your suggestion, we trained SentZero with three different random seeds. The table below reports the mean ± standard deviation, alongside the best baseline on each dataset (from Table 1).

OpenICXR14PadChestCXD10CheXpertCXD10 (grounding)MS-CXR (grounding)
Best baseline0.8710.8190.8530.8250.9230.6560.886
SentZero (3 seeds)0.886 ± 0.0020.834 ± 0.0030.862 ± 0.0010.843 ± 0.0010.903 ± 0.0000.722 ± 0.0080.906 ± 0.012

On five benchmarks (OpenI, CXR14, PadChest, CXD10 classification, and CXD10 grounding), the mean exceeds the best baseline by more than four standard deviations. On MS-CXR, it exceeds the best baseline by about 1.7 standard deviations. Wherever SentZero improves over the previous best baseline, the improvement is therefore consistent across seeds and not an artifact of a single run.

Bootstrap confidence intervals for grounding. We computed 95% confidence intervals from 10,000 bootstrap resamples for SentZero and the two baselines with publicly released checkpoints (RadZero and CoNNs). CXD10 contains 966 boxes from 542 images across 10 classes, and MS-CXR contains 167 phrase–box pairs.

ModelCXD10 [95% CI]MS-CXR [95% CI]
RadZero (released)0.6288 [0.5910, 0.6657]0.8503 [0.7964, 0.9042]
CoNNs (released)0.6200 [0.5803, 0.6593]0.8503 [0.7964, 0.9042]
SentZero0.7302 [0.6938, 0.7656]0.9222 [0.8802, 0.9581]

To test whether these gains are statistically significant, we also ran a paired bootstrap on the performance difference (i.e., SentZero minus baseline). Each cell reports the mean difference with its 95% CI and the bootstrap p-value.

CXD10MS-CXR
ComparisonMean difference [95% CI], p-valueMean difference [95% CI], p-value
SentZero (ours) vs. RadZero+0.1014 [+0.0672, +0.1361], p < 0.0001+0.0719 [+0.0180, +0.1257], p = 0.006
SentZero (ours) vs. CoNNs+0.1102 [+0.0733, +0.1485], p < 0.0001+0.0719 [+0.0180, +0.1257], p = 0.008

All four intervals exclude zero, with p < 0.01 in every case. Therefore, SentZero's grounding gains over both baselines significantly on both datasets. (Note that the identical MS-CXR values in the two comparisons are not an error: RadZero and CoNNs achieve the same score on this dataset on bootstrapping, so their differences from SentZero and the corresponding intervals coincide. Their p-values are different.)


[W2 & Q3] Hyperparameters for false negative loss

We report zero-shot test results for the loss coefficient λ_fn and the patch selection ratio M in Tables 5–6. We do not claim that any single hyperparameter setting is superior to the others; these tables are provided to illustrate the overall trends and to keep our experiments transparent. Since some value must be chosen, we report the one that gives the most stable results across datasets.

As you pointed out, in real-world deployment, hyperparameter selection on a held-out set can make the process more refined and robust. We therefore re-evaluated the results across hyperparameters on a held-out set separate from training. For this purpose, we used the MIMIC-CXR test split, which provides physician-verified labels for 13 major findings (5,159 cases × 13 findings = 67,067 labels) [1, 2]. This split is not used for training or for any of our zero-shot benchmarks. Held-out selection picks the same configuration (λ_fn = 0.01, M = 20%), so the main results are unchanged. We have added the tables below to Table 7 in Appendix B of the revised manuscript.

λ_fn (M = 20%)AUROCAUPRC
0.01 (final)0.79390.4093
0.030.78910.4055
0.050.78800.4054
M (λ_fn = 0.01)AUROCAUPRC
10%0.78920.4022
20% (final)0.79390.4093
30%0.79080.4063

Revisions in response to the “Suggestions” section

1) Row 1 → row 7 deltas. We now report these deltas in the abstract and in Sec. 4.3, separately from the Table 1 comparison.

2) MS-CXR accuracy by phrase length. We added this breakdown to the Appendix D of revised manuscript. Pointing accuracy decreases only slightly as phrases get longer and remains above 90% even for the longest group:

Phrase length (words)PhrasesHitsPointing accuracy95% CI (Wilson)
< 4524994.23%84.36–98.02
4–5625791.94%82.47–96.51
> 5534890.57%79.75–95.90
All16715492.22%87.14–95.39

3) Typo and citations. We removed the duplicated phrase in Sec. 3.2 and added the missing citation for Qwen3-Next-80B. GLINT has been accepted to NeurIPS 2026, so we updated its citation and its venue in Table 1.


References

[1] Yang, Lin, et al. "Advancing multimodal medical capabilities of Gemini." arXiv preprint arXiv:2405.03162 (2024).

[2] Park, K., Sayres, R., Sellergren, A., Pollard, T., Jamil, F., Kohlberger, T., Lau, C., & Kiraly, A. (2025). Application of Med-PaLM 2 in the refinement of MIMIC-CXR labels (version 1.0.0). PhysioNet. RRID:SCR_007345. https://doi.org/10.13026/7wmp-jx90

AuthorsOpenReview ↗

Follow-up on Our Response

Dear Reviewer 84no,

Thank you again for your careful review and concrete suggestions. We are writing to politely remind you that our responses to all reviews have been posted, and we would like to check whether our response has addressed your concerns.

Following your suggestions, we:

  • W1 & Q1: clarified what row 1 of Table 2 is. It is a modified reimplementation of RadZero trained in our setup (RAD-DINO-MAIRA-2 at 518 px, LoRA, four transformer blocks), with epochs, batch size, resolution and text encoder matched to RadZero. The abstract and Sec. 4.3 now report the row 1 → row 7 deltas (up to +3.4 pp classification, +8.0 pp grounding) separately from the Table 1 comparison. These gains are comparable to the improvements that recent SOTA models have achieved over their predecessors (please refer to Table 1 and the table in our response to [W2]).
  • W2 & Q2: ran three seeds and bootstrap tests. On five of seven benchmarks, the three-seed mean exceeds the best baseline by more than four standard deviations. Paired bootstrap shows grounding gains over RadZero and CoNNs of +10.1 and +11.0 pp on CXD10 and +7.2 pp on MS-CXR (p < 0.01 in all four comparisons).
  • W2 & Q3: Shows the results according to λ_fn and M on a held-out set. The MIMIC-CXR test split, which is not used for training or for any benchmark, picks the same configuration (λ_fn = 0.01, M = 20%), so the main results are unchanged.
  • Suggestions: added MS-CXR accuracy by phrase length (above 90% even for the longest group), removed the duplicated phrase in Sec. 3.2 and added the Qwen3-Next-80B citation.

If any of these points remains unresolved, please let us know as early as possible. Given the limited time in the discussion period, the sooner we receive your feedback, the better equipped we will be to run any additional experiments you might request. We are ready to engage in further discussion immediately.

Thank you again for your time and effort in evaluating our work.

 

Best regards,

The Authors

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response and Kind Reminder

Dear Area Chair and Reviewers,

We thank all reviewers for their time and thoughtful feedback. With about 48 hours left in the discussion period, we are writing to politely remind you that our responses to all reviews and a revised manuscript have been posted.

We are encouraged that all the reviewers found SentZero to be a well-motivated sentence-centric framework with strong zero-shot results. Below we summarize the strengths noted across the reviews. Each weakness and question is addressed in our individual response to the corresponding reviewer, and we kindly ask you to refer to those responses.

[Strengths]

1. Strong zero-shot multi-task performance (Reviewers 84no, gpDh, Xa7C). All reviewers noted that SentZero performs well across the classification and grounding benchmarks, with a large gain in grounding over prior methods.

2. Simple and transferable abstract-level mapping (Reviewers 84no, gpDh). The reviewers found that adding (Topic, Presence)-level sentences alongside the original detailed phrases is a sensible way to supervise at two granularities, at the cost of one LLM pass and no architectural change.

3. Clear and compelling motivation (Reviewers 84no, Xa7C). The reviewers recognized the false-negative problem caused by clinically equivalent sentences recurring across studies as an interesting and important one, and noted that the sentence-level formulation supports classification and grounding in a single framework.

4. Comprehensive evaluation and design checks (Reviewers gpDh, Xa7C). The reviewers noted the component ablations, the alternative false-negative strategies, and the matched-size comparison for text conditioning.

We have carefully addressed every weakness and question, and we hope your concerns have been resolved. If any issue remains, we kindly ask that you let us know as early as possible. With only about 48 hours remaining, the sooner we receive your feedback, the better equipped we will be to run any additional experiments you might request.

We are ready to engage in further discussion immediately. Thank you again for your time and effort in evaluating our work.

Best regards,

The Authors

AuthorsOpenReview ↗

[Final Reminder] About one day remains in the author-reviewer discussion period (Nov 18 AoE)

Dear Reviewers,

We sincerely appreciate the time and care you have devoted to our submission, especially during such a busy period. As the discussion period closes at the end of November 18 (AoE), now about one day away, we would like to send one final, gentle reminder.

Your reviews have helped us clarify and strengthen the paper, and we have done our best to respond to every point thoroughly. We respectfully believe that our rebuttal addresses all of the concerns raised. For a summary, we kindly invite you to refer to our general response and to our individual reply to each review.

If any response remains unclear or any concern is still open, we would be very grateful if you could let us know. We are monitoring the system continuously and will reply right away, before the phase closes.

If you find that your concerns have been resolved, we would deeply appreciate it if you could update your review and rating accordingly. Even a brief acknowledgment would be of great help, as it would ensure that the final decision reflects the outcome of this discussion.

Thank you once again for your dedication and thoughtful evaluation of our work.

 

Best regards,

Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.