- Rating
- 3
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes SentZero for zero-shot chest X-ray classification and grounding through vision–language pretraining. It combines LLM-based abstract-level sentence augmentation, a patch-level false-negative mitigation objective based on shared cross-report statements, sentence-conditioned residual modulation, and multi-layer visual feature aggregation, achieving strong results across multiple public benchmarks.
Critical strengths
1.Clear and compelling motivation: Starting from the contrastive learning objective, the paper identifies the risk of pushing apart compatible image–text pairs when identical descriptions appear in other samples. This is an interesting and important problem, and the sentence-level formulation elegantly supports both zero-shot classification and grounding within a unified pretraining framework.
2.Comprehensive experimental evaluation: The paper evaluates the method on multiple chest X-ray benchmarks, including CheXpert. Detailed ablation studies further support the effectiveness of the proposed components.
Critical weaknesses
1.Attribution of the false-negative mitigation gains remains incomplete: Table 3 compares masking and positive relabeling, but these alternatives modify contrastive pair assignments, whereas the proposed approach introduces additional local image–text supervision. Without a control applying the same local alignment objective to original positive pairs, it remains unclear how much of the improvement comes from targeted false-negative handling rather than more general local supervision.
2.Limited validation of LLM choices: Although reusing RadZero’s LLaMA3-extracted phrases helps ensure comparability, the authors do not justify using Qwen for the subsequent structured extraction. Without comparisons across LLM choices, it remains unclear how sensitive the final performance is to the specific extraction models.
Questions for authors
1.Could the authors apply the same local image–text alignment loss to original positive pairs, keeping patch selection and loss settings unchanged, to distinguish the benefits of additional local supervision from targeted false-negative handling?
2.Could the authors evaluate alternative LLMs to assess the method’s robustness to the choice of extraction model? For example, could LLaMA also be used for structured extraction to maintain consistency across the two stages?
3.Although Table 3 evaluates false-negative masking, could the authors explain why excluding other positive sentences from the same sample is appropriate, whereas additionally excluding shared sentences across samples generally reduces performance? What mechanism explains the advantage of retaining these false-negative pairs while adding local attraction?
4.The ViT [CLS] output, which typically serves as a global image representation, does not appear to be directly used in the subsequent alignment or prediction stages. Could the authors explain this choice and whether combining the attention-weighted patch representation with the global [CLS] embedding could be beneficial?
5 comments
[W2 & Q2] Choice of LLM for abstract-level mapping
We reused RadZero's LLaMA3-extracted sentences for comparability, but did not use LLaMA3 for the second stage because it is now outdated. We instead chose Qwen3-Next-80B: its parameter count is close to that of Llama-3.3-70B-Instruct, and it is neither too old nor too recent, without being a frontier-scale model. Note that recent baselines such as GLINT have similarly moved to a newer model (gpt-oss-120b).
To test sensitivity to the choice of LLM for structured tuple generation, we repeated the abstract-level mapping with two other LLMs, 1) Llama-3.3-70B-Instruct (older than Qwen3-Next) and 2) gpt-oss-120b (similar release period but more parameters), and retrained the full model for each. As shown in the following table, our method is not highly sensitive to the mapping LLM.
| Mapping LLM | OpenI | CXR14 | PadChest | CXD10 | CheXpert | CXD10 (grounding) | MS-CXR (grounding) |
|---|---|---|---|---|---|---|---|
| Qwen3-Next-80B (main) | 0.8891 | 0.8389 | 0.8609 | 0.8433 | 0.9037 | 0.7315 | 0.9222 |
| Llama-3.3-70B-Instruct | 0.8849 | 0.8354 | 0.8641 | 0.8410 | 0.9070 | 0.7078 | 0.9281 |
| gpt-oss-120b | 0.8842 | 0.8445 | 0.8615 | 0.8501 | 0.9032 | 0.7046 | 0.8982 |
[Q4] Use of the [CLS] token
We did not use [CLS] because we wanted a single representation that serves both classification and grounding. Grounding is read directly from the patch attention weights, and adding a global token to the image representation would instead disperse the attention weights rather than concentrate them on the relevant patches. In addition, with a ViT backbone followed by bidirectional transformer blocks, global context is already mixed into every patch embedding.
We tested the reviewer's suggestion by combining the [CLS] embedding with the attention-weighted patch representation, which are as follows. Adding [CLS] did not help: classification is unchanged or slightly lower, consistent with the reasoning above.
| OpenI | CXR14 | PadChest | CXD10 | CheXpert | CXD10 (grounding) | MS-CXR (grounding) | |
|---|---|---|---|---|---|---|---|
| With [CLS] token | 0.8830 | 0.8336 | 0.8614 | 0.8418 | 0.9025 | 0.7128 | 0.9162 |
| Ours (no [CLS] token) | 0.8891 | 0.8389 | 0.8609 | 0.8433 | 0.9037 | 0.7315 | 0.9222 |
Follow-up on Our Response
Dear Reviewer Xa7C,
Thank you again for your thoughtful review and concrete suggestions. We are writing to politely remind you that our responses to all reviews have been posted, and we would like to check whether our response has addressed your concerns.
Following your suggestions, we:
- W1 & Q1: ran the control you proposed. Applying our proposed patch-level loss for false negative handling to the original positive pairs, with patch selection and loss settings unchanged, lowers all seven benchmarks.
- Q3: explained why retaining false-negative pairs with local attraction works better than masking or relabeling. Findings are focal, so relabeling pulls unrelated patches toward the sentence, and masking removes the repulsion those patches still need. Our loss keeps the pair assignments and attracts only the top-M% patches. This is now in Sec. 4.4.
- W2 & Q2: repeated the abstract-level mapping with Llama-3.3-70B-Instruct and gpt-oss-120b and retrained the full model for each.
- Q4: tested combining the [CLS] embedding with the attention-weighted patch representation. It gives no gain in classification and lowers grounding.
If any of these points remains unresolved, please let us know as early as possible. Given the limited time in the discussion period, the sooner we receive your feedback, the better equipped we will be to run any additional experiments you might request. We are ready to engage in further discussion immediately.
Thank you again for your time and effort in evaluating our work.
Best regards,
The Authors
We thank the reviewer for the thoughtful and constructive comments. We particularly appreciate the recognition of our model’s strong performance over the baselines across multiple benchmarks, a strength highlighted by all reviewers. We are also grateful for the positive assessment of the motivation behind our proposed components and of our evaluation design. Universal encoders that achieve strong zero-shot performance across multiple chest X-ray tasks still remain rare, and we would be grateful if the reviewer could take these results into account in the final assessment.
We address each point of weakness and question you argued below. We use W for weaknesses and Q for questions. The numbers (1, 2, 3) follow the order of the main headings in the review. For a more natural flow, we answer Q3 directly after Q1, since both concern the false-negative loss, and then Q2 and Q4.
As suggested, we applied the false negative patch loss Lfn to the original positive pairs instead of the false negative pairs. However, this did not achieve higher performance, indicating that the gain from our proposed loss does not simply come from additional local supervision. Combined with the motivation and working hypothesis behind our false negative handling, which we explain in our response to [Q3] below, this result supports the validity of the proposed loss.
The design of our proposed false negative patch loss follows from the focal nature of radiographic findings, which we discussed with a medical expert during study design. For instance, in a sentence such as "There is opacity", only a few patches contain the evidence; the remaining patches do not show opacity. If a false negative pair is directly converted into a positive (false negative transition) or excluded from the loss computation (false negative masking), the following issues arise and can in fact lead to detrimental results.
Since the image representation is a similarity-weighted sum over all patches, any change to the contrastive logit of a pair acts on all patches of that image in one direction. As a result, the two methods above can misguide the global pull–push geometry of all patches from the same image sample.
Our loss mechanism. Our proposed loss leaves the contrastive pair assignments unchanged, and therefore preserves the global pull–push geometry. Additional attraction is applied only to the top-M% patches that are most similar to the shared sentence, which are the patches that should not be pushed away from it. Therefore, unlike the two methods above, our loss largely preserves the global contrastive objective while selectively mitigating the effect on only those patches whose similarity could be reduced by potential false negatives. In addition, following the original formulation of the MP-NCE loss, we exclude the other positives from the denominator when computing it.
We have incorporated the above content into Section 4.4 of the revised manuscript.