- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper studies weakly supervised slice-level localization in 3D CT. It first shows that an image-agnostic Centered Gaussian (CG) baseline outperforms existing MIL methods at localization on three RSNA CT datasets (head, chest, abdomen). It then proposes Normal Guidance (NG), which fits a Gaussian reference to the mean and variance of the current attention (with stop-gradient) and penalizes the KL divergence between them; for TransMIL, it is applied per head (MHNG). All experiments use a frozen ImageNet ViT-B/16 encoder and a linear head. NG/MHNG significantly exceed CG on Head and Chest, are statistically tied on Abdomen, and keep scan-level classification competitive. The paper also introduces “best-in-class ceilings” trained with slice labels as reference points.
Critical strengths
- The evaluation is large-scale and rigorous: over 4M slices across three public datasets, patient-level splits, three partitions, and bootstrap significance tests. Model selection uses only scan-level labels, so the weak-supervision setting is strictly respected.
- Extending the CG finding to multiple body parts is itself a valuable negative result, signaling that the localization ability of current MIL attention is overstated.
- The method is simple, pooling-agnostic, and reproducible. The limitations are discussed candidly, including the unimodality assumption, evaluation on positive bags only, and the attention-is-not-explanation issue.
Critical weaknesses
- The gains may come mainly from a positional prior rather than from image content. NG’s reference is centered at the attention’s own mean, so it is not center-biased by construction. The apparent center bias likely comes from initialization: near-uniform initial attention has E[J] ≈ S/2, which yields a centered, broad reference.
- The baselines are not strong enough; in particular, simple content-aware baselines are missing. Per-bag AUROC is rank-based, so CG is effectively “rank slices by distance to the middle slice.” In head CT, the middle slices are often exactly the ones containing the most brain tissue. The following baselines should be added: scoring each slice by organ or tissue coverage (e.g., via thresholding or TotalSegmentator); a weakly supervised positional prior, i.e., MIL with only the position j/S as instance feature.
- The weak encoder limits the generality of the conclusions. With a frozen ImageNet ViT, scan-level AUROC on Chest and Abdomen is only about 0.66–0.68, far below what is typically achievable on these datasets. With such a weak content signal, it is expected that positional priors dominate. Please verify whether CG remains strong, and whether NG still helps, with CT-specific or medically pretrained encoders (e.g., a CT foundation model, RadImageNet, or fine-tuned 2D/3D models).
- There is no evidence for the multimodal capability of the multi-head variant. MHNG is motivated by multiple non-adjacent ROIs, but there is no quantitative evaluation on multi-ROI scans. The appendix visualizations (e.g., Figs. C.6 and C.10) show that the head-averaged attention remains unimodal. On Abdomen, MHNG (0.634) is below NG (0.663) and even below CG.
Questions for authors
What are the localization AUROCs of NG, MHNG, and CG when stratified by ROI position and count?
3 comments
Response 2 of 2: Encoder evaluations and multi-ROI discussion
3. Encoder
With a frozen ImageNet ViT, scan-level AUROC on Chest and Abdomen is only about 0.66–0.68, far below what is typically achievable
We don’t doubt that some better encoder might raise performance here. But this has been harder to achieve than we expected.
We’re not sure what exactly is “typically achievable” in the reviewer’s mind, since no specific work is cited. Several tasks are supported on these datasets. Before assessing whether any other performance is comparable, we’d like to verify that the class definitions, 2D slice resolution (224x224 pixels), and train/test splits roughly match our settings.
To try to investigate improved encoders and gauge the overall difficulty, we tried two different options, each surveyed below. We plan to add both to the manuscript soon.
Attempt 1: MedSAM (updated Wed 2026-11-18)
We ran experiments using frozen MedSAM (Ma et al. 2024) as the image encoder in our MIL pipeline. This is an encoder trained on medical images to perform segmentation tasks, using the popular “Segment Anything” approach. In MedSAM’s training set, there were over half million image-mask pairs for the CT modality. CT is the 2nd most common modality in its training set (see their Fig 2a).
Below are results for all datasets
Head CT bag-level AUROC
- ABMIL with ViT (Tab 2): 0.919 +/- 0.009
- ABMIL with MedSAM: 0.938+/- 0.009
- NG with MedSAM: 0.973 +/- 0.006
Abdomen CT bag-level AUROC
- ABMIL with ViT (Tab 2): 0.639 +/- 0.044
- ABMIL with MedSAM: 0.676 +/- 0.025
- NG with MedSAM: 0.676 +/- 0.019
Chest CT bag-level AUROC
- ABMIL with ViT: 0.664 ± 0.017
- ABMIL with MedSAM: 0.649 ± 0.010
- NG with MedSAM: 0.652 ± 0.011
Interpretation: Swapping ViT to MedSAM helps bag-level classification improve modestly on 2 datasets but there is a slight drop on Chest. For ABMIL, the AUROC increases by 0.019 on Head CT and 0.037 on Abdomen CT. The scope of change is modest at best, so we suggest that using a medical foundation model doesn’t dramatically make Chest or Abdomen tasks better. We think this is evidence that our submission should be considered as a reasonably competitive effort, not something "far below" the norm.
Attempt 2: Modern VLMs for medical images
To gauge the difficulty of Chest and Abdomen tasks, we ran QoQ-Med-VL-7B (Dai et al. NeurIPS ‘25, https://arxiv.org/abs/2506.00711) zero-shot on each slice with a reasoning budget of 512 tokens; if it has not answered by then, we close its reasoning and force an answer. The slice score is the probability of "Yes" versus "No".
Localization results for QoQ on Chest and Abdomen tasks, alongside our NG and other MIL methods, are in the table below.
| Method | Chest CT | Abdomen CT |
|---|---|---|
| Centered Gaussian | 0.850 ± 0.001 | 0.642 ± 0.022 |
| ABMIL | 0.558 ± 0.086 | 0.526 ± 0.041 |
| Normal Guidance (ours) | 0.866 ± 0.005 | 0.663 ± 0.006 |
| TransMIL | 0.585 ± 0.077 | 0.579 ± 0.031 |
| Multi-Head NG (ours) | 0.869 ± 0.002 | 0.634 ± 0.007 |
| QoQ-Med-VL-7B (zero-shot) | 0.702 ± 0.005 | 0.547 ± 0.010 |
| Best-in-Class Ceiling | 0.871 ± 0.004 | 0.717 ± 0.011 |
Our NG methods beat zero-shot QoQ by at least 0.08 AUROC points on both Chest and Abdomen.
QoQ-Med struggles with localization on the Chest and Abdomen tasks, even though it processes higher resolution (512x512 slices) and uses a much larger model (7B parameters) than our approach does. We think this is decent evidence that the Chest and Abdomen tasks are difficult for specialized medical imaging VLMs, not just difficult for MIL with frozen VITs.
QoQ-Med’s training data includes many kinds of medical images from the CLIMB collection (https://arxiv.org/abs/2503.07667), One source of images is the RSNA Pulmonary Embolism dataset behind our Chest CT task. Thus, this score may be optimistic for QoQ-Med (it includes data QoQ-Med saw during training).
4. Need evaluation on multi-ROI scans
Thanks for the suggestion. We now include an evaluation of localization for multiple ROI and single ROI scans in App. C.2. See also our "New Table: Localization AUROC by ROI location and count" response to another reviewer.
Please keep in mind that multi-ROI scans are less common in all datasets. Use caution before trying to infer too much from the small test sets of some multi-ROI rows.
We agree that the current numbers do not show that MHNG offers a huge benefit over standard NG on multi-ROI scans, at least with the present encoder. We will revise the main paper’s text to be more candid about this.
We still think the primary contribution of this work is realizing how poor current MIL methods do relative to CG even when the data is mostly single-ROI scans, and developing NG to beat the image-free CG baseline. We hope this paper might inspire future work to better handle the multi-ROI case. MHNG should be reasonably capable, but likely needs a better encoder to drive the specific heads to localize well.
Complete Results with MedSAM encoder
For reference, here are complete results using the MedSAM encoder discussed above in "Response 2 of 2". We used the same evaluation protocol (average over 3 train/test partitions, same hyperparameter search) as described in our paper.
Localization AUROC with MedSAM
Comparable to Tab. 1 in main paper
| Head CT | Chest CT | Abdomen CT | |
|---|---|---|---|
| MedSAM + ABMIL | 0.779 +/- 0.021 | 0.699 +/- 0.037 | 0.587 +/- 0.011 |
| MedSAM + ABMIL + Normal Guidance | 0.884 +/- 0.001 | 0.756 +/- 0.030 | 0.602 +/- 0.023 |
Bag-level Classification AUROC with MedSAM
Comparable to Tab. 2 in main paper
| Head CT | Chest CT | Abdomen CT | |
|---|---|---|---|
| MedSAM + ABMIL | 0.938 +/- 0.009 | 0.649 +/- 0.010 | 0.676 +/- 0.025 |
| MedSAM + ABMIL + Normal Guidance | 0.973 +/- 0.006 | 0.652 +/- 0.011 | 0.675 +/- 0.017 |
Response 1 of 2: Where gains come from / request for more baselines
We thank the reviewer for their time and their constructive comments. We have attached a revised PDF that addresses several requests (see new evaluations on multiple ROIs in App C.2.). Look for magenta colored text indicating revised content.
Our specific replies to various concerns and questions are provided below.
1. Do Gains come from image content or position?
NG beats the CG baseline on 2 real CT tasks in Tab. 1, while tying on the 3rd task (confirmed with bootstrap significance tests, see App. B.4). NG’s superiority on two tasks over a baseline that represents only a positional prior suggests that NG’s ability to use image content is providing a meaningful signal.
Yes, this is by design and we agree this is a useful inductive bias early in training.
We will adjust text to remind readers about the usefulness of such initialization.
Yes, we agree. CG is designed to be a simple baseline that captures an essential property of the data at hand. We think it is an important finding that many existing well-published methods from past MIL work using attention and transformers cannot seem to beat this simple baseline.
In contrast, our NG, which combines image signal with regularization that prefers the center-focused, unimodal nature of the data, can outperform CG at localization (see analysis around Tab.1 ).
2. Baselines to consider
Our claimed contributions (Sec. 1) are specifically about relative performance of our methods to modern MIL methods:
To validate these claims, our results in Tab 1 and 2 already compare to many baselines from modern competitor MIL methods, including Additive MIL (NeurIPS ‘22), AEM (MICCAI ‘25), Smooth Operator (NeurIPS ‘24), AA regularization from AFFNet (2026 journal paper), and CIA MIL (recent MIDL ‘26 paper).
The additional comparisons requested seem interesting, but somewhat out of scope for the contributions our work claims since they don’t involve modern deep MIL approaches.
We’re not sure what the “thresholding” baseline suggested here is supposed to be; as far as we know, there are no signals provided in the ground-truth labels of our datasets that can be easily thresholded to obtain organ coverage estimates.
Given the limited time available in the response period, we weren’t able to get TotalSegmentor installed or working; we instead focused effort on other comparisons (evaluation of ROI by location, comparisons to the AffNet paper suggested by the AC).
Even if we had organ labels, we’re not sure this would help much in the Abdomen CT task we consider. That task is about detection of any injury to several possible major organs in the abdomen, not an organ-specific task.
We will mention TotalSegmentor as a possible route for future work.
Again, we ran out of time to try out this baseline, sorry.
Squinting at Fig 1 (which shows how instance labels relate to the j/S position across datasets), our educated guess is that using only the position as a feature would yield performance likely on par with the centered gaussian (CG) baseline, since both Fig 1 empirical histograms and CG have a center-focus and unimodal nature.