acceptodds
Under review as a conference paper at ICLR 2027

Normal Guidance is what Attention Needs for Multiple Instance Localization in CT Scans

Abstract

We consider training classifiers for 3D medical images using only one binary label for the entire volume rather than a label for each 2D slice. In such weakly supervised settings, can we learn accurate classifiers for slice-level predictions? Attention-based multiple instance learning (MIL) can produce an attention score for every slice. Yet recent work demonstrates that a simple center-focused baseline that ignores image content can outperform attention-based and transformer-based MIL at slice-level classification of 3D brain scans. We show this baseline also outperforms existing MIL at slice-level classification of thoracic and abdominal CT scans. Motivated by this baseline, we propose Normal Guidance, a regularization technique that encourages the learned attention distribution to follow a bell-shaped curve. Across three medical imaging datasets totaling over 4 million 2D slices, we show our Normal Guidance enables attention-based and transformer-based MIL methods to deliver better slice-level localization than the state-of-the-art while remaining competitive at whole-scan classification.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Deep mulitple instance learning, localization, inductive bias, medical imaging

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_YNKBOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies weakly supervised slice-level localization in 3D CT. It first shows that an image-agnostic Centered Gaussian (CG) baseline outperforms existing MIL methods at localization on three RSNA CT datasets (head, chest, abdomen). It then proposes Normal Guidance (NG), which fits a Gaussian reference to the mean and variance of the current attention (with stop-gradient) and penalizes the KL divergence between them; for TransMIL, it is applied per head (MHNG). All experiments use a frozen ImageNet ViT-B/16 encoder and a linear head. NG/MHNG significantly exceed CG on Head and Chest, are statistically tied on Abdomen, and keep scan-level classification competitive. The paper also introduces “best-in-class ceilings” trained with slice labels as reference points.

Critical strengths
  1. The evaluation is large-scale and rigorous: over 4M slices across three public datasets, patient-level splits, three partitions, and bootstrap significance tests. Model selection uses only scan-level labels, so the weak-supervision setting is strictly respected.
  2. Extending the CG finding to multiple body parts is itself a valuable negative result, signaling that the localization ability of current MIL attention is overstated.
  3. The method is simple, pooling-agnostic, and reproducible. The limitations are discussed candidly, including the unimodality assumption, evaluation on positive bags only, and the attention-is-not-explanation issue.
Critical weaknesses
  1. The gains may come mainly from a positional prior rather than from image content. NG’s reference is centered at the attention’s own mean, so it is not center-biased by construction. The apparent center bias likely comes from initialization: near-uniform initial attention has E[J] ≈ S/2, which yields a centered, broad reference.
  2. The baselines are not strong enough; in particular, simple content-aware baselines are missing. Per-bag AUROC is rank-based, so CG is effectively “rank slices by distance to the middle slice.” In head CT, the middle slices are often exactly the ones containing the most brain tissue. The following baselines should be added: scoring each slice by organ or tissue coverage (e.g., via thresholding or TotalSegmentator); a weakly supervised positional prior, i.e., MIL with only the position j/S as instance feature.
  3. The weak encoder limits the generality of the conclusions. With a frozen ImageNet ViT, scan-level AUROC on Chest and Abdomen is only about 0.66–0.68, far below what is typically achievable on these datasets. With such a weak content signal, it is expected that positional priors dominate. Please verify whether CG remains strong, and whether NG still helps, with CT-specific or medically pretrained encoders (e.g., a CT foundation model, RadImageNet, or fine-tuned 2D/3D models).
  4. There is no evidence for the multimodal capability of the multi-head variant. MHNG is motivated by multiple non-adjacent ROIs, but there is no quantitative evaluation on multi-ROI scans. The appendix visualizations (e.g., Figs. C.6 and C.10) show that the head-averaged attention remains unimodal. On Abdomen, MHNG (0.634) is below NG (0.663) and even below CG.
Questions for authors

What are the localization AUROCs of NG, MHNG, and CG when stratified by ROI position and count?

3 comments
AuthorsOpenReview ↗

Response 1 of 2: Where gains come from / request for more baselines

We thank the reviewer for their time and their constructive comments. We have attached a revised PDF that addresses several requests (see new evaluations on multiple ROIs in App C.2.). Look for magenta colored text indicating revised content.

Our specific replies to various concerns and questions are provided below.

1. Do Gains come from image content or position?

The gains may come mainly from a positional prior rather than from image content.

NG beats the CG baseline on 2 real CT tasks in Tab. 1, while tying on the 3rd task (confirmed with bootstrap significance tests, see App. B.4). NG’s superiority on two tasks over a baseline that represents only a positional prior suggests that NG’s ability to use image content is providing a meaningful signal.

The apparent center bias likely comes from initialization: near-uniform initial attention has E[J] ≈ S/2, which yields a centered, broad reference.

Yes, this is by design and we agree this is a useful inductive bias early in training.

We will adjust text to remind readers about the usefulness of such initialization.

CG is effectively “rank slices by distance to the middle slice.” In head CT, the middle slices are often exactly the ones containing the most brain tissue.

Yes, we agree. CG is designed to be a simple baseline that captures an essential property of the data at hand. We think it is an important finding that many existing well-published methods from past MIL work using attention and transformers cannot seem to beat this simple baseline.

In contrast, our NG, which combines image signal with regularization that prefers the center-focused, unimodal nature of the data, can outperform CG at localization (see analysis around Tab.1 ).

2. Baselines to consider

Our claimed contributions (Sec. 1) are specifically about relative performance of our methods to modern MIL methods:

    1. Centered Gaussian baseline outperforms modern MIL at localization
    1. Our NG enables attention-based and transformer-based MIL to beat the CG baseline for localization

To validate these claims, our results in Tab 1 and 2 already compare to many baselines from modern competitor MIL methods, including Additive MIL (NeurIPS ‘22), AEM (MICCAI ‘25), Smooth Operator (NeurIPS ‘24), AA regularization from AFFNet (2026 journal paper), and CIA MIL (recent MIDL ‘26 paper).

The additional comparisons requested seem interesting, but somewhat out of scope for the contributions our work claims since they don’t involve modern deep MIL approaches.

scoring each slice by organ or tissue coverage (e.g., via thresholding or TotalSegmentator)

We’re not sure what the “thresholding” baseline suggested here is supposed to be; as far as we know, there are no signals provided in the ground-truth labels of our datasets that can be easily thresholded to obtain organ coverage estimates.

Given the limited time available in the response period, we weren’t able to get TotalSegmentor installed or working; we instead focused effort on other comparisons (evaluation of ROI by location, comparisons to the AffNet paper suggested by the AC).

Even if we had organ labels, we’re not sure this would help much in the Abdomen CT task we consider. That task is about detection of any injury to several possible major organs in the abdomen, not an organ-specific task.

We will mention TotalSegmentor as a possible route for future work.

weakly supervised positional prior, i.e., MIL with only the position j/S as instance feature

Again, we ran out of time to try out this baseline, sorry.

Squinting at Fig 1 (which shows how instance labels relate to the j/S position across datasets), our educated guess is that using only the position as a feature would yield performance likely on par with the centered gaussian (CG) baseline, since both Fig 1 empirical histograms and CG have a center-focus and unimodal nature.

AuthorsOpenReview ↗

Response 2 of 2: Encoder evaluations and multi-ROI discussion

3. Encoder

With a frozen ImageNet ViT, scan-level AUROC on Chest and Abdomen is only about 0.66–0.68, far below what is typically achievable

We don’t doubt that some better encoder might raise performance here. But this has been harder to achieve than we expected.

We’re not sure what exactly is “typically achievable” in the reviewer’s mind, since no specific work is cited. Several tasks are supported on these datasets. Before assessing whether any other performance is comparable, we’d like to verify that the class definitions, 2D slice resolution (224x224 pixels), and train/test splits roughly match our settings.

To try to investigate improved encoders and gauge the overall difficulty, we tried two different options, each surveyed below. We plan to add both to the manuscript soon.

Attempt 1: MedSAM (updated Wed 2026-11-18)

We ran experiments using frozen MedSAM (Ma et al. 2024) as the image encoder in our MIL pipeline. This is an encoder trained on medical images to perform segmentation tasks, using the popular “Segment Anything” approach. In MedSAM’s training set, there were over half million image-mask pairs for the CT modality. CT is the 2nd most common modality in its training set (see their Fig 2a).

Below are results for all datasets

Head CT bag-level AUROC

  • ABMIL with ViT (Tab 2): 0.919 +/- 0.009
  • ABMIL with MedSAM: 0.938+/- 0.009
  • NG with MedSAM: 0.973 +/- 0.006

Abdomen CT bag-level AUROC

  • ABMIL with ViT (Tab 2): 0.639 +/- 0.044
  • ABMIL with MedSAM: 0.676 +/- 0.025
  • NG with MedSAM: 0.676 +/- 0.019

Chest CT bag-level AUROC

  • ABMIL with ViT: 0.664 ± 0.017
  • ABMIL with MedSAM: 0.649 ± 0.010
  • NG with MedSAM: 0.652 ± 0.011

Interpretation: Swapping ViT to MedSAM helps bag-level classification improve modestly on 2 datasets but there is a slight drop on Chest. For ABMIL, the AUROC increases by 0.019 on Head CT and 0.037 on Abdomen CT. The scope of change is modest at best, so we suggest that using a medical foundation model doesn’t dramatically make Chest or Abdomen tasks better. We think this is evidence that our submission should be considered as a reasonably competitive effort, not something "far below" the norm.

Attempt 2: Modern VLMs for medical images

To gauge the difficulty of Chest and Abdomen tasks, we ran QoQ-Med-VL-7B (Dai et al. NeurIPS ‘25, https://arxiv.org/abs/2506.00711) zero-shot on each slice with a reasoning budget of 512 tokens; if it has not answered by then, we close its reasoning and force an answer. The slice score is the probability of "Yes" versus "No".

Localization results for QoQ on Chest and Abdomen tasks, alongside our NG and other MIL methods, are in the table below.

MethodChest CTAbdomen CT
Centered Gaussian0.850 ± 0.0010.642 ± 0.022
ABMIL0.558 ± 0.0860.526 ± 0.041
Normal Guidance (ours)0.866 ± 0.0050.663 ± 0.006
TransMIL0.585 ± 0.0770.579 ± 0.031
Multi-Head NG (ours)0.869 ± 0.0020.634 ± 0.007
QoQ-Med-VL-7B (zero-shot)0.702 ± 0.0050.547 ± 0.010
Best-in-Class Ceiling0.871 ± 0.0040.717 ± 0.011

Our NG methods beat zero-shot QoQ by at least 0.08 AUROC points on both Chest and Abdomen.

QoQ-Med struggles with localization on the Chest and Abdomen tasks, even though it processes higher resolution (512x512 slices) and uses a much larger model (7B parameters) than our approach does. We think this is decent evidence that the Chest and Abdomen tasks are difficult for specialized medical imaging VLMs, not just difficult for MIL with frozen VITs.

QoQ-Med’s training data includes many kinds of medical images from the CLIMB collection (https://arxiv.org/abs/2503.07667), One source of images is the RSNA Pulmonary Embolism dataset behind our Chest CT task. Thus, this score may be optimistic for QoQ-Med (it includes data QoQ-Med saw during training).

4. Need evaluation on multi-ROI scans

Thanks for the suggestion. We now include an evaluation of localization for multiple ROI and single ROI scans in App. C.2. See also our "New Table: Localization AUROC by ROI location and count" response to another reviewer.

Please keep in mind that multi-ROI scans are less common in all datasets. Use caution before trying to infer too much from the small test sets of some multi-ROI rows.

We agree that the current numbers do not show that MHNG offers a huge benefit over standard NG on multi-ROI scans, at least with the present encoder. We will revise the main paper’s text to be more candid about this.

We still think the primary contribution of this work is realizing how poor current MIL methods do relative to CG even when the data is mostly single-ROI scans, and developing NG to beat the image-free CG baseline. We hope this paper might inspire future work to better handle the multi-ROI case. MHNG should be reasonably capable, but likely needs a better encoder to drive the specific heads to localize well.

AuthorsOpenReview ↗

Complete Results with MedSAM encoder

For reference, here are complete results using the MedSAM encoder discussed above in "Response 2 of 2". We used the same evaluation protocol (average over 3 train/test partitions, same hyperparameter search) as described in our paper.

Localization AUROC with MedSAM

Comparable to Tab. 1 in main paper

Head CTChest CTAbdomen CT
MedSAM + ABMIL0.779 +/- 0.0210.699 +/- 0.0370.587 +/- 0.011
MedSAM + ABMIL + Normal Guidance0.884 +/- 0.0010.756 +/- 0.0300.602 +/- 0.023

Bag-level Classification AUROC with MedSAM

Comparable to Tab. 2 in main paper

Head CTChest CTAbdomen CT
MedSAM + ABMIL0.938 +/- 0.0090.649 +/- 0.0100.676 +/- 0.025
MedSAM + ABMIL + Normal Guidance0.973 +/- 0.0060.652 +/- 0.0110.675 +/- 0.017
Reviewer_Sq7yOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Normal Guidance (NG), an attention-based regularization approach for improving localization in weakly supervised multiple instance learning (MIL). NG encourages learned attention distributions to follow a Gaussian shape and is further extended to Multi-Head Normal Guidance (MHNG). Experiments on three CT datasets (Head, Chest, Abdomen) demonstrates NG’s effectiveness while maintaining competitive classification performance.

Critical strengths
  1. This paper includes extensive experiments on three public datasets with expert annotation.
  2. The findings in this paper may bring important insights to the community.
Critical weaknesses
  1. The novelty is limited. Attention-based regularization has already been proposed in previous works, such as counterfactual intervention for attention (CIA) and cluster-to-conquer. The Related Work section should more clearly distinguish the proposed method from previous studies. In addition, Section 2 emphasizes that “Normal Guidance encourages a univariate learned attention distribution to follow a bell-shaped curve.” The authors should provide explanations about why a bell-shaped curve is needed for better multiple instance localization.

  2. Figure 1 is hard to understand. The last sentence in the caption says “Normal Guidance improves …”; however, there’s no comparison between Normal Guidance and others in Figure 1. Furthermore, Figure 1 is the result of ONE CT scan. I am concerned about the general performance. Also, the annotation for the blue dash line is missing.

  3. The comparisons in Table D.1 is not sufficient. The results show experiments on Head CT. Results on the other datasets should also be provided to show better transparency.

  4. The qualitative results in Figures C.5, C.9, and C.13 suggest that Normal Guidance struggled to localize the ROIs near the scan boundaries. Although Appendix C provides examples under different ROI configurations, the paper lacks a quantitative evaluation stratified by ROI locations.

Questions for authors
  • Since rir_i and aia_i are computed by the model itself, will model training at an early phase be affected by initial unstable rir_i and aia_i?
2 comments
AuthorsOpenReview ↗

Response 1 of 2: Related work, Why bell curve? and Comments on Fig 1 and App D

We thank the reviewer for their constructive comments. We have attached a revised PDF that addresses several requests (see new related work table now in Tab H.1).

Our specific replies to various concerns and questions are provided below.

Related work

Related Work section should more clearly distinguish the proposed method from previous studies

We have revised language in Sec. 2 to highlight differences from other methods. We also now include a new table (Tab. H1) comparing related work in App H, to help situate our proposed NG regularization method for attention against alternatives. (This is referenced in Sec 2, so readers know where to find it).

While other approaches have tried to regularize attention to meet other goals (e.g. be closer to a uniform distribution), no other approach we’ve seen directly steers attention to follow a unimodal bell-shaped curve. We select a bell-curve to match the empirical statistics of the instance-level labels of the CT scan problems we consider.

Why bell-curve?

The authors should provide explanations about why a bell-shaped curve is needed for better multiple instance localization.

In Sec. 1, we state the following as high-level reasons why we favor a bell-shaped curve: “This reference distribution provides 3 helpful inductive biases during training: smoothness, center-focus, and unimodality”

To expand on these reasons, the following text was in Sec. 3 of our submission: “We hypothesize this construction has 3 useful inductive biases: smoothness, center-focus, and unimodality. In our later CT tasks, positive slices are more likely to occur close to the middle position (see Fig. 3) and most scans have just one contiguous region of interest or ROI (64% of Chest CT, over 84% for Head and Abdomen; see Tab. A.2). Thus, the way NG favors smooth unimodal attention near the center is desirable for our 3D CT tasks.” NG’s inductive biases are then discussed further in App. G.2. We also provide evidence that NG is not outperformed by more flexible guidance that allows multimodal attention via a mixture of Gaussian processes (App. G.3).

In our revision, we added the paragraph header Why Bell Curve? in Sec. 3 to help readers find this justification, in case it got lost below the details of how NG is implemented.

Comments on Figure 1

“there’s no comparison between Normal Guidance and others in Figure 1.”

Both the legend and the caption mention ABMIL, a competitor method lacking attention regularization. We compare our Normal Guidance regularization method directly to ABMIL in this figure.

We revised caption wordings to make this more clear.

Figure 1 is the result of ONE CT scan. I am concerned about the general performance.

We provide quantitative results comparing localization performance in Tab. 1. These results represent aggregate performance over hundreds of CT scans in a held out test set (e.g., over 3600 for Head CT, over 700 for Abdomen CT).

We also provide qualitative visuals of the attention distribution predicted by various methods on several more CT scans in App. C.

Also, the annotation for the blue dash line is missing.

The blue dashed line represents the reference “bell curve” we use as a goal for the learned attention. This reference r_i is constructed by applying Eq 3 to the shaded blue attention distribution a_i visualized in this figure, which comes from a learned ABMIL model.

We have updated the caption in the revised figure to make the intent clear.

Comparisons of different ways to construct the NG reference curve (App D)

The comparisons in Table D.1 is not sufficient. The results show experiments on Head CT. Results on the other datasets should also be provided

Tab. D.1 broadly suggests that there are several ways to construct the Normal Guidance reference bell curve that yield similar performance in terms of bag-level classification and localization. We expect that trying these variants on other datasets like Chest CT would similarly lead to all methods performing within a few points of one another on both metrics.

During this busy response period, we had many requests for experiments. We rated other requested work (such as evaluation of multi-head NG and comparisons to the regularization in AFFNet) as more important, so we chose to prioritize them in this limited-time window.

Even without additional experiments testing variants of NG, we think the main claims in the manuscript are sufficiently supported.

AuthorsOpenReview ↗

Response 2 of 2: Localization results by ROI location and count / Stability of Parameter Estimates

New Table: Localization AUROC by ROI location and count

We show the mean AUROC ± std over 3 test partitions, with the best mean value per row in bold. The n column indicates the number of positive test scans per partition. Same results are also available in new App C.2 in the revised PDF.

DatasetROI binnCGABMILABMIL + NGTransMILTransMIL + MHNG
HeadSingle ROI, center1156/1174/11560.918 ± 0.0020.763 ± 0.0510.918 ± 0.0010.826 ± 0.0330.922 ± 0.003
Single ROI, near edge228/226/2300.565 ± 0.0040.631 ± 0.0600.676 ± 0.0040.653 ± 0.0220.653 ± 0.006
Multi ROI, center8/9/80.835 ± 0.0120.667 ± 0.0810.847 ± 0.0240.765 ± 0.0520.831 ± 0.004
Multi ROI, near edge89/80/830.715 ± 0.0070.658 ± 0.0450.741 ± 0.0060.693 ± 0.0180.731 ± 0.004
ChestSingle ROI, center192/202/2050.913 ± 0.0030.564 ± 0.0870.907 ± 0.0070.586 ± 0.0680.915 ± 0.004
Single ROI, near edge37/25/400.565 ± 0.0170.531 ± 0.0940.726 ± 0.0220.592 ± 0.1080.704 ± 0.015
Multi ROI, center61/54/500.891 ± 0.0080.567 ± 0.0930.881 ± 0.0070.579 ± 0.0710.886 ± 0.005
Multi ROI, near edge79/88/740.788 ± 0.0190.545 ± 0.0770.814 ± 0.0070.583 ± 0.0920.815 ± 0.010
AbdomenSingle ROI, center18/28/220.864 ± 0.0420.530 ± 0.0940.811 ± 0.0300.618 ± 0.0600.824 ± 0.030
Single ROI, near edge26/17/290.421 ± 0.0180.520 ± 0.0370.506 ± 0.0260.546 ± 0.0230.436 ± 0.015
Multi ROI, center5/1/10.851 ± 0.1060.518 ± 0.1610.859 ± 0.0940.705 ± 0.0450.865 ± 0.061
Multi ROI, near edge7/11/100.613 ± 0.0610.562 ± 0.0670.647 ± 0.0600.560 ± 0.0690.624 ± 0.064

Paper needs quantitative evaluation stratified by ROI locations

We have added the requested results in new appendix section, “C.2 Quantitative Localization Evaluation”. These are repeated in the table above for convenience.

The table examines ROI localization across several specific data subsets:

  • single ROI, centered
  • single ROI, near edge (aka not centered)
  • 2 or more ROI, centered
  • 2 or more ROI, near edge

Here, “centered” means all ROI centers occur in the middle third of the scan. Not centered means that any ROI occurs in the first or last third of the scan, near the edge.

As expected, CG localizes less well when the ROI is near the scan boundary. On single-ROI near-edge scans, NG improves on CG by 0.08 to 0.16 AUROC on all three datasets. On multi-ROI near-edge scans, this improvement is about 0.03. These results suggest that NG's gains are not explained by a center prior alone. Edge ROIs remain the hardest case for CG, NG and MHNG. MHNG shows no clear advantage over NG on multi-ROI scans.

Please keep in mind that multi-ROI scans are relatively rare in all datasets. Single ROI scans are far more common (see App. A). Also, use caution before trying to infer too much from the small test sets of some multi-ROI rows.

Stability of parameter estimates throughout training

Since E[J] and Var[J] are computed by the model itself, will model training at an early phase be affected by initial unstable values?

We do agree that using early attention values to inform the parameters has a small risk that the initial values may be noisy and unstable across early iterations, before the attention module has been sufficiently informed by the data. We have not yet seen this substantially hurt eventual performance across the 3 real CT datasets we assess.

We have revised our Limitations section to acknowledge this potential concern. One way to avoid potential instability would be to avoid using the regularization term in the loss until attention values stabilize after a few passes through the data set.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.