- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes a training-free pipeline for 2D (slice-wise) Brain MRI tumor localization from pretrained vision models, named SymmetryBreak. The method consists of the following steps: the contralateral pairs of two hemisphere is partitioned by mid-point slice (at location W/2, given W the width of the feature map(s)). Then, latent feature statistics statistics are calculated through low-discrepancy locations. Then standard deviations from this latent feature statistics statistics provides a hint of the abnormal hemisphere. Finally, a thresholding is applied to obtain the localization mask for brain tumors. The paper evaluates the approach on three public datasets.
Critical strengths
The proposed method is easy to follow and understand. The method itself is easy to apply with good details of implementation. The motivation of the design is clear and in most cases the proposed method works well. In some cases it outperforms existing learning-based methods on Dice score and AUPRC.
The idea of utilizing frozen vision models (e.g., MedSAM) is interesting for such localization tasks. I also appreciate the objective discussion raised by the authors.
Critical weaknesses
- The proposed method includes quite a few assumptions that is appearing too empirical. And the inference process is mainly based these empirical assumptions with a large-pretrained vision segmentation model. For example, from sec 3.2, the pipeline starts from a partition that devides the image (feature) along the width-dimension evenly. The bilateral latent symmetry breaking, despite its name, it essentially a pixel-wise cosine distance map from left and right hemisphere. However, I would like to point out that this relies on both well affine registration to an atlas or good bilateral references, of which has been done at the stage of dataset preparation in BraTs.
The following abnormality score, is measured by standard deviation to the image latent feature statistics statistics such as mean. However, I found the methodology seems to be heavily relying on the hyperparameter choice of these rules, e.g., the region used for calculating the ‘healthy reference’ and segmentation threshold. This leaves the work rather empirical and insufficiently established.
- I also found the proposed methodology needs more careful validation beyond the given metrics and the failure cases in appendix do not seem to be easily resolved given the nature of the method of hard-choice of hemispheres and thresholding-based segmentation from feature maps. Please see questions for a detailed comment.
Questions for authors
-
In the main tables, the authors uses DSC and AUPRC for evaluating the accuracy. I would like to ask whether DSC is a proper and fair metric? Since the proposed method can be tuned with multiple hyperparameters such as the threshold of segmentation, would this be also beneficial if the same search is applied to other baseline methods?
-
The use of vision foundation models, especially, segmentation foundation models is reasonable. However, does the data leakage happen: could the author investigate this like whether MedSAM has been trained on this kind of task at least brain segmentation?
-
However, from Fig.3, it seems that the SB, due to the hard-threshold on a certain hemisphere, fails to segment the correct tumor. I would like to ask whether the assumption of locating the tumor in one hemisphere is too strict as some abnormality can happen across the midpoint of brain? Overall, does the proposed method has the ability to deal with non-linear or even non-midpoint cases?
-
From the failure cases, it seems that the proposed method struggles a lot on the small tumor part: I would like to ask whether this could be fixed without retraining anything?
5 comments
Response to Reviewer Ub3c (2/3)
3. Is DSC fair, and would the same search help the baselines? (Q1)
We consistently used a fixed segmentation threshold of (, without performing method-specific threshold optimization. To investigate whether threshold tuning could substantially affect the reported results, we conducted an additional experiment on BraTS-2021 T1 dataset, searching for the optimal threshold for each method over ( {0.1, 0.2, , 0.9}), using the same tuning split and selection criterion.
As shown in the table, threshold optimization results in only marginal improvements in DSC across all methods (at most 0.5 percentage points). In particular, SymmetryBreak achieves a DSC of 69.6% at the default threshold of 0.5, compared with 69.8% using the optimized threshold. These results indicate that the fixed threshold of 0.5 already provides near-optimal segmentation performance for our method.
| Method | DSC, default (0.5) | DSC, tuned | AUPRC |
|---|---|---|---|
| f-AnoGAN | 28.8 | 28.9 | 34.1 |
| DenoisingAE | 39.4 | 39.9 | 47.5 |
| AnoDDPM | 61.8 | 62.1 | 73.4 |
| AutoDDPM | 67.4 | 67.4 | 79.3 |
| SymmetryBreak | 69.6 | 69.8 | 81.3 |
We therefore retain the fixed threshold of 0.5 in the main experiments to ensure a consistent evaluation protocol.
4. Foundation-model leakage (Q2)
MedSAM's training collection includes BraTS, so we cannot rule out overlap between its pretraining data and some of our evaluation cases. Rather than asserting that overlap is unlikely, we ran a controlled experiment to test whether our results depend on it.
Leakage control. We replaced the MedSAM encoder with backbones that differ in pretraining exposure, keeping the pipeline, hyperparameters, and evaluation protocol fixed. Experiments use BraTS-Africa T2:
| Backbone | Medical pretraining | BraTS masks seen | DSC | AUPRC |
|---|---|---|---|---|
| MedSAM ViT-B | Yes (segmentation) | Yes | 57.6 | 66.6 |
| BiomedCLIP | Yes (image-text only) | No | 55.3 | 65.1 |
| SAM ViT-B | No | No | 54.5 | 63.7 |
| DINOv2 ViT-B | No | No | 46.9 | 54.4 |
Findings.
-
Memorization does not explain the results. SAM ViT-B has never seen medical images or segmentation masks, yet it reaches 54.5 DSC and 63.7 AUPRC, within 3.1 DSC and 2.9 AUPRC of MedSAM. The method's core behavior therefore does not require exposure to the test distribution.
-
Pretraining exposure plausibly helps, modestly. Backbones with medical pretraining score higher, and we do not dismiss this. The MedSAM-SAM difference of about 3 DSC is an upper bound on what leakage or domain pretraining could contribute in our setting, and it does not alter our conclusions relative to the baselines. BiomedCLIP, which saw no segmentation masks, falls between the two, suggesting that domain-relevant features help generally rather than through memorized labels.
-
Backbone quality matters. DINOv2 degrades substantially (46.9 DSC), indicating that the method benefits from features that preserve fine spatial structure and that not every general-purpose encoder is suitable. We do not claim backbone-agnosticism; we claim the approach is not contingent on medical-domain exposure.
Task novelty. MedSAM was trained for supervised, prompt-based segmentation. Our setting is unsupervised anomaly detection via left-right asymmetry, with no labels or prompts at inference, so the model was never optimized for this objective on any data it saw.
We will (i) document the pretraining exposure of each backbone, and (ii) discuss residual leakage risk as a limitation in our camera ready manuscript.
Response to Reviewer Ub3c (3/3)
5. Small tumors and the hard hemisphere choice (Q3, Q4)
We acknowledge that hard hemisphere gating may impose a restrictive assumption when abnormalities extend across the brain midline or when the hemisphere-specific evidence is ambiguous. In the original implementation, the hard gating mechanism selects one hemisphere based on the estimated evidence, which may suppress relevant information from the opposite hemisphere.
To investigate this limitation, we considered a soft-gating strategy that replaces the hard argmax operation with temperature-controlled softmax weighting:
where controls the smoothness of the hemisphere selection. Unlike hard gating, this formulation allows contributions from both hemispheres, particularly when their corresponding evidence scores are similar. We use
This modification does not require retraining the network and provides a more flexible mechanism for handling ambiguous hemisphere assignments.
To investigate whether the small-tumor segmentation performance could be improved without retraining, we evaluated soft hemisphere gating as a post-processing modification.
Specifically, we compared the original hard-gating strategy against soft gating on the BraTS-2021 FLAIR dataset. For this evaluation, tumors were categorized according to their relative volume: small (less than 10%), medium (10–30%), and large (greater than 30%) of the total brain image size.
The quantitative results in terms of Dice Similarity Coefficient (DSC, %) are presented below:
| Variant | Small | Medium | Large |
|---|---|---|---|
| Hard gating | 39.4 | 78.5 | 82.1 |
| Soft gating | 52.2 | 75.3 | 78.9 |
The results demonstrate that soft gating substantially improves small-tumor segmentation, increasing DSC from 39.4% to 52.2% (an absolute improvement of 12.8 percentage points).
Response to Reviewer Ub3c (1/3)
We thank the reviewer for the careful review and constructive feedback. We address the comments below.
1. Reliance on alignment and registration (W1)
We agree that BraTS is co-registered, which simplifies midline alignment (as noted in Appendix E). SymmetryBreak relies on approximate within-subject left-right correspondence and does not require an external atlas. However, its sensitivity to misalignment and the potential benefit of alignment preprocessing had not been evaluated.
To address this, we designed a controlled experiment on BraTS-Africa T1 in which we first introduced random rigid misalignment to each slice through rotations of up to {2°, 5°, 10°} and horizontal translations of up to {2%, 5%, 10% } of image width. We then applied a rigid registration procedure to estimate and correct these transformations, restoring approximate midline alignment before evaluating SymmetryBreak. This experiment allows us to distinguish the performance degradation caused by misalignment from the effectiveness of alignment preprocessing.
We evaluate performance under three conditions: (1) original images, (2) randomly misaligned images, and (3) misaligned images with rigid registration. This comparison quantifies both the impact of misalignment and the extent to which alignment preprocessing restores SymmetryBreak's performance.
This evaluation is relevant because rigid alignment to an estimated mid-sagittal plane is a conventional preprocessing operation in neuroimaging pipelines. Our objective is not to claim that SymmetryBreak is inherently invariant to arbitrary geometric transformations, but rather to establish whether its reliance on approximate left-right correspondence can be addressed through a practical preprocessing step.
SymmetryBreak assumes approximate left-right anatomical correspondence. In settings with uncontrolled head positioning or substantial acquisition-related misalignment, an alignment preprocessing step may be necessary.
2. Dependence on hyperparameters and empirical rules (W2)
The sensitivity plots in Figs. 4–7 exhibit broad performance plateaus, suggesting that the proposed method does not rely on narrowly tuned hyperparameter values. To further investigate its sensitivity to dataset-specific tuning, we conducted cross-dataset hyperparameter selection experiments. Specifically, hyperparameters were selected using one dataset and evaluated on a different, out-of-sample dataset without further adjustment. This protocol assesses whether configurations selected on one dataset generalize to another.
The results are reported in the table below. The performance differences remain small, with maximum decreases of 0.7 percentage points in DSC and 0.9 percentage points in AUPRC on BraTS-Africa T2, and 0.3 percentage points in both metrics on BraTS-2019 T1ce.
For reference, we also report oracle configurations, where hyperparameters are selected and evaluated on the same target dataset. These configurations provide an indication of the potential benefit of dataset-specific optimization. Their improvements over the original paper configuration are modest (at most 0.7 percentage points in DSC and 0.4 percentage points in AUPRC).
Overall, these results suggest that the proposed method is relatively insensitive to the choice of hyperparameters and that configurations selected on one dataset transfer effectively to other datasets, reducing the need for dataset-specific empirical tuning.