SymmetryBreak: Resolving Bilateral Ambiguity in Training-Free Brain MRI Anomaly Localization

Localizing brain abnormalities from an unlabeled MRI without task-specific train- ing, lesion annotations, or external healthy-brain references remains challenging. Although the brain’s bilateral organization provides an intrinsic within-subject ref- erence, conventional contralateral comparison can identify where the hemispheres differ without determining which side is abnormal, often producing mirrored re- sponses. We introduce SymmetryBreak, a training-free framework that converts bilateral latent asymmetry into directional abnormality localization. Given a sin- gle MRI slice, a frozen vision encoder produces spatially organized latent rep- resentations, which are divided into corresponding left and right hemispheres through deterministic horizontal mirroring. SymmetryBreak first computes a bilat- eral cosine discrepancy map to identify regions of symmetry breaking. It then con- structs an image-internal reference from low-discrepancy correspondences and measures side-specific latent deviations from this reference. A shared-scale direc- tional comparison determines which hemisphere exhibits stronger abnormality, and a global hemisphere decision suppresses mirrored responses before construct- ing a localization map. SymmetryBreak requires no task-specific training or exter- nal healthy-brain atlases. Across BraTS 2019, BraTS 2021, and BraTS-Africa, SymmetryBreak achieves an average Dice Similarity Coefficient (DSC) of 67.4% and an AUPRC of 78.4%, outperforming the evaluated baselines. These results demonstrate that intrinsic anatomical symmetry, combined with image-internal directional inference, can provide a simple training-free paradigm for localized brain abnormality detection without an external normal reference.