Zero-Shot Medical Anomaly Segmentation under Domain Shift: A Prototype-Level Analysis

Zero-shot anomaly segmentation localizes abnormalities in unseen domains without target-domain supervision. Yet existing evaluations reveal little about how source-domain evidence manifests in unseen domains or why transfer fails. We introduce ProtoZSAD, which decomposes normal and abnormal representations into learned prototype queries that provide intermediate units for analyzing source-to-target transfer while supporting abnormal-region localization. Trained on MVTec and evaluated without target-domain adaptation on four medical datasets, ProtoZSAD achieves competitive abnormal-region localization across domains. Prototype-level interventions show that individual prototypes have distinct, domain-dependent effects on final predictions, while greater prototype response to normal anatomy relative to lesions is associated with lower Dice and increased false positives. A targeted Brain-domain intervention further reduces the implicated prototype responses and false positives while improving segmentation. Together, these results show how prototype-level analysis can identify when source-associated prototype responses remain aligned with true abnormalities and when they instead become aligned with normal target anatomy.