acceptodds
Under review as a conference paper at ICLR 2027

Zero-Shot Medical Anomaly Segmentation under Domain Shift: A Prototype-Level Analysis

18 views

Abstract

Zero-shot anomaly segmentation localizes abnormalities in unseen domains without target-domain supervision. Yet existing evaluations reveal little about how source-domain evidence manifests in unseen domains or why transfer fails. We introduce ProtoZSAD, which decomposes normal and abnormal representations into learned prototype queries that provide intermediate units for analyzing source-to-target transfer while supporting abnormal-region localization. Trained on MVTec and evaluated without target-domain adaptation on four medical datasets, ProtoZSAD achieves competitive abnormal-region localization across domains. Prototype-level interventions show that individual prototypes have distinct, domain-dependent effects on final predictions, while greater prototype response to normal anatomy relative to lesions is associated with lower Dice and increased false positives. A targeted Brain-domain intervention further reduces the implicated prototype responses and false positives while improving segmentation. Together, these results show how prototype-level analysis can identify when source-associated prototype responses remain aligned with true abnormalities and when they instead become aligned with normal target anatomy.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
zero-shot, medical, anomaly segmentation, interpretation

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_XPZAOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper presents ProtoZSAD, which combines CLIP visual features, normal and abnormal prototypes, balanced assignment, and gating mechanisms. The model is trained on industrial data and transferred to four medical datasets. Through prototype responses, branch removal, and interventions on brain images, the paper analyzes the model's anomalous responses to normal anatomical structures.

Critical strengths

The paper addresses a meaningful problem: false positives caused by normal anatomical structures in cross-domain anomaly segmentation. Prototype responses provide observable intermediate representations.

  1. Evaluation on four medical datasets, together with prototype removal and interventions on brain images with a control condition, provides evidence beyond heatmap visualization.
Critical weaknesses
  1. The added explanatory value of the prototype analysis is unclear. The analysis mainly describes associations between prototype responses and segmentation errors. It does not sufficiently establish what deeper, testable insights into cross-domain failure mechanisms are gained beyond conventional feature-response analysis.

  2. The rationale for key design choices and their individual contributions are insufficiently validated. In localized lesion segmentation, anomalous regions typically occupy only a small portion of the image. However, the paper uses balanced assignment that is independent of the lesion proportion, without sufficiently justifying its rationale or suitability. Moreover, optimal transport has already been used for balanced prototype assignment in PIRN (ICLR 2026) and prototype refinement in FastRef (CVPR 2026). The additional methodological contribution of the proposed combination remains unclear.

  3. Comparisons with related segmentation methods and simple baselines are insufficient. The paper's prototype matching is similar to a regularized segmentation head, yet there is no comparison with AnomalyVFM (CVPR 2026), which uses a segmentation decoder. A segmentation-head baseline under the same backbone and training conditions is also missing. The evidence is therefore insufficient to establish the additional benefits of the prototype design.

  4. Extensive equations are not accompanied by sufficient visual presentation and explanation. The method diagram provides limited detail, and the paper lacks illustrations that adequately explain the motivation, module mechanisms, and intervention effects. The main text devotes substantial space to equations and implementation details without corresponding intuitive analysis, making the overall method and analytical conclusions less clear.

Questions for authors

See above.

1 comment
AuthorsOpenReview ↗

Response to Reviewer XPZA

Response 1. Answer to: Unclear added explanatory value of prototype analysis. Our analysis not only describe association between prototype response and segmentation errors but also indicates which characteristics of an abnormality are transferred from the industrial domain to the medical domain. For example, in Section 4.2.1, Figure 2, and Table 2, we use the same prototype as a reference unit for comparing visual patterns between the source and the target. Especially in the absence of characterization via prototypical analysis, it is difficult to draw profound conclusions regarding transfer errors associated with anatomical structures that exhibit high contrast.

Response 2. Answer to: Insufficient justification and validation of key design choices and balanced assignment.

  • Individual component contribution. The purpose and contribution of each component have been addressed in our Response 4. of reviewer ELKd.

  • Balaned Assignment. The role of balanced assignment is to distribute patch evidence across prototypes and prevent a small subset of prototypes from dominating the representation. Importantly, these balanced assignment scores are not used directly as the final abnormal prediction. DCAG subsequently filters abnormal prototype responses using two conditions: the response must be sufficiently strong (HA gate) and must exceed competing normal-prototype evidence (NR gate). Therefore, balanced prototype utilization does not imply balanced normal/abnormal pixel prediction. Consistently, the resulting abnormal false-positive-outside rates are 3.74%3.74\%, 6.59%6.59\%, 10.81%10.81\%, and 8.46%8.46\% on Retina, Breast, Brain, and Lung, respectively. We have clarified this distinction between prototype assignment and final anomaly localization in the revised manuscript.

  • We do not claim optimal transport or balanced assignment as our main contributions, as what we claim in lines 45–51. These are well-known building blocks employed in numerous papers, extending beyond just PIRN and FastRef. While both PIRN and FastRef focus on enhancing performance for few-shot learning, we instead approach the concept of prototypes through the lens of cross-domain knowledge transfer for zero-shot learning.

Response 3. Answer to: Insufficient comparisons with related segmentation methods and simple baselines. We do not compare with simple baselines. Our cited paper like MRAD-CLIP and VisualAD are two SOTA methods in ICLR-2026 and CVPR-2026. And our other cited prior works, which are also SOTA methods in 2024 and 2025, also use same backbone and are described in detail in Appendix Section: Experiment Setup.

  • The lack of segmentation-head baseline can not seen as a critical weakness. The segmentation decoder module in AnomalyVFM is responsible for upscaling feature maps to the original image dimensions; it is essentially the same as the bilinear upsampling mechanism used in our paper, as well as in baselines such as AdaCLIP and VisualAD. The regularized component of AnomalyVFM essentially adds a regularization loss to the segmentation loss, aiming to address the issue of noisy labels when learning from synthetic data. We design prototype to capture multiple characteristic of abnormal patterns for cross domain understanding, instead of working as regularized condition.

Response 4. Answer to: Insufficient visual presentation and intuitive explanation of the method. We address a similar concern in our Response 2 to Reviewer ELKd. Please refer to that response for further details.

Reviewer_RSwSOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

ProtoZSAD uses learned normal/abnormal prototypes to study industrial-to-medical zero-shot transfer. The diagnostic perspective is worthwhile, but the current evidence supports an exploratory model-specific case study more strongly than a reproducible explanation of domain-shift failure.

Critical strengths

The paper distinguishes prototype activation, prediction relevance, and beneficial contribution. It reports five-seed segmentation results, prototype-removal experiments, and a BrainSmooth intervention control. The supplement acknowledges normal-image failures.

Critical weaknesses
  1. The main evaluation gives an incomplete picture of performance. Table 1 reports abnormal-only Dice-1. Supplement Table 6 shows Dice-0=0 across all four target domains and 6.93–14.38% false-positive area on normal images. Under the empty-mask convention, Dice-0=0 indicates non-empty predictions on normal images, not that every pixel is incorrect. Brain overall Dice is 18.51%, versus 54.86% for AdaCLIP. Include normal-image and overall results prominently, and compare source-calibrated operating points alongside threshold-free metrics. A common numerical threshold of 0.5 does not ensure comparable calibration.

  2. The analysis does not isolate domain shift from architectural bias. Prototype-based interpretation [1] and cross-domain zero-shot anomaly segmentation [2] are established directions; the contribution here is using shared prototypes to analyze transfer. Joint Sinkhorn balancing, spatial min–max normalization, and max aggregation are plausible alternative contributors to false positives. The Relative-gate variant does not isolate them. Test these choices with source-only selection, and align permutation-symmetric prototypes across seeds to establish whether response patterns and removal effects reproduce. Five-seed performance alone does not establish stable prototype interpretations.

  3. The interventions do not establish a consistent mechanism. BrainVent improves Dice-1 from 34.91% to 37.06%, but changes local texture, contrast, and boundaries together. BrainSmooth is useful; matched-area/contrast edits and held-out validation remain necessary. In the single-seed LungUniform experiment, Dice improves while p05/p07 contributions increase, so the Brain response-suppression explanation does not straightforwardly generalize. Confusion–performance correlations also require controls for lesion size and activation area because both measures use lesion masks. Separate post-hoc diagnosis from improvements to the unchanged zero-shot model.

  4. Presentation and code availability limit verification. Figure 1 does not distinguish source training from target inference, and its caption barely explains the operations. Figure 2 has small panels, no color scale, and no anatomical annotations; its caption omits independent normalization within each prototype/image, precluding direct comparisons of activation magnitude. Key evaluation conditions and limitations are scattered across the supplement. I could not locate a public code repository, and the supplied review materials contain no source-code archive. Providing training, evaluation, prototype-removal, and image-intervention scripts would enable verification of the central analyses.

References [1] Chen et al. This Looks Like That: Deep Learning for Interpretable Image Recognition. NeurIPS, 2019. [2] Zhou et al. AnomalyCLIP: Object-agnostic Prompt Learning for Zero-shot Anomaly Detection. ICLR, 2024.

Questions for authors
  1. Evaluation and calibration: Given the zero Dice-0 across all four target domains in Table 6, how should the abnormal-only Dice-1 gains in Table 1 be interpreted? Were the thresholds and gating hyperparameters selected exclusively on source-domain data? Do existing source-calibrated or threshold-free results support improved transfer beyond a different sensitivity–specificity trade-off?

  2. Stability of the prototype interpretation: Were the prototype-response patterns, removal effects, and confusion–performance associations reproduced across the five training seeds? If so, how were permutation-symmetric prototypes aligned? Please distinguish analyses replicated across seeds from those based on a single trained model.

  3. Interpretation of the interventions: BrainVent reduces the implicated prototype contributions while improving Dice, whereas LungUniform improves Dice despite increased p05/p07 contributions. What mechanism is supported consistently by these results? Does the existing evidence distinguish anatomy-specific confusion from effects of changing local contrast, texture, or the assignment and gating operations?

4 comments
AuthorsOpenReview ↗

Response to Reviewer RSwS (part 1)

Response 1. Answer to: Interpretation of Dice-1 gains, source-domain calibration, and threshold-free evidence. The zero Dice-0 together with improved Dice-1 reflects the operating behavior induced by our HA- and NR-gates: DCAG prioritizes strongly activated abnormal regions, resulting in higher lesion sensitivity but weaker normal-image specificity at the fixed 0.5 segmentation threshold. We therefore interpret the Dice-1 gains specifically as improved abnormal-region localization, rather than improved overall calibration.

  • Table 13 makes this trade-off explicit. The Relative variant recovers non-zero Dice-0 on Retina and Breast, but substantially decreases Dice-1, showing that modifying the decision mechanism shifts the sensitivity–specificity balance rather than resolving it. The threshold-free pAUC results are competitive but not consistently superior, so the current evidence does not support a claim of uniformly improved calibration across normal and abnormal images. All target datasets use the same source-trained model and configuration without target-domain adaptation.

Response 2. Answer to: Stability of the prototype interpretation and Separating domain-shift effects from architectural bias. The stability of prototype interpretation is addressed in our Response 2 to Reviewer zGqi. We clarify that the main segmentation results are averaged over five training seeds, whereas the prototype-level analyses reported in the main paper are based on a single trained model. Because prototype identities are permutation-symmetric, we do not assume that the same prototype index corresponds to the same visual pattern across seeds without explicit alignment. Consequently, directly averaging prototype-indexed quantities across seeds, such as prototype--GT IoU or pairwise prototype IoU in Table 3, would be misleading because prototype roles may be permuted between runs. To provide additional evidence, we include in Appendix C.4 the corresponding prototype-level analysis from an independently trained model with a different seed.

  • In addition, we evaluate BrainVent and LungUniform using MVFA, MRAD-CLIP, and VisualAD. Dice-1 improves after both interventions for all three methods, suggesting that confusion with high-contrast normal anatomical structures is not specific to ProtoZSAD, but represents a broader domain-shift failure mode. We add this result in Appendix C.3.
MethodDatasetDiceDice-0Dice-1pAUCN-FP AreaAbn. FPO
MVFA-baseBrain17.770.0033.5393.718.479.97
BrainVent20.250.0038.1994.809.109.00
Lung10.841.7518.1375.308.7512.01
LungUniform13.021.7522.0678.647.678.72
MRAD-CLIPBrain49.1071.8028.9095.200.702.20
BrainVent42.2053.8031.9094.900.902.80
Lung6.500.0011.6063.3015.8024.00
LungUniform8.600.0015.4069.0018.7022.30
VisualADBrain17.990.0033.9594.329.088.63
BrainVent18.220.0034.3793.6510.359.71
Lung8.420.0015.1961.859.9911.32
LungUniform9.590.0017.2866.188.578.65

All values are reported in percentage (%). Higher is better for Dice, Dice-0, Dice-1; lower is better for N-FP Area and Abn. FPO.

AuthorsOpenReview ↗

Response to Reviewer RSwS (part 2)

Response 3. Answer to: Interpretation of the intervention. The two interventions Brain and LungUniform provide different strengths of evidence. BrainVent gives the cleaner mechanistic result: ventricular modification substantially reduces p05/p07p_{05}/p_{07} contributions, lowers FPO, and improves Dice, while the BrainSmooth control does not reproduce this effect. This supports ventricle-specific prototype confusion rather than a generic smoothing effect.

  • LungUniform should be interpreted more cautiously. Although it reduces false positives in the modified mediastinal region and improves Dice, p05/p07p_{05}/p_{07} contributions slightly increase. Therefore, it supports the relevance of normal mediastinal/airway appearance to Lung errors, but does not establish that the improvement is mediated by suppressing these specific prototypes. One likely reason for this weaker separation is that Lung contains multiple repeated high-contrast vessel- and airway-related structures, while LungUniform is based on a simple heuristic procedure and does not make the entire intervention region fully homogeneous. Thus, residual high-contrast structures may remain and continue to activate these prototypes.

  • More generally, our claim is not that a prototype encodes a specific anatomical structure. Rather, the prototypes capture visual characteristics such as high contrast, edges, and intensity transitions, which can occur in both lesions and normal anatomical structures, which shown in Table 2.a and Figure 2.

Response 4. Answer to: Presentation, figure clarity, and code availability. We revised Figure 1 to clearly distinguish source-domain training from target-domain inference, clarify the HA/NR gating operations, and expanded its caption to explain Dual-Condition Abnormal Gate contribution. We also added a colorbar to Figure 2 and clarified that each heatmap is independently min–max normalized per prototype and image; colors therefore indicate relative spatial responses. In Figure 4, we further added annotations identifying the relevant normal anatomical structures in the Brain and Lung examples to make the analyzed failure regions easier to interpret. Implementation of prototype analysis details are provided in Appendix A.3 to A.6, and we will release the source code with the final version.

Thank you for the added experiments and clearer figures. They address some of my concerns. However, the best Dice found by testing different thresholds does not show how well a threshold chosen using only source data would work. The extra training run and image edits provide useful evidence, but do not yet show that the same explanation holds across runs and datasets. False positives on normal images also remain a concern. I therefore keep my rating.

Let me clarify further regarding your concerns.

  • The fixed threshold of t=0.5t=0.5 used in our main evaluation was selected using only the held-out MVTec source-domain validation set. This threshold was then fixed before any target-domain evaluation and applied unchanged to Retina, Breast, Brain, and Lung. Therefore, the main results already correspond to a source-calibrated operating point. The target-domain threshold sweep in Table 14 is used only as a post-hoc oracle sensitivity analysis and is not used for threshold selection or model tuning.

  • Regarding false positives on normal images, this behavior is not unique to ProtoZSAD; similar normal-image false-positive behavior is also observed in AA-CLIP, MRAD-CLIP, and VisualAD. More generally, this reflects a sensitivity–specificity trade-off associated with the operating behavior of different models

Reviewer_zGqiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

ProtoZSAD learns balanced normal/abnormal prototypes using masked supervision from MVTec AD’s test split, then evaluates medical-domain transfer at threshold 0.5 and analyzes prototype behavior.

Critical strengths

1.The paper usefully distinguishes spatial activation, prediction relevance, and segmentation utility through quantitative analyses and interventions.

2.Against AA-CLIP, ProtoZSAD achieves comparable or higher Dice-1 with lower normal-image false-positive area and abnormal FPO across all domains.

Critical weaknesses
  1. It is unclear what the prototype analysis explains beyond lesion response and size. With approximately balanced assignments, the confusion score satisfies C≈(1/K−m)/(1−f)C \approx (1/K - m)/(1 - f), where mm is the mean lesion assignment and ff is the lesion fraction. The score therefore largely restates these two quantities; it does not tell us which normal structures attract the response. Table 2's evidence statistics cover only the top-10 activation images per domain, and the paper does not show whether the prototype interpretations hold across seeds.

  2. The reported Dice gains do not establish a consistent localization advantage. The improvements over the strongest baselines (+2.24/+2.48/+0.96 pp) are similar in scale to the reported seed standard deviations (2.73/2.07/1.91). This does not establish significance or its absence, but uncertainty estimates for the differences are needed. Pixel AUROC is lowest on Retina and Brain, and Dice-0 is zero across all four domains. The paper also does not isolate how much per-image normalization and NR gating contribute to these false positives.

Questions for authors
  1. Does prototype confusion add information beyond lesion size, lesion response, and aggregated-map contrast?

  2. Are prototype roles stable across seeds after matching identities? How do the unablated L_PPC and L_PPD losses shape them?

  3. How is BrainSmooth matched to BrainVent, and what is the intervention uncertainty?

  4. Do gains persist under comparable false-positive constraints or clearly labeled threshold-swept F1-max?

2 comments
AuthorsOpenReview ↗

Response to Reviewer zGqi (part 1)

Response 1. Answer to: Whether prototype confusion adds information beyond lesion size, lesion response, and aggregated-map contrast. We agree that, because Sinkhorn approximately balances each prototype's total assignment, the confusion score is mathematically related to both lesion response and lesion fraction. We therefore do not interpret Table 4(a) as identifying which normal structure attracts a prototype, nor as an independent factor beyond lesion size and response. Rather, it is used as a compact image-level diagnostic showing that relatively stronger prototype allocation outside the lesion is associated with lower Dice and higher FPO.

  • The anatomical interpretation comes from a different part of the analysis: the prototype activation maps identify where the responses occur, and the BrainVent intervention tests whether modifying an implicated high-contrast normal structure changes prototype responses and segmentation behavior. Thus, Table 4(a) provides association, whereas the spatial analysis and intervention provide localization and structure-specific evidence.

  • The top-10 analysis in Sec. 4.2.1 is intended only to characterize the strongest, most representative responses of each prototype, not to establish population-level validity. The subsequent overlap, prototype-removal, and confusion analyses use the full set of abnormal images. The consistent of prototype interpretation across seeds is described detail in Response 2 below.

Response 2. Answer to: Prototype stability across seeds and the roles of LPPCL_{\mathrm{PPC}} and LPPDL_{\mathrm{PPD}}. We evaluated segmentation performance across five training runs. The prototype overlap and removal analyses were conducted using the reference model in Table 3 and independently repeated using another trained model with seed 123 in Table 17 (Appendix C.4). The main observations remain consistent across these two models: the prototype bank contains both redundant and complementary representations, spatial distinctiveness does not necessarily imply lesion relevance, and prototype involvement is distinct from segmentation utility. However, because prototype identities are permutation-symmetric, we do not assume that the same prototype index corresponds to the same functional role across independently trained models. For example, a role associated with p04p_{04} in one run may be represented by another prototype in a different run. We therefore compare recurring prototype-level behaviors rather than fixed prototype identities across seeds. Figures 2 and 3 illustrate the detailed prototype-response patterns of the reference model.

  • the role of LPPCL_{\mathrm{PPC}} and LPPDL_{\mathrm{PPD}}: We address a similar concern in our Response 4 to Reviewer ELKd. Please refer to that response for further details.

Response 3. Answer to: Matching between BrainSmooth and BrainVent and intervention uncertainty. BrainSmooth serves as a control for the generic effect of image modification: it applies the same inpainting/smoothing operation to a normal parenchymal region outside both the ventricle and lesion, whereas BrainVent specifically modifies the candidate ventricular region. Thus, the comparison tests whether the improvement is specific to modifying the implicated ventricular structure rather than simply smoothing normal brain tissue.

  • The main intervention uncertainty comes from ventricular localization. BrainVent identifies candidate regions heuristically using central location, adaptive intensity thresholds, and morphological filtering rather than a dedicated ventricle segmentation annotation. Therefore, the exact intervention boundary may vary across images.
AuthorsOpenReview ↗

Response to Reviewer zGqi (part 2)

Response 4. Answer to: Performance under comparable false-positive constraints or threshold-swept F1-max. We additionally performed a threshold sweep over t∈{0.3,0.4,0.5,0.6,0.7,0.8}t \in \{0.3, 0.4, 0.5, 0.6, 0.7, 0.8\} and report the oracle best Dice-1 for each method and target domain. Since Dice and F1 are equivalent for binary segmentation, this provides a threshold-swept F1/Dice-max diagnostic. The target-domain sweep is used solely for post-hoc evaluation, not for training or model selection.

MethodBrainBreastLungRetina-RESC
ProtoZSAD0.8 / 37.700.7 / 49.790.5 / 18.240.3 / 63.92
VisualAD0.6 / 37.300.5 / 45.330.6 / 15.860.3 / 60.29
AA-CLIP0.6 / 27.600.6 / 31.010.7 / 20.880.6 / 63.37
MRAD-CLIP0.4 / 31.010.8 / 40.660.7 / 14.720.3 / 61.78
AdaCLIP0.3 / 23.980.3 / 38.990.3 / 10.830.3 / 16.07
AnomalyCLIP0.3 / 27.340.3 / 30.290.3 / 9.780.3 / 3.86

Each entry reports best threshold / Dice-1 (%).

  • ProtoZSAD achieves the highest threshold-swept Dice-1 on three of the four target domains (Brain, Breast, and Retina-RESC), while AA-CLIP performs best on Lung. These results demonstrate that the localization advantage of ProtoZSAD is not limited to the fixed 0.5 operating threshold, although the optimal threshold varies across methods and domains. We have included the complete threshold-sweep results in Appendix C.1.

  • Moreover, we emphasize that Dice and IoU are standard overlap-based evaluation metrics in segmentation tasks. A probability threshold of 0.5, or its equivalent zero-logit threshold, is also used in widely adopted promptable segmentation implementations such as SAM (Kirillov et al., 2023) and MedSAM (Ma et al., 2024).

SAM: Alexander Kirillov et al., Segment Anything, ICCV 2023, pp. 4015–4026. https://openaccess.thecvf.com/content/ICCV2023/html/Kirillov_Segment_Anything_ICCV_2023_paper.html

MedSAM: Jun Ma et al., Segment anything in medical images, Nature Communications, 15, 654, 2024. https://www.nature.com/articles/s41467-024-44824-z

Reviewer_ELKdOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes ProtoZSAD, a zero-shot medical anomaly segmentation framework that builds on MVFA's multi-level CLIP visual adapters and introduces learnable normal and abnormal prototypes, balanced Sinkhorn assignments, and a dual-condition gating mechanism to localize anomalies. Trained exclusively on industrial images and evaluated on four unseen medical datasets without adaptation, the method achieves competitive segmentation performance and uses prototype-level analyses to investigate cross-domain transfer, including how learned abnormal patterns can mistakenly activate on normal anatomical structures.

Critical strengths
  1. Promising zero-shot cross-domain performance. The paper reports competitive medical anomaly segmentation performance despite training exclusively on industrial images, without any medical-domain fine-tuning. If reproducible, this is an interesting empirical result given the substantial differences between industrial defects and medical abnormalities.

  2. Prototype-level analysis of transfer failures. The paper goes beyond reporting segmentation metrics by analyzing how individual learned prototypes respond to medical images, including their tendency to confuse normal anatomical structures with abnormalities. The prototype-removal experiments and targeted interventions provide some evidence linking these responses to segmentation failures.

Critical weaknesses

Critical Weaknesses

  1. Insufficient explanation and validation of zero-shot cross-domain transfer. The model is trained exclusively on industrial images from MVTec AD, yet reports surprisingly strong segmentation performance on medical images, particularly retinal OCT images. Given the substantial differences between industrial defects and medical abnormalities, it is unclear how the learned prototypes and adapters can reliably distinguish pathological regions from normal anatomical structures without any medical-domain training. The authors should provide stronger empirical evidence explaining the source of this transferability, including controlled experiments on different source training domains and medical pathologies.

  2. Unnecessarily convoluted presentation. Section 3 (ProtoZSAD Architecture) uses overly complicated language and mathematical formulations to describe relatively simple operations, making the method unnecessarily difficult to follow. For example, the Dual-Condition Abnormal Gate (DCAG) essentially applies two sigmoid-based thresholds to abnormal prototype activations and their differences from normal prototype activations, followed by a minimum operation. This mechanism could be explained much more clearly and concisely.

  3. Limited methodological novelty. The proposed architecture largely reuses MVFA's multi-level CLIP visual adapters and combines them with existing techniques, including learnable prototypes, Sinkhorn-based balanced assignments, prototype regularization, and heuristic gating. The individual components are well established, and their combination does not appear to introduce a substantial methodological advance.

  4. Insufficient component-wise ablations. The paper does not adequately isolate the contributions of its architectural components, particularly DCAG and industrial-domain adapter training. Comparisons against simpler prototype aggregation strategies, alternative gating mechanisms, and the original MVFA architecture under the same training and evaluation protocol would help establish whether the proposed design is necessary.

Questions for authors

N/A

2 comments
AuthorsOpenReview ↗

Response to Reviewer ELKd1(part 1)

Response 1. Answer to: Insufficient explanation and validation of zero-shot cross-domain transfer. ProtoZSAD is trained with labeled MVTec images and does not learn medical semantics; instead, its prototypes capture transferable local visual patterns. As illustrated in Fig. 2, the same abnormal prototype (e.g., p07p_{07}) responds to compact high-contrast regions in both industrial defects and medical images, suggesting that transfer is driven by shared local appearance rather than pathology-specific semantics. This cross-domain setting is also consistent with prior work using similar vision backbones, where abnormality knowledge learned from supervised MVTec data is transferred zero-shot to medical domains through generic visual patterns. Our prototype-multiplicity study further supports this intuition: on Retina-RESC, one normal/one abnormal prototype achieves 47.46%47.46\% Dice-1, while four normal/four abnormal prototypes improve it to 60.01%60.01\%. Across Retina, Breast, Brain, and Lung, we further show that transfer becomes unreliable when the same source-associated cues also occur in normal target anatomy.

Response 2. Answer to: Unnecessarily convoluted presentation of Section 3 and DCAG. We agree that the previous Section 3 mixed the conceptual method with low-level implementation details. We have rewritten it around four stages: motivation, prototype assignment, DCAG, and training objective, retaining only four compact equation blocks in the main text. DCAG is now explained intuitively as two complementary checks: HA identifies relative spatial peaks, while NR requires the abnormal assignment to exceed the strongest normal assignment; their minimum acts as a soft conjunction, followed by a maximum across abnormal prototypes.

Response 3. Answer to: Limited methodological novelty. We agree that the individual components of ProtoZSAD, such as visual adapters, prototypes, Sinkhorn assignment, and prototype regularization, are not individually novel. Our methodological contribution instead lies in structuring these components into an intermediate prototype representation specifically designed to analyze zero-shot source-to-target transfer. Unlike prior ZSAD methods that mainly expose a final anomaly map, ProtoZSAD preserves prototype-specific responses between source-associated visual evidence and target predictions, allowing us to examine how such evidence is transferred, whether it affects the final prediction, and when it becomes misleading.

  • This representation leads to a concrete empirical insight: abnormal prototype responses can become aligned with normal anatomical structures, producing false positives under domain shift. Our correlation analysis and Brain intervention further validate this failure mode. We therefore view the main contribution as an analysis-oriented framework and the new transfer insights it enables, rather than the novelty of each individual architectural component.
AuthorsOpenReview ↗

Response to Reviewer ELKd1(part 2)

Response 4. Answer to: Insufficient component-wise ablations of prototype learning, DCAG, and MVFA. We add component-wise ablations of balanced assignment, PPC/PPD losses, the NR gate, the HA gate, and the original MVFA architecture under the same MVTec training/evaluation protocol. The complete results are reported in Table 15 of Appendix C.2.

  • Balanced assignment together with PPC/PPD losses is important for maintaining distinct prototype representations. Empirically, without these constraints, the prototypes tend to collapse into two coarse normal/abnormal groups, likely because Dice-based supervision mainly encourages foreground-background separation. Removing either balanced assignment or PPC/PPD losses therefore reduces Dice-1, particularly on Retina, Breast, and Brain.

  • For DCAG, the two gates are complementary: the HA gate retains strongly activated abnormal regions, while the NR gate suppresses responses that do not exceed competing normal evidence. Removing either gate degrades Dice-1 across the target domains, with especially large drops when removing HA on Breast and Brain.

  • We also add the original MVFA baseline under the same source-domain protocol. ProtoZSAD substantially improves Dice-1 on Retina and Breast while remaining comparable on Brain and Lung. We emphasize that our goal is to introduce a prototype-level intermediate representation for transfer analysis while preserving competitive segmentation performance, rather than to optimize segmentation performance at all costs. Full results and discussion are provided in Appendix C.2 (Table 15).

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response and Revision Summary

"Dear Area Chair and Reviewers,

We sincerely thank you for your time and constructive comments. Below, we summarize the main revisions and additional analyses made in response to the reviewers:

  • For Reviewer ELKd’s concerns regarding cross-domain transfer, methodological novelty, and insufficient ablations, we provided further prototype-level evidence explaining how visual characteristics learned from industrial defects transfer to medical images. We substantially simplified Section 3 and added component-wise ablations, including comparisons with the original MVFA baseline under the same training protocol.

  • For Reviewer XPZA’s concerns about interpretability and design choices, we further clarified the added explanatory value of prototype analysis, particularly its ability to identify transferable visual characteristics and diagnose failures caused by normal anatomical structures. We also clarified the distinction between balanced prototype assignment and final anomaly prediction, and explained the rationale for selecting the state-of-the-art baselines used in our comparisons.

  • For Reviewer zGqi’s concerns regarding prototype confusion, stability, and localization performance, we clarified the interpretation and limitations of the confusion analysis, discussed cross-seed prototype behavior and permutation symmetry, and explained the BrainSmooth control and intervention uncertainty. We additionally conducted threshold-sweep experiments to evaluate segmentation performance beyond the fixed threshold.

  • For Reviewer RSwS’s concerns regarding evaluation, domain shift, and intervention validity, we clarified the Dice-0/Dice-1 trade-off and the interpretation of threshold-free metrics. We also extended BrainVent and LungUniform interventions to existing baselines, providing additional evidence that confusion with normal anatomical structures represents a broader cross-domain failure mode rather than being specific to ProtoZSAD.

We have revised the manuscript and provided detailed point-by-point responses. The main changes are summarized as follows:

  • Fig. 1: Revised the architecture diagram to distinguish source-domain training from target-domain inference and added more explicit information on the dual-condition abnormal gate (DCAG). The figure caption now also clarifies the purpose of DCAG.

  • Sec. 3: Reorganized and simplified the methodology into four stages, with intuitive explanations of prototype assignment and the two complementary DCAG gates (HA and NR), while removing unnecessary mathematical complexity.

  • App. C.1: Added threshold-swept Dice/F1-max comparisons across six methods, showing that ProtoZSAD achieves the highest performance on three of four medical datasets.

  • App. C.2: Added component-wise ablations of balanced assignment, PPC/PPD losses, and both DCAG gates, together with an MVFA baseline under the same source-domain training protocol.

  • App. C.3: Extended BrainVent and LungUniform intervention experiments to MVFA, MRAD-CLIP, and VisualAD to examine whether the identified domain-shift failure mode generalizes to other methods.

  • App. C.4: Added an additional-seed prototype analysis to examine the consistency of the main prototype-level findings across independently trained models.

We appreciate the reviewers’ thoughtful feedback and hope these revisions strengthen the clarity, empirical validation, and interpretation of our contributions. Detailed point-by-point responses are provided below."

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.