acceptodds
Under review as a conference paper at ICLR 2027

Fast-ZSAD: Patch-wise Adaptive Inference for Efficient Zero-Shot Anomaly Detection

Abstract

Zero-shot anomaly detection (ZSAD) enables the detection and localization of anomalies in unseen categories; however, the substantial inference cost of existing Transformer-based ZSAD models limits their processing throughput and practical deployment in large-scale scenarios. Compared with conventional visual tasks, anomaly detection exhibits a highly imbalanced spatial importance distribution, with anomalous regions of interest typically occupying only a small fraction of an image, making it inefficient to uniformly process all patches through the full Transformer depth. In this paper, we propose Fast-ZSAD, a patch-wise adaptive inference framework that dynamically allocates computation to normal and anomalous patches, thereby substantially accelerating ZSAD models. Specifically, Fast-ZSAD employs an Anomaly-Aware Patch Router to estimate patch-wise anomaly evidence, enabling confidently normal patches to exit early while forwarding suspicious patches for deeper inference. During router training, we find that the highly imbalanced ratio between normal and anomalous patches causes the optimization to be overwhelmingly dominated by abundant normal patches. To this end, we introduce Contrast-Aware Regularization (CAR), which encourages compact normal representations while separating anomaly-related patches from the normal distribution, thereby improving routing robustness. Furthermore, during deep inference after patch routing, we observe that the number and spatial relationships of forwarded tokens differ from those under full-token inference, causing deep-layer attention to become less focused on anomaly-relevant regions. We therefore develop Depth-Spatial Feature Correction (DSFC), which refines forwarded representations by jointly exploiting local spatial support and cross-depth feature evolution. Extensive experiments across AACLIP, AD-DINOv3, and RareCLIP on industrial and medical anomaly detection benchmarks demonstrate a favorable performance–efficiency trade-off: Fast-ZSAD achieves a 1.69× average inference speedup on AACLIP while improving the average mScore from 70.58 to 70.61, and further delivers 1.58× and 1.84× average speedups on AD-DINOv3 and RareCLIP, respectively.

Primary area:
optimization
Keywords:
Zero-Shot Anomaly Detection, Efficient Inference, Adaptive Computation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_wmyeOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper aims to reduce the computational overhead of Transformer-based zero-shot anomaly detection (ZSAD) models. Based on the observation that anomalies usually cover only a small portion of an image and normal patches stabilize in earlier layers, the authors propose Fast-ZSAD. The method places an Anomaly-Aware Patch Router at layer 12 to exit confident normal patches early, while sending a fixed ratio (ρ=0.3\rho = 0.3) of suspicious patches through the deeper layers. To handle class imbalance during router training, the authors introduce Contrast-Aware Regularization. They also propose Depth-Spatial Feature Correction (DSFC) to mitigate feature drift in deeper layers by combining multi-depth representations using local spatial support and cross-depth feature transitions.

Critical strengths

This paper is well-written and very easy to follow. Evaluations across AACLIP, AD-DINOv3, and RareCLIP on industrial and medical datasets report average speedups of 1.58x–1.84x while largely preserving detection scores.

Critical weaknesses
  1. The method fixes the deep-inference budget to a constant ratio of ρ=0.3\rho = 0.3. For completely normal images, still pushing 30% of the tokens through the remaining 12 Transformer layers wastes considerable compute. On the other hand, for anomalous images, while the Complete Miss Rate (CMR) at ρ=0.3\rho = 0.3 is 0% on MVTec AD, the Ground-Truth Anomaly Patch Recall (GT Recall@30%) is only 89.95% on AACLIP and 86.72% on AD-DINOv3. This means 10% to 13% of actual defect patches are exited early at layer 12 and never get processed by the deeper layers. Using a fixed Top-KK selection rather than a score-based threshold seems driven mainly by GPU batching constraints rather than actual image content.
  2. DSFC combines four hand-crafted terms: routing density, shallow feature coherence, transition magnitude, and neighborhood transition consistency. The final fusion weights rely on several normalization scalers, affine cosine mappings, and an exponential scaling hyperparameter. As shown in Fig. 9, performance degrades noticeably once λsd>3.5\lambda_{sd} > 3.5, suggesting that the formulation is quite sensitive to hyperparameter tuning.
  3. Fig. 3 argues that sparse deep inference causes severe "prediction drift," which distorts features and increases false positives in deeper layers. However, Eq. 15 and Eq. 16 assign exponentially higher weights to deeper features that have larger transition distances (d^il\hat{d}_i^l). If deeper layers are indeed prone to unfaithful drift, giving higher weights to layers with larger feature shifts seems to work against the paper's own motivation.
  4. In Table 5 (AD-DINOv3 on MVTec AD), the Naive Adaptive baseline gets 75.08 mScore, whereas adding the Map Router drops the score to 52.44 (-22.64 points) and raises CMR to 43.80% (Table 7). Distilling from the full model's own predictions causes the router to miss defects in almost half of the anomalous images. Simply attributing this drop to "uniform map supervision is insufficient" without analyzing the root cause (such as gradient imbalance, softmax saturation, or adapter interference) leaves an important failure mode unexplained.
  5. The experiments only compare against token reduction methods borrowed from general vision tasks (ATS, DTOP, Cropr), missing two straightforward baselines: (i) simply lowering the input resolution from 518×518518 \times 518 to 384×384384 \times 384 or 448×448448 \times 448, which gives a similar 1.5×1.5\times–2.0×2.0\times speedup without changing the architecture; and (ii) directly taking predictions from layer 16 or layer 20 for all tokens without dynamic routing. Without these comparisons, it is hard to tell how much benefit comes specifically from patch-wise sparse routing.
Questions for authors
  1. In a single-image streaming setup (Batch Size = 1), how does Fast-ZSAD perform if patch selection is based on an absolute anomaly score threshold rather than a fixed Top-30% ratio? What is the actual runtime and skip rate on exclusively normal validation sets?
  2. How does Fast-ZSAD compare in mScore and latency against the full baseline executed at reduced input resolutions (384×384384 \times 384 or 448×448448 \times 448), or against early-terminating the entire model uniformly at layer 16?
  3. Provide head-to-head empirical results against KeepAD (Hou et al., 2026) under identical hardware and evaluation protocols.
  4. In Table 8, a non-uniform late-biased prior was hand-crafted for AD-DINOv3. What happens if this exact prior is applied to AACLIP and RareCLIP, or conversely, if the uniform prior is enforced on AD-DINOv3 across all datasets?
5 comments

We sincerely thank the reviewer for the valuable and constructive comments, which help us improve our work. We address the concerns as follows.

W1: Fixed Deep-Inference Ratio and Anomaly-Patch Coverage.

Thank you for raising this important concern. Fast-ZSAD adopts a fixed computational budget while adaptively selecting anomaly-relevant patches for deeper inference. The default (ρ=0.3\rho=0.3) is chosen based on the accuracy–efficiency trade-off in Fig. 7(a), providing predictable computation despite some redundancy on fully normal images.

Importantly, early-exited patches retain anomaly predictions learned through full-depth map distillation and CAR (Eq. 6), rather than raw shallow-layer scores. Thus, GT Recall@30% measures anomalous-patch coverage for deep refinement, not the final anomaly detection recall. Tables 6–7 further show that the learned router improves GT Recall over direct shallow-score routing from 85.98% to 89.95% on AACLIP and from 78.48% to 86.72% on AD-DINOv3.Our batch-1 experiments in Q1 further show that fixed Top-30% routing better preserves detection accuracy than threshold-based routing at comparable normal-image latency, supporting its use as an accuracy–efficiency choice rather than merely a GPU batching constraint.

W2: DSFC Design and Hyperparameter Sensitivity.

Thank you for the constructive comments. We conduct leave-one-out ablations to evaluate the four DSFC signals on AACLIP and AD-DINOv3, and further examine the sensitivity to λsd\lambda_{sd} on AD-DINOv3.

Individual Signal Ablation.

DSFC ConfigurationAACLIP mScore ↑
Without DSFC74.32
w/o Routing density76.10
w/o Feature coherence76.23
w/o Transition magnitude76.38
w/o Transition consistency76.12
Full DSFC76.51

The ablation results suggest that these signals jointly contribute to the effectiveness of DSFC.

Sensitivity to λsd\lambda_{sd}.

λsd\lambda_{sd}AD-DINOv3 mScore ↑
078.49
0.578.44
178.45
278.46
378.54
3.2578.53
3.578.45
478.25
578.25

On AD-DINOv3, mScore varies by less than 0.30 points across λsd∈[0,5]\lambda_{sd} \in [0,5], remaining relatively stable over [0,3.5][0,3.5]. The degradation at larger values is less pronounced than on AACLIP (Fig. 9), indicating backbone-dependent sensitivity rather than a consistently narrow stable range.

W3: Clarification of Prediction Drift and Cross-Depth Feature Weighting.

Thank you for this insightful comment. Prediction drift and cross-depth feature evolution represent two distinct phenomena. Prediction drift refers to the discrepancy between sparse-token and full-token inference at the same depth (Fig. 3), whereas cross-depth evolution measures representation changes across successive layers. As shown in Fig. 1(b), anomalous patches exhibit larger cross-layer score variations, motivating us to exploit this feature evolution.

Importantly, DSFC does not determine fusion weights solely based on transition magnitude. As defined in Eq. (15), the depth signal is

Dil=d^ilκil,D_i^l = \hat{d}_i^l \kappa_i^l,

where d^il\hat{d}_i^l measures the normalized transition magnitude and κil\kappa_i^l captures its consistency with neighboring patch transitions, reducing the influence of large but locally inconsistent changes. The fusion weights are then computed as

wil=exp⁡(λsdSiDil)∑q∈LDexp⁡(λsdSiDiq)w_i^l = \frac{\exp(\lambda_{sd} S_i D_i^l)}{\sum_{q \in \mathcal{L}_D} \exp(\lambda_{sd} S_i D_i^q)}

Together with local spatial support SiS_i, these signals jointly determine the fusion weights, rather than simply favoring larger cross-depth changes. While they cannot eliminate all prediction drift, they enable spatially informed multi-depth feature fusion.The effectiveness of this design is supported by our ablation in Fig. 6, where removing depth modeling reduces mScore from 76.51 to 75.20 on AACLIP and from 78.53 to 77.05 on AD-DINOv3, demonstrating the benefit of incorporating cross-depth information into feature correction.

W4: Diagnosing the Map-only Router Degradation on AD-DINOv3.

Thank you for highlighting this issue. The Map-only Router degradation on AD-DINOv3 is related to sigmoid saturation caused by the large similarity scaling factor in the original AD-DINOv3 anomaly scoring formulation. Specifically, the patch-level anomaly probability is computed as

{\exp(100z_i^\top t_a)+\exp(100z_i^\top t_n)} = \sigma(m_i), \quad m_i=100z_i^\top(t_a-t_n),$$ where $z_i$ denotes the projected patch feature, and $t_a$ and $t_n$ are the abnormal and normal text embeddings, respectively. The Map-only Router minimizes the MSE between its predicted anomaly probability $p_i$ and the full-depth teacher prediction $q_i$: $$\mathcal{L}_{\mathrm{map}} = \frac{1}{N}\sum_{i=1}^{N}(p_i-q_i)^2$$ The corresponding gradient with respect to the student logit $m_i$ is $$\frac{\partial\mathcal{L}_{\mathrm{map}}}{\partial m_i} =\frac{2}{N}(p_i-q_i)p_i(1-p_i).$$ The large similarity scaling factor can push predictions toward saturated regions, where $p_i(1-p_i)$ approaches zero and suppresses corrective gradients. Moreover, under normal-dominated supervision, average MSE may favor low anomaly responses without preserving the teacher's relative patch ranking. Consequently, Map-only may converge to a degenerate low-response solution with unreliable patch discrimination despite a low distillation loss. CAR mitigates this issue by extracting relative anomaly evidence from standardized teacher predictions and introducing feature-space compactness and separation objectives that do not directly depend on saturated sigmoid gradients. Accordingly, CAR improves mScore from 52.44 to 76.64 and reduces CMR from 43.80% to 0% on AD-DINOv3 (Tables 5 and 7). >W5, Q2: Comparison with Resolution Scaling and Uniform Early Exiting. > Thank you for your valuable suggestion. We conduct additional comparisons on AACLIP across five benchmarks, including reduced input resolutions (384 × 384 and 448 × 448) and uniform early exiting at layers 16 and 20. Performance–Efficiency Comparison on AACLIP: | Method | MVTec AD mScore ↑ | Speedup ↑ | VisA mScore ↑ | Speedup ↑ | BTAD mScore ↑ | Speedup ↑ | MPDD mScore ↑ | Speedup ↑ | MedAD mScore ↑ | Speedup ↑ | |---|---:|---:|---:|---:|---:|---:|---:|---:|---:|---:| | AACLIP (518×518) | 76.31 | – | 64.24 | – | 81.47 | – | 63.38 | – | 67.52 | – | | AACLIP (384×384) | 73.99 | 1.62× | 62.54 | 1.55× | 76.30 | 1.63× | 60.58 | 1.58× | 66.21 | **1.69×** | | AACLIP (448×448) | 76.08 | 1.26× | 63.58 | 1.30× | **80.71** | 1.33× | 61.95 | 1.24× | 67.10 | 1.36× | | Early Exit (L16) | 71.98 | 1.28× | 57.65 | 1.33× | 72.41 | 1.38× | 53.99 | 1.27× | 62.43 | 1.41× | | Early Exit (L20) | 73.25 | 1.13× | 57.97 | 1.15× | 73.73 | 1.17× | 56.61 | 1.13× | 61.84 | 1.18× | | **Fast-ZSAD (Ours)** | **76.51** | **1.63×** | **64.55** | **1.70×** | 80.40 | **1.80×** | **62.76** | **1.71×** | **68.85** | 1.62× | The results show that Fast-ZSAD achieves a favorable performance–efficiency trade-off compared with these simple acceleration alternatives. For instance, on MVTec AD, Fast-ZSAD achieves 76.51 mScore at 1.63× speedup, compared with 73.99 at 1.62× for resolution reduction to 384 × 384. Across five benchmarks, Fast-ZSAD achieves 1.62×–1.80× speedups while largely preserving the original detection performance. In contrast, uniform early exiting incurs larger performance degradation with lower speedups in the evaluated settings. >Q1: Absolute-Threshold Routing under Batch-1 Streaming Inference. > Thank you for this suggestion. We compare fixed Top-30% and absolute-threshold routing on AACLIP under batch-1 inference, with results shown below. | Method | Full-set mScore ↑ | Full-set Latency (ms/image) ↓ | Normal-only Latency (ms/image) ↓ | Normal Patch Skip Rate ↑ | Full Deep-stage Skip Rate ↑ | |---|---:|---:|---:|---:|---:| | Teacher | 76.31 | 98.31 | 98.90 | 0% | 0% | | Top-30% | 76.51 | 71.13 | 70.51 | 70.05% | 0% | | Threshold −8 | 76.38 | 86.20 | 82.16 | 62.72% | 23.84% | | Threshold −6 | 75.30 | 77.90 | 70.38 | 81.36% | 43.07% | | Threshold 0.3 | 74.80 | 58.63 | 54.22 | 99.99% | 91.49% | *Full deep-stage skip rate denotes the percentage of normal images for which no patch is forwarded to the remaining Transformer blocks. All latency measurements are conducted with batch size = 1.* Threshold-based routing can skip the entire deep stage for more normal images, but at the cost of detection accuracy. Notably, Top-30% achieves higher mScore than Threshold −6 (76.51 vs. 75.30) at nearly identical normal-image latency, supporting its favorable accuracy–efficiency trade-off.

Q3: Clarification on the Comparison with KeepAD.

Thank you for this suggestion. A direct comparison with KeepAD is currently infeasible because its official repository provides only the README and license, without implementation code, checkpoints, or evaluation scripts. Moreover, KeepAD reports batch-1 FP32 throughput on an RTX 5090, whereas our experiments use batch size 32 on an RTX A6000, with different anomaly detectors and evaluation metrics, making direct comparisons of published efficiency numbers unfair. Methodologically, Fast-ZSAD accelerates existing ZSAD models through patch-wise adaptive inference, whereas KeepAD relies on a separately trained dense detector and progressive defect-preserving token pruning.

Q4: Generalization of Layer Priors across Models and Datasets.

Thank you for this valuable suggestion. We have conducted additional experiments by applying the late-biased prior to AACLIP and RareCLIP, and the uniform prior to AD-DINOv3 across five benchmarks. The results are shown below.

Cross-model and cross-dataset generalization of layer priors (mScore):

ModelPriorMVTec ADVisABTADMPDDMedADAvg.
AACLIPUniform76.5164.5580.4062.7668.8570.61
Late-biased75.5763.5778.1962.4166.5869.26
RareCLIPUniform87.9877.1185.1864.6967.3876.47
Late-biased86.6677.0784.4264.5966.7075.89
AD-DINOv3Uniform77.0568.9272.7267.1564.8670.14
Late-biased78.5370.5674.6868.5064.4171.34

The results show that the uniform prior performs better on AACLIP and RareCLIP, while the late-biased prior achieves higher average performance on AD-DINOv3. This indicates that different ZSAD models benefit from different layer-prior configurations, supporting our baseline-specific design.

Reviewer_81MnOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper accelerates Transformer-based zero-shot anomaly detectors by routing patches after layer 12. A two-layer MLP router, distilled from the full-depth anomaly map and regularized with a compactness/separation loss (CAR), sends the top 30% of patches through the remaining blocks. The other patches keep the router's prediction and are summarized by one average-pooled token. For the forwarded patches, DSFC fuses features from several depths with spatial and cross-depth weights. On AACLIP, AD-DINOv3 and RareCLIP the paper reports average speedups of 1.69×1.69\times, 1.58×1.58\times and 1.84×1.84\times, with the AACLIP average mScore going from 70.58 to 70.61.

Critical strengths
  1. Inference cost is a real obstacle for high-resolution ZSAD models, and the method is evaluated on three different backbones and on both industrial and medical data.
  2. Code is provided, the timing includes the router and correction overhead, and the appendix gives full per-metric tables and a router-recall analysis.
Critical weaknesses
  1. Novelty is limited. Letting easy tokens exit early with their intermediate predictions in a dense prediction task is what DToP does, and pooling the exited tokens into one token is the fused token of EViT (Liang et al., 2022, cited). CAR is a center-compactness loss plus a softplus margin, and DSFC is a weighted multi-layer feature fusion. The combination is reasonable engineering, but each part is known, and the experiments do not show that the ZSAD-specific choices matter beyond these generic tools (see 4 and 5).

  2. The accuracy claims hide localization losses. The mScore averages three image-level and three pixel-level metrics. For AACLIP, pixel AUROC drops on all five benchmarks (by 1.28 points on average, 2.51 on BTAD), and the +0.03 mScore comes from image-level gains. For AD-DINOv3 the average mScore falls by 1.74 and pixel AP by 3.08 (5.08 on BTAD), and RareCLIP loses 2.46 mScore on VisA. The abstract reports only speedups for these two backbones. No variance or number of runs is reported, so a +0.03 change cannot be read as an improvement.

  3. Hyperparameters are tuned on test data. ρ\rho and the routing layer are chosen from sensitivity plots on MVTec AD (Fig. 7), and λsd\lambda_{sd} the same way (Fig. 9), although MVTec AD is also an evaluation set in this cross-dataset protocol. The AD-DINOv3 layer prior [0.05, 0.125, 0.375, 0.45] is picked from six candidates by MVTec AD test mScore (Table 8). With the uniform prior that the main text describes, Fast-ZSAD on AD-DINOv3/MVTec AD gets 77.05, against 78.53 with the tuned prior and 79.23 for the full model. In the released code λsd\lambda_{sd} defaults to 3.25 for AACLIP, 0.35 for RareCLIP and 0.5 for AD-DINOv3. Only the first lies in the stable range shown in Fig. 9, and the per-backbone values are not reported in the paper.

  4. The comparisons are not at matched budgets, and the baselines look poorly configured. Cropr is slower than the unaccelerated model on AACLIP (0.74×0.74\times to 1.00×1.00\times), and DToP loses 23 mScore points on AACLIP/MVTec AD. Without accuracy-speed curves for each method, Tables 1-3 compare points at different speedups. Simple alternatives are missing: lower input resolution, a smaller backbone, ToMe (cited and training-free) and half precision. Timing is done in FP32 on an A6000, where TF32 or FP16 alone would likely give a speedup of similar size; Fast-ZSAD should be reported on top of such a setting.

  5. One ablation reference point looks broken. On AD-DINOv3 the map-distilled router reaches 17.06% GT recall and 19.85% teacher recall at ρ=0.3\rho = 0.3 (Table 7), worse than random selection (about 30%), and its mScore (52.44) is far below naive routing (75.08). A router trained with MSE to imitate the teacher map should not rank patches worse than chance, so this suggests a training or implementation problem, and it inflates the gain credited to CAR (+24.2). On AACLIP, routing by the baseline's own layer-12 score already gives 85.98% GT recall, against 89.95% for the full router (Table 6). "Naive Adaptive" (61.08 on AACLIP/MVTec AD) and ATS (75.46) presumably both route at layer 12 and keep layer-12 predictions for exited patches, yet they differ by 14 points. The naive variant is never defined.

  6. Reporting and reproducibility. With a fixed 518×518518\times518 input and a fixed 30% forwarding ratio, Fast-ZSAD has the same per-image FLOPs on every dataset, yet its reported speedups range from 1.51×1.51\times to 1.77×1.77\times on AD-DINOv3. Please give FLOPs, absolute latency and throughput with variance. The RareCLIP test loop processes images one at a time inside each batch, which does not match the stated batch size of 32. The released RareCLIP code only supports MVTec AD, VisA and BTAD, so its MPDD and MedAD results cannot be reproduced from it. The AACLIP and AD-DINOv3 baselines are re-trained by the authors, and AD-DINOv3 uses an image-level scoring rule borrowed from RareCLIP, so please also compare these baselines with the numbers in their original papers.

Questions for authors
  1. Can you show accuracy-speed curves for Fast-ZSAD and each baseline, including ToMe, lower resolution and FP16, at matched speedups?
  2. What are the results when ρ\rho, the routing layer, λsd\lambda_{sd} and the layer prior are chosen on the training dataset only?
  3. Why does the map-only router on AD-DINOv3 rank patches worse than random, and what exactly is "Naive Adaptive"?
  4. Can the main tables report image-level and pixel-level metrics separately, with variance over several router-training seeds?
4 comments

We sincerely thank the reviewer for the valuable and constructive comments, which help us improve our work. We address the concerns as follows.

W1: Concerns about Methodological Novelty.

Thank you for your comments. Fast-ZSAD addresses two ZSAD-specific challenges: normal-dominated router optimization and sparse-inference-induced feature drift. Existing methods are not directly suited to ZSAD. DToP does not explicitly account for the highly imbalanced anomaly evidence in ZSAD, making its routing decisions less reliable for rare anomaly-critical patches. EViT merges less important tokens without preserving their individual spatial predictions, making it unsuitable for precise anomaly localization in ZSAD. Fast-ZSAD addresses these limitations through CAR, which introduces anomaly-evidence-guided regularization, and DSFC, which adaptively fuses multi-depth features based on spatial support and cross-depth feature evolution, while preserving dense anomaly localization.

W2: Accuracy–Efficiency Trade-off and Performance Stability.

Thank you for your comments. We would like to clarify that the primary objective of Fast-ZSAD is to achieve a favorable performance–efficiency trade-off. While some pixel-level metrics decrease on certain benchmarks under adaptive inference, improvements are also observed in both image-level and pixel-level metrics. For example, on AACLIP with MVTec AD, I-AUROC improves from 88.59 to 89.92 and I-F1 from 90.23 to 91.65. On MedAD, P-AP improves from 34.50 to 36.22 and P-F1 from 38.51 to 40.33. Compared with existing token acceleration methods, Fast-ZSAD better preserves detection and localization performance at comparable speedups. For instance, on AACLIP with MVTec AD, Fast-ZSAD achieves a P-AUROC of 89.66 at 1.63× speedup, substantially outperforming DToP (67.90 at 1.62×). Similar advantages are observed across different ZSAD backbones and benchmarks (Tables 1–3 and Appendix D.1). To provide a more transparent evaluation, we report image-level and pixel-level metrics separately, together with standard deviations over five independent router-training seeds, in our response to Q4.

W3, Q2: Concerns about Test-Set Hyperparameter Tuning.

Thank you for your valuable comments. To address the concern about test-set-based hyperparameter tuning, we conducted additional experiments using a source-domain validation protocol. Specifically, we split the VisA data into 90% for router training and 10% for validation, without using the MVTec AD test set for hyperparameter selection. Following our experimental setting, we tune ρ\rho, the routing layer, and λsd\lambda_{sd} on AACLIP, and the layer prior on AD-DINOv3, using the corresponding VisA validation splits.

Sensitivity to ρ\rho (AACLIP):

ρ\rhomScore ↑Speedup ↑
0.178.341.77×
0.381.321.52×
0.581.771.31×
0.782.091.14×
0.982.331.09×

Sensitivity to Routing Layer (AACLIP):

Routing LayermScore ↑Speedup ↑
L1281.321.52×
L1681.731.29×
L2077.511.13×

Sensitivity to λsd\lambda_{sd} (AACLIP):

λsd\lambda_{sd}mScore ↑
0.080.26
0.580.21
1.080.50
2.080.83
3.081.33
3.2581.328
3.580.16
4.080.07
5.080.02

Sensitivity to Layer Prior (AD-DINOv3):

Prior SchemeLayer PriormScore ↑
Late-biased[0.05, 0.125, 0.375, 0.45]82.39
Decreasing[0.40, 0.30, 0.20, 0.10]80.13
Up-down[0.15, 0.35, 0.35, 0.15]82.23
Uniform[0.25, 0.25, 0.25, 0.25]81.44
Down-up[0.35, 0.15, 0.15, 0.35]80.33
Increasing[0.10, 0.20, 0.30, 0.40]82.08

The results support the original choices of ρ=0.3\rho=0.3, routing layer L12, and the Late-biased Prior, while the validation-selected λsd\lambda_{sd} for AACLIP changes slightly from 3.25 to 3.0. These selections are based on source-domain validation rather than target test performance. We then apply the validation-selected configurations to all five evaluation benchmarks without further hyperparameter tuning, and report the resulting mScores below.

Fast-ZSAD on AACLIP with Validation-Selected Hyperparameters:

MetricMVTec ADVisABTADMPDDMedAD
mScore ↑76.4764.3280.4164.2468.91

Fast-ZSAD on AD-DINOv3 with Validation-Selected Hyperparameters:

MetricMVTec ADVisABTADMPDDMedAD
mScore ↑78.5469.8674.4868.7764.46

W4, Q1: Accuracy–Speed Trade-offs and Additional Baselines.

Thank you for your valuable suggestion. We conducted additional experiments to evaluate the accuracy–speed trade-offs across different acceleration settings, including comparisons with additional acceleration baselines. Representative results are shown below, with the complete accuracy–speed curves provided in Appendix C.1 (Fig. 8).

Accuracy–speed trade-off comparisons on AACLIP with MVTec AD:

MethodSpeedup (×)mScore (%)
AACLIP1.0076.31
ATS1.2875.46
ATS1.3873.66
DToP1.4154.98
DToP1.6253.13
ToMe1.3047.13
Cropr0.9560.08
AACLIP (384×384)1.6273.99
AACLIP (ViT-B/16)1.7966.83
AACLIP (AMP FP16)1.1676.21
Fast-ZSAD1.4176.72
Fast-ZSAD1.6376.51
Fast-ZSAD1.7775.63
Fast-ZSAD + AMP FP161.7076.47

As shown above and in Fig. 8, Fast-ZSAD achieves a favorable accuracy–speed trade-off across different inference budgets. Notably, at nearly identical speedups (1.63× vs. 1.62×), Fast-ZSAD achieves an mScore of 76.51, outperforming DToP (53.13) and the reduced-resolution baseline (73.99). Similarly, Fast-ZSAD achieves 75.63 at 1.77× speedup, compared with 66.83 for the smaller backbone at 1.79×. These results demonstrate that Fast-ZSAD better preserves detection performance at comparable inference speeds.

Regarding FP16 inference, we adopt PyTorch automatic mixed precision (AMP FP16), as full FP16 may introduce numerical errors in precision-sensitive operations, while certain CUDA kernels still perform intermediate computations in FP32. Under this setting, AACLIP achieves 1.16× speedup with an mScore of 76.21, while Fast-ZSAD combined with AMP FP16 achieves 1.70× speedup with an mScore of 76.47, demonstrating their complementary acceleration benefits.

W5, Q3: Map-Only Router Degradation and Clarification of Naive Adaptive.

Thank you for your careful examination. The poor performance of the Map-only Router on AD-DINOv3 is related to gradient saturation induced by the large similarity scaling factor in the original AD-DINOv3 anomaly scoring formulation. Specifically, the router computes patch-level anomaly probabilities as

{\exp(100z_i^\top t_a)+\exp(100z_i^\top t_n)} = \sigma(m_i), \quad m_i=100z_i^\top(t_a-t_n),$$ where $z_i$ is the projected patch representation, and $t_a$ and $t_n$ denote the abnormal and normal text embeddings. The Map-only Router minimizes the MSE between its prediction $p_i$ and the full-depth teacher prediction $q_i$: $$\mathcal{L}_{\mathrm{map}} =\frac{1}{N}\sum_{i=1}^{N}(p_i-q_i)^2, \quad \frac{\partial\mathcal{L}_{\mathrm{map}}}{\partial m_i} =\frac{2}{N}(p_i-q_i)p_i(1-p_i).$$ The large inherited scaling factor can cause student predictions to enter saturated regions, where $p_i(1-p_i)$ approaches zero, resulting in vanishing corrective gradients. Under normal-dominated source-domain supervision, average MSE may further favor uniformly low anomaly responses without preserving the teacher's relative patch ranking. These effects can lead to degenerate optimization and unreliable Top-K routing, plausibly explaining the poor recall in Table 7, although they do not fully explain its below-random performance. CAR mitigates this issue by using relative anomaly evidence from standardized teacher predictions and feature-space compactness and separation objectives, avoiding direct dependence on saturated sigmoid gradients. Regarding Naive Adaptive, it is a training-free baseline that selects the Top-30% patches using layer-12 anomaly scores for deeper inference, while retaining layer-12 predictions for exited patches. Unlike Naive Adaptive, ATS selects tokens using CLS attention and value-vector norms through inverse transform sampling and allows selected tokens to attend to all tokens at the pruning boundary.

W6: Efficiency Reporting and Baseline Reproducibility.

Thank you for your constructive comments. We provide additional experimental results and clarifications regarding efficiency measurement and baseline reproducibility.

Efficiency Measurement and Reporting:

With a fixed input resolution of 518 × 518 and a 30% patch forwarding ratio, Fast-ZSAD has consistent theoretical computation across datasets. The complete AD-DINOv3 baseline requires 740.110 GFLOPs per image, while Fast-ZSAD requires 475.136 GFLOPs. Although the speedup ratios vary across datasets, these variations correspond to only minor differences in measured inference latency under our evaluation setting. These small latency variations may be influenced by GPU runtime conditions, workload scheduling, and measurement fluctuations, despite identical theoretical FLOPs. We report the absolute latency and throughput, together with their standard deviations and variances.

DatasetLatency: AD-DINOv3Latency: OursThroughput: AD-DINOv3Throughput: Ours
MVTec AD52.68±1.54 (2.37)33.34±1.33 (1.77)18.98±0.55 (0.31)29.99±1.20 (1.43)
VisA54.09±2.07 (4.28)34.19±1.04 (1.08)18.49±0.71 (0.50)29.25±0.89 (0.79)
BTAD53.24±1.14 (1.30)34.31±1.05 (1.10)18.78±0.40 (0.16)29.15±0.89 (0.80)
MPDD53.69±1.24 (1.54)33.97±1.11 (1.23)18.63±0.43 (0.19)29.44±0.96 (0.93)
MedAD54.17±2.01 (4.04)32.24±1.03 (1.06)18.46±0.68 (0.47)31.02±0.99 (0.98)

Latency is measured in ms/image and throughput in images/s. Values are reported as mean ± SD (variance).

Reproducibility of RareCLIP:

RareCLIP maintains an online prototype memory bank that is sequentially updated using previously processed images. We follow this original inference procedure for both RareCLIP and Fast-ZSAD to ensure a fair comparison. For MPDD and MedAD, we reuse the dataset loading and evaluation interfaces from AA-CLIP without modifying RareCLIP's detection mechanism. We will release the complete evaluation code after the review.

Comparison with the Original Baselines:

We appreciate the reviewer's suggestion to compare our reproduced baselines with the originally reported results. Since the original AA-CLIP and AD-DINOv3 papers do not provide publicly available pretrained checkpoints, we reproduced both baselines following the training protocols and configurations specified in their respective papers. We compare their performance with the originally reported results below.

DatasetAA-CLIP I-AUROCAA-CLIP P-AUROCAD-DINOv3 P-AUROCAD-DINOv3 P-F1
MVTec AD90.50 / 88.5991.90 / 91.4391.60 / 92.5750.10 / 53.59
VisA84.60 / 78.1595.50 / 94.9195.60 / 95.8337.70 / 36.72
BTAD94.80 / 95.2897.00 / 96.1293.50 / 95.3651.70 / 51.51
MPDD75.10 / 71.1596.70 / 95.5996.20 / 92.9438.80 / 39.32

Each cell reports Original Paper / Our Retrained Baseline (%). Only metrics available in the original papers are included.

Neither original paper reports results on MedAD, so a direct comparison is unavailable. Finally, the original AD-DINOv3 paper focuses on pixel-level anomaly detection and does not explicitly specify an image-level scoring rule. We therefore adopt the image-level scoring strategy from RareCLIP to enable image-level evaluation.

Q4: Image-Level and Pixel-Level Metrics with Variance Across Seeds.

Thank you for your valuable suggestion. We report the image-level and pixel-level metrics as mean ± standard deviation over five independent router-training seeds for AACLIP, AD-DINOv3, and RareCLIP, as shown below.

Results on AACLIP (mean ± std):

DatasetI-AUROC ↑I-AP ↑I-F1 ↑P-AUROC ↑P-AP ↑P-F1 ↑
MVTec AD89.86 ± 0.1495.36 ± 0.1091.58 ± 0.1189.67 ± 0.2645.49 ± 0.1047.11 ± 0.07
VisA77.94 ± 0.3882.21 ± 0.3279.83 ± 0.3694.82 ± 0.0623.82 ± 0.1329.42 ± 0.24
BTAD93.34 ± 0.3396.75 ± 0.1191.85 ± 0.1893.98 ± 0.5351.71 ± 0.4355.54 ± 0.32
MPDD71.99 ± 0.3577.12 ± 0.1878.87 ± 0.3494.88 ± 0.3324.74 ± 0.1630.69 ± 0.11
MedAD74.83 ± 0.1184.57 ± 0.1382.78 ± 0.0794.72 ± 0.2836.40 ± 0.1140.37 ± 0.05

Results on AD-DINOv3 (mean ± std):

DatasetI-AUROC ↑I-AP ↑I-F1 ↑P-AUROC ↑P-AP ↑P-F1 ↑
MVTec AD91.93 ± 0.1396.48 ± 0.0992.73 ± 0.2090.65 ± 0.4649.76 ± 0.1350.74 ± 0.09
VisA88.38 ± 0.2290.17 ± 0.5884.96 ± 0.4093.53 ± 0.6130.20 ± 0.2536.31 ± 0.38
BTAD89.40 ± 0.4587.56 ± 0.4386.46 ± 0.4093.42 ± 0.1844.51 ± 0.3748.35 ± 0.23
MPDD77.36 ± 0.3583.28 ± 0.2682.51 ± 0.2293.90 ± 0.1533.52 ± 0.4039.60 ± 0.29
MedAD67.36 ± 0.1473.81 ± 0.1079.30 ± 0.1094.06 ± 0.3835.53 ± 0.3137.35 ± 0.37

Results on RareCLIP (mean ± std):

DatasetI-AUROC ↑I-AP ↑I-F1 ↑P-AUROC ↑P-AP ↑P-F1 ↑
MVTec AD98.15 ± 0.1998.83 ± 0.2097.37 ± 0.2098.15 ± 0.0869.69 ± 0.0765.91 ± 0.40
VisA91.95 ± 0.1693.44 ± 0.2989.04 ± 0.1798.29 ± 0.0842.37 ± 0.2547.42 ± 0.21
BTAD95.15 ± 0.2698.76 ± 0.3295.79 ± 0.1397.35 ± 0.1161.69 ± 0.4461.13 ± 0.23
MPDD71.32 ± 0.2477.19 ± 0.3180.94 ± 0.3596.60 ± 0.2029.83 ± 0.1132.41 ± 0.14
MedAD69.52 ± 0.1679.06 ± 0.1781.55 ± 0.1396.05 ± 0.1236.47 ± 0.3941.58 ± 0.22
Reviewer_uAMsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper proposes Fast-ZSAD, a patch-wise adaptive inference framework for accelerating Transformer-based zero-shot anomaly detection. A learned router estimates patch-level anomaly evidence at an intermediate layer, allowing confidently normal patches to exit early while suspicious patches proceed to deeper layers. To make this work, the authors introduce a contrast-aware regularization to counteract the normal-patch imbalance during router training, and a depth-spatial feature correction module to mitigate representation drift caused by sparse deep inference. Experiments across several ZSAD backbones and industrial/medical benchmarks report moderate inference speedups with largely preserved detection and localization accuracy.

Critical strengths
  • Inference efficiency for ZSAD is genuinely underexplored, and the observation that anomaly evidence is spatially sparse (Fig. 1a) while normal-patch scores stabilize across layers is a sensible empirical basis for patch-wise adaptive computation.
  • The method is instantiated on three different ZSAD architectures and evaluated on both industrial and medical benchmarks, which goes beyond the single-backbone evaluation common in this area.
Critical weaknesses
  • The headline speedup is only ~1.6–1.8×, achieved with three new modules, an extra training stage (router distillation on a source dataset), and multiple hyperparameters (ρ, routing layer r, λ_sd, entropy factor α). Notably, the paper's own baselines sometimes match or beat this trade-off trivially: DToP on RareCLIP (Table 3) reaches 1.06× speedup with essentially no accuracy loss, and on AACLIP the method's mScore improvement over the baseline (70.58 → 70.61 average) is well within run-to-run noise — no variance or significance analysis is reported anywhere. The claim of "improving performance while accelerating" is therefore not convincingly supported.

  • All latency numbers come from a single RTX A6000 with no statement about batch size, warm-up, or whether the reported ratios are wall-clock or FLOP-based. Dynamic token-count methods are notoriously hard to realize as wall-clock speedups due to gather/scatter overhead and variable-length attention; the paper gives no evidence the measured speedups hold under batched, production-like inference (e.g., per-image speedup at batch 1 says little about throughput, which is the paper's stated motivation). The fact that Cropr shows speedups below 1× on AACLIP (Table 1) without any explanation further undermines confidence in the measurement protocol.

  • The router is trained on VisA and evaluated on other datasets (and vice versa). While this follows a cross-dataset generalization protocol, the paper does not discuss the practical implication: deploying Fast-ZSAD on a truly unseen category requires an available labeled/curated anomaly dataset for router training and full-depth baseline predictions as supervision. This is a meaningful deployment burden for an "efficiency" method and weakens the central practicality argument. A router trained only with distillation from the frozen baseline on unlabeled data (a much more deployable setting) is not evaluated.

  • CAR is a center-pull/separation objective closely related to standard contrastive and one-class (e.g., Deep SVDD-style) regularizers; DSFC is an adaptive multi-layer feature fusion in the spirit of hypercolumns/FPN-style aggregation with heuristic spatial-density weighting. Neither the "anomaly responsibility" weighting (Eq. 10–13) nor the DSFC weighting (Eq. 14–17) is derived from a principled objective; they are heuristic combinations validated only empirically, and the entropy-based image-level factor α is justified only descriptively.

Questions for authors

Please refer to weaknesses.

4 comments

We sincerely thank the reviewer for the valuable comments, which have helped us improve our work. We address your concerns as follows.

W1: Performance–Efficiency Trade-off, Complexity, and Stability.

Thank you for your thoughtful comments. We would like to clarify the performance–efficiency trade-off and computational overhead Fast-ZSAD.

Regarding the performance–efficiency trade-off. The primary advantage of Fast-ZSAD lies in achieving a favorable trade-off between inference speed and anomaly detection performance, rather than maximizing speedup alone. We provide a comparison with representative acceleration methods on RareCLIP.

Accuracy–efficiency comparison on RareCLIP (MVTec AD):

MethodSpeedup (×) ↑mScore (%) ↑
RareCLIP (Original)1.0088.06
ATS1.7976.50
DToP1.0679.24
Cropr1.2178.36
Fast-ZSAD (Ours)1.8687.98

As shown above, Fast-ZSAD achieves a 1.86× speedup with only a 0.08-point mScore decrease. In contrast, ATS achieves a comparable 1.79× speedup but incurs an 11.56-point drop. DToP achieves only a 1.06× speedup with an 8.82-point drop, while Cropr achieves a 1.21× speedup with a 9.70-point drop. These results demonstrate that Fast-ZSAD achieves a more favorable accuracy–efficiency trade-off, substantially accelerating inference while largely preserving detection performance.

On method complexity and computational overhead. Although Fast-ZSAD introduces additional components, their computational overhead is relatively small, as shown in the following latency breakdown.

MethodBackbone (ms)Adapter (ms)Router (ms)DSFC (ms)Other (ms)Total (ms)Speedup (×)
AACLIP (Original)78.682.000.000.001.1381.811.00
Fast-ZSAD (Ours)45.041.590.551.931.0850.191.63

As shown above, Fast-ZSAD reduces backbone latency by 33.64 ms, while the additional Router MLP and DSFC together introduce only 2.48 ms of computation. Consequently, the end-to-end inference latency decreases from 81.81 ms to 50.19 ms, achieving a 1.63× speedup. This demonstrates that the computational savings from adaptive inference substantially outweigh the overhead introduced by our lightweight components.

Regarding experimental stability. We conducted additional experiments on AACLIP using five independent random seeds across all five evaluation benchmarks. The results are reported as mean ± standard deviation below.

Results on AACLIP (mean ± std):

DatasetI-AUROC ↑I-AP ↑I-F1 ↑P-AUROC ↑P-AP ↑P-F1 ↑
MVTec AD89.86 ± 0.1495.36 ± 0.1091.58 ± 0.1189.67 ± 0.2645.49 ± 0.1047.11 ± 0.07
VisA77.94 ± 0.3882.21 ± 0.3279.83 ± 0.3694.82 ± 0.0623.82 ± 0.1329.42 ± 0.24
BTAD93.34 ± 0.3396.75 ± 0.1191.85 ± 0.1893.98 ± 0.5351.71 ± 0.4355.54 ± 0.32
MPDD71.99 ± 0.3577.12 ± 0.1878.87 ± 0.3494.88 ± 0.3324.74 ± 0.1630.69 ± 0.11
MedAD74.83 ± 0.1184.57 ± 0.1382.78 ± 0.0794.72 ± 0.2836.40 ± 0.1140.37 ± 0.05

The small standard deviations across five benchmarks demonstrate the stability of Fast-ZSAD across different random seeds.

W2: Inference Measurement Protocol and Practical Speedup.

Thank you for your valuable comments. We would like to clarify our inference measurement protocol and the observed Cropr results.

As detailed in Appendix B.2, all experiments are conducted on a single NVIDIA RTX A6000 GPU using FP32 precision, a batch size of 32 unless otherwise specified, and five warm-up iterations before timing. We measure actual end-to-end wall-clock inference time rather than estimating speedups from theoretical FLOPs.

Regarding Cropr, its speedups below 1× on AACLIP (Table 1) can be attributed to additional computational and memory-access overhead from token importance scoring, sorting, gathering, and repeated feature and index reorganization. Since AACLIP requires multi-layer features for dense anomaly localization, pruned tokens must be restored and reordered at the corresponding feature extraction layers to preserve spatial correspondence. Furthermore, Cropr restores the complete token sequence before the final Transformer layer, limiting the computational savings from token pruning. As a result, these overheads can outweigh the computational savings, leading to a 0.74× speedup on AACLIP (MVTec AD). In contrast, although Fast-ZSAD introduces additional computational overhead through patch routing, token selection, and DSFC, these components are lightweight, and the computational savings substantially outweigh the added overhead, resulting in a 1.63× speedup on AACLIP.

W3: Router Training Requirements and Practical Deployment.

Thank you for highlighting the practical deployment implications of router training. We would like to clarify that although the source datasets contain ground-truth annotations, our router training does not use any image-level labels or anomaly masks. Instead, the router is trained using anomaly-map predictions generated by the frozen baseline as supervision, without requiring manual annotations. Once trained, the router can be directly applied to unseen target categories without additional training, and the teacher model is not required during inference.

W4: Design Rationale and Validation of CAR and DSFC.

Thank you for your valuable comments. We clarify the design motivations of CAR and DSFC in addressing the challenges of patch-wise adaptive inference.

For CAR, our key observation is the severe mismatch between anomaly evidence and optimization gradients. As shown in Fig. 2, the top 5% of patches account for approximately 82% of anomaly evidence but only 31% of gradient magnitude, causing router training to be dominated by abundant normal patches. CAR addresses this imbalance through anomaly-evidence-guided responsibilities, where patch-level weights emphasize anomaly-relevant regions and the image-level factor α adjusts anomaly-related regularization based on the concentration of anomaly evidence across patches. Notably, α is equivalent to the normalized KL divergence between the teacher's spatial anomaly distribution and a uniform distribution, providing an information-theoretic interpretation of the image-level weighting.

For DSFC, our key observation is that sparse deep inference alters the spatial context of forwarded patches, leading to representation deviation and anomaly-response drift (Fig. 3). To address this issue, DSFC jointly considers local spatial support and cross-depth feature evolution to adaptively determine the contribution of features from different depths for each forwarded patch.

After reading the rebuttal, I find the additional timing results and baseline comparisons helpful, and my concern about the need for manual annotations has been resolved. However, my main concern about the methodological contribution remains.

The ablations support the effectiveness of CAR and DSFC, but their advantages over simpler alternatives are still not sufficiently established. In particular, the authors attribute the severe degradation of the map-only router on AD-DINOv3 to sigmoid saturation, without testing whether directly addressing this optimization issue would recover much of the performance. This leaves uncertainty about how much of the large gain attributed to CAR depends on the specific regularization proposed. Similarly, the DSFC ablations show that its components contribute, but provide limited evidence that the particular weighting design is necessary beyond simpler multi-depth fusion.

I acknowledge the practical acceleration benefits, but the rebuttal does not sufficiently resolve my concerns about novelty and the justification of the specific designs. I therefore maintain my original score.

Dear Reviewer uAMs,

Thank you for your further feedback and for recognizing our practical acceleration benefits. We would like to further clarify the motivations and contributions of CAR and DSFC.

Regarding CAR. CAR is designed to address normal-dominated router optimization in ZSAD, rather than specifically to resolve sigmoid saturation in AD-DINOv3. CAR leverages relative anomaly evidence to encourage compact normal representations and separate anomaly-related patches from the normal distribution. Its effectiveness is also demonstrated on AACLIP, where CAR improves the mScore from 69.46 to 74.32 (Table 4), despite the absence of severe Map-only Router degradation. Furthermore, Fig. 5 demonstrates the complementary contributions of the compactness and separation objectives on both models. These results support the effectiveness of CAR in addressing the more general challenge of normal-dominated router optimization in ZSAD.

Regarding DSFC. DSFC is specifically designed to address representation deviation under sparse deep inference. Fixed layer weighting assigns the same fusion weights to all patches, regardless of their spatial context and cross-depth feature evolution. In contrast, DSFC jointly considers local spatial support, transition magnitude, and transition consistency to adaptively determine the fusion weights for individual patches. The spatial support regulates the strength of the depth adjustment, while transition consistency reduces the influence of large but locally inconsistent feature changes. As shown in Fig. 6, removing depth modeling reduces the mScore from 76.51 to 75.20 on AACLIP and from 78.53 to 77.05 on AD-DINOv3. Our additional individual signal ablations further demonstrate the complementary contributions of the four signals on both models.

We hope these clarifications further explain the task-specific motivations and empirical evidence underlying our methodological designs. Thank you again for your valuable feedback.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Rebuttal Summary of Submission 102

We sincerely thank all reviewers for their valuable and constructive comments. Below is a summary of our responses during the rebuttal phase.

We are encouraged by the reviewers' recognition of the significance, motivation, and experimental validation of our work:

1. Significance and motivation. The reviewers acknowledged that inference efficiency for ZSAD is “genuinely underexplored” (uAMs) and that inference cost is “a real obstacle for high-resolution ZSAD models” (81Mn). Reviewer uAMs also recognized our observations on anomaly sparsity and normal-patch score stability as “a sensible empirical basis for patch-wise adaptive computation.”

2. Comprehensive experimental validation. The reviewers acknowledged our extensive evaluation across three ZSAD models on both industrial and medical benchmarks (uAMs, 81Mn, wmye), which goes beyond the single-backbone evaluation common in this area (uAMs).

Following the reviewers' constructive suggestions, we validated our hyperparameter choices using a validation split from the source domain and evaluated the selected configurations across all five benchmarks without further tuning (81Mn-W3/Q2). We also provided further analyses of hyperparameter sensitivity and layer priors (wmye-W2/Q4), and clarified the router training protocol (uAMs-W3) and baseline comparisons (uAMs-W2, wmye-Q3). Below, we summarize our responses to the two major concerns raised by the reviewers.

1. Concerns about the performance–efficiency trade-off and practical efficiency (uAMs, 81Mn, wmye).

  • Performance–efficiency trade-off. We provided comparisons with existing token acceleration methods (including ToMe), reduced input resolutions, a smaller backbone, and uniform early exiting (uAMs-W1, 81Mn-W4/Q1, wmye-W5/Q2), with accuracy–speed trade-off curves in Appendix C.1 (Fig. 8). These results demonstrate that Fast-ZSAD achieves a favorable performance–efficiency trade-off compared with the evaluated alternatives.
  • Efficiency validation. We clarified the end-to-end wall-clock measurement protocol described in Appendix B.2 (uAMs-W2) and provided additional component-wise latency breakdowns (uAMs-W1), together with FLOPs, absolute latency, throughput, and runtime variability (81Mn-W6). These results demonstrate that the computational savings of Fast-ZSAD substantially outweigh the overhead introduced by its additional components, resulting in practical end-to-end acceleration.
  • Performance stability. We reported the means and standard deviations of all six evaluation metrics over five independent router-training seeds across three ZSAD models and five benchmarks (uAMs-W1, 81Mn-W2/Q4), further demonstrating the stability of Fast-ZSAD.
  • Routing strategy. We clarified that patches exiting early retain their learned anomaly predictions (wmye-W1). Additional experiments with a batch size of 1 comparing fixed ratio and absolute threshold routing further support our choice of routing strategy (wmye-Q1).

2. Concerns about methodological novelty and design rationale (uAMs, 81Mn, wmye).

Fast-ZSAD is an adaptive inference framework tailored to ZSAD, where dense anomaly predictions must be preserved while reducing computation (81Mn-W1). To this end, it addresses two challenges insufficiently considered by generic token acceleration methods: normal-dominated router optimization and representation deviation under sparse deep inference.

  • CAR. We clarified how CAR mitigates normal-dominated router optimization by leveraging anomaly evidence to encourage compact normal representations and separate anomaly-related patches from the normal distribution (uAMs-W4). We also analyzed the potential causes of Map-only Router degradation (81Mn-W5/Q3, wmye-W4). These analyses, together with the ablation results in Tables 5 and 7, demonstrate the effectiveness of CAR in improving routing quality and downstream detection performance.
  • DSFC. We clarified the design of DSFC for ZSAD, which combines spatial support with feature evolution across layers to compensate for representation deviation under sparse deep inference (uAMs-W4, 81Mn-W1). We also explained the distinction between prediction drift and feature evolution across layers (wmye-W3). Additional ablation studies on AACLIP and AD-DINOv3 further support the contribution of each of the four signals in DSFC (wmye-W2).

We are encouraged by the positive feedback from Reviewer wmye, who explicitly stated: “The extensive explanations and analyses clarify and address most of my concerns, so I decide to improve my original score.” While we have not yet received further feedback from Reviewers uAMs and 81Mn, we believe that our responses substantially address the reviewers' concerns and that the additional experiments and analyses further strengthen our work. Finally, we sincerely thank all reviewers for their insightful feedback and the ACs for their time, effort, and consideration throughout the review process.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.