acceptodds
Under review as a conference paper at ICLR 2027

Rethinking Structural Anomaly Detection: From Decision Boundaries to Projection Operators

Abstract

Many anomaly detection methods either rely on estimating a probability density or learning an enclosing decision boundary, implicitly assuming that normal data occupies a region of non-zero volume in the ambient space. In contrast, structural anomaly detection deals with perceptual data such as images, where normal samples concentrate near a low-dimensional manifold and anomalies correspond to deviations outside its support. This view reveals a mismatch between the volumetric bias of existing methods and the geometric structure of the data, often resulting in degraded performance in practice. To address this mismatch, we formalize the task as learning a projection operator onto the manifold of normal samples and define a sample as anomalous if it is altered by this projection. This formulation naturally integrates the inductive bias of manifold-supported data and reframes anomaly detection in terms of a projection residual, thereby resolving issues arising from modeling degenerate distributions. Notably, it provides a unifying interpretation of reconstruction-based methods by explaining their success and failure in terms of projection quality. In particular, it explains the strong generalization ability of projection-aligned models to unseen anomalies as a consequence of their contraction behavior that suppresses off-manifold variation. Empirically, we demonstrate that such models substantially outperform a range of boundary-based methods and achieve state-of-the-art performance while producing segmentation masks with fine, accurate boundaries.

Primary area:
other topics in machine learning (i.e., none of the above)
Keywords:
visual anomaly detection, structural anomalies, projection operator, reconstruction, manifold learning

Reviews

Among:
Shape:
1234this paper, Bottom 25%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.75 over 4 reviews, above 25% of the others.
Reviewer_R4exOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

This paper establishes theoretical foundations for detecting off-manifold anomalies and proposes an iterative reconstruction framework to replace anomalous pixels with normal pixel counterparts. Specifically, it shows that the Bayesian optimal reconstruction operator exactly recovers uncorrupted normal pixels and replaces anomalous pixels with the posterior mean of the corresponding normal pixels before corruption. Building upon on this observation, they introduce iterative reconstruction during inference to suppress anomalous information during entire reconsturction process. They evaluate proposed methodology on MVTecAD and VisA, demonstrating that their proposed PAE achieve competitive performance compared with previous reconsturction-based methods.

Critical strengths

S1: The paper is well-written and easy to follow.

S2: This paper revisits prevalent reconstruction-based AD approaches in terms of manifold projection. The established theoretical grounding helps readers to understand how reconstruction-based AD works and its benefits compared with other boundary-based methods.

S3: Experimental results show that the proposed methodology achieves better performance than boundary-based approaches and remains competitive compared with previous reconstruction-based methods.

Critical weaknesses

W1: It is unclear whether the statement in L291–292 follows from Theorem 1. As f^*_{N(x_hat)}(x_hat) = x_hat_{N(x_hat)} in Eq. 7 only proves the forward implication of Eq. 5, it is necessary to show that f^*_{A(x_hat)}(x_hat) = E_{p(x | x_hat)}[x_{A(x_hat)}] also proves the reverse implication of Eq. 5. This may require an additional assumption beyond the scope of Theorem 1.

W2: As they introduce an extra segmentation module to extract foreground regions, which is not widely used in the field of SAD, it is better to include results without such a segmentation module for a fair comparison.

W3: The current experimental evaluation lacks comparison between widely used methodologies. Because the proposed PAE relies on pseudo-anomalies generated using DTD texture datasets, it should be compared with other DTD-related AD methods: TransFusion [1], PRN [2], DeSTSeg [3]. Also, as this paper focuses on the "corruption-then-recover" paradigm of AD, it would be better to add more comparisons against diffusion-based methods, including DeCoDiff [4], DiffusionAD [5].

W4: While the proposed PAE achieves better performance on MVTecAD and VisA, it is still unclear whether this approach scales to more difficult and large-scale datasets, including MANTA [6], Real-IAD [7].

W5: The current experimental analysis does not seem to sufficiently support the claim in this paper. Specifically, there is no analysis of corruption strategies in the provided manuscript. Because weaker corruption may induce identical shortcuts and degrade detection performance, such a corruption strategy plays a key role for reconstruction-based AD. Also, more implementation details should be provided, including feature extractors, number of iterations during inference, and other details.

References

[1] Fučka+, "TransFusion – A Transparency-Based Diffusion Model for Anomaly Detection." ECCV'24, https://doi.org/10.1007/978-3-031-72761-0_6.

[2] Zhang+, "Prototypical Residual Networks for Anomaly Detection and Localization." CVPR'23, https://doi.org/10.1109/CVPR52729.2023.01562.

[3] Zhang+, "DeSTSeg: Segmentation Guided Denoising Student-Teacher for Anomaly Detection.", CVPR'23, https://doi.org/10.1109/CVPR52729.2023.00381.

[4] Beizaee+, "Correcting Deviations from Normality: A Reformulated Diffusion Model for Multi-Class Unsupervised Anomaly Detection", CVPR'25, https://doi.org/10.1109/CVPR52734.2025.01778.

[5] Zhang+, "DiffusionAD: Norm-Guided One-Step Denoising Diffusion for Anomaly Detection", TPAMI, https://doi.org/10.1109/TPAMI.2025.3570494.

[6] Fan+, "MANTA: A Large-Scale Multi-View and Visual-Text Anomaly Detection Dataset for Tiny Objects", CVPR'25, https://ieeexplore.ieee.org/xpl/conhome/11091818/proceeding.

[7] Wang+, "Real-IAD: A Real-World Multi-View Dataset for Benchmarking Versatile Industrial Anomaly Detection", CVPR'24, https://doi.org/10.1109/CVPR52733.2024.02159.

Questions for authors

All questions mostly correspond to the weakness section.

Q1: Could you clarify whether L291-L292 can be directly derived from Theorem 1 or still requires additional assumptions?

Q2: How about detection performance on VisA without an extra segmentation module?

Q3: Could you compare the proposed method with an additional 2 or 3 methods as listed in W3?

Q4: How about performance on large-scale datasets as mentioned in W4?

Q5: What kind of corruption strategy best suits reconstruction-based AD?

Q6: Could we extend the proposed framework without pseudo-anomalies? i.e., fully unsupervised AD.

Q7: Could you provide some efficiency numbers for the proposed PAE? It would be better to compare efficiency with other methods, as the proposed PAE introduces recurrent updates during inference.

In summary, this paper is well-written, and it provides a clear explanation of the success of the prevalent reconstruction-based AD methods. However, the proposed method, PAE, seems to be limited in some kind of corruption strategy and lacks broader empirical verification. If the authors have effectively addressed all of my concerns/questions, I would be willing to increase my score.

Reviewer_v9mzOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a method for structural anomaly detection (SAD) that differs from the prevailing boundary-based and reconstruction-based approaches. By formulating SAD as the learning of a projection operator onto the manifold of normal data, this paper derives the key inductive biases required for detection and localization. Experimental results demonstrate that the resulting projection autoencoder achieves performance comparable to state-of-the-art methods on industrial anomaly detection datasets.

Critical strengths
  • The theoretical derivation is highly detailed.
  • The paper is clearly written.
Critical weaknesses

It lacks enough support from diverse scenarios and ablation studies.

Questions for authors
  • This paper introduces a general strategy for improving anomaly detection. However, the method should also apply to anomaly detection tasks of medical imaging. The scenes are mentioned by the authors in lines 36–37.

  • There is a lack of ablation studies on key parameters and modules. In addition, the analysis of computational cost is not reported.

  • The foreground segmentation preprocessing was adopted in this method. But, no ablation study was conducted for this preprocessing module. In addition, it is not mentioned whether the preprocessing was used for these baseline models.

1 comment
  • Scope and medical imaging. We mention medical imaging solely to illustrate the practical relevance of structural anomaly detection. The paper's claims require no empirical validation in that domain, and treating the absence of medical imaging experiments as a critical weakness is therefore unjustified. The proofs establish the theoretical results, and the industrial defect detection benchmarks provide sufficient empirical support for the paper's claims.
  • Ablations and computational cost. Our main contribution is a theoretical framework for a family of reconstruction methods. The benchmark results serve to demonstrate competitive performance. A systematic ablation was outside the scope of this study and would duplicate analyses in prior work. We will report computational costs in the paper.
  • Foreground preprocessing. To address annotation omissions in VisA, we used foreground segmentation to restrict PAE’s predictions to the inspected objects. In selected VisA categories, anomalous background patterns are left unlabeled, yet the evaluation protocol still scores those pixels. Methods that ignore these patterns can therefore score higher without localizing object defects any better. Foreground masking was the simplest workaround, though applying it only to PAE limits direct comparability with the published baseline scores. Reporting scores with and without masking would quantify the change, but interpreting it would require a substantial sample-by-sample visual inspection to distinguish detections of unlabeled background anomalies from genuine false positives. We will clarify these details in the paper.
Reviewer_3L4JOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper reframes structural anomaly detection as learning an approximate projection onto the manifold of normal data. It motivates fixed-point behavior and minimal correction support as useful inductive biases, trains an autoencoder on normal images and synthetically corrupted versions, and scores anomalies using the reconstruction residual. It also analyzes when repeated application can contract corrupted inputs toward a fixed point. Experiments on MVTec AD and VisA report strong image-level and pixel-level AUROC, with a training setup based on a category-specific modified U-Net and synthetic corruptions. The method is competitive on several metrics, although its VisA region-overlap score is notably lower than several competing methods.

Critical strengths
  1. The paper gives a coherent geometric account of why reconstruction residuals can detect structural deviations and relates the account to explicit fixed-point and minimal-correction conditions. This is a useful conceptual contribution beyond reporting another benchmark score.
  2. The evaluation covers both MVTec AD and VisA and reports image-level detection as well as several localization metrics. The parameter and metric sensitivity analysis helps show where results are stable and where they depend on evaluation choices.
Critical weaknesses
  1. The theoretical characterization does not by itself guarantee that the practical U-Net trained with the stated corruption objective is a projection or a contraction. Those properties depend on assumptions about the normal manifold, corruption directions, uniqueness of anomalous support, and model capacity; the experiments offer examples but limited direct measurement of these properties.
  2. Aggregate AUROC is strong, but localization quality is mixed: on VisA, the reported P-PRO for PAE is 84.8, below the listed competing methods, despite high pixel AUROC and average precision. This matters because structural anomaly detection includes identifying defect regions, not only ranking pixels.
Questions for authors

Can you quantify fixed-point error and contraction toward normal data for the trained models on held-out normal and anomalous samples, and show whether iterative application improves the reported detection/localization metrics over one pass? This would connect the practical model more directly to the theoretical claims.

1 comment
  1. The theoretical characterization specifies the properties of the desired operator; it does not guarantee that a finite-capacity U-Net trained on finitely sampled corruptions satisfies them exactly. The corruption objective provides supervision toward this behavior. The experiments demonstrate its practical utility, while direct measurement of projection and contraction properties would provide additional evidence.

  2. We report multiple metrics because they capture different aspects of localization. Evaluating a method’s quality by focusing on one metric alone therefore requires caution. First, as discussed in the paper, annotation inaccuracies in VisA can substantially affect pixel-level evaluation. Second, PRO emphasizes region coverage, whereas pixel average precision accounts directly for precision. Methods achieving higher P-PRO but lower pixel average precision are therefore consistent with these different emphases, rather than demonstrating uniformly better localization. Third, integrated P-PRO also evaluates score rankings through a threshold sweep; it does not inherently provide a better assessment of segmentation quality. Our qualitative comparison with GLASS illustrates this distinction: PAE produces sharper, more precisely delineated anomaly masks despite its lower P-PRO score.

Q1. Yes, we could compute the fixed-point error on normal inputs. However, reporting that number would add little here because we have no reference for deciding how small the error must be to count as practical preservation. Visual inspection strongly suggests that the model behaves like the identity mapping in practice, since normal inputs and their reconstructions are indistinguishable to the human eye. Appendix Figure 7 provides several examples.

Regarding contraction, the proposed Inductive Bias I requires the autoencoder’s output to differ from its input in anomalous regions. This input–output difference is the detection signal, regardless of how close the reconstruction is to the normal manifold. Once a single pass separates anomalous pixels from normal pixels, further contraction can improve reconstruction fidelity, as illustrated in Figure 3, without being expected to improve scores based on pixel rankings. It can still produce sharper anomaly maps that are easier to interpret visually. Figure 8 in Appendix demonstrate the qualitative difference between GLASS an PAE.

Reviewer_E276OpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper interprets structural anomaly detection as learning a projection onto normal data, with reconstruction residuals used for detection and localization. It proposes fixed-point consistency and minimal correction support as desirable properties, analyses denoising and iterative reconstruction, and trains a projecting autoencoder using synthetic texture corruptions. Experiments on MVTec AD and VisA report competitive detection AUROC and improved pixel-level average precision.

Critical strengths
  1. The idea to distinguish between detecting an anomalous image and modifying only its defective regions is useful.

  2. Minimal correction support offers an intuitive objective for explaining why accurate reconstruction alone may not yield accurate localization.

Critical weaknesses
  1. The writing and exposition in this paper require further refinement. In particular, the authors should clearly summarize the main contributions and related work.

  2. Although Proposition 1 correctly notes that a continuous feature map pulls an open acceptance region back to an open input-space region (Appendix A.1), a continuous PAE with a positive residual threshold also accepts an open neighborhood around each exactly reconstructed normal point. Thus, the practical thresholded method also accepts some off-manifold inputs. This does not negate the value of learning geometry-aligned residuals, but it undermines the categorical claim that projection resolves a structural mismatch inherent to boundary methods; the meaningful comparison is the geometry and robustness of the learned acceptance regions at comparable normal-data acceptance rates.

  3. Section 4 uses an independently trained foreground-segmentation U-Net for selected VisA categories, but does not specify its supervision, data split, affected categories, or whether baselines receive equivalent masks.

  4. Equation (1) explicitly adopts the loss of Bauer et al. (2024), yet Table 1 does not include that method as a direct baseline.

  5. The experiments are limited in scope, and the improvements over the baselines are not significant.

Questions for authors
  1. How should the claimed advantage over volumetric acceptance regions be interpreted when positive residual thresholds are used? Could the comparison instead be framed at matched normal-data acceptance rates?

  2. Which VisA categories use foreground masks? How is the segmentation model trained, and do the baselines receive the same preprocessing? How do the scores change without this preprocessing, and how much of the P-PRO deficit can be attributed to the annotation issue illustrated in the paper?

  3. In Table 1, why does the proposed PAE method show significant improvement only on P-AP? Does this indicate a preference of PAE for this metric or setting?

Comments on the paper

2 comments

Dear Reviewers,

Please respond to the authors rebuttal. Thank you.

Regards, AC

Thank you for providing additional details. However, these responses did not adequately address most of my concerns. In particular, please provide a revised manuscript that reflects these answers (e.g., Q1, Q7). Also, most of my questions remain unclear and need further verification through additional empirical results. Therefore, I'm inclined to maintain my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.