- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper explains the well-known trade-off of reconstruction-based anomaly detection (AD) from the perspective of manifold learning based on the Pursuit of Subspaces (PoS) hypothesis, revealing that reconstruction alone does not suffice for AD, due to meet preference and join blindness. To avoid these failures, they propose straightforward modifications to the training objective, termed “Dynamic Push and Pull” and “nested manifold carving”. While the former encourages the model to avoid identical mappings by adding a denoising objective, the latter enforces the model to reconstruct recursively in the previously learned latent space. Their experiments cover AD tasks, including MNIST, CIFAR-10, and Arrhythmia, and also consider detecting unseen image degradation and more general representation learning on ECG data.
Critical strengths
S1: Theoretical interpretation of reconstruction-based AD. Under the PoS hypothesis, this paper explains well-known failure cases for reconstruction-based AD. Existing works only provide a rough explanation of these failures: excess capacity leads to identical mappings, while less capacity leads to poor reconstruction for nominal samples. This work instead gives an explanation through a manifold perspective, offering an intuition for how learned deep neural networks fall into such failures.
S2: Novelty. Proposed methodologies composed of push-and-pull and nested learning have a certain novelty in the field of reconstruction-based AD. Also, this paper provides a guarantee that a push-and-pull successfully avoids prevalent failure modes in Lemma 1.
S3: Thorough experimental ablations. They ablate the performance of the proposed method on several standard AD benchmarks and verify with diverse methodological designs. Such a broad range of ablations makes the contribution of each module clear.
Critical weaknesses
W1: Connection to diffusion models and comparison. The proposed push-and-pull resembles an invention of the diffusion-based AD in terms of its denoising procedure for avoiding identical mappings. Recent works [1-5] demonstrate that diffusion-based AD can yield striking AD performance in a reconstruction-based manner. Therefore, it should be encouraged to organize the relationships or differences between them and compare against them to understand the strength of the proposed method.
W2: Evaluations on practical AD benchmarks. Current experimental settings and benchmarks seem to be biased toward OOD detection literature. These settings measure the capability for detecting semantic outliers rather than tiny, local anomalous features. To measure the practical AD performance of these methods, it is suggested to evaluate on widely used image AD benchmarks, such as MVTecAD [6], and VisA [7]. Also, recent AD works adopt pre-trained feature extractors such as DINOv2 [8] to leverage their general capabilities on various vision tasks. Could the proposed method be effective in such a latent space?
W3: technical details. An apparent difference between diffusion-based AD is the adoption of "dynamic push margin" as in Eq. 2, and it consistently improves performance across several OOD detection datasets as shown in Fig. 4. While it serves as an extra separation signal for AD, this paper also adds a masking procedure in its MNIST experiments in Tab. 1. However, such a masking procedure is not stated in their methodology section and lacks more detailed information on this in the main manuscript.
References
[1] Baizaee+, "Correcting Deviations from Normality: A Reformulated Diffusion Model for Multi-Class Unsupervised Anomaly Detection", CVPR'25, https://arxiv.org/abs/2503.19357.
[2] Pintilie+, "Time Series Anomaly Detection using Diffusion-based Models", ICDM'23, https://arxiv.org/abs/2311.01452.
[3] Zhang+, "DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detection", TPAMI'25, https://arxiv.org/abs/2303.08730.
[4] Liu+, "Unsupervised Out-of-Distribution Detection with Diffusion Inpainting", ICML'23, https://arxiv.org/abs/2302.10326.
[5] Zhang+, "Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model", ICCV'23, https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Unsupervised_Surface_Anomaly_Detection_with_Diffusion_Probabilistic_Model_ICCV_2023_paper.pdf.
[6] Bergmann+, "MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection", CVPR'19, https://openaccess.thecvf.com/content_CVPR_2019/papers/Bergmann_MVTec_AD_--_A_Comprehensive_Real-World_Dataset_for_Unsupervised_Anomaly_CVPR_2019_paper.pdf.
[7] Zou+, "SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation", ECCV'22, https://arxiv.org/abs/2207.14315.
[8] Oquab+, "DINOv2: Learning Robust Visual Features without Supervision", TMLR'23, https://arxiv.org/abs/2304.07193.
Questions for authors
Q1: How about the relationship between the proposed method and prevalent diffusion-based AD?
Q2: How about the performance of the proposed method compared with these diffusion-based AD methods?
Q3: How about the performance on the practical AD benchmarks?
Q4: Could the proposed method be effective with extra feature extractors?
Q5: Does this paper incorporate the masking procedure for all main experiments? If so, please state it explictly in the methodological or experimental setup sections.
If the author addressed concerns (W1-3) and corresponding questions (Q1-5), I'm willing to increase my score. For the rebuttal, my major concern is its performance on practical AD benchmarks beyond OOD detection datasets. While this paper states, "we compare state-of-the-art unsupervised anomaly detection methods" in L132, it seems to lack a comparison with SoTA diffusion-based methods on practical AD benchmarks.
9 comments
Answers to Reviewer JLuL, Part II: Q3 and Q2 (Responses Continue)
Q3: How does the method perform on practical AD benchmarks?
We evaluated the method on the two requested benchmarks, MVTec AD and VisA, and report both image and pixel AUROC. All our results use the final checkpoint and require no anomalous validation data. The 0.83M model uses the same architecture as our RGB HQ image experiments. It is trained from scratch and uses only reconstruction scores, without a pretrained backbone, segmentation head, or auxiliary detection loss.
| Method | Params (M) | Synth. | Seg. head | Sel. | MVTec I | MVTec P | VisA I | VisA P |
|---|---|---|---|---|---|---|---|---|
| Reconstruction score, trained from scratch | ||||||||
| DRAEM Recon. AE† | 69.1 | yes | no | last | 83.9 | 89.7 | - | - |
| DRAEM Recon. AE + MSGMS† | 69.1 | yes | no | last | 90.7 | 93.4 | - | - |
| RIAD† | - | no | no | last | 91.7 | 94.2 | - | - |
| InTra† | 55 | no | no | val | 95.0 | 96.6 | - | - |
| Ours, reconstruction score, trained from scratch | ||||||||
| DAE, α = 0 | 0.83 | yes | no | last | 92.10 | 92.88 | 92.84 | 95.77 |
| AE-PP, α = 1 | 0.83 | yes | no | last | 92.42 | 93.28 | 92.76 | 96.36 |
| AE-PP, α = 2 | 0.83 | yes | no | last | 92.16 | 92.88 | 93.28 | 96.35 |
| Reconstruction plus segmentation head trained on synthetic anomalies | ||||||||
| DRAEM† | 97.4 | yes | yes | last | 98.0 | 97.3 | - | - |
| DRAEM‡ | 97.4 | yes | yes | last | 97.90 | 97.04 | 90.18 | 93.65 |
| DiffusionAD† | 160.0 | yes | yes | best | 99.7 | 98.7 | 98.8 | 98.9 |
| DiffusionAD‡ | 160.0 | yes | yes | last | 98.52 | 97.50 | 96.98 | 98.38 |
I and P denote image and pixel AUROC. Params denotes trained parameters. Synth. indicates training with synthetic anomalies. † denotes results reported in the original papers. ‡ denotes our reruns; the DRAEM result is averaged over three seeds. Sel. denotes checkpoint selection. Bold marks the best result among our models.
Within the same reconstruction score setting, namely training from scratch without a segmentation head, AE-PP reaches 92.42 MVTec image AUROC. This exceeds both DRAEM reconstruction variants, at 83.9 and 90.7, and RIAD, at 91.7. Only InTra is higher, at 95.0, and it uses validation checkpoint selection. On pixel AUROC, AE-PP exceeds the basic DRAEM reconstruction branch and nearly matches DRAEM with MSGMS, at 93.28 versus 93.4, while remaining below RIAD and InTra. Thus, within this reconstruction only group, our 0.83M AE-PP ranks second on image AUROC, surpassing both DRAEM reconstruction variants and RIAD.
On VisA, AE-PP with reaches 93.28 image and 96.35 pixel AUROC. It exceeds our rerun of the complete DRAEM detector, at 90.18 and 93.65, even though DRAEM uses 97.4M parameters and a separate segmentation head. The matched DAE comparison isolates the contribution of Push and Pull. AE-PP improves MVTec by 0.32 image and 0.40 pixel AUROC, and VisA by 0.44 image and 0.58 pixel AUROC.
Q2: How does performance compare with diffusion based AD?
The complete DiffusionAD detector remains numerically stronger, but it uses a substantially different pipeline with 160M trained parameters, synthetic anomaly masks, and a separate segmentation head. Our model contains 0.83M trained parameters, meaning that DiffusionAD has approximately 193 times as many trained parameters. Our model is trained from scratch and evaluated using the final checkpoint with direct reconstruction scoring. At inference, our model requires a single autoencoder forward pass followed by reconstruction scoring, whereas DiffusionAD evaluates its diffusion reconstructor at two predefined noise scales and then applies a separate segmentation network. Our pipeline therefore requires fewer network evaluations, while we restrict our quantitative efficiency comparison to parameter count.
Thus, AE-PP achieves competitive performance in a much smaller reconstruction only setting designed to isolate the PoS derived Push and Pull principle. Incorporating this principle into diffusion reconstructors and segmentation pipelines is an important direction for future work.
Answers to Q4 and Q5 continue in the following comments.
Answers to Reviewer JLuL, Part III: Q4 and Q5 (Responses Continue)
Q4: Is the method effective with pretrained feature extractors?
Yes. To test whether Push and Pull remains effective with a strong pretrained representation, we combine AE-PP with FITYMI. FITYMI fine-tunes an ImageNet-21K ViT-B/16 with synthetic anomalies. As in FITYMI + TOLL, we add one loss term to this training. Ours trains a small autoencoder on the ViT feature with push–pull (perturbations in image space, margin in feature space) and does not modify the backbone’s training; the anomaly score is FITYMI’s kNN score on the autoencoder output. This is the nested-carving construction applied to a pretrained feature space.
| Method | Pretrained backbone | Params (M) | Synth. | CIFAR-10 | CIFAR-100 |
|---|---|---|---|---|---|
| FITYMI† | ViT-B/16, IN-21K | 85.8 | yes | 98.41 | 96.46 |
| FITYMI + TOLL† | ViT-B/16, IN-21K | 85.8 | yes | 98.78 | 97.11 |
| FITYMI‡ | ViT-B/16, IN-21K | 85.8 | yes | 98.48 | 96.48 |
| FITYMI + AE-PP‡ | ViT-B/16, IN-21K | 86.7 | yes | 99.32 | 98.11 |
Results are one-class AUROC percentages averaged over ten seeds. † denotes results reported by TOLL; its FITYMI baseline is TOLL’s rerun using the released FITYMI code and synthetic samples. ‡ denotes our rerun using the final checkpoint.
Our FITYMI rerun closely reproduces the TOLL rerun, reaching 98.48 versus 98.41 on CIFAR-10 and 96.48 versus 96.46 on CIFAR-100. Adding AE-PP improves our matched FITYMI baseline by 0.84 points on CIFAR-10 and 1.63 points on CIFAR-100, reaching 99.32 and 98.11. It also exceeds FITYMI + TOLL by 0.54 and 1.00 points, respectively.
This experiment also bears on the second category of predictions in the paper, the representation dynamics of Section 3.3: as a representation becomes more compact in the PoS sense, nominal components separate relatively while off-union anomalies move toward them, and Section 4.2 confirmed both effects in all 16 Push and Pull configurations (Table 2; metrics in Appendix D). A nested stage will compact whatever range it is handed. What varies is how much the latent-stage anomaly score has to work with: directions the earlier stage already removed are gone, but directions it kept can still be exposed. For a pretrained backbone, that geometry points to a gain, as we explain next.
Why nesting helps here, and what remains untested: A pretrained, discriminatively fine-tuned ViT is not compact in the PoS sense: its feature space retains anomaly-relevant directions alongside nominal ones, as the Stage-0 range does in Figure 3. The nested push–pull stage carves this inherited range and exposes those directions, which is the mechanism validated on MNIST (Section 4.1, Table 13) and is consistent with the gain above. The opposite regime, in which the backbone is itself trained compactly and may remove anomaly directions before the nested stage sees them (Section 3.3), is not tested here; it requires training the backbone with push–pull from scratch and then nesting, which we identify as future work and will add to the extended discussion.
Together, the FITYMI experiment shows that Push and Pull improves a strong ImageNet-21K-pretrained vision transformer, and the geometric argument above predicts when nesting on a pretrained backbone should help. Extending the construction to pretrained backbones on practical AD benchmarks is left for future work.
Q5: Is the masking procedure used in all main experiments?
No, only in the controlled MNIST study of Section 4.1, where it is one of the four ablated factors. We will add the following sentence to Section 4.1 in the revision:
“Masking denotes masked reconstruction: random patches of the input are masked to form , and the network reconstructs the full unmasked image from , with the loss taken over the whole image; in masked push–pull configurations, the additive perturbation is applied to and the margin is computed from the total deviation between the perturbed masked image and the clean image.”
The benchmarks of Sections 4.3 and 4.4 and the FITYMI experiment use the perturbations in Table 20 without masking. The mask parameters will be stated in Appendix C.
Answers to Reviewer JLuL, Part I (Responses Continue in Following Comments)
We thank the reviewer for the careful reading, positive assessment of the theoretical framework, and constructive suggestions. We agree that addressing these points will significantly improve the paper. In this rebuttal, we answer Q1, Q2, Q3, Q4, and Q5, together with the related work packages, individually and in full. The answers, clarifications, and new results are provided here in continuation comments. By November 17, we will also upload a consolidated revision incorporating these changes, the rewritten abstract, and three revised Introduction paragraphs.
Before answering the questions, we clarify the paper's intended position. This work is not intended to introduce another application specific anomaly detection algorithm. Its broader aim is to help move machine learning from a predominantly empirical discipline toward a predictive science. We demonstrate a concrete cycle from geometric postulates to predictions made before training, and from those predictions to controlled experiments.
The original PoS hypothesis proposed nonlinear orthogonal projection onto a union of submanifolds as an ideal operation and qualitatively conjectured failure cases. This paper advances that program by providing an explicit mathematical dictionary for predicting join blindness and meet preference, deriving separate predictions about learning dynamics, and testing both categories experimentally. It also proposes Dynamic Push and Pull and Nested Manifold Carving as practical approximations of the ideal operation.
Anomaly detection is therefore a measurable testbed in which the predictions can be isolated and falsified, not the endpoint of the contribution. To make this unambiguous, we have rewritten the abstract and edited three Introduction paragraphs. The complete replacement text is provided in the following continuation comments and will be included in the revised manuscript.
Q1: How about the relationship between the proposed method and prevalent diffusion based anomaly detection?
We thank the reviewer for highlighting this connection. From the PoS perspective, the reconstruction components of the cited diffusion methods follow the same broad Pull only principle as denoising autoencoders. They corrupt an input and restore it toward the nominal distribution through reconstruction, noise prediction, or deviation prediction. Unlike an ordinary denoising autoencoder, diffusion models learn a time conditioned process and generally apply multiple reverse steps.
In our formulation, α=0 is the closest counterpart to these denoising objectives, whereas α>0 adds an explicit Push margin separating nominal reconstructions from perturbed samples. Nested Manifold Carving applies this principle recursively in previously learned latent spaces.
The cited pipelines also differ computationally. DeCo Diff performs iterative correction in a pretrained VAE latent space. The time series models use multiple reverse steps. LMD performs iterative inpainting and repeated reconstructions. DiffAD uses 50 reverse steps, synthetic anomalies, and a separate segmentation network. These procedures increase inference cost, limiting real time industrial use.
DiffusionAD avoids an iterative reverse chain through one step reconstruction. Its complete pipeline reconstructs at two predefined noise scales and then applies a separate segmentation network. Its training objective also combines noise prediction with a segmentation loss based on synthetic anomalies and their masks.
Our purpose is to isolate the learning dynamics predicted by PoS. We therefore use reconstruction scores directly, without a task specific segmentation network, adversarial loss, or other auxiliary detection loss. These components are complementary and can be combined with Push and Pull in future work.
The revision will cite all five suggested papers and add an extended discussion comparing their objectives, inference procedures, computational requirements, and auxiliary components with our formulation.
Our answers to Q2, Q3, Q4, and Q5 continue in the following comments.