acceptodds
Under review as a conference paper at ICLR 2027

When Does Unsupervised Learning Succeed or Fail? A PoS Perspective on Reconstruction-Based Anomaly Detection

Abstract

Reconstruction-based unsupervised learning can fail in two opposing ways: a model may reconstruct anomalies too accurately or discard valid nominal variation. Using the Pursuit of Subspaces hypothesis, we characterize these failures through the meet, union, and join geometries induced by the nominal components. Excess learned range produces join blindness, while insufficient capacity produces meet preference and loss of nominal fidelity. We show that the compact nominal union is optimal among nominal faithful ranges and generally requires a nonlinear reconstruction map. Based on this geometry, we introduce Dynamic Push and Pull, which learns from controlled perturbations without anomaly labels, and nested manifold carving, which applies the same principle recursively in latent space. Experiments confirm the predicted changes in latent geometry across every tested Push and Pull configuration. The proposed methods improve reconstruction-based anomaly detection across standard benchmarks and unseen image degradations, while also improving pretrained ECG representations for downstream classification. These results connect reconstruction failures to identifiable geometric conditions and provide practical mechanisms for learning compact representations.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
unsupervised anomaly detection, one-class classification, deep anomaly detection, representation learning, autoencoders, manifold learning, geometric deep learning, reconstruction-based anomaly detection

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_JLuLOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper explains the well-known trade-off of reconstruction-based anomaly detection (AD) from the perspective of manifold learning based on the Pursuit of Subspaces (PoS) hypothesis, revealing that reconstruction alone does not suffice for AD, due to meet preference and join blindness. To avoid these failures, they propose straightforward modifications to the training objective, termed “Dynamic Push and Pull” and “nested manifold carving”. While the former encourages the model to avoid identical mappings by adding a denoising objective, the latter enforces the model to reconstruct recursively in the previously learned latent space. Their experiments cover AD tasks, including MNIST, CIFAR-10, and Arrhythmia, and also consider detecting unseen image degradation and more general representation learning on ECG data.

Critical strengths

S1: Theoretical interpretation of reconstruction-based AD. Under the PoS hypothesis, this paper explains well-known failure cases for reconstruction-based AD. Existing works only provide a rough explanation of these failures: excess capacity leads to identical mappings, while less capacity leads to poor reconstruction for nominal samples. This work instead gives an explanation through a manifold perspective, offering an intuition for how learned deep neural networks fall into such failures.

S2: Novelty. Proposed methodologies composed of push-and-pull and nested learning have a certain novelty in the field of reconstruction-based AD. Also, this paper provides a guarantee that a push-and-pull successfully avoids prevalent failure modes in Lemma 1.

S3: Thorough experimental ablations. They ablate the performance of the proposed method on several standard AD benchmarks and verify with diverse methodological designs. Such a broad range of ablations makes the contribution of each module clear.

Critical weaknesses

W1: Connection to diffusion models and comparison. The proposed push-and-pull resembles an invention of the diffusion-based AD in terms of its denoising procedure for avoiding identical mappings. Recent works [1-5] demonstrate that diffusion-based AD can yield striking AD performance in a reconstruction-based manner. Therefore, it should be encouraged to organize the relationships or differences between them and compare against them to understand the strength of the proposed method.

W2: Evaluations on practical AD benchmarks. Current experimental settings and benchmarks seem to be biased toward OOD detection literature. These settings measure the capability for detecting semantic outliers rather than tiny, local anomalous features. To measure the practical AD performance of these methods, it is suggested to evaluate on widely used image AD benchmarks, such as MVTecAD [6], and VisA [7]. Also, recent AD works adopt pre-trained feature extractors such as DINOv2 [8] to leverage their general capabilities on various vision tasks. Could the proposed method be effective in such a latent space?

W3: technical details. An apparent difference between diffusion-based AD is the adoption of "dynamic push margin" as in Eq. 2, and it consistently improves performance across several OOD detection datasets as shown in Fig. 4. While it serves as an extra separation signal for AD, this paper also adds a masking procedure in its MNIST experiments in Tab. 1. However, such a masking procedure is not stated in their methodology section and lacks more detailed information on this in the main manuscript.

References

[1] Baizaee+, "Correcting Deviations from Normality: A Reformulated Diffusion Model for Multi-Class Unsupervised Anomaly Detection", CVPR'25, https://arxiv.org/abs/2503.19357.

[2] Pintilie+, "Time Series Anomaly Detection using Diffusion-based Models", ICDM'23, https://arxiv.org/abs/2311.01452.

[3] Zhang+, "DiffusionAD: Norm-guided One-step Denoising Diffusion for Anomaly Detection", TPAMI'25, https://arxiv.org/abs/2303.08730.

[4] Liu+, "Unsupervised Out-of-Distribution Detection with Diffusion Inpainting", ICML'23, https://arxiv.org/abs/2302.10326.

[5] Zhang+, "Unsupervised Surface Anomaly Detection with Diffusion Probabilistic Model", ICCV'23, https://openaccess.thecvf.com/content/ICCV2023/papers/Zhang_Unsupervised_Surface_Anomaly_Detection_with_Diffusion_Probabilistic_Model_ICCV_2023_paper.pdf.

[6] Bergmann+, "MVTec AD — A Comprehensive Real-World Dataset for Unsupervised Anomaly Detection", CVPR'19, https://openaccess.thecvf.com/content_CVPR_2019/papers/Bergmann_MVTec_AD_--_A_Comprehensive_Real-World_Dataset_for_Unsupervised_Anomaly_CVPR_2019_paper.pdf.

[7] Zou+, "SPot-the-Difference Self-Supervised Pre-training for Anomaly Detection and Segmentation", ECCV'22, https://arxiv.org/abs/2207.14315.

[8] Oquab+, "DINOv2: Learning Robust Visual Features without Supervision", TMLR'23, https://arxiv.org/abs/2304.07193.

Questions for authors

Q1: How about the relationship between the proposed method and prevalent diffusion-based AD?

Q2: How about the performance of the proposed method compared with these diffusion-based AD methods?

Q3: How about the performance on the practical AD benchmarks?

Q4: Could the proposed method be effective with extra feature extractors?

Q5: Does this paper incorporate the masking procedure for all main experiments? If so, please state it explictly in the methodological or experimental setup sections.

If the author addressed concerns (W1-3) and corresponding questions (Q1-5), I'm willing to increase my score. For the rebuttal, my major concern is its performance on practical AD benchmarks beyond OOD detection datasets. While this paper states, "we compare state-of-the-art unsupervised anomaly detection methods" in L132, it seems to lack a comparison with SoTA diffusion-based methods on practical AD benchmarks.

9 comments
AuthorsOpenReview ↗

Answers to Reviewer JLuL, Part I (Responses Continue in Following Comments)

We thank the reviewer for the careful reading, positive assessment of the theoretical framework, and constructive suggestions. We agree that addressing these points will significantly improve the paper. In this rebuttal, we answer Q1, Q2, Q3, Q4, and Q5, together with the related work packages, individually and in full. The answers, clarifications, and new results are provided here in continuation comments. By November 17, we will also upload a consolidated revision incorporating these changes, the rewritten abstract, and three revised Introduction paragraphs.

Before answering the questions, we clarify the paper's intended position. This work is not intended to introduce another application specific anomaly detection algorithm. Its broader aim is to help move machine learning from a predominantly empirical discipline toward a predictive science. We demonstrate a concrete cycle from geometric postulates to predictions made before training, and from those predictions to controlled experiments.

The original PoS hypothesis proposed nonlinear orthogonal projection onto a union of submanifolds as an ideal operation and qualitatively conjectured failure cases. This paper advances that program by providing an explicit mathematical dictionary for predicting join blindness and meet preference, deriving separate predictions about learning dynamics, and testing both categories experimentally. It also proposes Dynamic Push and Pull and Nested Manifold Carving as practical approximations of the ideal operation.

Anomaly detection is therefore a measurable testbed in which the predictions can be isolated and falsified, not the endpoint of the contribution. To make this unambiguous, we have rewritten the abstract and edited three Introduction paragraphs. The complete replacement text is provided in the following continuation comments and will be included in the revised manuscript.

Q1: How about the relationship between the proposed method and prevalent diffusion based anomaly detection?

We thank the reviewer for highlighting this connection. From the PoS perspective, the reconstruction components of the cited diffusion methods follow the same broad Pull only principle as denoising autoencoders. They corrupt an input and restore it toward the nominal distribution through reconstruction, noise prediction, or deviation prediction. Unlike an ordinary denoising autoencoder, diffusion models learn a time conditioned process and generally apply multiple reverse steps.

In our formulation, α=0\alpha=0 is the closest counterpart to these denoising objectives, whereas α>0\alpha>0 adds an explicit Push margin separating nominal reconstructions from perturbed samples. Nested Manifold Carving applies this principle recursively in previously learned latent spaces.

The cited pipelines also differ computationally. DeCo Diff performs iterative correction in a pretrained VAE latent space. The time series models use multiple reverse steps. LMD performs iterative inpainting and repeated reconstructions. DiffAD uses 50 reverse steps, synthetic anomalies, and a separate segmentation network. These procedures increase inference cost, limiting real time industrial use.

DiffusionAD avoids an iterative reverse chain through one step reconstruction. Its complete pipeline reconstructs at two predefined noise scales and then applies a separate segmentation network. Its training objective also combines noise prediction with a segmentation loss based on synthetic anomalies and their masks.

Our purpose is to isolate the learning dynamics predicted by PoS. We therefore use reconstruction scores directly, without a task specific segmentation network, adversarial loss, or other auxiliary detection loss. These components are complementary and can be combined with Push and Pull in future work.

The revision will cite all five suggested papers and add an extended discussion comparing their objectives, inference procedures, computational requirements, and auxiliary components with our formulation.

Our answers to Q2, Q3, Q4, and Q5 continue in the following comments.

AuthorsOpenReview ↗

Answers to Reviewer JLuL, Part II: Q3 and Q2 (Responses Continue)

Q3: How does the method perform on practical AD benchmarks?

We evaluated the method on the two requested benchmarks, MVTec AD and VisA, and report both image and pixel AUROC. All our results use the final checkpoint and require no anomalous validation data. The 0.83M model uses the same architecture as our RGB HQ image experiments. It is trained from scratch and uses only reconstruction scores, without a pretrained backbone, segmentation head, or auxiliary detection loss.

MethodParams (M)Synth.Seg. headSel.MVTec IMVTec PVisA IVisA P
Reconstruction score, trained from scratch
DRAEM Recon. AE†69.1yesnolast83.989.7--
DRAEM Recon. AE + MSGMS†69.1yesnolast90.793.4--
RIAD†-nonolast91.794.2--
InTra†55nonoval95.096.6--
Ours, reconstruction score, trained from scratch
DAE, α = 00.83yesnolast92.1092.8892.8495.77
AE-PP, α = 10.83yesnolast92.4293.2892.7696.36
AE-PP, α = 20.83yesnolast92.1692.8893.2896.35
Reconstruction plus segmentation head trained on synthetic anomalies
DRAEM†97.4yesyeslast98.097.3--
DRAEM‡97.4yesyeslast97.9097.0490.1893.65
DiffusionAD†160.0yesyesbest99.798.798.898.9
DiffusionAD‡160.0yesyeslast98.5297.5096.9898.38

I and P denote image and pixel AUROC. Params denotes trained parameters. Synth. indicates training with synthetic anomalies. † denotes results reported in the original papers. ‡ denotes our reruns; the DRAEM result is averaged over three seeds. Sel. denotes checkpoint selection. Bold marks the best result among our models.

Within the same reconstruction score setting, namely training from scratch without a segmentation head, AE-PP reaches 92.42 MVTec image AUROC. This exceeds both DRAEM reconstruction variants, at 83.9 and 90.7, and RIAD, at 91.7. Only InTra is higher, at 95.0, and it uses validation checkpoint selection. On pixel AUROC, AE-PP exceeds the basic DRAEM reconstruction branch and nearly matches DRAEM with MSGMS, at 93.28 versus 93.4, while remaining below RIAD and InTra. Thus, within this reconstruction only group, our 0.83M AE-PP ranks second on image AUROC, surpassing both DRAEM reconstruction variants and RIAD.

On VisA, AE-PP with α=2\alpha=2 reaches 93.28 image and 96.35 pixel AUROC. It exceeds our rerun of the complete DRAEM detector, at 90.18 and 93.65, even though DRAEM uses 97.4M parameters and a separate segmentation head. The matched DAE comparison isolates the contribution of Push and Pull. AE-PP improves MVTec by 0.32 image and 0.40 pixel AUROC, and VisA by 0.44 image and 0.58 pixel AUROC.

Q2: How does performance compare with diffusion based AD?

The complete DiffusionAD detector remains numerically stronger, but it uses a substantially different pipeline with 160M trained parameters, synthetic anomaly masks, and a separate segmentation head. Our model contains 0.83M trained parameters, meaning that DiffusionAD has approximately 193 times as many trained parameters. Our model is trained from scratch and evaluated using the final checkpoint with direct reconstruction scoring. At inference, our model requires a single autoencoder forward pass followed by reconstruction scoring, whereas DiffusionAD evaluates its diffusion reconstructor at two predefined noise scales and then applies a separate segmentation network. Our pipeline therefore requires fewer network evaluations, while we restrict our quantitative efficiency comparison to parameter count.

Thus, AE-PP achieves competitive performance in a much smaller reconstruction only setting designed to isolate the PoS derived Push and Pull principle. Incorporating this principle into diffusion reconstructors and segmentation pipelines is an important direction for future work.

Answers to Q4 and Q5 continue in the following comments.

AuthorsOpenReview ↗

Answers to Reviewer JLuL, Part III: Q4 and Q5 (Responses Continue)

Q4: Is the method effective with pretrained feature extractors?

Yes. To test whether Push and Pull remains effective with a strong pretrained representation, we combine AE-PP with FITYMI. FITYMI fine-tunes an ImageNet-21K ViT-B/16 with synthetic anomalies. As in FITYMI + TOLL, we add one loss term to this training. Ours trains a small autoencoder on the ViT feature with push–pull (perturbations in image space, margin in feature space) and does not modify the backbone’s training; the anomaly score is FITYMI’s kNN score on the autoencoder output. This is the nested-carving construction applied to a pretrained feature space.

MethodPretrained backboneParams (M)Synth.CIFAR-10CIFAR-100
FITYMI†ViT-B/16, IN-21K85.8yes98.4196.46
FITYMI + TOLL†ViT-B/16, IN-21K85.8yes98.7897.11
FITYMI‡ViT-B/16, IN-21K85.8yes98.4896.48
FITYMI + AE-PP‡ViT-B/16, IN-21K86.7yes99.3298.11

Results are one-class AUROC percentages averaged over ten seeds. † denotes results reported by TOLL; its FITYMI baseline is TOLL’s rerun using the released FITYMI code and synthetic samples. ‡ denotes our rerun using the final checkpoint.

Our FITYMI rerun closely reproduces the TOLL rerun, reaching 98.48 versus 98.41 on CIFAR-10 and 96.48 versus 96.46 on CIFAR-100. Adding AE-PP improves our matched FITYMI baseline by 0.84 points on CIFAR-10 and 1.63 points on CIFAR-100, reaching 99.32 and 98.11. It also exceeds FITYMI + TOLL by 0.54 and 1.00 points, respectively.

This experiment also bears on the second category of predictions in the paper, the representation dynamics of Section 3.3: as a representation becomes more compact in the PoS sense, nominal components separate relatively while off-union anomalies move toward them, and Section 4.2 confirmed both effects in all 16 Push and Pull configurations (Table 2; metrics in Appendix D). A nested stage will compact whatever range it is handed. What varies is how much the latent-stage anomaly score has to work with: directions the earlier stage already removed are gone, but directions it kept can still be exposed. For a pretrained backbone, that geometry points to a gain, as we explain next.

Why nesting helps here, and what remains untested: A pretrained, discriminatively fine-tuned ViT is not compact in the PoS sense: its feature space retains anomaly-relevant directions alongside nominal ones, as the Stage-0 range does in Figure 3. The nested push–pull stage carves this inherited range and exposes those directions, which is the mechanism validated on MNIST (Section 4.1, Table 13) and is consistent with the gain above. The opposite regime, in which the backbone is itself trained compactly and may remove anomaly directions before the nested stage sees them (Section 3.3), is not tested here; it requires training the backbone with push–pull from scratch and then nesting, which we identify as future work and will add to the extended discussion.

Together, the FITYMI experiment shows that Push and Pull improves a strong ImageNet-21K-pretrained vision transformer, and the geometric argument above predicts when nesting on a pretrained backbone should help. Extending the construction to pretrained backbones on practical AD benchmarks is left for future work.

Q5: Is the masking procedure used in all main experiments?

No, only in the controlled MNIST study of Section 4.1, where it is one of the four ablated factors. We will add the following sentence to Section 4.1 in the revision:

“Masking denotes masked reconstruction: random patches of the input are masked to form sms_m, and the network reconstructs the full unmasked image from sms_m, with the loss taken over the whole image; in masked push–pull configurations, the additive perturbation is applied to sms_m and the margin is computed from the total deviation between the perturbed masked image and the clean image.”

The benchmarks of Sections 4.3 and 4.4 and the FITYMI experiment use the perturbations in Table 20 without masking. The mask parameters will be stated in Appendix C.

Reviewer_nyUPOpenReview ↗
Rating
1
Confidence
4(+1)
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies reconstruction-based unsupervised learning from the Pursuit of Subspaces (PoS) perspective. It identifies two failure modes, join blindness and meet preference, and argues that an ideal model should learn a compact nonlinear union of nominal submanifolds. Based on this view, the authors propose Dynamic Push and Pull to separate perturbed samples from the nominal manifold, and Nested Manifold Carving to further refine representations in latent space. Experiments on several anomaly-detection benchmarks and ECG representation learning show improvements over reconstruction-based baselines.

Critical strengths
  1. A coherent geometric interpretation of reconstruction-based anomaly detection. The distinction between join blindness and meet preference provides an intuitive unified explanation for two opposite failure regimes of reconstruction models. The paper does more than simply report anomaly-detection performance: it derives qualitative predictions for representation geometry and attempts to measure these predictions experimentally.
  2. The controlled experiments connect the proposed interpretation to observable representation behavior. The MNIST study, the comparison between Pull-only/denoising and Push-and-Pull objectives, and the representation-dynamics analysis make the empirical story more complete than a standard benchmark-only anomaly-detection paper. In particular, the attempt to validate predicted geometric changes rather than relying solely on downstream AUROC is a positive aspect of the work.
Critical weaknesses
  1. The methodological novelty of Push-and-Pull appears limited relative to existing synthetic-anomaly and perturbation-based anomaly/OOD learning. The practical mechanism ultimately relies on generating perturbed versions of nominal samples and using them as negative or off-manifold examples. The paper itself acknowledges that methods such as CutPaste, DRAEM, NSA, and CSI already learn discriminative anomaly criteria from synthetic anomalies or distribution-shifted views, and that perturbation-to-clean reconstruction corresponds directly to the proposed Pull component. The main algorithmic addition is therefore an explicit separation margin through the Push term. More broadly, closely related ideas of constructing proxy negative samples and enforcing separation have already been explored in anomaly and OOD detection, e.g., [1] and [2]. Although the geometric interpretation in the present paper is different, I am not convinced that the resulting learning algorithm represents a sufficiently substantial advance beyond this existing line of work. More importantly, the empirical improvements may largely originate from injecting strong prior knowledge through the chosen perturbations. For example, CIFAR uses phase scrambling, patch shuffling, CutPaste, and channel shuffling, while different datasets use different manually selected transformations. It is therefore unclear how much of the performance gain is due specifically to the PoS-derived geometric principle rather than conventional pseudo-anomaly supervision.

  2. The image anomaly-detection evaluation is not sufficiently representative of the current state of the field. The primary standard image experiments remain MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. While MNIST is useful as a controlled toy environment for verifying the proposed geometry, I find these benchmarks insufficient for establishing the practical relevance of a new unsupervised anomaly-detection method in 2027. Recent anomaly-detection literature commonly evaluates on substantially more realistic benchmarks such as MVTec AD, MVTec LOCO, and VisA, where anomalies involve local structural, logical, and texture defects. Existing synthetic-anomaly approaches have already been evaluated in these settings. Furthermore, modern pretrained visual representations, e.g., DINO/DINOv2-based anomaly detectors, provide strong reference points that are absent from the current comparison. Without such experiments, it is difficult to assess whether the proposed representation-learning principle remains beneficial beyond relatively low-resolution one-class classification settings. This concern is strengthened by the evaluation protocol: although training uses only nominal samples, checkpoints are selected using a validation set containing both nominal and anomalous examples. Therefore, the reported setting is not fully anomaly-agnostic at the model-selection stage. A label-free selection protocol would provide stronger evidence for the claimed unsupervised setting.

[1] Generating and Reweighting Dense Contrastive Patterns for Unsupervised Anomaly Detection. [2] Revisiting Energy-Based Model for Out-of-Distribution Detection.

Questions for authors

See weaknesses.

33 comments
AuthorsOpenReview ↗

Answers to Reviewer nyUP, Part I: Position and Contribution (Responses Continue in Following Comments)

We thank the reviewer for the careful reading and for recognizing two central strengths: the coherent geometric explanation of reconstruction failures and the controlled experiments connecting that explanation to observable representation dynamics. We also thank the reviewer for the detailed concerns about how the AD results should be positioned relative to the current state of the art, and we agree that addressing these concerns will further improve the paper. In this rebuttal, we will answer each concern individually and in full through continuation comments. By November 17, we will also upload a consolidated revision incorporating these clarifications, the new results, the rewritten abstract, and three revised Introduction paragraphs.

Before addressing the individual concerns, we would first like to clarify the paper’s intended position. This work is not intended to introduce another application specific anomaly detection algorithm. Its broader aim is to help move machine learning from a predominantly empirical discipline toward a predictive science. We demonstrate a concrete cycle from geometric postulates to predictions made before training, and from those predictions to controlled experiments.

The original PoS hypothesis proposed nonlinear orthogonal projection onto a union of submanifolds as an ideal operation and qualitatively conjectured possible failure cases. This paper advances that program by providing an explicit mathematical dictionary for predicting join blindness and meet preference, deriving a separate category of predictions concerning learning dynamics, and testing both categories experimentally. It also introduces Dynamic Push and Pull and Nested Manifold Carving as practical approximations of the ideal operation.

Anomaly detection is therefore one measurable case study in which these predictions can be isolated and falsified, not the endpoint of the contribution. Section 4.4 extends the evaluation beyond low resolution one-class benchmarks by learning a submanifold of high quality RGB images and testing unseen realistic degradations. Section 4.5 goes beyond anomaly detection altogether through unsupervised pretraining on unlabelled ECG recordings followed by frozen-encoder downstream classification. These experiments demonstrate that the proposed framework concerns general unsupervised representation learning rather than a single AD application.

To make this position unambiguous, we have rewritten the abstract and edited three Introduction paragraphs to clarify the paper’s objective, prediction categories, and contributions. These changes will be included in the consolidated revised manuscript.

Our detailed answers, clarifications, and new experimental results continue in the following comments.

AuthorsOpenReview ↗

Answers to Reviewer nyUP, Part II: Q1, Practical Relevance Beyond Controlled Benchmarks (Responses Continue)

Answers to the questions

Q1: “The image anomaly-detection evaluation is not sufficiently representative of the current state of the field. The primary standard image experiments remain MNIST, Fashion-MNIST, CIFAR-10, and CIFAR-100. While MNIST is useful as a controlled toy environment for verifying the proposed geometry, I find these benchmarks insufficient for establishing the practical relevance of a new unsupervised anomaly-detection method in 2027.”

We agree that MNIST alone would not establish practical relevance. However, MNIST is intentionally used as a controlled environment in which the predicted failure geometry and representation dynamics can be directly measured; it is not the paper’s sole evidence of practical relevance. The paper already includes two substantially different settings that are important to this concern.

First, Section 4.4 learns a submanifold of high quality RGB images using only clean images and evaluates the detection of unseen degradations across 16 settings. Rain and underwater degradations are absent from training, while REDS and GoPro test transfer from parametric training blur to realistic motion blur. To position this experiment against methods designed specifically for image degradation assessment, we additionally evaluated five standard no-reference image-quality metrics: BRISQUE, NIQE, MUSIQ, CLIP-IQA, and MANIQA.

SetNormalAE-PoSBest NR-IQASelected metric
Kodak24, σ = 25own GT1.0001.000four of five
REDS blur 0.2own GT0.6000.633MANIQA
REDS blur 0.4own GT0.7810.808MANIQA
REDS blur 0.6own GT0.9550.947MANIQA
REDS blur 0.8own GT0.9970.993MUSIQ
REDS blur 1.0own GT1.0000.997MUSIQ
GoProown GT0.9850.986MUSIQ
GoProDIV8K1.0000.996MANIQA
Rain Test100DIV8K0.9590.652BRISQUE
Rain100LDIV8K0.8930.477BRISQUE
Rain100HDIV8K0.9570.921BRISQUE
Rain Test1200DIV8K0.9920.801CLIP-IQA
Rain Test2800DIV8K0.9880.568CLIP-IQA
Underwater highDIV8K0.9890.908CLIP-IQA
Underwater mediumDIV8K0.9870.939CLIP-IQA
Underwater lowDIV8K0.9800.941CLIP-IQA

All entries are AUROC. The NR-IQA metrics use the pyIQA defaults, with lower predicted quality treated as more anomalous. “Own GT” denotes the corresponding dataset’s clean ground-truth images, while DIV8K denotes clean DIV8K images as the nominal set. “Best NR-IQA” gives the five competing metrics an oracle advantage by selecting the strongest metric separately for each setting.

AE-PoS nevertheless wins 12 of the 16 settings and ties one. On blur, where these quality metrics are specifically strong, AE-PoS is competitive. On rain and underwater degradation, AE-PoS wins all eight settings by 0.04 to 0.42 AUROC. Several individual NR-IQA metrics fall below 0.5 and incorrectly rate degraded images as higher quality than clean images.

Crucially, AE-PoS does not require selecting a degradation-specific quality metric. A single 1.67M-parameter reconstruction model and one anomaly score cover all 16 settings, without a pretrained backbone or task-specific detection head. Thus, this experiment is not merely a geometric proof of concept. The current model can already serve as a lightweight no-reference degradation detector, particularly when the degradation type is unknown or heterogeneous.

Second, Section 4.5 moves beyond anomaly detection altogether. It pretrains on 164,972 unlabelled ECG recordings, freezes the encoder, and evaluates linear diagnostic classifiers. Nested representation learning improves all five downstream classification tasks and outperforms the compared pretrained ECG models on each task. This provides a direct test of the framework as general unsupervised representation learning rather than as an AD-specific algorithm.

Finally, in response to the reviewer’s request, we have added the practical MVTec AD and VisA benchmarks. These results, including comparisons with modern reconstruction and segmentation pipelines, are reported in our answer to Q2.

Our answer to Q2 continues in the following comment.

AuthorsOpenReview ↗

Answers to Reviewer nyUP, Part III: Q2, MVTec AD and VisA Results (Responses Continue)

Q2: Evaluation on practical anomaly detection benchmarks

We agree that evaluation beyond low resolution one class classification is important. We therefore added two practical industrial benchmarks identified by the reviewer, MVTec AD and VisA, and report both image and pixel AUROC. All our results use the final checkpoint and require no anomalous validation data. The 0.83M model uses the same architecture as our RGB HQ image experiments. It is trained from scratch and uses only reconstruction scores, without a pretrained backbone, segmentation head, or auxiliary detection loss.

MethodParams (M)Synth.Seg. headSel.MVTec IMVTec PVisA IVisA P
Reconstruction score, trained from scratch
DRAEM Recon. AE†69.1yesnolast83.989.7--
DRAEM Recon. AE + MSGMS†69.1yesnolast90.793.4--
RIAD†-nonolast91.794.2--
InTra†55nonoval95.096.6--
Ours, reconstruction score, trained from scratch
DAE, α = 00.83yesnolast92.1092.8892.8495.77
AE-PP, α = 10.83yesnolast92.4293.2892.7696.36
AE-PP, α = 20.83yesnolast92.1692.8893.2896.35
Reconstruction plus segmentation head trained on synthetic anomalies
DRAEM†97.4yesyeslast98.097.3--
DRAEM‡97.4yesyeslast97.9097.0490.1893.65
DiffusionAD†160.0yesyesbest99.798.798.898.9
DiffusionAD‡160.0yesyeslast98.5297.5096.9898.38

I and P denote image and pixel AUROC. Params denotes trained parameters. Synth. indicates training with synthetic anomalies. † denotes results reported in the original papers. ‡ denotes our reruns; the DRAEM result is averaged over three seeds. Sel. denotes checkpoint selection. Bold marks the best result among our models.

Within the same reconstruction score setting, namely training from scratch without a segmentation head, AE-PP reaches 92.42 MVTec image AUROC. This exceeds both DRAEM reconstruction variants, at 83.9 and 90.7, and RIAD, at 91.7. Only InTra is higher, at 95.0, and it uses validation checkpoint selection. On pixel AUROC, AE-PP exceeds the basic DRAEM reconstruction branch and nearly matches DRAEM with MSGMS, at 93.28 versus 93.4. Thus, within this reconstruction only group, our 0.83M AE-PP ranks second on image AUROC, surpassing both DRAEM reconstruction variants and RIAD.

On VisA, AE-PP with α = 2 reaches 93.28 image and 96.35 pixel AUROC. It exceeds our rerun of the complete DRAEM detector, at 90.18 and 93.65, even though DRAEM uses 97.4M trained parameters and a separate segmentation head. The matched DAE comparison isolates the contribution of Push and Pull: AE-PP improves MVTec by 0.32 image and 0.40 pixel AUROC, and VisA by 0.44 image and 0.58 pixel AUROC.

These results demonstrate that the proposed learning principle remains beneficial beyond low resolution one class classification, including industrial images containing local texture and structural defects.

Our response on compatibility with pretrained visual representations, evaluated using the ImageNet-21K pretrained ViT-B/16 model from FITYMI, continues in the following comment.

Reviewer_Rt9NOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies reconstruction based anomaly detection problem. It introduces dynamic push and pull, that learns from controlled perturbations without anomaly labels. It also applies nested manifold carving which applies this principle recursively in latent space. The experiments show promising results.

Critical strengths

The paper explains the idea well and in sufficient detail.

Dynamic push and pull learning dynamics is interesting in the problem of unsupervised learning, where training data is usally only in-domain.

Critical weaknesses

Evaluation of the approach is limited. Particularly, the evaluation on MNIST and CIFAR-10 does not necessarily mean generalizability across complicated one-class classification problems such as industrial and medical domains. More advanced datasets for such problems are highly encouraged, for example, MVTech or similar.

The approch is somewhat similar to augmentation methods, which also evaluate on anomaly detection for the similar effect. The methods lacks such detailed analysis. For example, see Table 17 of [1] and compared methods.

[1] DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models, CVPR 2024

Questions for authors

In addition to the weaknesses highlighted, I wonder how the approach will work in video-based datasets where complexity is manifold. For example, ped2 dataset or more complex one-class versions of ucf-crime or xd-violance datasets.

7 comments
AuthorsOpenReview ↗

Answers to Reviewer Rt9N, Part I: Position and Practical Evaluation (Responses Continue)

We thank the reviewer for the careful reading and for recognizing the interest of Dynamic Push and Pull for unsupervised learning with only in-domain training data. We address both weaknesses and the video question individually in this response and the continuation comment.

Before addressing them, we clarify the paper’s intended scope. This work is not intended to introduce another application-specific anomaly-detection algorithm. Its broader aim is to help move machine learning toward a predictive science by demonstrating a complete cycle from geometric postulates, to predictions made before training, to controlled experimental tests. The paper predicts both failure cases and learning dynamics, then introduces Dynamic Push and Pull and Nested Manifold Carving as practical approximations of the ideal PoS operation. Anomaly detection is a measurable testbed for these predictions, while the ECG experiment tests the framework as general unsupervised representation learning. The revised abstract and three revised Introduction paragraphs make this position explicit.

Q1: “Evaluation of the approach is limited. Particularly, the evaluation on MNIST and CIFAR-10 does not necessarily mean generalizability across complicated one-class classification problems such as industrial and medical domains. More advanced datasets for such problems are highly encouraged, for example, MVTech or similar.”

We agree that MNIST and CIFAR alone would not establish practical relevance. They are used as controlled environments in which the predicted failure geometry and learning dynamics can be measured directly. The submitted paper already includes two substantially different settings. Section 4.4 learns a submanifold of high-quality RGB images and detects unseen blur, rain, underwater degradation, and noise across 16 settings. Section 4.5 pretrains on 164,972 unlabelled ECG recordings, freezes the encoder, and evaluates five downstream diagnostic tasks, moving beyond anomaly detection to medical representation learning.

In addition, we evaluated the requested practical industrial benchmarks, MVTec AD and VisA:

MethodParams (M)Synth.Seg.Sel.MVTec IMVTec PVisA IVisA P
Reconstruction score, trained from scratch
DRAEM Recon. AE†69.1yesnolast83.989.7--
DRAEM Recon. AE + MSGMS†69.1yesnolast90.793.4--
RIAD†-nonolast91.794.2--
InTra†55nonoval95.096.6--
Ours, reconstruction score, trained from scratch
DAE, α=0\alpha=00.83yesnolast92.1092.8892.8495.77
AE-PP, α=1\alpha=10.83yesnolast92.4293.2892.7696.36
AE-PP, α=2\alpha=20.83yesnolast92.1692.8893.2896.35
Reconstruction plus segmentation head
DRAEM‡97.4yesyeslast97.9097.0490.1893.65
DiffusionAD‡160.0yesyeslast98.5297.5096.9898.38

I and P denote image and pixel AUROC. † denotes published results; ‡ denotes our reruns, with DRAEM averaged over three seeds. All our results use the final checkpoint and no anomalous validation data.

Within the matched reconstruction-only setting, our 0.83M AE-PP reaches 92.42 MVTec image AUROC, exceeding both DRAEM reconstruction variants and RIAD. On VisA, it reaches 93.28 image and 96.35 pixel AUROC, exceeding our rerun of the complete 97.4M-parameter DRAEM detector. DiffusionAD remains stronger, but uses 160M trained parameters, synthetic anomaly masks, and a separate segmentation network. Thus, the added results establish practical industrial relevance while preserving the controlled, reconstruction-only setting needed to test the PoS-derived principle.

Our answers concerning augmentation methods and video continue in the following comment.

AuthorsOpenReview ↗

Answers to Reviewer Rt9N, Part II: Augmentation Methods and Video Extension

Q2: “The approch is somewhat similar to augmentation methods, which also evaluate on anomaly detection for the similar effect. The methods lacks such detailed analysis. For example, see Table 17 of [1] and compared methods. [1] DiffuseMix: Label-Preserving Data Augmentation with Diffusion Models, CVPR 2024.”

We thank the reviewer for pointing us to DiffuseMix, which we will cite and discuss. We understand this question as asking whether the observed improvement could arise from conventional data augmentation and how the method compares with the augmentation baselines reported in Table 17.

Table 17 of DiffuseMix compares Mixup, CutMix, AugMix, Outlier, PixMix, and DiffuseMix across five safety metrics for supervised CIFAR-100 classifiers, including anomaly detection. These methods use label-preserving augmentation during multi-class classifier training. Our setting is different: each model is trained only on nominal samples, without semantic or anomaly labels, and anomaly detection is performed through reconstruction. Consequently, the numerical results in Table 17 cannot be compared directly with our one-class AUROC values under an equivalent protocol.

Nevertheless, the central question of whether our gain comes from ordinary augmentation can be tested directly. We fix the images, architecture, perturbations, and training budget, and vary only how the perturbations enter the objective:

ObjectiveMVTec IVisA I
Plain AE69.0069.61
AE + perturbed data as ordinary augmentation62.2064.22
Pull-only DAE, α=0\alpha=092.1092.84
Push and Pull, α=1\alpha=192.4292.76
Push and Pull, α=2\alpha=292.1693.28

Under ordinary augmentation, the perturbed images are reconstructed as themselves, and performance falls below the plain AE. Under Pull, the same perturbed samples are mapped back to their paired clean samples, removing the perturbed directions from the learned range. Push then adds a severity-adaptive separation constraint. Across paired runs, Push outperforms Pull-only in 92/100 MNIST, 82/100 Fashion-MNIST, 84/100 CIFAR-10, and 149/200 CIFAR-100 cases. Thus, the improvement cannot be explained by exposure to augmented samples alone; it depends on how the PoS-derived objective uses those samples.

We are also attempting an analogous comparison with the augmentation families reported in DiffuseMix under a common one-class protocol. If completed before the discussion closes on November 18, we will report the results in a continuation comment. Otherwise, the revised manuscript will cite DiffuseMix, explain the protocol distinction above, and identify a unified one-class comparison with Mixup, CutMix, AugMix, PixMix, and DiffuseMix as future work.

Q3: “In addition to the weaknesses highlighted, I wonder how the approach will work in video-based datasets where complexity is manifold. For example, ped2 dataset or more complex one-class versions of ucf-crime or xd-violance datasets.”

The geometric formulation extends naturally from image manifolds to spatiotemporal clip manifolds. Dynamic Push and Pull can be applied with a video autoencoder, 3D convolutional network, or video transformer by replacing image perturbations with controlled spatial and temporal perturbations, such as local appearance corruption, frame masking, temporal shuffling, speed variation, and motion inconsistency. Nested Manifold Carving can then operate on the learned spatiotemporal latent representation, with reconstruction scores aggregated at frame, region, or clip level.

We have not yet evaluated this extension on Ped2, UCF-Crime, or XD-Violence, so we do not claim empirical video performance in the present paper. We will add this extension and its distinct temporal-design requirements to the future-work discussion.

AuthorsOpenReview ↗

Answers to Reviewer Rt9N, Part III: Completed Augmentation-Generator Comparison

Answers to Reviewer Rt9N, Part III: Completed Augmentation-Generator Comparison

We completed the comparison promised in Part II. We adapted the image-generation procedures from Mixup, CutMix, AugMix, PixMix, and DiffuseMix to provide the perturbed input s~n\tilde{s}_n for the same AE-PP objective. This experiment does not reproduce their original supervised classifier-training methods; it isolates the effect of generator choice within our one-class reconstruction protocol.

All results use CIFAR-100 with 20 nominal superclasses and 10 seeds. Every row uses the final checkpoint without anomalous validation data. The 65.65 control is therefore the final-checkpoint AE-PP result under this protocol, whereas the 68.56 result in the submitted paper uses validation-based checkpoint selection following the matched TOLL protocol.

Generator adapted within AE-PPAUROC (%)Δ\Delta vs. controlWins / 200
Our image-space mixture (control)65.65−-−-
Mixup adaptation57.73−7.92-7.9211
CutMix adaptation59.30−6.35-6.3517
AugMix adaptation60.34−5.30-5.3028
PixMix adaptation68.88+3.24+3.24155
DiffuseMix adaptation60.58−5.06-5.0626

Wins denote paired improvements over the control across the 200 superclass-seed runs. PixMix is the only tested generator that improves the control, increasing final-checkpoint AUROC from 65.65 to 68.88 and winning 155 of 200 paired comparisons. This result confirms that the perturbation family matters. One plausible interpretation is that PixMix’s structurally complex fractal and feature-visualization content probes off-component directions more broadly, although this experiment does not establish that explanation or identify a universally optimal generator.

This comparison evaluates generator choice within AE-PP and does not isolate the Push term, because all rows use the Push-and-Pull objective. Isolating Push for each generator would additionally require matched Pull-only (α=0\alpha=0) runs. Together with the objective ablation reported in Part II, the experiments separate two questions: under fixed perturbations, performance depends on how the objective uses them; under a fixed objective, performance also depends on which perturbations are sampled.

Finally, these one-class reconstruction results are not directly comparable with the multiclass OOD results in DiffuseMix Table 17. We will cite DiffuseMix, explain this protocol distinction, and include this generator ablation in the revised appendix.

Comments on the paper

7 comments
AuthorsOpenReview ↗

General Response: Scope of the Paper and Revised Abstract

To clarify the paper’s scope, objective, and contributions, we provide the revised abstract and three revised Introduction paragraphs below. These changes will also appear in the consolidated revised manuscript.

Revised Abstract

Modern machine learning remains largely empirical: architectures and training strategies are commonly discovered through experimentation and explained only afterwards. This work investigates whether a postulate-based geometric theory can instead generate falsifiable predictions and design principles before training. Using the Pursuit of Subspaces (PoS) hypothesis, we instantiate a complete postulate-to-prediction-to-experiment cycle for reconstruction-based unsupervised learning. We derive two distinct categories of predictions. First, failure predictions identify excess learned range as \emph{join blindness}, which permits accurate reconstruction of anomalies, and insufficient capacity as \emph{meet preference}, which suppresses valid nominal variation. Second, learning-dynamics predictions describe how representations should change as geometric compaction improves: nominal components become relatively more separated, while off-union samples move toward the nominal union. We show that the compact nominal union is optimal among nominal-faithful ranges and generally requires a nonlinear reconstruction map. To approximate this ideal geometry, we introduce Dynamic Push and Pull, which excludes probed off-union directions, and Nested Manifold Carving, which applies the same principle recursively in frozen latent spaces. Controlled experiments validate both predicted representation changes in every tested Push-and-Pull configuration. Further experiments demonstrate improvements on standard one-class benchmarks, detection of unseen image degradations, and unsupervised ECG pretraining. Anomaly detection provides a measurable testbed for these geometric predictions, while the ECG study evaluates their relevance beyond detection. The results provide evidence that geometric postulates can generate testable hypotheses and guide objective design in unsupervised representation learning.

The three revised Introduction paragraphs continue in the following comments.

AuthorsOpenReview ↗

General Response (cont.): Revised Introduction, Paragraphs 1–2

Revised Introduction, Paragraph 1

Modern neural networks achieve extraordinary performance, yet architectures and training strategies are still discovered largely through empirical search and explained only afterwards. A central missing ingredient is a predictive mathematical foundation: a compact set of principles from which one can state, before training, which geometric operation a module should perform, which designs should therefore succeed, and under which conditions they should fail. Existing theories each account for part of this picture. Spline theory characterizes piecewise-affine networks through the input-space partitions they induce (Balestriero & Baraniuk, 2018; Humayun et al., 2023), whose number can grow exponentially with depth (Montúfar et al., 2014); rate reduction constructs one architecture forward from an information-theoretic objective (Chan et al., 2022); manifold and tangent-space methods characterize local data geometry without specifying which directions a module should remove (Rifai et al., 2011; Cohen et al., 2020); geometric deep learning designs equivariant architectures from input symmetries (Bronstein et al., 2021); and neural population geometry links representation geometry to classification capacity (Chung et al., 2018). Each is rigorous within its scope, but none states, before training, which geometric operation a module should perform and under which conditions it should fail. Simon et al. (2026) argue that deep learning now needs exactly such a falsifiable, prospectively useful theory. The recently proposed Pursuit of Subspaces (PoS) hypothesis provides a postulate-based geometric foundation for representation learning (Yamac et al., 2026).

Revised Introduction, Paragraph 2

The original PoS work described the ideal behavior of a neural module as nonlinear orthogonal projection onto a compact union of submanifolds and explained how such ideal modules could be extended compositionally without losing the learned structure. It also qualitatively anticipated failure regimes arising when the learned representation departs from this ideal geometry. However, it did not provide an explicit mathematical dictionary connecting these departures to observable failure modes and measurable representation changes, nor did it specify practical learning objectives that drive an ordinary reconstruction network toward the ideal compact union.

The final revised Introduction paragraph continues in the following comment.

AuthorsOpenReview ↗

General Response (cont.): Revised Introduction, Paragraph 3

Revised Introduction, Paragraph 3

This work takes that next step by presenting a concrete example of the complete cycle

postulates⟶predictions⟶controlled experiments⟶design principles.\text{postulates} \longrightarrow \text{predictions} \longrightarrow \text{controlled experiments} \longrightarrow \text{design principles}.

We use reconstruction-based unsupervised learning as the case study because reconstruction error makes deviations from the predicted projection geometry directly observable. From the PoS postulates, we derive two distinct categories of predictions. The first concerns failure: excess learned range produces join blindness, allowing off-union samples to be reconstructed as accurately as nominal data, whereas insufficient capacity produces meet preference and removes component-specific nominal directions. The second concerns learning dynamics: as a representation approaches the PoS ideal, nominal components should become relatively more separated, while samples outside the union should move toward nominal components. We derive these predictions from the postulates and then test them directly in controlled representation-space experiments.

The PoS ideal specifies what geometry should be learned but not how an ordinary network can be driven toward it. We therefore derive two practical learning strategies. Dynamic Push and Pull uses controlled perturbations to probe and exclude off-union directions while preserving nominal observations. Nested Manifold Carving freezes each learned representation and recursively applies the same compaction principle in its latent space. These strategies are not introduced as another task-specific anomaly-detection pipeline; they are theory-derived realizations used to test whether the predicted geometry can be induced in practice. Anomaly detection provides a particularly transparent measurement setting, while the evaluation on unseen image degradations and downstream ECG classification tests whether the learned representation principles extend beyond that setting.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.