acceptodds
Withdrawn from ICLR 2027

Unified Anomaly Understanding and Generation through Causal Diffusion Inference

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Existing anomaly understanding and anomaly generation methods are usually treated as isolated tasks and rely on separate models, which limits knowledge sharing and scalability. In this paper, we investigate how to unify anomaly understanding and anomaly generation in a single latent diffusion model. Motivated by causal effects, we propose a novel factual-counterfactual generation framework, which enables us to capture the pure anomaly-related response as the direct causal effect of the defect reference on the generation process and suppress background interference by subtracting the counterfactual normal-reference effect from the total attention response. Extensive experiments validate our method's superiority in both anomaly generation and downstream detection tasks. Compared with specialized models, it slashes the FID generation score by roughly 50 points and boosts downstream detection performance by around 20% in PRO and 10% in max-F1. The codes will be public after acceptance.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Anomaly generation, Anomaly detection, Diffusion model

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.25 over 4 reviews, above 2% of the others.
Reviewer_VoknOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

Causal-Diff aims to unify anomaly generation and detection in a latent diffusion model. It contrasts attention to defect and normal reference images to guide generation and produce anomaly maps, with boundary-aware mask supervision. The paper evaluates generation on MVTec, VisA, Polyp, and Floor Dirty; detection on MVTec and VisA; and synthetic-data training of downstream detectors. It reports gains over Anomagic and several detection baselines, including lower average FID on MVTec (186.93 to 53.11) and improved downstream results on multiple datasets.

Critical strengths
  • Unifying anomaly generation and detection through reference-conditioned diffusion is a worthwhile direction with potential practical value.
  • Experiments span industrial, medical, and indoor datasets, including downstream detection using synthetic data.
Critical weaknesses
  • The attention difference in Eqs. 4–7 is not shown to estimate the causal effect defined in Eq. 1. The two attention maps use different key sets and softmax normalizations, so their difference does not isolate a causal effect. The experiments do not directly validate this mechanism.
  • The experimental evidence for generation is limited: the comparison is largely against Anomagic, with too few competing methods to establish a comprehensive state-of-the-art advantage. The reported per-category FID scores are also difficult to interpret with the small number of real anomaly samples available in MVTec AD and VisA. The paper should report the real/generated sample counts and protocol, and quantify variability (e.g., repeated estimates or confidence intervals); otherwise the FID ranking may be unstable.
  • Detection training uses pixel-level anomaly-mask supervision, but the training data, splits, reference-image protocol, and baseline settings are not sufficiently specified. The appendix also describes different UNet initializations for the reference and generation branches without clarifying whether their parameters are shared. These gaps prevent assessing the central claims and reproducing the comparisons.
Questions for authors
  • What causal estimand do Eqs. 4–7 compute, given the different softmax key sets? If this is a heuristic attention contrast, what controlled ablation demonstrates its benefit over matched-normalization or single-reference alternatives?
  • Please specify the detection training data and mask sources, train/test splits, reference-image selection, and baseline protocols. Are the reference and generation UNet parameters tied, copied, or independently updated?
Reviewer_U51zOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper focuses on unifying anomaly understanding and generation, two tasks traditionally handled by separate models due to their conflicting objectives. The authors propose Causal-Diff, a unified latent diffusion framework that leverages factual-counterfactual attention to simultaneously generate anomalies and localize defects through shared parameters. Specifically, the method isolates anomaly-related responses by contrasting defective and normal references, while introducing causal gating and boundary-aware supervision to improve generation and detection performance. While experiments report improvements in generation fidelity and downstream detection across multiple datasets, the proposed causal mechanism primarily relies on heuristic attention subtraction rather than rigorous causal identification. Consequently, the results validate effective attention-based background suppression rather than a principled causal framework with broad generalizability.

Critical strengths

The paper attempts to combine anomaly generation and detection within a single diffusion framework. Using the attention responses from the generation process for anomaly localization is an interesting direction, as it avoids training a separate detection head.

Critical weaknesses
  1. The motivation for unifying anomaly localization and generation is unclear. The paper does not justify why these two distinct tasks should share a single model or demonstrate clear advantages over separate task-specific models.

  2. The causal foundation of the proposed method is not rigorous. Although the paper introduces formal definitions of causal effects, it does not establish how the attention difference in Eq. (7) corresponds to these effects. The use of causal terminology therefore lacks sufficient theoretical support.

  3. The computation of the causal attention difference is questionable. The factual and counterfactual branches use different key sets and softmax normalization denominators, making their attention probabilities not directly comparable. The paper provides little justification for subtracting these responses to identify anomalies.

  4. The anomaly detection pipeline is insufficiently specified. The framework requires a background image and an inpainting mask, but the paper does not explain how these inputs are obtained during detection. This raises concerns about whether the method can localize unknown anomalies without additional spatial information.

  5. The experimental comparisons are outdated for anomaly localization. The paper compares against older methods and omits stronger recent baselines, such as AnomalyCLIP and AdaCLIP. This makes it difficult to assess the competitiveness of the proposed framework on either task.

Questions for authors
  1. The selection of reference images is unclear. The method relies on contrasting normal and defective references, yet the paper does not specify how these references are sampled or matched during training and inference. The sensitivity of the results to different reference choices is also not investigated.

  2. The computational efficiency of the unified framework is not adequately evaluated. Maintaining two reference banks and computing separate attention paths introduces additional complexity, but the paper does not report inference latency or memory consumption. It is therefore difficult to assess the practical benefit of sharing parameters across detection and generation.

  3. The generation of anomaly masks is an important part of anomaly synthesis, yet the paper does not explain how the inpainting masks are obtained. This missing detail raises concerns about the reproducibility and practical applicability of the proposed generation pipeline.

  4. The amount of defective training data is not specified. The paper does not report the number or proportion of anomaly images used for training. Since the method relies on defect references and ground-truth anomaly masks, its effectiveness under limited anomaly data remains unclear.

Reviewer_E6DcOpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The goal of this paper is to combine anomaly detection and synthetic anomaly generation into one unified model. To do so it combines a pre-trained VAE, forming as a feature extractor, with a diffusion transformer operating in the latent (feature) space. In its main (generative mode), it takes a background image, a mask indicating where to insert the anomaly, and reference anomaly and "normal" images. It begins by processing the reference images and extracting their latent embeddings at each layer of the model. It then runs the model twice: one where it is allowed to the saved defect hidden states and one where it is not. By measuring the difference in how much attention is paid to each part, it then constructs a gate which combines the hidden states of both, with the goal of removing spurious and non-causal information. This proceeds all through the network to obtain the final image. In its detection mode, it instead aggregates these differences into a scalar score. It is unclear to me to what extent these two modes actually leverage a single unified model. It is also not obvious why, conceptually, synthetic anomalous data generation and anomaly detection should be unified.

From a methodological point of view its claimed contribution is the use of attention patterns to deduce causal relationships in the network.

The method is validated primarily in terms of generative metrics on standard downstream benchmarks. The experimental setting is not fully specified, making it hard to judge how the baselines were evaluated and whether the purported differences are significant.

Critical strengths

None

Critical weaknesses

The method is rather involved and includes several non-obvious design decisions that are not expanded upon in the paper, rendering it impossible to verify its soundness. Its core methodological contribution is the use of attention patterns to derive causal relationships in data, which is not a novel idea as far as this reviewer knows.

Questions for authors
  • Eq (2): How is ss set? What is the purpose of rescaling the embeddings given by the (fixed) VAE?
  • What is the intuition for why utu_t needs to take a downsampled mask m↓m_\downarrow when the masking information is already present in zmz^m? How is the downsampling performed?
  • What is the motivation for averaging the attention patterns over the heads before taking differences, instead of the other way around?
  • Since you apply ReLUs to form the Δ\Deltas, the input to the sigmoid in Eq (8) is bounded by [0,∞)[0, \infty). Its output---that is, the gate---is thus bounded by [1/2,1)[1/2, 1). In other words, you never get an output houth^{out} where the counterfactual path has a weight greater than 1/21/2. Is that intentional? What is the motivation for this?
  • Suggestion: You can probably do both the causal and counterfactual runs with a single forward pass by just constructing the attention mask appropriately.
Reviewer_8G94OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper uses a diffusion inpainting model to do two tasks simultaneously- 1. Generate anomalies, 2. Detect anomalies. The approach is the following:

  1. Two reference branches - encodes a defect and a reference image into two feature banks.
  2. Attention - the generation UNet's queries attend twice, once for a factual pass (with self, defect, normal) and one for a counterfactual pass (self, normal).
  3. Causal effect - It defines the causal effect △=ReLU\triangle = \mathrm{ReLU}(attention on defect tokens in the factual pass − attention on normal tokens in the counterfactual pass). Then △\triangle is used to blend the factual and counterfactual hidden states. △\triangle is supervised against the ground truth defect mask with a class balanced, boundary weighted focal loss. In detection, △\triangle is averaged over queries and layers into an anomaly map on the defect reference image. The max of that is the image level score.

The paper compares against Anomagic in MVTec, VisA generation, localization, and polyp/faeces segmentation. The detection gains come in PRO and F1. AUROC improves only by 0.2 - 1.5 points. The generation gains vary substantially by category. The downstream segmentation results show useful evidence of the generated data being useful.

Critical strengths

S1. The paper presents an interesting idea of combining anomaly detection and generation.

S2. Reading the reference-to-generation attention as a detection map is a clever solution.

S3. Data augmentation gains on faeces and polyps are substantial and practical.

Critical weaknesses

W1. The gate mathematics as written is confusing. Over NN tokens, the mean is at most 1/N1/N, which gives g∈[0.5,0.5+1/4N]g \in [0.5, 0.5 + 1/4N]. For large NN this is basically a constant interval. The authors should clarify where the gains in Table 5 come from, and make requisite fixes.

W2. The detection comparison is very weak. Causal-Diff is trained with ground truth defect masks and needs a paired normal reference at test time. The baselines are zero shot and mask free, which makes it a very unfair comparison.

W3. Anomagic is the only main baseline, and it is a zero-shot generator, while Causal-Diff is trained on in-domain defects and masks. There needs to be comparisons with in-domain trained generators.

W4. There are important ablations that need to be added. For example, a comparison of factual only attention vs the causal difference for detection. Also, ablating on removing the supervision loss entirely.

Questions for authors

Q1. How are the defect and normal references paired? How sensitive is detection to misalignment or to a random normal reference?

Q2. What are the sample sizes for FID and KID, and what is the variance across seeds?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.