acceptodds
Withdrawn from ICLR 2027

Finer Understanding, Finer Control: Towards Controllable and Generalizable Anomaly Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Industrial anomaly generation has emerged as an effective approach to alleviate defect-data scarcity, yet existing methods, while generalizing to unseen object categories, remain limited in precisely controlling fine-grained anomaly attributes. In this work, we study zero-shot controllable anomaly generation, which requires both accurate attribute control and strong cross-category generalization. To avoid disturbing the pre-trained generalizable generative prior through direct generator training, our key insight is to cast controllable generation as a visual understanding problem with the generator kept frozen, and this can be inherently supported by leveraging a unified multimodal understanding–generation model (UMM). Based on it, we present the first Controllable and Generalizable Anomaly Generation framework, comprising grounded attribute understanding, on-policy generation alignment, and inference-time corrective steering. In particular, in the UMM, we first establish reliable anomaly attribute understanding and then use the grounded attribute space to align condition signal for the frozen generator via reinforcement learning. Our method successfully enables precise control over position, size, shape, and orientation on zero-shot object categories. Extensive cross-dataset experiments demonstrate state-of-the-art controllability and visual quality. We further show that stronger attribute understanding leads to more faithful generation and improves targeted augmentation for downstream anomaly detection.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
anomaly generation, anomaly detection, unified models

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_dYyqOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
1
Read review
Paper summary

To achieve finer control while preserving good generalization in anomaly generation, this work proposes Controllable and Generalization Anomaly Generation (CGAG). This method formulates controllable generation as a visual understanding problem and employs a fixed pretrained generator to retain the generalization prior. Specifically, it introduces grounded attribute understanding, on-policy generation alignment, and test-time attribute steering to achieve these goals. Empirical evaluation under the ViaA to MVTec AD setting demonstrates improved anomaly controllability and effectiveness in the downstream detection task.

Critical strengths
  1. Anomaly generation is a crucial and urgent problem for industrial detection scenarios, especially in the era of LLMs.

  2. The paper is well structured and easy to follow.

  3. The empirical results in Table 2 demonstrate the effectiveness of the generated anomalies.

Critical weaknesses
  1. The research motivation is not clearly articulated. The authors should explain why finer controllability is important for industrial anomaly generation.

  2. Based on the training recipe, a well-trained model constructs a good mapping between an input (In,c,D)(I_n, c, D) and a generated anomaly IaI_a. To encourage generation diversity, varying cc and DD is inevitable, which introduces new problems for the generation process: (1) How should these DD be designed, and why would such designs be reasonable? How many are sufficient? These problems impede the deployment in the real-world detection tasks.

  3. The scope of empirical evaluation is limited, mainly to benchmarks and experimental settings (leave-one-out would be better). Moreover, for anomaly generation, the most important evaluation is to verify the effectiveness of generated anomalies on different detection algorithms under scarce real anomalous samples. The proposed method conducts this experiment on only one detection algorithm.

  4. The ablation study in Table 3 indicates that the positive gains from Grounding and Alignment are quite limited, suggesting an inconsistency between the method design and the empirical evaluation.

Questions for authors
  1. In Table 2, what is the detailed difference between generic and targeted augmentation?

  2. Based on the proposed method, if instruction e can be regarded as an anomaly generation prompt? If yes, can the authors compare the contributions of different prompt construction strategies?

Reviewer_GryKOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper studies controllable anomaly generation across unseen object categories without target-domain parameter updates and proposes the CGAG framework. Built upon InternVL-U, CGAG keeps the visual generation module frozen and updates the understanding-side LoRA adapters through a two-stage GRPO procedure. First, it grounds attribute understanding using real anomalies and geometric attributes derived from their masks. Second, it optimizes the generation-conditioning policy using attribute rewards computed by an EMA copy of the model on self-generated images. During inference, gradient-based steering is further applied to the generation latents using a soft attribute discrepancy objective. The paper trains on VisA and evaluates cross-dataset transfer to MVTec AD, measuring controllability over position, size, shape, and orientation, and investigates targeted data augmentation for detector hard cases.

Critical strengths
  1. The motivation and overall framework are clear, the problem is well-defined, and the proposed approach has practical value.
  2. The proposed method achieves some improvements in anomaly generation quality over the baselines in the cross-dataset setting.
  3. The ablation studies provide a comprehensive evaluation of the individual components.
Critical weaknesses
  1. The self-evaluative feedback mechanism may lead to reward hacking. Both the generation policy and its training reward rely on the same attribute model, potentially allowing the model's predictions to match the requested attributes without ensuring that the generated anomalies actually exhibit those attributes.
  2. The core visual-validity reward rtexr_{\mathrm{tex}} is not fully defined. This lack of detail makes the reward design difficult to reproduce and its effectiveness difficult to assess.
  3. The evaluation does not adequately support claims of multi-attribute controllability. Appendix E explicitly states that each of the 36,500 generation test prompts specifies only one attribute. Therefore, high average scores for the four attributes individually do not demonstrate that a single generated image can simultaneously satisfy multiple constraints.
  4. The paper does not explain how the generated anomaly masks used to compute the core controllability metrics are obtained. This omission raises concerns about the reproducibility and reliability of the evaluation protocol.
  5. The computational overhead of inference-time steering is not evaluated. Repeated latent-gradient updates may increase inference latency and computational cost, but the paper reports neither inference speed nor the associated overhead.
Questions for authors
  1. How are the generated anomaly masks and object foreground masks obtained?
  2. Could the authors provide complete definitions of rtexr_{\mathrm{tex}} and τtex\tau_{\mathrm{tex}}, as well as details of the feature extractor and the construction of the target region?
  3. Could the authors report quantitative results for joint control of two, three, and four attributes, respectively?
  4. What is the end-to-end inference speed of the anomaly detection pipeline?
Reviewer_U9f4OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes CGAG, a framework for controllable industrial anomaly generation on unseen object categories. It aims to synthesize defects with requested positions, sizes, shapes, and orientations from normal images, without requiring anomaly reference images or user-supplied masks at inference. The method builds on InternVL-U, freezes its visual generator, and adapts the shared semantic backbone through two GRPO stages. The first stage grounds attribute predictions using geometric labels extracted from real anomaly masks. The second stage samples editing instructions and aligns generated images with the requested attributes using an exponential-moving-average evaluator and a visual-validity reward. During inference, a differentiable attribute objective further steers latent states through gradient updates. Experiments train on VisA and evaluate transfer to MVTec AD, reporting improved geometric alignment relative to the underlying model. The paper also uses diagnosed detector failures to guide targeted synthesis, obtaining stronger downstream results than generic augmentation under a matched synthetic-data budget. The approach presents a coherent connection between visual understanding and controllable generation, while its evaluation protocol and the attribution of gains remain important points for clarification.

Critical strengths

1.The problem formulation address a useful distinction between producing plausible anomalies and producing anomalies with specified geometric properties. Sections 3.2–3.5 present a coherent way to connect these goals through a common attribute space, reusing understanding for both generation feedback and inference guidance. Keeping the visual generator fixed and requiring neither anomaly reference images nor user-supplied masks at inference makes the proposed interface attractive for generating defects on new object categories.

2.The comparison with the underlying InternVL-U model provides concrete evidence of improved control. In Table 1, average alignment increases from 73.65 to 75.88, with improvements in all four attributes, while IS increases from 2.10 to 2.18 and the reported IC-L remains 0.38. Table 3 further separates training-stage adaptation from inference steering, and Figures 5–7 illustrate how requested attributes affect generated defects. These analyses make the reported gains easier to examine.

3.The downstream experiment connects controllability to a practical application beyond image synthesis. Sections 4.1 and 4.3 specify the same detector architecture, synthetic-data budget, normal-image pool, defect types, and training schedule for generic and targeted augmentation. Under this reported setup, Table 2 shows targeted CGAG improving hard-case I-ACC from 79.13 to 82.39 and P-PRO from 84.66 to 88.17. Controlling these factors is a useful step toward isolating the benefit of the requested attribute distribution.

Critical weaknesses
  1. The controllability evaluation needs a clearer account of its reference labels and what the reported scores measure. Section 4.1 derives realized attributes from generated anomaly masks, but the manuscript does not explain how those masks and object foregrounds are obtained or validated. This matters because every geometric metric depends on them. The scoring rule assigns partial credit to mismatched attributes, so alignment is not an exact request-satisfaction rate. Moreover, the generation test protocol specifies one controlled attribute per prompt; averaging these results does not establish simultaneous compositional control, although qualitative examples of joint requests are provided. An explicit mask protocol, handling of absent or inapplicable defects, and exact-match results for a limited set of joint requests would make the central evaluation substantially more interpretable.

  2. The downstream protocol does not clearly separate diagnosing failures from evaluating generalization after refinement. Section 4.3 constructs a hard-case pool using PatchCore, WinCLIP, and DRAEM predictions, uses its attribute configurations to direct synthesis, and reports performance on a fixed hard-case subset. It is unclear whether these are distinct development and evaluation samples, or whether the reported recovery concerns the same diagnosed cases. The latter can be a useful adaptation setting, but it supports a different conclusion from improvement on unseen failures. Please specify the split, pool size, selection procedure, and data used for checkpoint selection, and distinguish recovery of known cases from performance on an untouched evaluation set.

  3. The reliability and implementation of the generation feedback require further substantiation. The implication in Equation (3) depends on grounded attribute predictions remaining accurate on generated images, whereas Section 3.4 uses an EMA of the adapter being updated during alignment. Figure 3 reports understanding improvements, but does not establish evaluator fidelity on this evolving generated distribution. Evaluating predictions against independently obtained masks on generated examples, or comparing with a fixed Stage I evaluator, would directly address this premise. Reproducibility is also limited by the visual-validity reward introduced in Section 3.4. The manuscript illustrates successful and failed generations, but does not specify the feature extractor, spatial aggregation, and threshold defining rtex. These definitions are necessary to assess the feedback mechanism.

  4. The evidence supports an association between understanding and control more clearly than it isolates understanding as the source of the gains. Figure 3(a) contains only four model points, including the adapted model and its own backbone, while shared-backbone adaptation changes generation conditioning as well as understanding. Table 3 is informative: grounding plus alignment improves average control by 0.66 points, whereas steering alone improves it by 1.45 points. The reported inference configuration allows up to ten gradient updates at each of five sampling steps, but their runtime cost is not reported. A comparison controlling inference compute, together with uncertainty estimates for these modest differences, would help establish the contribution of learned understanding beyond additional inference optimization.

Questions for authors

See above

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.