acceptodds
Under review as a conference paper at ICLR 2027

Revisiting Batch Normalization for Semi-Supervised Anomaly Detection on Tabular Datasets

Abstract

Batch Normalization (BN) is widely used to accelerate convergence and provide implicit regularization, yet its behavior remains largely unexplored in semi-supervised anomaly detection (SAD) for tabular data. We identify a previously overlooked failure mode, termed distribution collapse, in which the batch-to-batch variability of normalized normal representations progressively diminishes under anomaly-oversampled training. Because SAD repeatedly oversamples a limited set of labeled anomalies, their influence on shared BN statistics can make the standardized relation between normal and abnormal samples increasingly consistent across mini-batches, thereby reducing representation-level stochasticity and weakening a source of BN's implicit regularization. To address this issue while retaining the optimization benefits of BN, we propose Normal-Only Batch Normalization (NoBa), which computes normalization statistics exclusively from normal samples and removes the influence of oversampled anomalies from statistic estimation. Extensive experiments across eleven tabular datasets and five SAD models show that NoBa consistently improves upon conventional BN and alternative normalization methods while retaining BN's fast convergence.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Semi-Supervised Anomaly Detection, Batch Normalization, Tabular Data, Anomaly Oversampling, Implicit Regularization

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_8L6hOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper says that in tabular SAD, oversampling a small pool of labeled anomalies causes something called “distribution collapse”. Shared BN statistics make the normalized mean of normal samples nearly constant across batches, which removes part of BN’s batch noise. So their method NoBa computes BN statistics only from normal samples, then centers them at the Pearson mode, and applies them to all samples. Their experiments show consistent average gains over BN and other normalizers.

The method is practical and the gains over BN look real. But thhe proposed mechanism is not established. Both propositions are algebraic identities, a simpler explanation fits the evidence. The comparisons also have issues that make the gains hard to attribute.

Critical strengths

Some strengths I can see:

  1. This paper has a simple, drop-in method. Labels are already available during SAD training, so computing BN statistics from normal-labeled samples adds nothing at inference time. Inference uses running statistics exactly as standard BN does. It works with any backbone without changing the objective.

  2. The evaluation is broad and good. The paper covers many settings with sweeps over learning rate, batch size and number of labeled anomalies. And NoBa beats BN on average in every configuration.

  3. Table 2 crosses reference set (all vs normal-only) with centering statistic (mean, median, mode). This is a clean ablation as it separates the two components clearly and show that the reference set accounts for most of the gains.

Critical weaknesses

There are also a lot of weaknesses, which outweigh the strengths in my opinion:

  1. FIgures 3 and 7 appear inconsistent with the definitions. Under mean-centered, normal-only BN, the normal-sample z should be exactly zero. But the “Unsup BN normal samples” panel shows high-frequency variation around ±1. Either a different quantity is plotted (post-affine, post-activation, or using running statistics?), or something else is going on.

  2. The collapse mechanism is asserted but not shown. I don't see anything saying Var(q_B) shrinks because anomalies are reused. With 60 anomalies and 256 anomaly slots per batch, each anomaly appears about 4× per batch from the first step. Reuse is constant over training, so I think it cannot drive a decline over epochs.

  3. Many issues in the comparisons I can see: a) Gains are concentrated in two datasets. On DevNet, thyroid alone contributes about 0.034 of the 0.053 average gain over w/out BN, and thyroid plus campaign account for about 90%. NoBa is worse than w/out BN on 5 of 11 datasets. b) Despite saying 10 seeds, there are no standard deviations or significance tests, and many margins are 0.01–0.02. This is a minor point, but still. c) The w/out BN baseline uses ReLU, while BN-based variants use ELU. So it is not the same.

  4. I have a fair concern about this paper's novelty, is it actually new? I'm not familiar with the literature here, but the paper does not convince me.

  5. The paper also does not provide any code, it's not a must but it doesnt help either.

  6. Some more minor points: Figure 2 is hard to interpret. In (a, b), cross-pi comparisons are confounded by the pi^2 factor. Panel (c) is 11 points with no statistics, and the layer measured is unspecified.

Questions for authors

Some questions for the authors, in addition to the weaknesses above:

  1. What exactly is plotted as z̄ in Figures 3 and 7, and why is it non-zero for mean-centered, normal-only normalization?
  2. How does w/out BN perform with ELU? Are input features standardized before the network?
  3. Can you report standard deviations and significance tests (e.g., Wilcoxon or critical-difference diagrams), and the average gains excluding thyroid?
  4. Why do the Table 2 numbers differ from Table 1?
6 comments
AuthorsOpenReview ↗

Response to W1–W2: Figure Consistency and the Mechanism of Distribution Collapse

Thank you for the careful and constructive review. Your reading of the figures, including the appendix, identified a genuine error on our side and helped us improve the paper. In response, we revised the figures for clarity, refined the proposition characterizing the collapse mechanism, and added experiments examining the performance and statistical significance of NoBa. We address each point below and would welcome any further questions or concerns regarding the revised theoretical analysis and empirical evidence.

W1 & Q1. Figure Consistency and Representation Measurements

In the unsupervised BN normal-sample panel of the original Figure 3, we plotted the batch mean μ instead of the mean normalized activation z̄. The panel did not match its label or the definition. We corrected this and redesigned Figure 3. The new figure shows z̄ computed from the normalized activations before the affine transformation. For both BN and NoBa, it reports the mean over all, abnormal, and normal samples. The former appendix Figure 7 has been merged into the revised Figure 3. The panels now agree with the definitions. For BN, the all-sample mean is zero by construction, and the variability appears in the normal and abnormal panels. For NoBa, the normal-sample mean is zero by construction, and the variability appears in the all-sample and abnormal panels.


W2. Mechanism of distribution collapse.

We agree that abnormal sample reuse remains constant throughout training and therefore cannot independently explain the progressive reduction in batch-to-batch variability. Accordingly, we have revised Section 3.2 to explain this phenomenon through the interaction between SAD objectives and conventional BN.

Distribution collapse can arise from SAD training itself and can be further amplified by anomaly oversampling. Let Δ=μabnorm−μnorm\Delta=\mu^{\mathrm{abnorm}}-\mu^{\mathrm{norm}} denote the normal–abnormal mean gap and q=Δ/σallq=\Delta/\sigma^{\mathrm{all}} its standardized counterpart under conventional BN. The batch-to-batch variability of normalized normal representations at epoch tt is defined as

bt=VarB(t)(zˉNall)=π2VarB(t)(q)b_t=\mathrm{Var}_{B}^{(t)}(\bar z_N^{all})=\pi^2 \mathrm{Var}_{B}^{(t)}(q)

where zˉNall\bar z_N^{\mathrm{all}} is the mean normalized activation of normal samples and π\pi is the abnormal ratio.

To characterize the attenuation of this variability, we introduce the saturation ratio s=π(1−π)q2s=\pi(1-\pi)q^2, which measures the contribution of the normal–abnormal mean gap to the shared BN variance. Our new Proposition 1 establishes that, when Δ\Delta has a consistent sign across mini-batches,

bt≤π1−π(1−St),b_t\leq\frac{\pi}{1-\pi}(1-S_t),

where St=EB(t)[s]S_t=\mathbb E_{\mathcal B}^{(t)}[\sqrt{s}] denotes the average square-root saturation. As saturation increases, the upper bound on batch-to-batch variability decreases and converges to zero as St→1S_t\to1.

The asymmetric SAD objective encourages normal scores to concentrate around a common reference and abnormal scores to exceed a separation margin. These effects can reduce the within-group variance relative to the squared mean gap as representations evolve, thereby increasing saturation. Furthermore, increasing π\pi up to 0.5 increases both the between-group contribution to the shared BN variance and the relative emphasis on abnormal separation in the SAD objective. Thus, anomaly oversampling can amplify saturation-induced variability attenuation even when abnormal sample reuse remains constant throughout training. See Section 3.2 and the Appendix B.1 for a detailed discussion of the distribution collapse mechanism and the proof of Proposition 1.

AuthorsOpenReview ↗

Response to W3. Comparison experiments

W3 & Q2,3. Comparison experiments

We address the three concerns (a) concentration of gains, (b) statistics, and (c) activation function.

(a) Concentration of gains. We report the results without thyroid and campaign. NoBa still exceeds both baselines for all five backbones over the remaining nine datasets, although the margin over w/o BN is small for DevNet (+0.006) and PReNet (+0.001), whose gains are concentrated mainly in these two datasets.

MethodDevNetDSADPReNetFeaWADCADAvg.
w/o BN0.635 ± 0.0070.660 ± 0.0080.615 ± 0.0120.612 ± 0.0160.614 ± 0.0060.627 ± 0.007
BN0.578 ± 0.0280.659 ± 0.0170.589 ± 0.0290.548 ± 0.0260.621 ± 0.0110.599 ± 0.011
NoBa0.641 ± 0.0170.679 ± 0.0100.616 ± 0.0200.629 ± 0.0160.647 ± 0.0100.643 ± 0.010

To assess consistency beyond average performance, we additionally conduct a dataset-level rank analysis over the eleven datasets. NoBa achieves the best average rank at both contamination levels, and the Friedman test indicates significant differences among the three methods at both 00% and 22% contamination (χ2=9.455\chi^2=9.455, p=0.0089p=0.0089). The post-hoc Nemenyi test shows that NoBa ranks significantly better than conventional BN at both settings (p=0.008p=0.008), whereas its difference from w/o BN is not significant, consistent with the fact that NoBa does not uniformly outperform w/o BN on every dataset.

Contam.w/o BNBNNoBaNoBa vs. BNNoBa vs. w/o BN
0%2.1822.5451.2730.0080.083
2%1.8182.7271.4550.0080.670

The value of NoBa therefore does not lie in universal dominance over w/o BN. Under oversampling, conventional BN helps on some datasets but substantially degrades performance on others, whereas NoBa reduces this dataset-dependent variability in the usefulness of BN.Even where the gain over w/o BN is marginal, NoBa reduces the need to decide in advance whether BN will be beneficial for a given dataset.

(b) Statistical reliability. To assess statistical reliability, we conducted Wilcoxon and paired t-tests over 10 random seeds, applied to the 11-dataset mean AUPR (Appendix G). We also report the mean AUPR and standard deviation over seeds for each dataset and backbone (Appendix K). At 0% contamination, NoBa exceeds both w/o BN and BN for all five backbones, with p ≤ 0.003 in every comparison (table below). At 2% contamination, the gain over BN is significant under both tests for DevNet, PReNet, FeaWAD, and CAD. DSAD is significant under the paired t-test only (Wilcoxon p = 0.084). The gain over w/o BN is significant under both tests for DevNet, PReNet, and FeaWAD. The results support the statistical reliability of the gains under clean supervision and the advantage over conventional BN for most backbones under contamination.

Contam.BackboneWilcoxon vs. w/o BNPaired t-test vs. w/o BNWilcoxon vs. BNPaired t-test vs. BN
0%DevNet0.002< 0.0010.002< 0.001
0%DSAD0.002< 0.0010.0020.003
0%PReNet0.002< 0.0010.0020.001
0%FeaWAD0.002< 0.0010.002< 0.001
0%CAD0.002< 0.0010.002< 0.001
2%DevNet0.0270.0200.002< 0.001
2%DSAD0.0370.0590.0840.032
2%PReNet0.004< 0.0010.0200.018
2%FeaWAD0.002< 0.0010.002< 0.001
2%CAD0.6950.8390.002< 0.001

(c) Activation function. The w/o BN baseline uses ReLU, whereas the BN-based variants use ELU. We therefore evaluated w/o BN with ELU as well, so that all methods share the same activation (Section 4.7). ELU improves w/o BN for DSAD (0.615 to 0.640), FeaWAD (0.603 to 0.622), and PReNet (0.571 to 0.580). It changes DevNet and CAD by at most 0.003. Under the matched activation, NoBa exceeds w/o BN for all five backbones. The gain of NoBa is not attributable to the activation function alone.

Backbonew/o BN + ReLUw/o BN + ELUNoBa (ELU)
DevNet0.591 ± 0.0080.590 ± 0.0080.644 ± 0.012
DSAD0.615 ± 0.0090.640 ± 0.0080.670 ± 0.012
PReNet0.571 ± 0.0120.580 ± 0.0120.624 ± 0.018
FeaWAD0.603 ± 0.0110.622 ± 0.0120.633 ± 0.011
CAD0.613 ± 0.0100.616 ± 0.0060.653 ± 0.008
AuthorsOpenReview ↗

Response to W4–W6 & Questions: Novelty, Code Availability, Figure 2, and Additional Clarifications

W4. Novelty: Why this is new

  • An overlooked problem in SAD. Tabular SAD research has focused on architectures and objectives, and normalization has been treated as a fixed component. Most methods omit BN. We study BN as a component that interacts with how SAD is trained.
  • A newly defined failure mode, shown theoretically and empirically. SAD relies on oversampling, so anomalies form a large share of every mini-batch and enter the BN statistics. We define the resulting distribution collapse, in which the batch-to-batch variability of the normalized normal representation is suppressed. We explain why it occurs and show it in experiments.
  • Reliable use of BN under oversampling. The benefit of BN varies across datasets and backbones, so neither using nor removing it is uniformly preferable. NoBa reduces the need to choose between BN and w/o BN for each dataset.
  • Compatibility with other modules. NoBa can be combined with other components such as the class-balanced loss. Under the same loss, NoBa improves over BN for all five backbones (Section 4.6).
  • A new research direction. Class imbalance in SAD reaches the normalization layer, not only the loss and the sampler. This motivates further study of normalization in SAD.

W5. Code availability

The implementation of NoBa is ready and will be released on GitHub. To reduce ambiguity in reimplementation, Appendix A describes how NoBa computes its normalization statistics during training and at test time (Algorithm 1). NoBa modifies only the normalization statistics. It can be applied to other SAD models without changes to the sampler or the loss.


W6. Figure 2 Clarifications

We revised Figure 2 following your comments. In Figures 2(a) and 2(b), the original y-axis was VarB(zˉN)\mathrm{Var}_B(\bar z_N), which contains the factor π2\pi^2. Curves with different π\pi were therefore confounded. The y-axis is now VarB(q)\mathrm{Var}_B(q), which removes this factor and allows a direct comparison across π\pi. In Figure 2(c), we added the Pearson correlation coefficient (r=−0.745r=-0.745) over the 11 datasets. All quantities in Figure 2 are measured at the last layer of DevNet. This is now stated in the main text and in the figure caption.


Q1. What is plotted as z̄ in Figures 3 and 7? Please see our response to W1. The original panel plotted the batch mean μ instead of zˉz̄. The revised Figure 3 plots zˉz̄ computed from the normalized activations before the affine transformation, and Figure 7 has been merged into it.


Q2. w/o BN with ELU, and input standardization. Please see our response to W3(c) for the w/o BN baseline with ELU. NoBa exceeds w/o BN with ELU across all five backbones. Input features are standardized before being passed to the network using StandardScaler. The mean and standard deviation are estimated exclusively from the training set, and the same training-derived statistics are applied to the test set.


Q3. Standard deviations, significance tests, and gains without thyroid. Please see our response to W3 (a) and (b). Appendix K reports standard deviations and Appendix G reports the Wilcoxon and paired t-tests. Excluding thyroid and campaign, NoBa still exceeds both baselines for all five backbones.


Q4. Differences between Table 1 and Table 2

The differences arise from the BN implementation. Table 1 uses PyTorch's built-in torch.nn.BatchNorm1d. Table 2 uses a BN module that we implemented with PyTorch operations, in order to control which samples enter the normalization statistics (normal and abnormal, or normal only) and which centering statistic is used (mean, median, or mode). The two implementations perform similarly. Between the BN in Table 1 and the normal & abnormal + mean setting in Table 2, the difference in average AUPR is at most 0.007 for every backbone. The custom module therefore reproduces the behavior of the standard BN, and it is a suitable basis for the ablation in Table 2.

Reviewer_yfV4OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper revisits the role of Batch Normalization (BN) in semi-supervised anomaly detection (SAD) on tabular data. The authors identify a phenomenon termed distribution collapse, where repeatedly oversampled anomalies reduce the batch-to-batch variability of normalized normal representations, potentially weakening BN's implicit regularization. To address this issue, they propose Normal-Only Batch Normalization (NoBa), which estimates normalization statistics exclusively from normal samples while applying the resulting transformation to both normal and abnormal samples. The method further incorporates Pearson-mode-based centering to better accommodate skewed feature distributions.

The authors provide theoretical analyses of representation variability under conventional BN and NoBa. They evaluate the proposed method on 11 tabular datasets across five representative SAD models. Experimental results demonstrate consistent improvements in average AUPR over conventional BN and alternative normalization methods, with additional analyses examining anomaly oversampling, batch size, learning rate, labeled anomaly counts, and label contamination.

Critical strengths
  1. NoBa achieves consistent improvements across 11 datasets and five SAD models. Extensive experiments under different settings further demonstrate its effectiveness and robustness.
  2. NoBa can be easily integrated into existing SAD models without modifying their training objectives. The ablation study confirms that normal-only normalization is the primary source of performance gains.
Critical weaknesses

The core idea of estimating normalization statistics from selected samples is closely related to prior work, including Selective Batch Normalization [1] and Masked Batch Normalization [2]. More importantly, Keras 3.6.0 [3] already supports masked batch normalization, which can implement the mean-based core of NoBa by selecting normal samples for statistic estimation while normalizing all samples. Although NoBa additionally introduces Pearson-mode centering, the paper does not sufficiently clarify its methodological novelty over these existing approaches, nor does it discuss the closely related prior work.

[1] Zhang, Linfeng et al. “Auxiliary Training: Towards Accurate and Robust Models.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 369-378.

[2] Takayama, Natsuki et al. “Masked Batch Normalization to Improve Tracking-Based Sign Language Recognition Using Graph Convolutional Networks.” 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (2021): 1-5.

[3] https://github.com/keras-team/keras/blob/v3.6.0/keras/src/layers/normalization/batch_normalization.py

Questions for authors

See Weaknesses

2 comments
AuthorsOpenReview ↗

Response to W1: Relation to Prior Work

We thank the reviewer for raising an important concern regarding the novelty of NoBa in relation to prior normalization methods that compute statistics from selected samples. This comment prompted us to clarify the distinction between NoBa and existing approaches such as Selective BN [1] and Masked BN [2], particularly in terms of why samples are excluded and how the resulting statistics are applied. We have added the relevant prior work to the revised manuscript and more explicitly articulated why the resulting formulation is specific to anomaly-oversampled tabular SAD.

Difference from prior work. Selective BN computes separate statistics for clean and corrupted images. Its purpose is robustness to corruption. Mixing the two groups in one set of statistics would let the distribution shift of corrupted inputs distort the normalization of clean inputs. Masked BN excludes padding from the statistics. Its purpose is to handle variable-length sequences, where padded positions are not real observations and would distort the statistics. In both methods, selection removes a source of contamination or invalidity from the normalization reference. Neither method addresses our problem. Selective BN targets a group that has a different distribution but is not the object of learning. Masked BN targets padding, which does not exist in tabular data. In tabular SAD, the anomalies that affect the statistics are valid supervision that the objective requires. Normalizing them with separate statistics would give normal and abnormal samples different references and remove the shared reference that SAD relies on to separate them.

NoBa selects samples for a different purpose. The labeled anomalies it excludes are not noise. They remain in training. NoBa excludes them from the statistics because their oversampled presence suppresses the batch-to-batch variability of the normalized normal representation. This distribution collapse can weaken the implicit regularization of BN. A single reference estimated from normal samples is applied to both normal and abnormal samples. NoBa thus separates the samples that supervise learning from the samples that define the normalization reference. Masked BN does not apply to this problem, since tabular data contains no padding. Selective BN can be applied by normalizing normal and abnormal samples with separate statistics. It does not resolve the problem, because the anomalies still determine the statistics used for the abnormal group, and normal and abnormal samples no longer share one reference. The table below shows that NoBa achieves higher AUPR than this variant.

BackboneSelectiveBNNoBa
DevNet0.245 ± 0.0560.644 ± 0.012
DSAD0.296 ± 0.0330.670 ± 0.012
PReNet0.294 ± 0.0440.624 ± 0.018
FeaWAD0.352 ± 0.0420.633 ± 0.011
CAD0.203 ± 0.0610.653 ± 0.008

We agree that the masking functionality in Keras can be used to implement the mean-based statistic selection of NoBa. Our contribution is therefore not the masking operation itself, but identifying why anomaly oversampling creates a normalization-specific failure mode in tabular SAD and formulating a shared normal-derived reference that addresses this interaction.

[1] Zhang, Linfeng et al. “Auxiliary Training: Towards Accurate and Robust Models.” 2020 IEEE/CVF Conference on Computer Vision and Pattern Recognition (CVPR) (2020): 369-378.

[2] Takayama, Natsuki et al. “Masked Batch Normalization to Improve Tracking-Based Sign Language Recognition Using Graph Convolutional Networks.” 2021 16th IEEE International Conference on Automatic Face and Gesture Recognition (FG 2021) (2021): 1-5.

AuthorsOpenReview ↗

Response to W1: Novelty

Why this is new for tabular SAD. Tabular SAD relies on oversampling to exploit scarce labeled anomalies. Under conventional BN, these anomalies enter the normalization statistics. We identify this interaction and characterize it. Once the classes separate, the statistics are shifted toward the anomalies, which suppresses the batch-to-batch variability of the normalized normal representation (distribution collapse). The strength of this effect depends on the standardized normal-abnormal gap, and therefore on the dataset and the model. We are not aware of prior SAD work that identifies this failure mode of BN or the resulting need for a normal-only reference.

Contribution to the field.

  1. A previously overlooked interaction between anomaly oversampling and BN, with an analysis of when it occurs. It opens the study of how class imbalance reaches the BN layer in SAD.
  2. A normalization that preserves the optimization benefits of BN under oversampling, without dataset-specific selection rules or backbone-specific objectives.
  3. Evidence from eleven datasets and five SAD models that neither using nor removing BN is uniformly preferable.
  4. Compatibility with other modules such as the class-balanced loss. Under the same loss, NoBa improves over BN for all five backbones (Section 4.6).

Pearson-mode centering. Normal-only statistics do not remove the skewness of tabular activations, and the mean shifts toward the long tail. We therefore center with the existing Pearson approximation, which combines the mean and the median and requires no density estimation. This is a complementary design choice, not a new mode estimator.

Reviewer_o3apOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper addresses a problem of BN in semi-supervised anomaly detection on tabular data when the labeled anomalies are oversampled. The authors argue that when the same anomalies are repeatedly included in the BN statistics, the batch-to-batch variability of the normal representations decreases and the implicit regularization effect of BN becomes weaker. To mitigate this, they propose NoBa, which computes the BN statistics using only normal samples. NoBa also uses the Pearson mode instead of the mean as the centering statistic. Experiments on several datasets and models show that NoBa outperforms standard BN and other normalization methods.

Critical strengths
  1. It is new to address the interaction between oversampling and BN in semi-supervised anomaly detection on tabular data. In particular, the paper makes the problem concrete by showing that using BN causes a large performance drop on the fraud dataset.

  2. Since the method only changes which samples are used to compute the normalization statistics, it can be easily applied to existing models.

  3. The paper compares various normalization methods across several backbones and datasets. The ablation study also separates the effect of each component.

Critical weaknesses
  1. The results are reported without standard deviations or any statistical significance test, so it is hard to judge how reliable they are. The authors should report per-dataset standard deviations over the seeds and run a significance test. Also, the improvement of NoBa over w/o BN comes mostly from a few datasets, and under 2% contamination the gain is small. For example, for DevNet, if thyroid and campaign are excluded, the average improvement on the remaining datasets is very small. Under 2% contamination, NoBa even performs worse than w/o BN in some cases.

  2. There is no experiment with reweighting. The paper mentions both oversampling and reweighting as common practice, but only oversampling is used in the experiments that support the main claim. I think the case of using BN together with reweighting should be tested. In addition, the relationship between NoBa and improvements on the sampling side is unclear. For example, it would be good to check whether there is any additional gain when both are used together.

  3. The training setup of the backbones is different from the original papers. This paper applies 1:1 oversampling to all backbones, but DSAD originally uses reweighting instead of oversampling, and it also uses BN. So the BN problem shown in this paper may not appear under the original training setup, and may instead be caused by forcing oversampling.

  4. BN and NoBa use ELU as the activation function, while w/o BN uses ReLU. As a result, the performance difference between with and without BN is mixed with the effect of the activation function. For a careful evaluation, results that separate these two factors are needed.

  5. Only two contamination ratios (0% and 2%) are considered. A more detailed sensitivity analysis is needed. For reference, related work such as DevNet covers a wider range of contamination ratios. Also, CAD, one of the backbones, was designed for unlabeled data contaminated at the natural anomaly ratio, and its paper actually criticizes the common practice of keeping the anomaly ratio below 2%. However, this paper evaluates CAD mainly under the 0% contamination setting.

Questions for authors
  1. DevNet is set up with three hidden layers in this paper, but the default setting in the original DevNet paper is one hidden layer. The original paper also reports that the three-hidden-layer version performs worse, because there are only a few labeled anomalies. Does the improvement of NoBa still hold with the default DevNet setting?

  2. The anomaly ratio of the fraud dataset is only 0.17%, so there are not enough anomalies to build 2% contamination by injecting real anomalies. How exactly were 2% of the normal-labeled training samples contaminated? (e.g., replaced with anomalies, or anomalies added)

  3. In the batch size experiment in Figure 9, is the number of batches per epoch also fixed at 20? If so, the amount of training data also increases as the batch size grows, so it is hard to interpret the performance drop with larger batch sizes as a result of reduced mini-batch stochasticity.

16 comments
AuthorsOpenReview ↗

Response to W1: Statistical Reliability, Gain Concentration, and Label Contamination

We thank the reviewer for the careful and constructive evaluation. The comments prompted us to strengthen the empirical validation of NoBa, particularly regarding statistical reliability, the concentration of performance gains, robustness to label contamination, and the interaction between anomaly oversampling and alternative balancing strategies. We have added the corresponding analyses and experiments to the revised manuscript and address each point below.

W1 statistical reliability, concentration of gains, 2% contamination

1. Statistical reliability. We report the mean AUPR and seed-wise standard deviations for each dataset, backbone, and normalization type in Appendix K, and the Wilcoxon and paired t-test results (10 seeds) in Appendix G. At 0% contamination, NoBa outperforms both w/o BN and BN for all five backbones (all p ≤ 0.003). At 2% contamination, the gain over BN is significant under both tests for DevNet, PReNet, FeaWAD, and CAD (DSAD: paired t-test only, Wilcoxon p = 0.084), and the gain over w/o BN is significant for DevNet, PReNet, and FeaWAD.

Contam.BackboneWilcoxon vs. w/o BNPaired t-test vs. w/o BNWilcoxon vs. BNPaired t-test vs. BN
0%DevNet0.002< 0.0010.002< 0.001
0%DSAD0.002< 0.0010.0020.003
0%PReNet0.002< 0.0010.0020.001
0%FeaWAD0.002< 0.0010.002< 0.001
0%CAD0.002< 0.0010.002< 0.001
2%DevNet0.0270.0200.002< 0.001
2%DSAD0.0370.0590.0840.032
2%PReNet0.004< 0.0010.0200.018
2%FeaWAD0.002< 0.0010.002< 0.001
2%CAD0.6950.8390.002< 0.001

We also conducted a dataset-level analysis, ranking the three methods on each of the 11 datasets by the 10-seed mean AUPR averaged over the five backbones (lower is better). The Friedman test is significant at both ratios (χ² = 9.455, p = 0.0089). Per backbone, it is significant for DevNet, FeaWAD, and CAD, but not for DSAD (p = 0.234) and PReNet (p = 0.178). For DSAD, BN itself performs well because its loss limits the influence of oversampled anomalies (see W3), so the methods are hard to separate. For PReNet, the gain over w/o BN is driven by thyroid and campaign (+0.001 without them), so it is not consistent across datasets.

Contam.w/o BNBNNoBaNoBa vs. BN (Nemenyi p)NoBa vs. w/o BN (Nemenyi p)
0%2.1822.5451.2730.0080.083
2%1.8182.7271.4550.0080.670

Its difference from w/o BN is not significant, although its average rank is better and the Nemenyi test has limited power with 11 datasets. The seed-level tests support the gain over both baselines at 0% contamination, whereas the dataset-level analysis supports the gain over BN (diagrams in Appendix G).

2. Concentration of gains. Excluding thyroid and campaign, NoBa still exceeds both w/o BN and BN for all five backbones at 0% contamination. The margin over w/o BN is clear for DSAD, FeaWAD, and CAD (+0.019, +0.017, +0.033). It is marginal for DevNet (+0.006) and PReNet (+0.001), whose gains come mainly from thyroid and campaign. The preferable baseline varies across datasets. BN yields the largest gains on thyroid and campaign, whereas it degrades considerably relative to w/o BN on fraud. NoBa does not outperform both baselines on every dataset. Its contribution lies in reducing the dependence of performance on this choice. By excluding oversampled anomalies from the normalization statistics, NoBa preserves the benefit of BN on datasets where BN is advantageous, mitigates the degradation induced by anomaly oversampling, and tends to maintain performance close to that of the better baseline across datasets.

MethodDevNetDSADPReNetFeaWADCADAvg.
w/o BN0.635 ± 0.0070.660 ± 0.0080.615 ± 0.0120.612 ± 0.0160.614 ± 0.0060.627 ± 0.007
BN0.578 ± 0.0280.659 ± 0.0170.589 ± 0.0290.548 ± 0.0260.621 ± 0.0110.599 ± 0.011
NoBa0.641 ± 0.0170.679 ± 0.0100.616 ± 0.0200.629 ± 0.0160.647 ± 0.0100.643 ± 0.010

3. 2% contamination.

NoBa is designed to remove the distortion that oversampled labeled anomalies induce in BN statistics. It does not target mislabeled anomalies in the normal-labeled set. At 2% contamination, NoBa falls below w/o BN for some backbones, which is attributable to how the two methods handle contaminated labels. Because NoBa estimates its normalization statistics from normal-labeled samples, true anomalies among them enter the mean and variance and affect every sample normalized with these statistics. The relevant comparison is thus with conventional BN, and NoBa achieves a higher average AUPR than BN for all five backbones at 2% contamination and at every ratio from 0% to 10% (Appendix F).

AuthorsOpenReview ↗

Response to W2–W3: Oversampling, Reweighting, and Backbone Training Settings

W2 Oversampling, reweighting

To examine the interaction with reweighting, we compare π=0.5\pi=0.5 oversampling with a lower abnormal ratio (π=0.1\pi=0.1) combined with loss reweighting, over eleven datasets and five backbones (Section 4.6). We adopt the class-balanced loss based on the effective number of samples [1]. With reweighting at π=0.1\pi=0.1, conventional BN outperforms w/o BN for all five backbones. It also improves over its π=0.5\pi=0.5 counterpart for all five (e.g., DevNet 0.571 to 0.627, FeaWAD 0.548 to 0.605). Under the same sampling and loss, replacing BN with NoBa yields a further gain for all five backbones (+0.007 to +0.041). NoBa at π=0.5\pi=0.5 without reweighting exceeds BN at π=0.1\pi=0.1 with reweighting for DevNet, PReNet, FeaWAD, and CAD, and equals it for DSAD. Combining NoBa with reweighting at π=0.1\pi=0.1 gives the highest average AUPR of NoBa for DevNet, DSAD, PReNet, and FeaWAD. For CAD, π=0.5\pi=0.5 without reweighting is higher (0.653 vs. 0.643). NoBa's gain also persists in most settings when dropout is applied. This indicates that its benefit does not merely compensate for insufficient explicit regularization, and that NoBa can be combined with existing regularization strategies. In summary, adjusting the sampling ratio and reweighting can mitigate the degradation of conventional BN, whereas NoBa directly addresses the statistics-estimation problem of BN even under training conditions that make full use of abnormal supervision, and acts complementarily to existing sampling and regularization strategies.

[1] Cui, Yin, et al. "Class-balanced loss based on effective number of samples." 2019 IEEE/CVF conference on computer vision and pattern recognition (CVPR). IEEE, 2019.

BackboneBN typeπ=0.5π=0.1 + class-balanced loss
DevNetw/o BN0.591 ± 0.0080.553 ± 0.022
BN0.571 ± 0.0250.627 ± 0.026
NoBa0.644 ± 0.0120.659 ± 0.010
DSADw/o BN0.615 ± 0.0090.618 ± 0.012
BN0.649 ± 0.0220.670 ± 0.020
NoBa0.670 ± 0.0120.683 ± 0.008
PReNetw/o BN0.571 ± 0.0120.576 ± 0.009
BN0.580 ± 0.0330.609 ± 0.022
NoBa0.624 ± 0.0180.650 ± 0.009
FeaWADw/o BN0.603 ± 0.0110.577 ± 0.013
BN0.548 ± 0.0290.605 ± 0.017
NoBa0.633 ± 0.0110.643 ± 0.011
CADw/o BN0.613 ± 0.0100.594 ± 0.009
BN0.624 ± 0.0070.636 ± 0.005
NoBa0.653 ± 0.0080.643 ± 0.006

W3. Backbone training settings differ from the original implementations

The original DSAD protocol does not use this abnormal batch ratio. DSAD was developed mainly for image anomaly detection, where its network architectures include BN. We retain BN for DSAD in the tabular setting and compare it with w/o BN and NoBa. For all backbones, we apply oversampling with a fixed abnormal batch ratio of π=0.5\pi=0.5 so that normalization methods are compared under a controlled training setting. The DSAD objective is

1n+m∑i=1n∥ϕ(xi)−c∥22+ηn+m∑j=1m(∥ϕ(x~j)−c∥22)y~j\frac{1}{n+m}\sum_{i=1}^{n}\lVert \phi(x_i)-c\rVert_2^2+\frac{\eta}{n+m}\sum_{j=1}^{m}\left(\lVert \phi(\tilde{x}_j)-c\rVert_2^2\right)^{\tilde{y}_j}

where nn is the number of unlabeled samples, mm is the number of labeled samples (y~j=+1\tilde{y}_j=+1 for labeled normal samples and y~j=−1\tilde{y}_j=-1 for labeled anomalies), and η\eta controls the relative contribution of the labeled and unlabeled terms. The objective does not reweight normal and abnormal samples by class.

DSAD is also the backbone on which BN shows little of the degradation observed elsewhere. BN outperforms w/o BN for DSAD (0.649 vs. 0.615 at π=0.5\pi=0.5), and its performance varies little with the abnormal batch ratio π\pi (Appendix C). We attribute this to the abnormal term of the loss. For labeled anomalies (y~=−1\tilde{y}=-1) the term is the inverse of the squared distance to the center, so its gradient shrinks as anomalies move away from cc during training. The influence of the abnormal gradient therefore decreases over training, which acts similarly to a class-balanced weight and makes BN robust to oversampling. The effect of oversampling on BN depends on the backbone and the dataset.

The key point of the DSAD result is that NoBa retains its gain even where the benefit of BN is clearly visible. NoBa achieves an average AUPR of 0.670 against 0.649 for BN. By estimating normalization statistics exclusively from normal samples, it preserves the benefits of BN while preventing labeled anomalies from affecting the batch statistics, which provides a normalization strategy tailored to tabular SAD.

AuthorsOpenReview ↗

Response to W4–W5: ELU Activation Function and Additional Contamination Ratio

W4. ELU Activation Function

w/o BN uses ReLU, whereas BN and NoBa use ELU. The effect of normalization is therefore confounded with that of the activation function. To separate the two, we add a w/o BN baseline with ELU to Figure 8 in Section 4.7 and compare all methods under the same activation. Replacing ReLU with ELU improves w/o BN for DSAD (0.615 to 0.640), FeaWAD (0.603 to 0.622), and PReNet (0.571 to 0.580). It leaves DevNet and CAD essentially unchanged (changes within 0.003). With ELU in both, NoBa achieves a higher average AUPR than w/o BN for all five backbones (+0.054 for DevNet, +0.030 for DSAD, +0.044 for PReNet, +0.011 for FeaWAD, +0.037 for CAD). The gain of NoBa is not explained by the choice of activation function alone.

Backbonew/o BN + ReLUw/o BN + ELUNoBa (ELU)
DevNet0.591 ± 0.0080.590 ± 0.0080.644 ± 0.012
DSAD0.615 ± 0.0090.640 ± 0.0080.670 ± 0.012
PReNet0.571 ± 0.0120.580 ± 0.0120.624 ± 0.018
FeaWAD0.603 ± 0.0110.622 ± 0.0120.633 ± 0.011
CAD0.613 ± 0.0100.616 ± 0.0060.653 ± 0.008

W5. Additional Contamination Ratio

To evaluate NoBa under a broader range of label contamination, we extend the contamination ratio from 0% and 2% to 5% and 10% and report the results in Figure 14 of Appendix F, over eleven datasets and five backbones. NoBa achieves a higher average AUPR than conventional BN for all five backbones at every ratio. Relative to w/o BN, NoBa remains superior across the evaluated range for PReNet and FeaWAD. For DevNet, it performs better at 0% and 2% contamination, becomes comparable at 5%, and falls below w/o BN at 10%. For DSAD and CAD, w/o BN performs better under contaminated supervision. Because NoBa estimates its normalization statistics from normal-labeled samples, contamination of this set affects the statistics directly, whereas w/o BN uses no batch statistics. Nevertheless, NoBa avoids the distribution collapse of conventional BN, which is consistent with its higher AUPR than BN across all ratios.

Comments on the paper

2 comments
AuthorsOpenReview ↗

General Response: Summary of Revisions

We thank all reviewers for their constructive comments. The revised manuscript changes the main paper and the appendix as follows.

Main paper

  • Section 3.2 (revised). The former Proposition 1 is now Definition 1. A new Proposition 1 formalizes the mechanism of distribution collapse.
  • Section 2.2 (revised). Selective BN is added to the discussion of related BN variants.
  • Figure 2 (revised). The y-axis of panels (a, b) is now Var_B(q), and panel (c) reports the Pearson correlation coefficient.
  • Figure 3 (revised). The previous measurement of zˉz̄ for unsupervised BN normal samples is replaced by the zˉz̄ of all, abnormal, and normal samples under semi-supervised BN and NoBa.
  • Section 4.6 (new). Class-balanced loss versus normal-only statistics.
  • Section 4.7 (new). ELU versus ReLU.

Appendix

  • Appendix B.1 (revised). Proof of the new Proposition 1.
  • Appendix C (new). Abnormal batch ratio (Figures 9 and 10).
  • Appendix D (new). Dropout (Figure 11).
  • Appendix F (new). Contamination ratio (Figure 14).
  • Appendix G (new). Statistical significance (Table 4, Figure 15).
  • Appendix J (revised). Add contamination generation procedure.
  • Appendix K (revised). Standard deviations added to Tables 7, 8, 9, and 10.

Detailed responses to each reviewer follow in the individual comments.

AuthorsOpenReview ↗

General Response: Additional Appendix Revisions

We have further revised the manuscript by adding Appendix K, which provides an analysis of distribution collapse during SAD training.

  • Appendix K (new). Analysis of Saturation and Batch-to-Batch Variability
  • Appendix L (revised, previously Appendix K). Standard deviations added to Tables 7–10.

The previous Appendix K, which reports the standard deviations in Tables 7–10, has consequently been moved to Appendix L. This change was made to place the additional analysis before the extensive experimental tables. We clarify this renumbering to avoid confusion with references in our earlier rebuttal responses. We will further reorganize the Appendix in a subsequent revision.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.