- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper studies how deep one-class classification (DOCC) training affects unseen normal and anomalous inputs. It decomposes representation changes into initialization, accumulated updates transferred through an initial gradient-coupling kernel, and a bounded remainder. Under stability and kernel nondegeneracy assumptions, it establishes contraction of normal training representations and a sufficient condition for anomaly detectability. It further characterizes the coupling under Gaussian initialization and extends the displacement analysis to Deep SAD. MNIST experiments examine these approximations and their implications for detection.
Critical strengths
Clear analytical framework. The trajectory decomposition explicitly separates errors from local linearization, Jacobian drift, and weight decay, providing a unified description of training effects on unseen inputs.
Transparent evaluation of approximation limits. The experiments examine how approximation quality varies with training time, width, and depth, including cases where it deteriorates. The limitations acknowledge unresolved displacement directions and differences between normal and anomalous responses.
Critical weaknesses
Limited experimental coverage. Experiments use only MNIST, with the main detection and anomaly-supervision results focusing on digit 3. Please report these results across all ten tasks and include CIFAR-10 or another data domain to assess whether the proposed explanations extend beyond this setting.
Initialization and training effects are insufficiently separated. Report initial detection performance and distance distributions, then quantify displacement and final miss rates across initial-distance groups. Also report the fraction of test anomalies satisfying the detection sufficient condition to demonstrate its practical informativeness.
Theoretical novelty remains unclear. DOCC can be viewed as squared-loss regression with a constant target (c), making its kernel-driven dynamics and training-residual contraction closely related to existing NTK analyses. Please identify which results require substantive new arguments rather than direct specialization, and clarify the technical advances in the finite-time error bounds.
Anomaly-specific insights remain limited. The same dynamics apply to normal and anomalous inputs, but the analysis does not establish when their coupling, displacement directions, or final scores differ systematically. A complete detection theory is unnecessary, but the paper should clarify what its framework adds to understanding normal–anomaly separation beyond generic training dynamics, ideally through controlled experiments.
Questions for authors
Please address the concerns raised in the weaknesses above.
4 comments
Response to Reviewer 4Tgg (2/2)
W3. Theoretical novelty remains unclear. DOCC can be viewed as squared-loss regression with a constant target c, making its kernel-driven dynamics and training-residual contraction closely related to existing NTK analyses. Please identify which results require substantive new arguments rather than direct specialization, and clarify the technical advances in the finite-time error bounds.
A: Thank you for your comment. First, we would like to emphasize that our main contribution is to explain the behavior of unseen inputs in DOCC: Theorem 1 shows how normal-training updates are transferred to an unseen input through gradient coupling, which explains the success and failure cases of detection. The same viewpoint is then extended to explain the effect of anomaly supervision.
Second, the finite-time result under the independent Gaussian initialization used in the paper, now established in Appendix E of the revision, is not a direct special case of existing analyses:
(1) Zhou et al. (2026a): their local derivative and linearization bounds rely on a cancellation that holds only for symmetric initialization, so they do not apply here.
(2) Zou et al. (2020): they work under independent initialization but provide perturbation ingredients rather than the local derivative and linearization bounds required here.
(3) Combining the two: this still does not directly yield our result, because the argument of Zhou et al. breaks exactly at the cancellation step. Our Lemma 9 replaces the role of this cancellation with a new backward-product stability bound, which yields the required bounds with the same dependence up to logarithmic factors, and hence the finite-time error bound under independent initialization.
Given these bounds, Theorem 2 follows standard NTK-type arguments and Theorem 3 is a direct consequence of Theorem 1.
W4. Anomaly-specific insights remain limited. The same dynamics apply to normal and anomalous inputs, but the analysis does not establish when their coupling, displacement directions, or final scores differ systematically. A complete detection theory is unnecessary, but the paper should clarify what its framework adds to understanding normal–anomaly separation beyond generic training dynamics, ideally through controlled experiments.
A: Thank you for this comment. Indeed, our framework provides a concrete mechanism for normal–anomaly separation: unseen inputs follow the same training dynamics, but they can respond differently because their coupling to the normal training samples differs. By Theorem 4, under Gaussian initialization, inputs that are more similar to the training normals receive stronger transfer of normal-training updates.
To examine this mechanism directly, we added controlled experiments on MNIST and Fashion-MNIST comparing held-out normals and anomalies at the same initial distance from the center (see Figure 9 and Table 4 in Appendix G.3). We find that anomalies are more weakly coupled to the training normals, undergo less center-directed displacement, and remain farther from the center after training. After controlling for initial distance, further conditioning on coupling strength substantially reduces the remaining separation in final distance (AUROC from 0.89 to 0.78 on MNIST and from 0.76 to 0.62 on Fashion-MNIST).
We have also clarified this point in the new Discussion section (Section 6): the contraction of unseen inputs is selective rather than uniform, and coupling to the training normals is a key source of this selectivity.
Thank you for the careful responses and additional experiments. I appreciate the authors' efforts. However, I still have some reservations about the practical explanatory value and broader applicability of the theoretical framework. Therefore, I will maintain my original score of 2 .
Thank you for reading our responses and for acknowledging the additional experiments. We would like to briefly address the two remaining reservations you mention.
Practical explanatory value. The framework makes statements about individual unseen inputs, and we tested their implications directly across all ten tasks of both datasets:
- It closely tracks the trained detector. At , the first-order description stays within 0.01 AUROC of DOCC up to a step between 78k and 200k (out of 200k) on every task (Appendix G.3.1).
- It identifies a mechanism behind normal–anomaly separation. At the same initial distance, held-out normals contract by 47–56% and anomalies by 27–38% on MNIST (48–57% versus 35–41% on Fashion-MNIST), consistent with their different coupling to the training normals (Table 2 and Figure 9).
- It gives a testable prediction under intervention. When all labeled anomalies are drawn from MNIST digit 7, Deep SAD recovers 98% of the missed 7s but only 47% of the other anomalies, compared with 70% and 79%, respectively, under a uniform labeled set (Table 7).
These findings also point to concrete directions for method design: they identify gradient coupling as a quantity that can be shaped, motivating selective contraction through representation design and coupling-aware selection or weighting of labeled anomalies, as discussed in Section 6.
Broader applicability. Theorems 1–3 are established under general local-regularity and coupling conditions (Theorems 5–7 in the appendix), rather than assumptions specific to ReLU networks or MNIST. The ReLU/Gaussian setting is used to verify these conditions and obtain an explicit kernel. Section 6 explicitly states the scope of the current guarantees and which extensions, including adaptive optimization and substantial feature learning, require separate analysis.
If a specific concern remains with these results or with the stated scope of the framework, we would be glad to clarify it. We would be grateful if you could take these points into account in your final assessment.
Response to Reviewer 4Tgg (1/2)
W1. Limited experimental coverage. Experiments use only MNIST, with the main detection and anomaly-supervision results focusing on digit 3. Please report these results across all ten tasks and include CIFAR-10 or another data domain to assess whether the proposed explanations extend beyond this setting.
A: Thank you for the suggestion. As requested, we now report both the detection and the anomaly-supervision results on all ten MNIST tasks, and we repeat the same experiments on a second dataset, Fashion-MNIST (see Table 1 in Appendix G.3). Each of the ten one-vs-rest tasks is run separately, taking each class in turn as the normal class. The results are consistent with those reported in the original submission. The table below reports the averages over the ten tasks of each dataset at m=8192, where the DOCC columns correspond to detection and the Deep SAD columns to anomaly supervision. The results for the other widths are given in Table 1.
The first four columns report the AUROC at initialization, of DOCC, of its first-order prediction and of Deep SAD. All quantities except the initialization AUROC are evaluated at the corresponding Figure 5 step. "Missed" is the fraction of test anomalies not detected by DOCC at a 5% false-alarm rate, and "recovered" is the fraction of those missed anomalies that are detected by Deep SAD.
(1) All ten MNIST tasks. DOCC improves on initialization for every digit and every width, and the first-order prediction closely tracks the DOCC result. Deep SAD recovers 78% of the anomalies missed by DOCC (51–97% across digits) while losing about 1% of the anomalies detected by DOCC.
(2) A second dataset. The same qualitative picture holds on Fashion-MNIST, where Deep SAD recovers 72% of the anomalies missed by DOCC.
W2. Initialization and training effects are insufficiently separated. Report initial detection performance and distance distributions, then quantify displacement and final miss rates across initial-distance groups. Also report the fraction of test anomalies satisfying the detection sufficient condition to demonstrate its practical informativeness.
A: Thank you for the suggestion. We have added such experiments to the revised manuscript (see Figure 8 and Tables 2 and 3 in Appendix G.3). They separate the effect of initialization from that of training. The numbers below are for m=8192 at the final training step T.
(1) Initial detection performance and distance distributions. The distributions of the initial distance of held-out normals and anomalies are shown in Figure 8(a). At initialization they overlap, with an AUROC of 0.72 on MNIST and 0.81 on Fashion-MNIST. Training raises the AUROC to 0.90 and 0.87, respectively.
(2) Displacement and final miss rates across initial-distance groups. We group the test inputs into ten bins by initial distance. For compactness, the table below reports four representative bins on MNIST. The full ten-bin results are given in Table 2 of Appendix G.3.
Contraction is the fraction of the initial distance lost during training. The miss rate decreases and the false-alarm rate increases with the initial distance, and in every bin normals contract more than anomalies (47–56% versus 27–38% over the ten bins, and 48–57% versus 35–41% on Fashion-MNIST).
(3) Fraction of test anomalies satisfying the sufficient condition. Since ET contains unspecified absolute constants and is a worst-case bound, we cannot meaningfully instantiate it as a numerical certificate at the widths considered. We therefore report an empirical diagnostic in which ET is replaced by the measured remainder ∥ρT(x)∥2, the quantity that ET bounds. In this diagnostic, the condition of Theorem 3 is satisfied by 3% of the test anomalies on MNIST, and the direction-aware condition of the new Corollary 1 by 31%, which is about half of the detected anomalies (7% and 33% on Fashion-MNIST, see Table 3). The measured remainder also decreases with the width, faster than the rate in our bound (Table 5).