acceptodds
Under review as a conference paper at ICLR 2027

What Drives the Inlier-Memorization Effect? A Theory of Outlier Detection via Early Training Dynamics

Abstract

Outlier detection (OD) aims to identify anomalous instances by learning the underlying structure of normal data (inliers), and is particularly challenging in fully unsupervised settings where no information about anomalies is available during training. Recent advances have leveraged the inlier-memorization (IM) effect, a phenomenon in which deep models memorize inlier patterns earlier than those of outliers, as a powerful signal for distinguishing outliers. However, despite its empirical success, the theoretical understanding of the IM effect remains limited. In this work, we present a theoretical study of the IM effect. Focusing on a simple autoencoder with a fixed decoder and an encoder trained by full-batch gradient descent, we show that, under suitable regularity conditions, the IM effect emerges over a range of training iterations. Our results characterize when such an interval exists and which outliers can be separated from inliers throughout the interval. Further, we characterize how data distribution and weight initialization affect the strength and persistence of the IM effect, and provide empirical support for these theoretical predictions through simulation studies. Guided by these insights, we study compact representations and EMA-based initialization as two practical strategies for enhancing the IM effect. Experiments on real-world data further evaluate their individual and combined effects on the performance of IM-based outlier detection methods.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
Outlier detection, Autoencoder, Gradient descent, Inlier-memorization effect

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_XH38OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper investigates the inlier-memorization (IM) effect in unsupervised outlier detection: a model may learn shared inlier structure before fitting outlier-specific deviations. It studies a bias-free, single-hidden-layer autoencoder with a fixed decoder and an encoder optimized by full-batch gradient descent. The core argument compares actual training with a reference trajectory using nearest-cluster-center inputs. Theorem 3.1 establishes, under explicit regularity and interval conditions, a nonempty interval of cluster-consistent reconstructions; Theorem 3.2 and Corollary 3.3 then give sufficient geometric conditions under which an outlier has larger reconstruction error than every inlier. Simulations examine cluster compactness, cluster imbalance, contamination and initialization. Motivated by these results, the authors test pretrained compact representations and an EMA warm-up on ODIM and ALTBI, reporting modest average improvements on ADBench datasets.

Critical strengths
  1. A well-targeted theoretical question. Unlike prior IM-based detectors emphasizing exploitation or stopping of early training, the submission directly asks when cluster-consistent reconstruction and provable inlier/outlier loss separation can occur. The explicit connection between the center-tracking bound Bν(t), separation threshold 2Bν(t)+εc, and a certified training interval is an intelligible conceptual contribution (Sec. 3; Appendix A.3.2).

  2. A reasonably broad attempt to test qualitative predictions. The paper studies four controlled factors and checks both an idealized fixed-decoder regime and more practical joint-training dynamics (Sec. 4.1; Appendix B.3). It also reports individual/combined EMA and representation changes across two established methods and a large benchmark collection, rather than evaluating only a new bespoke detector (Sec. 4.2; Tables B.3–B.9).

Critical weaknesses

1.Clarify the feasibility and practical significance of the theoretical sufficient conditions. Theorem 3.1 requires a positive margin (c > 0), the interval condition in Eq. (A.24), and uniform Jacobian regularity under Assumption A.2. While these conditions are mathematically meaningful, their feasibility in representative experimental settings remains unclear. The authors are encouraged to provide an illustrative example with estimates of the relevant theoretical quantities and compare the predicted interval with observed training dynamics. A clearer discussion of the restrictiveness of these assumptions would help establish the practical significance of the theoretical guarantees.

2.Strengthen the connection between the theoretical analysis and the proposed practical strategies. The theoretical analysis considers a fixed-decoder autoencoder trained with full-batch gradient descent, whereas the practical experiments involve more general training settings. Although the authors provide supporting simulations, the connection between the theoretical quantities and the proposed representation and EMA strategies could be more explicit. Reporting how these strategies affect representative measures of cluster compactness and initialization quality on selected datasets would help substantiate their theoretical motivation.

3.Further examine performance variability and cases of degradation. The proposed strategies yield modest average improvements but exhibit considerable variability across datasets. For instance, the AUROC of ODIM on shuttle decreases from 0.982 to 0.709 after applying the combined modifications (Table B.6). Moreover, the average AUPRC of ODIM decreases from 0.334 to 0.322 (Table B.7). A brief analysis of such cases, together with a summary of dataset-level improvements and degradations and their variability across random seeds, would help clarify the reliability and applicability of the proposed strategies.

4.Further assess the robustness of the geometric diagnostic analysis. The geometric diagnostic (\sigma) in Section 4.2 provides an interesting empirical connection to Corollary 3.3. However, it is a post-hoc measure based on estimated cluster statistics rather than a direct verification of the theoretical separation condition. Given that the correlation analysis involves only eleven datasets, a simple robustness check, such as a rank-based correlation or leave-one-dataset-out analysis, would help strengthen the interpretation of the reported relationship.

5.Clarify the construction and robustness of TabPFN representations in the unsupervised setting. Appendix B.1 describes extracting TabPFN representations by treating the first feature column as a supervised target, or the second column when the first has negligible variance. Although this procedure does not directly use anomaly labels, the rationale for selecting an arbitrary feature as the prediction target remains unclear. The authors should clarify how this choice affects representation quality and, if feasible, examine whether the detection performance is sensitive to the selection of the target feature.

Questions for authors

Q1. Could the authors provide an illustrative example showing how the sufficient conditions of Theorem 3.1 relate to the empirically observed IM interval?

Q2. Which theory-relevant quantities remain most informative when extending the analysis from fixed-decoder training to practical joint-training settings?

Q3. Does EMA warm-up measurably improve the initial cluster-center reconstruction quality? How does it compare with a non-EMA warm-up under a similar computational budget?

Q4. Could the authors discuss the substantial performance degradation of ODIM on shuttle and clarify the dataset characteristics under which the proposed strategies may be less effective?

Q5. Section 4.1 and Appendix B.2 appear to use different definitions of the empirical IM window. Could the authors clarify which definition was used to generate the reported results?

6 comments
AuthorsOpenReview ↗

Response to Reviewer XH38 (1/4)

We thank the reviewer for the helpful comments and suggestions. We address the concerns point-by-point as below.


W1: Clarify the feasibility and practical significance of the theoretical sufficient conditions. Theorem 3.1 requires a positive margin (c > 0), the interval condition in Eq. (A.24), and uniform Jacobian regularity under Assumption A.2. While these conditions are mathematically meaningful, their feasibility in representative experimental settings remains unclear. The authors are encouraged to provide an illustrative example with estimates of the relevant theoretical quantities and compare the predicted interval with observed training dynamics. A clearer discussion of the restrictiveness of these assumptions would help establish the practical significance of the theoretical guarantees.

Response

Thank you for this important suggestion.

  • The constants in Bν(t)B_\nu(t) are defined over a whole neighborhood of the initialization and cannot be computed exactly, so we approximated Bν(t)B_\nu(t) on the original simulation data by measuring its three terms directly.

  • Setting. We used the original Figure 2 data with εc=1.0\varepsilon_c=1.0 (n=2,000n=2{,}000, K=3K=3, p=2p=2) and one of its original initializations, fixed the decoder, and trained the encoder by full-batch gradient descent with learning rate 10−410^{-4} for 600,000600{,}000 updates.

  • Bν(t)B_\nu(t) approximation. In Eq. (A.19) (Appendix A.3.2), Bν(t)B_\nu(t) is the sum of three terms: the input-replacement error, the gap between the actual and reference trajectories, and the reference reconstruction error. We measured each term directly during training and summed them, i.e., we obtained an upper bound on the distance between each reconstruction and its center. Taking this bound over 80%80\% or 90%90\% of the samples gives the approximated B^ν(t)\widehat B_\nu(t).

  • Result. The table reports the intervals in which B^ν(t)<δ/2=0.1092\widehat B_\nu(t)<\delta/2=0.1092, together with the observed distance and the AUROC over each interval.

    Fraction of samplesEmpirical bound rangeObserved distance quantileUpdates with B^ν(t)<δ/2\widehat B_\nu(t)<\delta/2Length (% of 600,000 updates)AUROC
    80%0.08808–0.108090.06465–0.08892250,000–600,000350,000 (58.3%)0.971–0.974
    90%0.10089–0.107120.07490–0.08674290,000–600,000310,000 (51.7%)0.972–0.974

    The approximated bound stays below δ/2\delta/2 for more than half of training (310,000310{,}000 updates, 51.7%51.7\%, for 90%90\% of the samples), and the AUROC stays high (0.9720.972–0.9740.974) throughout this interval. That is, as the theory states, the IM effect appears while the reconstructions stay close to their centers.

  • To directly illustrate the feasibility of the sufficient conditions, we also considered a separate synthetic setting with two tight Gaussian clusters in R3\mathbb R^3, each containing 5,0005{,}000 inliers, with center separation δ=0.99\delta=0.99 and one outlier near each center. We trained a fixed-decoder autoencoder with H=64H=64 using full-batch gradient descent. Bounds valid throughout the prescribed neighborhood give αc=6.997\alpha_c=6.997, βc=7.001\beta_c=7.001, and Lc=0.351L_c=0.351. All conditions of Theorem 3.1 are satisfied: the margin is c=0.482>0c=0.482>0, and Eq. (A.24) holds (81.8<959.981.8<959.9), producing the certified center-tracking interval [32,53][32,53]. Over this interval, the theoretical bound is Bν(t)∈[0.024,0.040]B_\nu(t)\in[0.024,0.040], while the observed maximum center-tracking error remains below 9.4×10−59.4\times10^{-5}. Full details are provided in Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer XH38 (2/4)


W2: Strengthen the connection between the theoretical analysis and the proposed practical strategies. The theoretical analysis considers a fixed-decoder autoencoder trained with full-batch gradient descent, whereas the practical experiments involve more general training settings. Although the authors provide supporting simulations, the connection between the theoretical quantities and the proposed representation and EMA strategies could be more explicit. Reporting how these strategies affect representative measures of cluster compactness and initialization quality on selected datasets would help substantiate their theoretical motivation.

Response

Thank you for this valuable suggestion.

  • EMA versus ordinary warm-up. In the simulation (εc=1.0\varepsilon_c=1.0), both methods reduced the mean initial cluster-center reconstruction error R~0\widetilde R_0 from 32.232.2 to similar values after 500500 updates, with 2.32.3 for ordinary warm-up and a slightly lower 2.22.2 for EMA. Since the true centers are unknown in the eleven real-world datasets, we compare AUROC and AUPRC using the same TabPFN representations and identical warm-up updates.

    DatasetALTBI standardALTBI EMAODIM standardODIM EMA
    ALOI0.542 / 0.0360.542 / 0.0370.534 / 0.0420.532 / 0.041
    annthyroid0.601 / 0.0910.597 / 0.0900.539 / 0.0840.518 / 0.079
    campaign0.601 / 0.1730.599 / 0.1750.577 / 0.1880.568 / 0.183
    landsat0.515 / 0.2200.510 / 0.2110.553 / 0.2230.580 / 0.236
    magic.gamma0.650 / 0.5300.648 / 0.5270.639 / 0.5390.636 / 0.532
    mammography0.764 / 0.0540.757 / 0.0530.747 / 0.0520.718 / 0.052
    PageBlocks0.841 / 0.3480.853 / 0.3430.864 / 0.3920.902 / 0.425
    shuttle0.928 / 0.3550.928 / 0.4250.890 / 0.2460.901 / 0.300
    SpamBase0.628 / 0.5370.627 / 0.5360.614 / 0.5210.611 / 0.526
    Wilt0.391 / 0.0400.391 / 0.0400.394 / 0.0400.394 / 0.040
    yeast0.510 / 0.3440.512 / 0.3460.509 / 0.3450.509 / 0.346
    Mean difference (EMA − standard)-0.0004 / +0.0051+0.0008 / +0.0079

    With EMA, mean AUROC changed little, while mean AUPRC increased modestly (+0.005+0.005 for ALTBI and +0.008+0.008 for ODIM). ODIM’s two largest AUPRC gains occurred on PageBlocks and shuttle, which also had the highest σ\sigma values after TabPFN (0.6510.651 and 0.5470.547, respectively). We conjecture that EMA may help preserve inlier-cluster structure during warm-up, making outliers that lie relatively far from those centers easier to detect.

  • Effect of the TabPFN representation. For the eleven datasets, we applied the procedure of Section 4.2 to the raw input and the TabPFN representation of the same samples, and computed σ\sigma and the relative inlier spread ε^c/δ^\widehat\varepsilon_c/\widehat\delta.

    Datasetσ\sigma before TabPFNσ\sigma after TabPFNΔσ\Delta\sigma (after − before)ε^c/δ^\widehat\varepsilon_c/\widehat\delta: before → after
    ALOI0.2240.313+0.0887.14 → 1.22
    annthyroid0.4910.360-0.1316.87 → 1.01
    campaign0.5770.327-0.2502.01 → 0.98
    landsat0.4850.273-0.2122.81 → 1.10
    magic.gamma0.7030.325-0.3784.94 → 1.42
    mammography0.5860.278-0.3085.45 → 1.13
    PageBlocks1.1250.651-0.4736.00 → 1.78
    shuttle3.7290.547-3.1811.71 → 1.10
    SpamBase0.3220.018-0.3048.16 → 1.05
    Wilt0.2570.505+0.2485.94 → 1.10
    yeast0.2540.361+0.1071.43 → 1.17

    TabPFN makes the inlier clusters more compact on all eleven datasets (mean relative spread 4.77→1.194.77\to1.19) and reduces σ\sigma on average (0.80→0.360.80\to0.36). We have added both comparisons to Appendix B.4.

AuthorsOpenReview ↗

Response to Reviewer XH38 (3/4)


W3: Further examine performance variability and cases of degradation. The proposed strategies yield modest average improvements but exhibit considerable variability across datasets. For instance, the AUROC of ODIM on shuttle decreases from 0.982 to 0.709 after applying the combined modifications (Table B.6). Moreover, the average AUPRC of ODIM decreases from 0.334 to 0.322 (Table B.7). A brief analysis of such cases, together with a summary of dataset-level improvements and degradations and their variability across random seeds, would help clarify the reliability and applicability of the proposed strategies.

Response

Thank you for the suggestion.

  • Dataset-level differences over the 57 datasets. Using the full-precision means over three seeds underlying Tables B.6–B.7, we computed the dataset-level AUROC difference (modified − original), also without shuttle.

    Method / metricDatasetsMean differenceIncrease / tie / decrease
    ALTBI / AUROC57+0.009132 / 2 / 23
    ODIM / AUROC57+0.005632 / 0 / 25
    ALTBI / AUROC56 (without shuttle)+0.009231 / 2 / 23
    ODIM / AUROC56 (without shuttle)+0.010632 / 0 / 24

    Both methods improve the mean AUROC, and improve it on 3232 of the 5757 datasets. Without shuttle, the mean AUROC gain of ODIM nearly doubles (+0.0056→+0.0106+0.0056\to+0.0106), so much of the variability comes from the single large drop on shuttle.

    • In addition, the variation across seeds is small. Rerunning ODIM on the raw input with three data seeds, the AUROC varies by at most 0.0010.001 on shuttle and campaign and by 0.0140.014 on SpamBase, which is small compared with the differences between datasets.
    • For reference, ODIM's stopping rule is a heuristic and can select an unsuitable iteration. ODIM stops with a patience rule and reports the scores of the single check at which the separation between two Gaussian components fitted to the mini-batch losses is largest. This criterion is heuristic and fairly sensitive to the loss trajectory, so it occasionally stops outside the period in which the outliers are well separated. Changing the representation changes this trajectory, which we believe causes the drop on shuttle.
    • In contrast, ALTBI averages the scores over iterations and is less sensitive to this choice. As discussed in Remark 3.1, neither the optimal stopping iteration nor Tν\mathcal T_\nu is known in practice, and averaging the reconstruction errors over training iterations retains the separation obtained within Tν\mathcal T_\nu while reducing the dependence on a single stopping iteration. This is why, with the same TabPFN representation, ODIM drops to 0.7090.709 while ALTBI keeps AUROC 0.9850.985 (Table B.6).
    • Moreover, shuttle is already well separated. Shuttle has the largest σ\sigma in Table B.3 (4.044.04), so a new representation has little room to improve detection. We have added this discussion to Appendix B.4.

W4: Further assess the robustness of the geometric diagnostic analysis. The geometric diagnostic (\sigma) in Section 4.2 provides an interesting empirical connection to Corollary 3.3. However, it is a post-hoc measure based on estimated cluster statistics rather than a direct verification of the theoretical separation condition. Given that the correlation analysis involves only eleven datasets, a simple robustness check, such as a rank-based correlation or leave-one-dataset-out analysis, would help strengthen the interpretation of the reported relationship.

Response

Thank you for the suggestion.

  • We computed the Spearman correlation and Kendall's τb\tau_b between σ\sigma and the AUROC change on the eleven datasets, and repeated each after leaving out one dataset at a time.

    AnalysisODIMALTBI
    Spearman (p)-0.809 (0.003)-0.882 (0.00033)
    Kendall τb\tau_b (p)-0.673 (0.003)-0.782 (0.00033)
    Leave-one-out Spearman[-0.891, -0.745][-0.964, -0.842]
    Leave-one-out Kendall τb\tau_b[-0.778, -0.600][-0.911, -0.733]

    Both rank correlations are significantly negative, and they remain negative after leaving out any single dataset, including shuttle. That is, the relation between σ\sigma and the gain does not depend on a particular dataset. We have added these results to Appendix B.4.

Reviewer_JSriOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies an autoencoder with a fixed decoder and an encoder trained by full-batch gradient descent. It establishes a training interval during which reconstructions remain close to cluster centers, yielding reconstruction-error separation for sufficiently distant outliers. The authors examine the effects of data geometry and initialization, then evaluate pretrained representations and EMA-based initialization in ODIM and ALTBI.

Critical strengths

S1. The comparison between reference and actual training trajectories provides a substantive theoretical result: sufficient conditions for a nonempty interval and outlier separation within it.

S2. The main simulations use a fixed decoder and full-batch gradient descent, matching the theoretical setting. The four-way ablation on eleven tabular datasets also helps distinguish the performance contributions of EMA and pretrained representations.

Critical weaknesses

W1. The theoretical guarantee lacks a quantitative connection to the observed IM window. The nonempty interval depends on conditions including Eq. A.24, but the experiments neither verify these conditions nor evaluate (B_\nu(t)) against the observed reconstruction-to-center distances. The simulations support several qualitative trends, but leave unclear whether the tested settings satisfy the theorem’s assumptions and whether its bounds are informative. The relationship between the certified interval and the empirical IM window defined by an AUROC threshold remains untested.

W2. The practical improvements do not establish the proposed mechanism. The four-way ablation measures each component’s effect on detection performance, but does not directly show that EMA extends the IM window by reducing the initial cluster-center reconstruction error. A matched-budget warm-up without parameter averaging is missing, leaving the effects of additional training and averaging unresolved. Likewise, the correlation between data geometry and performance gains does not establish that pretrained representations improve detection through the mechanism described by the theory.

W3. The robustness of the overall gains remains unclear. Average AUROC improves across the 57 datasets, but average AUPRC increases only from 0.350 to 0.352 for ALTBI and decreases from 0.334 to 0.322 for ODIM. On shuttle, ODIM’s AUPRC drops from 0.944 to 0.302 (Table B.7). The main benchmark results report means over three seeds without paired uncertainty estimates. The substantial regressions also require an account of when these modifications fail.

The center-tracking analysis is the strongest part of the paper. My main reservation is that the experiments do not yet connect the theoretical conditions, observed training dynamics, and detection gains. The empirical gains are also mixed, with limited evidence on their robustness. I therefore lean toward weak rejection. Quantitative validation of the theoretical interval and controls that isolate the effect of EMA would materially affect my assessment.

Questions for authors

Q1. Can you verify the interval conditions in one synthetic setting, evaluate (B_\nu(t)), and jointly show the certified interval, observed center-tracking error, and empirical IM window?

Q2. With input representations and training budget held fixed, does EMA still improve over ordinary warm-up? Do the initial reconstruction error and IM duration change as hypothesized?

Q3. Can you report paired confidence intervals for the performance differences and investigate the large regression on shuttle?

6 comments
AuthorsOpenReview ↗

Response to Reviewer JSri (1/3)

We thank the reviewer for the helpful comments and suggestions. We address the concerns point-by-point as below.


W1. The theoretical guarantee lacks a quantitative connection to the observed IM window. The nonempty interval depends on conditions including Eq. A.24, but the experiments neither verify these conditions nor evaluate (B_\nu(t)) against the observed reconstruction-to-center distances. The simulations support several qualitative trends, but leave unclear whether the tested settings satisfy the theorem’s assumptions and whether its bounds are informative. The relationship between the certified interval and the empirical IM window defined by an AUROC threshold remains untested.

Response

Thank you for this important suggestion.

  • The constants in Bν(t)B_\nu(t) are defined over a whole neighborhood of the initialization and cannot be computed exactly, so we approximated Bν(t)B_\nu(t) on the original simulation data by measuring its three terms directly.

  • Setting. We used the original Figure 2 data with εc=1.0\varepsilon_c=1.0 (n=2,000n=2{,}000, K=3K=3, p=2p=2) and one of its original initializations, fixed the decoder, and trained the encoder by full-batch gradient descent with learning rate 10−410^{-4} for 600,000600{,}000 updates.

  • Bν(t)B_\nu(t) approximation. In Eq. (A.19) (Appendix A.3.2), Bν(t)B_\nu(t) is the sum of three terms: the input-replacement error, the gap between the actual and reference trajectories, and the reference reconstruction error. We measured each term directly during training and summed them, i.e., we obtained an upper bound on the distance between each reconstruction and its center. Taking this bound over 80%80\% or 90%90\% of the samples gives the approximated B^ν(t)\widehat B_\nu(t).

  • Result. The table reports the intervals in which B^ν(t)<δ/2=0.1092\widehat B_\nu(t)<\delta/2=0.1092, together with the observed distance and the AUROC over each interval.

    Fraction of samplesEmpirical bound rangeObserved distance quantileUpdates with B^ν(t)<δ/2\widehat B_\nu(t)<\delta/2Length (% of 600,000 updates)AUROC
    80%0.08808–0.108090.06465–0.08892250,000–600,000350,000 (58.3%)0.971–0.974
    90%0.10089–0.107120.07490–0.08674290,000–600,000310,000 (51.7%)0.972–0.974

    The approximated bound stays below δ/2\delta/2 for more than half of training (310,000310{,}000 updates, 51.7%51.7\%, for 90%90\% of the samples), and the AUROC stays high (0.9720.972–0.9740.974) throughout this interval. That is, as the theory states, the IM effect appears while the reconstructions stay close to their centers. We have added these results to Appendix A.4.

  • To complement these empirical results and demonstrate that the sufficient conditions can hold jointly, we constructed a separate synthetic example: two tight Gaussian clusters in R3\mathbb R^3, each with 5,0005{,}000 inliers, with center separation δ=0.99\delta=0.99, and one outlier near each center. We used a fixed-decoder autoencoder with H=64H=64 and full-batch gradient descent. Bounding the Jacobian quantities over the prescribed neighborhood gives αc=6.997\alpha_c=6.997, βc=7.001\beta_c=7.001, and Lc=0.351L_c=0.351. All conditions of Theorem 3.1 hold, with margin c=0.482>0c=0.482>0 and Eq. (A.24) satisfied (81.8<959.981.8<959.9), yielding the certified center-tracking interval [32,53][32,53]. Throughout this interval, the observed maximum center-tracking error stays below 9.4×10−59.4\times10^{-5}, while Bν(t)∈[0.024,0.040]B_\nu(t)\in[0.024,0.040]. This example demonstrates the feasibility of the sufficient conditions, complementing the empirical analysis on the original simulation data. Details are provided in Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer JSri (2/3)


W2. The practical improvements do not establish the proposed mechanism. The four-way ablation measures each component’s effect on detection performance, but does not directly show that EMA extends the IM window by reducing the initial cluster-center reconstruction error. A matched-budget warm-up without parameter averaging is missing, leaving the effects of additional training and averaging unresolved. Likewise, the correlation between data geometry and performance gains does not establish that pretrained representations improve detection through the mechanism described by the theory.

Response

Thank you for this valuable suggestion.

  • EMA versus warm-up without EMA. On the TabPFN representations of the eleven datasets, we compared the EMA warm-up with a warm-up that takes exactly the same updates but keeps the last weights (AUROC / AUPRC).

    DatasetALTBI standardALTBI EMAODIM standardODIM EMA
    ALOI0.542 / 0.0360.542 / 0.0370.534 / 0.0420.532 / 0.041
    annthyroid0.601 / 0.0910.597 / 0.0900.539 / 0.0840.518 / 0.079
    campaign0.601 / 0.1730.599 / 0.1750.577 / 0.1880.568 / 0.183
    landsat0.515 / 0.2200.510 / 0.2110.553 / 0.2230.580 / 0.236
    magic.gamma0.650 / 0.5300.648 / 0.5270.639 / 0.5390.636 / 0.532
    mammography0.764 / 0.0540.757 / 0.0530.747 / 0.0520.718 / 0.052
    PageBlocks0.841 / 0.3480.853 / 0.3430.864 / 0.3920.902 / 0.425
    shuttle0.928 / 0.3550.928 / 0.4250.890 / 0.2460.901 / 0.300
    SpamBase0.628 / 0.5370.627 / 0.5360.614 / 0.5210.611 / 0.526
    Wilt0.391 / 0.0400.391 / 0.0400.394 / 0.0400.394 / 0.040
    yeast0.510 / 0.3440.512 / 0.3460.509 / 0.3450.509 / 0.346
    Mean difference (EMA − standard)-0.0004 / +0.0051+0.0008 / +0.0079

    On average, EMA warm-up produces little change in AUROC, with somewhat larger numerical gains in AUPRC (+0.005+0.005 for ALTBI and +0.008+0.008 for ODIM). Notable gains occur on PageBlocks and shuttle, which also have the two largest σ\sigma values after TabPFN (0.6510.651 and 0.5470.547, respectively). We conjecture that greater relative separation of outliers from inlier-cluster centers may make EMA warm-up more effective, although this relationship remains to be verified.

  • Effect of the TabPFN representation. For the eleven datasets, we applied the procedure of Section 4.2 to the raw input and the TabPFN representation of the same samples, and computed σ\sigma and the relative inlier spread ε^c/δ^\widehat\varepsilon_c/\widehat\delta.

    Datasetσ\sigma before TabPFNσ\sigma after TabPFNΔσ\Delta\sigma (after − before)ε^c/δ^\widehat\varepsilon_c/\widehat\delta: before → after
    ALOI0.2240.313+0.0887.14 → 1.22
    annthyroid0.4910.360-0.1316.87 → 1.01
    campaign0.5770.327-0.2502.01 → 0.98
    landsat0.4850.273-0.2122.81 → 1.10
    magic.gamma0.7030.325-0.3784.94 → 1.42
    mammography0.5860.278-0.3085.45 → 1.13
    PageBlocks1.1250.651-0.4736.00 → 1.78
    shuttle3.7290.547-3.1811.71 → 1.10
    SpamBase0.3220.018-0.3048.16 → 1.05
    Wilt0.2570.505+0.2485.94 → 1.10
    yeast0.2540.361+0.1071.43 → 1.17

    TabPFN makes the inlier clusters more compact on all eleven datasets (mean relative spread 4.77→1.194.77\to1.19) and reduces σ\sigma on average (0.80→0.360.80\to0.36). We have added both comparisons to Appendix B.4.

AuthorsOpenReview ↗

Response to Reviewer JSri (3/3)


W3. The robustness of the overall gains remains unclear. Average AUROC improves across the 57 datasets, but average AUPRC increases only from 0.350 to 0.352 for ALTBI and decreases from 0.334 to 0.322 for ODIM. On shuttle, ODIM’s AUPRC drops from 0.944 to 0.302 (Table B.7). The main benchmark results report means over three seeds without paired uncertainty estimates. The substantial regressions also require an account of when these modifications fail. The center-tracking analysis is the strongest part of the paper. My main reservation is that the experiments do not yet connect the theoretical conditions, observed training dynamics, and detection gains. The empirical gains are also mixed, with limited evidence on their robustness. I therefore lean toward weak rejection. Quantitative validation of the theoretical interval and controls that isolate the effect of EMA would materially affect my assessment.

Response

Thank you for the suggestion.

  • Dataset-level differences over the 57 datasets. Using the full-precision means over three seeds underlying Tables B.6–B.7, we computed the dataset-level AUROC difference (modified − original), also without shuttle.

    Method / metricDatasetsMean differenceIncrease / tie / decrease
    ALTBI / AUROC57+0.009132 / 2 / 23
    ODIM / AUROC57+0.005632 / 0 / 25
    ALTBI / AUROC56 (without shuttle)+0.009231 / 2 / 23
    ODIM / AUROC56 (without shuttle)+0.010632 / 0 / 24

    Both methods improve the mean AUROC, and improve it on 3232 of the 5757 datasets. Without shuttle, the mean AUROC gain of ODIM nearly doubles (+0.0056→+0.0106+0.0056\to+0.0106), so much of the variability comes from the single large drop on shuttle.

    • For reference, ODIM's stopping rule is a heuristic and can select an unsuitable iteration. ODIM stops with a patience rule and reports the scores of the single check at which the separation between two Gaussian components fitted to the mini-batch losses is largest. This criterion is heuristic and fairly sensitive to the loss trajectory, so it occasionally stops outside the period in which the outliers are well separated. Changing the representation changes this trajectory, which we believe causes the drop on shuttle.
    • In contrast, ALTBI averages the scores over iterations and is less sensitive to this choice. As discussed in Remark 3.1, neither the optimal stopping iteration nor Tν\mathcal T_\nu is known in practice, and averaging the reconstruction errors over training iterations retains the separation obtained within Tν\mathcal T_\nu while reducing the dependence on a single stopping iteration. This is why, with the same TabPFN representation, ODIM drops to 0.7090.709 while ALTBI keeps AUROC 0.9850.985 (Table B.6).
    • Moreover, shuttle is already well separated. Shuttle has the largest σ\sigma in Table B.3 (4.044.04), so a new representation has little room to improve detection. We have added this discussion to Appendix B.4.

Q1. Can you verify the interval conditions in one synthetic setting, evaluate Bν(t)B_\nu(t), and jointly show the certified interval, observed center-tracking error, and empirical IM window?

Response

  • Please see our response to W1.

Q2. With input representations and training budget held fixed, does EMA still improve over ordinary warm-up? Do the initial reconstruction error and IM duration change as hypothesized?

Response

  • Please see our response to W2.

Q3. Can you report paired confidence intervals for the performance differences and investigate the large regression on shuttle?

Response

  • Please see our response to W3.
Reviewer_N4XMOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper analyzes why reconstruction-based unsupervised anomaly detectors may fit inliers before outliers. Its formal model is a bias-free, one-hidden-layer autoencoder with a fixed decoder and an encoder trained by full-batch gradient descent. The authors compare actual training with a reference problem that replaces each input by its assigned cluster center but retains the original observation as the reconstruction target. In the reference dynamics, cluster-shared residuals contract while within-cluster deviations are not fitted in the same way. A trajectory-stability argument then yields a conditional interval during which every reconstruction stays near its assigned center. A separate geometric requirement is needed to guarantee that a given outlier's reconstruction error exceeds every inlier's. The bounds suggest effects of cluster compactness, size balance, contamination, and initialization. The experiments vary these factors in two-dimensional simulations, repeat qualitative tests under joint training with mini-batch Adam, and study EMA warm-up and TabPFN representations in ODIM and ALTBI on ADBench. The real-data results include component analyses and AUROC/AUPRC results over 57 datasets.

Critical strengths
  1. The paper identifies a concrete training-dynamics mechanism rather than merely restating that inliers are learned first. Its reference problem gives samples in the same cluster a shared center as input while retaining their individual observations as reconstruction targets. This separates cluster-shared residuals, which contract along the reference trajectory, from sample-specific residuals, which are not fitted in the same way. A trajectory-stability argument then connects the reference behavior to training on the observed inputs. The analysis explains why a model may temporarily reconstruct common cluster structure without yet fitting an outlier’s individual deviation.
  2. The paper carefully separates cluster-consistent reconstruction from actual anomaly-score separation. Theorem 3.1 establishes a conditional interval in which reconstructions remain close to their assigned centers. Since that property alone does not imply successful detection, Theorem 3.2 adds an outlier-distance condition before concluding that a qualifying outlier has a larger reconstruction error than every inlier; Corollary 3.3 supplies a condition valid throughout the interval. This layered result makes clear which outliers are covered and when, avoiding the common but unjustified leap from center tracking to universal anomaly detection.
Critical weaknesses
  1. The central theoretical assumption has an underexplained architectural restriction. Assumption A.2(i) requires the cluster-center Jacobian to have a nonzero response in every output direction. Because the decoder is fixed, however, every encoder-induced output change lies in the decoder’s column space. The assumption therefore requires the decoder to have full row rank and the hidden width to be at least the input dimension, in addition to the stated requirement that it be at least the number of clusters. A fixed-decoder bottleneck autoencoder cannot satisfy the premise. This does not invalidate the conditional results, but it makes the claimed mildness and practical reach of their assumptions less convincing, particularly when the real-data models fall outside the formal setup.
  2. The experiments do not show whether the theorem’s separation guarantee is non-vacuous in the tested regimes. Theorem 3.2 concerns a qualifying outlier whose reconstruction error exceeds that of every inlier during a certified interval. The simulations instead identify an empirical window using AUROC above 0.9. A high AUROC neither establishes that the theorem’s sufficient conditions hold nor reveals how many outliers receive its pointwise guarantee. The paper does not report, even for its controlled synthetic data, the interval certified by its own bound or the fraction of outliers satisfying the separation condition. Consequently, the observed trends may be consistent with the theory without demonstrating that the proved mechanism actually accounts for them. This gap is consequential because the paper’s explanatory contribution depends on connecting its formal result to the phenomenon it measures.
Questions for authors
  1. Under its assumptions, Theorem 3.1 guarantees a contiguous training interval in which every reconstruction stays within a distance of half the minimum inter-center separation from its assigned center. In the controlled simulations of Figures 2–4, how long are the observed intervals satisfying this all-sample condition as within-cluster variation, cluster imbalance, and outlier rate change?
  2. Theorem 3.2 gives strict error separation for qualifying outliers, while Section 4.1 identifies empirical IM windows using AUROC of at least 0.9. How does the proportion of outliers whose reconstruction error exceeds the maximum inlier error evolve during those windows?
  3. Table 1 suggests a negative association between initial cluster-centered reconstruction error and IM-window length across three random initializations. With the data and decoder held fixed, does this association persist on average across a larger set of encoder initializations?
  4. Figure 5 shows that adding TabPFN after EMA substantially reduces ODIM’s AUROC on shuttle, while ALTBI remains stable. What explains the different responses of the two methods to the same representation?
5 comments
AuthorsOpenReview ↗

Response to Reviewer N4XM (1/2)

We thank the reviewer for the helpful comments and suggestions. We address the concerns point-by-point as below.


W1: Assumption A.2(i) implicitly requires a full-row-rank fixed decoder and hidden width at least the input dimension, excluding bottleneck autoencoders. Please clarify these restrictions and their practical implications.

Response

Thank you for the careful reading.

  • The rank and width requirements are already implied by Assumption A.2(i), and we have stated them explicitly. With the decoder A∈Rp×HA\in\mathbb R^{p\times H} fixed, every encoder-induced output change lies in the column space of AA, so Assumption A.2(i) requires rank(A)=p\mathrm{rank}(A)=p, i.e., H≥max⁡{K,p}H\ge\max\{K,p\}. We have stated this together with the existing condition H≥KH\ge K in Remark A.3, and revised Remark A.2 accordingly.
  • Our simulations satisfy this assumption. The simulations use p=2p=2, K=3K=3, and H=32H=32 (H=12H=12 in the imbalance experiment), so H≥max⁡{K,p}H\ge\max\{K,p\}. The fixed decoder at initialization has full row rank (eigenvalues of AA⊤AA^\top: 0.2870.287 and 0.4020.402), and the cluster-center Jacobian keeps full row rank throughout training, with smallest singular value 0.0100.010–0.0230.023 over the runs of Figure 2.
  • We fix the decoder to simplify the analysis. A decoder change ΔA\Delta A, with WW held fixed, changes each center output by ΔAϕ(Wμk)\Delta A\phi(W\mu_k). If {ϕ(Wμk)}k=1K\{\phi(W\mu_k)\}_{k=1}^K are linearly independent, these changes span all center-output directions without requiring H≥pH\ge p. Extending our proof would require this ability to remain well-conditioned throughout the neighborhood considered, so that no output direction becomes insensitive to parameter changes. We would also need training on actual inputs to stay close to reference training on center inputs as both AA and WW evolve. Indeed, the joint-training experiments in Appendix B.3 also exhibit the IM effect and qualitatively similar trends for the four factors in the settings studied, supporting the relevance of our theoretical insights beyond fixed-decoder training.

W2: High AUROC does not verify Theorem 3.2's sufficient conditions or pointwise separation guarantee. Please report the certified interval and the fraction of outliers satisfying the separation condition.

Response

Thank you for this important point.

  • Following your suggestion, we reran the simulations of Figures 2–4 and directly measured the outlier errors of Theorem 3.2. For each setting, we computed the interval in which at least 80%80\% of outliers exceed the 9595th percentile of inlier errors, and, within the AUROC window, the median fraction of outliers exceeding the maximum inlier error. The last column is the overlap between the two intervals, relative to the AUROC window (in iterations).

    SettingAUROC window lengthObserved separation interval lengthOutliers above every inlier (median fraction)Overlap / AUROC window
    εc=1.0\varepsilon_c=1.09,0308,8300.9297.8%
    εc=2.5\varepsilon_c=2.55,8705,3500.7291.1%
    εc=4.0\varepsilon_c=4.03,6602,4100.3465.8%
    nmax⁡:nmin⁡=1:1n_{\max}:n_{\min}=1:111,67012,3000.80100.0%
    nmax⁡:nmin⁡=11:1n_{\max}:n_{\min}=11:19,26010,7200.89100.0%
    nmax⁡:nmin⁡=27:1n_{\max}:n_{\min}=27:19,2407,1700.8177.6%
    ρ=0.05\rho=0.056,9306,4700.8293.3%
    ρ=0.1\rho=0.14,2204,0400.8295.6%
    ρ=0.15\rho=0.152,7952,8300.80100.0%

    The results shows that, the observed separation interval largely overlaps with the AUROC window (66%66\%–100%100\%) and gives the same message as Section 4.1. That is, the interval is longer for tighter, more balanced, and less contaminated clusters. The fraction of outliers above every inlier is also larger for tighter clusters (0.920.92, 0.720.72, 0.340.34). We have added these new results to Appendix A.4.

AuthorsOpenReview ↗

Response to Reviewer N4XM (2/2)


Q1: Under its assumptions, Theorem 3.1 guarantees a contiguous training interval in which every reconstruction stays within a distance of half the minimum inter-center separation from its assigned center. In the controlled simulations of Figures 2–4, how long are the observed intervals satisfying this all-sample condition as within-cluster variation, cluster imbalance, and outlier rate change?

Response

  • In the simulation settings of the paper (Figures 2–4), we trained for 40,00040{,}000 iterations, measured every 1010 iterations the fraction of all samples (inliers and outliers) whose reconstruction lies within δ/2\delta/2 of the assigned center, and report the interval in which this fraction is at least 80%80\% or 90%90\%.

    SettingAt least 80% within δ/2\delta/2At least 90% within δ/2\delta/2
    εc=1.0\varepsilon_c=1.0180–40,000210–40,000
    εc=2.5\varepsilon_c=2.5220–2,850None
    εc=4.0\varepsilon_c=4.0NoneNone
    nmax⁡:nmin⁡=1:1n_{\max}:n_{\min}=1:1470–40,000710–40,000
    nmax⁡:nmin⁡=11:1n_{\max}:n_{\min}=11:1480–40,000820–40,000
    nmax⁡:nmin⁡=27:1n_{\max}:n_{\min}=27:1390–40,000870–40,000
    ρ=0.05\rho=0.05200–40,000280–1,140
    ρ=0.1\rho=0.1200–3,740None
    ρ=0.15\rho=0.15220–640None

    As Section 4.1 shows, tighter clusters and lower outlier rates give longer intervals, and in the balance experiment the 90%90\% interval begins earlier for more balanced clusters (710710, 820820, and 870870 iterations, respectively). We have added these intervals to Appendix A.4.


Q2: Theorem 3.2 gives strict error separation for qualifying outliers, while Section 4.1 identifies empirical IM windows using AUROC of at least 0.9. How does the proportion of outliers whose reconstruction error exceeds the maximum inlier error evolve during those windows?

Response

  • Please see our response to W2.

Q3: Table 1 suggests a negative association between initial cluster-centered reconstruction error and IM-window length across three random initializations. With the data and decoder held fixed, does this association persist on average across a larger set of encoder initializations?

Response

Thank you for the suggestion.

  • Section 3.3 suggests earlier reference-dynamics entry for smaller R~0\widetilde R_0, motivating our analysis of its association with the empirical IM-window onset. We fixed the data and decoder of Table 1 (εc=4.0\varepsilon_c=4.0, H=32H=32, full-batch gradient descent), additionally drew 2020 encoder initializations, and repeated the same analysis. Five of them are shown below.

    ExampleR~0\widetilde R_0Onset T1T_1IM-window length
    117.4404,710
    217.8504,350
    330.2904,200
    433.11003,480
    540.61303,300

    A smaller R~0\widetilde R_0 is associated with an earlier onset of the IM window across the tested initializations. Over the 2020 initializations, the Spearman correlation between R~0\widetilde R_0 and T1T_1 is 0.820.82 (p<10−4p<10^{-4}). In addition, the IM window also becomes longer. We have added the scatter plot of R~0\widetilde R_0 against T1T_1 to Appendix A.4.


Q4: Figure 5 shows that adding TabPFN after EMA substantially reduces ODIM’s AUROC on shuttle, while ALTBI remains stable. What explains the different responses of the two methods to the same representation?

Response

  • ODIM selects a checkpoint for scoring using a heuristic based on the separation of two Gaussian components fitted to minibatch losses. Changing the representation can alter this criterion’s trajectory and may lead to a less suitable checkpoint.

  • ALTBI averages scores across iterations, potentially reducing sensitivity to checkpoint selection. As discussed in Remark 3.1, averaging preserves separation when positive margins within the IM interval outweigh adverse contributions outside it. This difference may contribute to the contrasting AUROC results on shuttle: 0.7090.709 for ODIM and 0.9850.985 for ALTBI (Table B.6).

  • Shuttle also offers limited room for improvement: ODIM already achieves AUROC 0.9820.982 on the original inputs, and shuttle has the largest σ\sigma in Table B.3 (4.044.04). We have added this discussion to Appendix B.4.

AuthorsOpenReview ↗

[Reminder: discussion period ends soon]

Dear reviewer N4XM,

Thank you again for your review. We understand that you are very busy, but as the discussion period ends soon, we would appreciate if you could take a look at our response. We hope and believe that it has addressed your concerns, and we would be grateful to know whether any other concerns remain.

Thanks, Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

[General Response]

To reviewers: We sincerely thank all reviewers for their careful reading and constructive comments. Following the review comments, we clarify the concerns and bridge the gap between our theory and numerical studies. Please also find our point-by-point responses below.


Summary of Changes

  • Remarks A.2–A.3. The decoder rank and width requirements and the role of the fixed decoder.
  • Appendix A.4. The observed separation intervals, the reconstruction-to-center intervals, the approximation of Bν(t)B_\nu(t) on the simulation data, and the initialization analysis.
  • Appendix B.4. The comparison of warm-ups with and without EMA on TabPFN, the geometry before and after TabPFN, the dataset-level differences on the benchmark, the rank correlations of σ\sigma, and the discussion of shuttle.
  • Appendices B.1 and B.4. The rationale and sensitivity of the TabPFN target.
  • Section 4.1. The IM-window definition aligned with Appendix B.2.
  • Remark 3.2. The quantities that remain relevant under joint training.

All changes are highlighted in blue in the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.