- Rating
- 3(+1)
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper investigates the inlier-memorization (IM) effect in unsupervised outlier detection: a model may learn shared inlier structure before fitting outlier-specific deviations. It studies a bias-free, single-hidden-layer autoencoder with a fixed decoder and an encoder optimized by full-batch gradient descent. The core argument compares actual training with a reference trajectory using nearest-cluster-center inputs. Theorem 3.1 establishes, under explicit regularity and interval conditions, a nonempty interval of cluster-consistent reconstructions; Theorem 3.2 and Corollary 3.3 then give sufficient geometric conditions under which an outlier has larger reconstruction error than every inlier. Simulations examine cluster compactness, cluster imbalance, contamination and initialization. Motivated by these results, the authors test pretrained compact representations and an EMA warm-up on ODIM and ALTBI, reporting modest average improvements on ADBench datasets.
Critical strengths
-
A well-targeted theoretical question. Unlike prior IM-based detectors emphasizing exploitation or stopping of early training, the submission directly asks when cluster-consistent reconstruction and provable inlier/outlier loss separation can occur. The explicit connection between the center-tracking bound Bν(t), separation threshold 2Bν(t)+εc, and a certified training interval is an intelligible conceptual contribution (Sec. 3; Appendix A.3.2).
-
A reasonably broad attempt to test qualitative predictions. The paper studies four controlled factors and checks both an idealized fixed-decoder regime and more practical joint-training dynamics (Sec. 4.1; Appendix B.3). It also reports individual/combined EMA and representation changes across two established methods and a large benchmark collection, rather than evaluating only a new bespoke detector (Sec. 4.2; Tables B.3–B.9).
Critical weaknesses
1.Clarify the feasibility and practical significance of the theoretical sufficient conditions. Theorem 3.1 requires a positive margin (c > 0), the interval condition in Eq. (A.24), and uniform Jacobian regularity under Assumption A.2. While these conditions are mathematically meaningful, their feasibility in representative experimental settings remains unclear. The authors are encouraged to provide an illustrative example with estimates of the relevant theoretical quantities and compare the predicted interval with observed training dynamics. A clearer discussion of the restrictiveness of these assumptions would help establish the practical significance of the theoretical guarantees.
2.Strengthen the connection between the theoretical analysis and the proposed practical strategies. The theoretical analysis considers a fixed-decoder autoencoder trained with full-batch gradient descent, whereas the practical experiments involve more general training settings. Although the authors provide supporting simulations, the connection between the theoretical quantities and the proposed representation and EMA strategies could be more explicit. Reporting how these strategies affect representative measures of cluster compactness and initialization quality on selected datasets would help substantiate their theoretical motivation.
3.Further examine performance variability and cases of degradation. The proposed strategies yield modest average improvements but exhibit considerable variability across datasets. For instance, the AUROC of ODIM on shuttle decreases from 0.982 to 0.709 after applying the combined modifications (Table B.6). Moreover, the average AUPRC of ODIM decreases from 0.334 to 0.322 (Table B.7). A brief analysis of such cases, together with a summary of dataset-level improvements and degradations and their variability across random seeds, would help clarify the reliability and applicability of the proposed strategies.
4.Further assess the robustness of the geometric diagnostic analysis. The geometric diagnostic (\sigma) in Section 4.2 provides an interesting empirical connection to Corollary 3.3. However, it is a post-hoc measure based on estimated cluster statistics rather than a direct verification of the theoretical separation condition. Given that the correlation analysis involves only eleven datasets, a simple robustness check, such as a rank-based correlation or leave-one-dataset-out analysis, would help strengthen the interpretation of the reported relationship.
5.Clarify the construction and robustness of TabPFN representations in the unsupervised setting. Appendix B.1 describes extracting TabPFN representations by treating the first feature column as a supervised target, or the second column when the first has negligible variance. Although this procedure does not directly use anomaly labels, the rationale for selecting an arbitrary feature as the prediction target remains unclear. The authors should clarify how this choice affects representation quality and, if feasible, examine whether the detection performance is sensitive to the selection of the target feature.
Questions for authors
Q1. Could the authors provide an illustrative example showing how the sufficient conditions of Theorem 3.1 relate to the empirically observed IM interval?
Q2. Which theory-relevant quantities remain most informative when extending the analysis from fixed-decoder training to practical joint-training settings?
Q3. Does EMA warm-up measurably improve the initial cluster-center reconstruction quality? How does it compare with a non-EMA warm-up under a similar computational budget?
Q4. Could the authors discuss the substantial performance degradation of ODIM on shuttle and clarify the dataset characteristics under which the proposed strategies may be less effective?
Q5. Section 4.1 and Appendix B.2 appear to use different definitions of the empirical IM window. Could the authors clarify which definition was used to generate the reported results?
6 comments
Response to Reviewer XH38 (2/4)
W2: Strengthen the connection between the theoretical analysis and the proposed practical strategies. The theoretical analysis considers a fixed-decoder autoencoder trained with full-batch gradient descent, whereas the practical experiments involve more general training settings. Although the authors provide supporting simulations, the connection between the theoretical quantities and the proposed representation and EMA strategies could be more explicit. Reporting how these strategies affect representative measures of cluster compactness and initialization quality on selected datasets would help substantiate their theoretical motivation.
Response
Thank you for this valuable suggestion.
-
EMA versus ordinary warm-up. In the simulation (), both methods reduced the mean initial cluster-center reconstruction error from to similar values after updates, with for ordinary warm-up and a slightly lower for EMA. Since the true centers are unknown in the eleven real-world datasets, we compare AUROC and AUPRC using the same TabPFN representations and identical warm-up updates.
Dataset ALTBI standard ALTBI EMA ODIM standard ODIM EMA ALOI 0.542 / 0.036 0.542 / 0.037 0.534 / 0.042 0.532 / 0.041 annthyroid 0.601 / 0.091 0.597 / 0.090 0.539 / 0.084 0.518 / 0.079 campaign 0.601 / 0.173 0.599 / 0.175 0.577 / 0.188 0.568 / 0.183 landsat 0.515 / 0.220 0.510 / 0.211 0.553 / 0.223 0.580 / 0.236 magic.gamma 0.650 / 0.530 0.648 / 0.527 0.639 / 0.539 0.636 / 0.532 mammography 0.764 / 0.054 0.757 / 0.053 0.747 / 0.052 0.718 / 0.052 PageBlocks 0.841 / 0.348 0.853 / 0.343 0.864 / 0.392 0.902 / 0.425 shuttle 0.928 / 0.355 0.928 / 0.425 0.890 / 0.246 0.901 / 0.300 SpamBase 0.628 / 0.537 0.627 / 0.536 0.614 / 0.521 0.611 / 0.526 Wilt 0.391 / 0.040 0.391 / 0.040 0.394 / 0.040 0.394 / 0.040 yeast 0.510 / 0.344 0.512 / 0.346 0.509 / 0.345 0.509 / 0.346 Mean difference (EMA − standard) -0.0004 / +0.0051 +0.0008 / +0.0079 With EMA, mean AUROC changed little, while mean AUPRC increased modestly ( for ALTBI and for ODIM). ODIM’s two largest AUPRC gains occurred on PageBlocks and shuttle, which also had the highest values after TabPFN ( and , respectively). We conjecture that EMA may help preserve inlier-cluster structure during warm-up, making outliers that lie relatively far from those centers easier to detect.
-
Effect of the TabPFN representation. For the eleven datasets, we applied the procedure of Section 4.2 to the raw input and the TabPFN representation of the same samples, and computed and the relative inlier spread .
Dataset before TabPFN after TabPFN (after − before) : before → after ALOI 0.224 0.313 +0.088 7.14 → 1.22 annthyroid 0.491 0.360 -0.131 6.87 → 1.01 campaign 0.577 0.327 -0.250 2.01 → 0.98 landsat 0.485 0.273 -0.212 2.81 → 1.10 magic.gamma 0.703 0.325 -0.378 4.94 → 1.42 mammography 0.586 0.278 -0.308 5.45 → 1.13 PageBlocks 1.125 0.651 -0.473 6.00 → 1.78 shuttle 3.729 0.547 -3.181 1.71 → 1.10 SpamBase 0.322 0.018 -0.304 8.16 → 1.05 Wilt 0.257 0.505 +0.248 5.94 → 1.10 yeast 0.254 0.361 +0.107 1.43 → 1.17 TabPFN makes the inlier clusters more compact on all eleven datasets (mean relative spread ) and reduces on average (). We have added both comparisons to Appendix B.4.
Response to Reviewer XH38 (3/4)
W3: Further examine performance variability and cases of degradation. The proposed strategies yield modest average improvements but exhibit considerable variability across datasets. For instance, the AUROC of ODIM on shuttle decreases from 0.982 to 0.709 after applying the combined modifications (Table B.6). Moreover, the average AUPRC of ODIM decreases from 0.334 to 0.322 (Table B.7). A brief analysis of such cases, together with a summary of dataset-level improvements and degradations and their variability across random seeds, would help clarify the reliability and applicability of the proposed strategies.
Response
Thank you for the suggestion.
-
Dataset-level differences over the 57 datasets. Using the full-precision means over three seeds underlying Tables B.6–B.7, we computed the dataset-level AUROC difference (modified − original), also without shuttle.
Method / metric Datasets Mean difference Increase / tie / decrease ALTBI / AUROC 57 +0.0091 32 / 2 / 23 ODIM / AUROC 57 +0.0056 32 / 0 / 25 ALTBI / AUROC 56 (without shuttle) +0.0092 31 / 2 / 23 ODIM / AUROC 56 (without shuttle) +0.0106 32 / 0 / 24 Both methods improve the mean AUROC, and improve it on of the datasets. Without shuttle, the mean AUROC gain of ODIM nearly doubles (), so much of the variability comes from the single large drop on shuttle.
- In addition, the variation across seeds is small. Rerunning ODIM on the raw input with three data seeds, the AUROC varies by at most on shuttle and campaign and by on SpamBase, which is small compared with the differences between datasets.
- For reference, ODIM's stopping rule is a heuristic and can select an unsuitable iteration. ODIM stops with a patience rule and reports the scores of the single check at which the separation between two Gaussian components fitted to the mini-batch losses is largest. This criterion is heuristic and fairly sensitive to the loss trajectory, so it occasionally stops outside the period in which the outliers are well separated. Changing the representation changes this trajectory, which we believe causes the drop on shuttle.
- In contrast, ALTBI averages the scores over iterations and is less sensitive to this choice. As discussed in Remark 3.1, neither the optimal stopping iteration nor is known in practice, and averaging the reconstruction errors over training iterations retains the separation obtained within while reducing the dependence on a single stopping iteration. This is why, with the same TabPFN representation, ODIM drops to while ALTBI keeps AUROC (Table B.6).
- Moreover, shuttle is already well separated. Shuttle has the largest in Table B.3 (), so a new representation has little room to improve detection. We have added this discussion to Appendix B.4.
W4: Further assess the robustness of the geometric diagnostic analysis. The geometric diagnostic (\sigma) in Section 4.2 provides an interesting empirical connection to Corollary 3.3. However, it is a post-hoc measure based on estimated cluster statistics rather than a direct verification of the theoretical separation condition. Given that the correlation analysis involves only eleven datasets, a simple robustness check, such as a rank-based correlation or leave-one-dataset-out analysis, would help strengthen the interpretation of the reported relationship.
Response
Thank you for the suggestion.
-
We computed the Spearman correlation and Kendall's between and the AUROC change on the eleven datasets, and repeated each after leaving out one dataset at a time.
Analysis ODIM ALTBI Spearman (p) -0.809 (0.003) -0.882 (0.00033) Kendall (p) -0.673 (0.003) -0.782 (0.00033) Leave-one-out Spearman [-0.891, -0.745] [-0.964, -0.842] Leave-one-out Kendall [-0.778, -0.600] [-0.911, -0.733] Both rank correlations are significantly negative, and they remain negative after leaving out any single dataset, including shuttle. That is, the relation between and the gain does not depend on a particular dataset. We have added these results to Appendix B.4.
Response to Reviewer XH38 (1/4)
We thank the reviewer for the helpful comments and suggestions. We address the concerns point-by-point as below.
Response
Thank you for this important suggestion.
The constants in Bν(t) are defined over a whole neighborhood of the initialization and cannot be computed exactly, so we approximated Bν(t) on the original simulation data by measuring its three terms directly.
Setting. We used the original Figure 2 data with εc=1.0 (n=2,000, K=3, p=2) and one of its original initializations, fixed the decoder, and trained the encoder by full-batch gradient descent with learning rate 10−4 for 600,000 updates.
Bν(t) approximation. In Eq. (A.19) (Appendix A.3.2), Bν(t) is the sum of three terms: the input-replacement error, the gap between the actual and reference trajectories, and the reference reconstruction error. We measured each term directly during training and summed them, i.e., we obtained an upper bound on the distance between each reconstruction and its center. Taking this bound over 80% or 90% of the samples gives the approximated Bν(t).
Result. The table reports the intervals in which Bν(t)<δ/2=0.1092, together with the observed distance and the AUROC over each interval.
The approximated bound stays below δ/2 for more than half of training (310,000 updates, 51.7%, for 90% of the samples), and the AUROC stays high (0.972–0.974) throughout this interval. That is, as the theory states, the IM effect appears while the reconstructions stay close to their centers.
To directly illustrate the feasibility of the sufficient conditions, we also considered a separate synthetic setting with two tight Gaussian clusters in R3, each containing 5,000 inliers, with center separation δ=0.99 and one outlier near each center. We trained a fixed-decoder autoencoder with H=64 using full-batch gradient descent. Bounds valid throughout the prescribed neighborhood give αc=6.997, βc=7.001, and Lc=0.351. All conditions of Theorem 3.1 are satisfied: the margin is c=0.482>0, and Eq. (A.24) holds (81.8<959.9), producing the certified center-tracking interval [32,53]. Over this interval, the theoretical bound is Bν(t)∈[0.024,0.040], while the observed maximum center-tracking error remains below 9.4×10−5. Full details are provided in Appendix A.4.