- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
The paper asks whether a pretrained tabular foundation model (TFM) can be reused for anomaly detection when the training data contain only normal samples. GradPAO learn pseudo-anomaly examples that supply the missing anomaly classes in the model's context. It uses the TFM's attention to group normal rows and moves selected rows away from their cluster centers to initialize these examples. Their feature values are then optimized through the frozen model using classification and attention overlap losses. Queries are scored by their predicted probability of belonging to any anomaly class. Experiments report strong aggregate performance across 57 ADBench and 690 public OddBench datasets, with TabICLv2 and TabPFN v3, component ablations, sensitivity studies, and runtime analysis. The contribution is an adaptation learned from normal data that changes the context supplied to an existing predictor.
Critical strengths
-
I really like the idea of using the pseudo-anomaly examples as a way of adapting the predictive model. The approach takes advantage of the TFM's pretrained knowledge while supplying/optimizing the missing anomaly classes and only requiring normal target data to do it. Using synthetic negatives and context optimization do have precedents, but I believe this combination represents an interesting and meaningful contribution. Additionally, I find it encouraging that results are strong across both TabICLv2 and TabPFNv3 as backbone. This supports the general use and application ofthis methodology across various TFMs (Sections 3.1-3.3 and Table 2).
-
I found the paper quite clear on a first read which was refreshing. Similarly, I feel its ablations support to main claims nicely and answer the natural questions a reader would have about this work. For example, with backbone and initialization fixed, ADBench's rAUC increases from 0.7589 without optimization to 0.7687 with separability and 0.7817 with attention diversity. This supports learning the examples beyond their initialization. Additionally, five seed main results and per-dataset tables add support, including the acknowledged AUROC case where OutFormer has a higher rescaled score (Tables 1 and 4, Appendices A.2 and A.6).
Critical weaknesses
-
It seems to me the attention heatmap leaves the diversity explanation open. Identical attention distributions could perhaps have small inner products when spread broadly over normal rows so low Gram entries may reflect flatter attention rather than different attended regions. The attention objective improves over input space diversity by 0.0056 rAUC without reported paired uncertainty, but I think the explanation would be more convincing with attention entropy, normalized similarity, and/or variation in the performance differences across seeds (Equation 7, Table 4, and Figure 3).
-
Additionally, I think the initialization needs a bit more implementation detail to make the method easier to reproduce, especially on large datasets. Appendix A.2 gives the optimization batch sizes, but the sample budget for constructing the initial attention map is less clear. The attention layer and preprocesing used at this stage also need clarification, along with how clusters containing fewer than eight rows or selected rows coinciding with their centroid are handled. These choices affect the initial pseudo-anomalies and the cost of constructing them. Although the default are listed, it would help to state when they were fixed and which data informed their selection (Section 3.2, Algorithm 1, and Appendices A.2 and A.4).
Questions for authors
-
Following weakness 1 above, it remains a bit unclear whether the diversity term leads to genuinely distinct behaviors, and I think further clarification would be helpful. For example, do the attention patterns remain distinct after accounting for differences in entropy and normalization? How consistent are the optimization and diversity gains across random seeds? Additionally, does incorporating multiple anomaly classes still provide benefits when the total number of pseudo-anomaly rows is held fixed?
-
Following weakness 2 above, could you explain in more detail how the initialization was implemented? More specifically, could you clarify the sample caps, attention layer, preprocessing, and small/degenerate cluster rules used for initialization, especially on large datasets? When were the defaults fixed, and which development data informed them?
4 comments
Rebuttal by Authors [2/2]
[Q1-3] Does incorporating multiple anomaly classes still provide benefits when the total number of pseudo-anomaly rows is held fixed?
[Response] We thank the reviewer for this insightful question, which helps isolate the effect of using multiple anomaly classes. To answer your question, we provide the results of varying the number of anomaly classes while fixing the total number of pseudo-anomalies at 72. As shown in the table below, the performance increases with the number of anomaly classes. Using 9 classes improves the rAUC by 0.095 over using a single class and by 0.033 over using 3 classes, and the average rank follows the same trend. Since the number of pseudo-anomalies is identical across the three settings, this gain comes from dividing the pseudo-anomalies into multiple classes rather than from the number of pseudo-anomalies. This supports our design of assigning a separate anomaly class to each cluster to represent diverse abnormal patterns, as described in Section 3.1.
| rAUC | Rank | |
|---|---|---|
| (9, 8) | 0.781 | 4.12 |
| (3, 24) | 0.748 | 4.61 |
| (1, 72) | 0.686 | 5.16 |
We note that a setting with more classes such as cannot be evaluated since most TFMs natively support at most 10 classes and one of them is reserved for the normal class, which limits to 9. Given the increasing trend with the number of anomaly classes, we expect that using more classes may further improve the performance as future TFMs support a large number of classes.
[W2/Q2] Could the authors provide more implementation details of attention-based initialization such as attention layer, preprocessing, clusters containing fewer than eight rows?
[Response] Thank you for pointing this out. To improve the reproducibility of our method, we provide the details of the preprocessing, the attention layer, and the handling of small clusters in the attention-based initialization, and we have incorporated them into Appendix A.2.
For the preprocessing, each feature is standardized using the mean and standard deviation of the train set, following the pipelines of ADBench and OddBench, and the forward pass then applies the built-in preprocessing of TabICLv2. To build the attention map, we use all normal training rows, or a random subset of 5,000 rows when more are available, and select a pool of up to 512 rows by farthest-point sampling. We then group the pool into clusters by agglomerative clustering on the attention map and select the least-attended rows in each cluster.
For the attention layer, we use the attention among the pooled rows in the first In-context Learning block of TabICLv2, averaged over its heads. As shown in the attention maps of different layers in Appendix A.4, the first layer shows block-diagonal patterns, where the normal samples attend mainly to the samples in their own cluster, whereas the deeper layers show vertical stripes, where a few rows receive more attention from all the other rows and the cluster patterns become less visible.
When a cluster contains fewer than eight rows, we reuse its rows and vary the displacement scale, so that the resulting pseudo-anomalies do not overlap and receive different gradient updates. Specifically, the displacement scale increases by 0.05 for each selected row, starting from , so that repeated rows are placed at different distances from the centroid.
Reviewer eDjW response to author rebuttal
Thank you for the detailed clarifications and additional experiments. I appreciate the evidence supporting the diversity objective and the consistency of the gains across seeds, as well as the additional implementation details. I am satisfied with the responses and will maintain my clear acceptance recommendation.
Dear Reviewer eDjW,
Thank you for your constructive feedback and for maintaining your recommendation. We are glad that the additional experiments and clarifications addressed your concerns, and we will incorporate them into the final version. If you have any further questions or suggestions, we would be happy to discuss them at any time.
Rebuttal by Authors [1/2]
Dear Reviewer eDjW,
We sincerely appreciate the reviewer’s time and insightful comments. Below, we address each comment in detail and will incorporate the corresponding analyses and clarifications into the revised manuscript.
[Response] Thank you for your valuable feedback. Following your suggestion, we computed the cosine similarity between the attention distributions of the pseudo-anomalies, but it was hard to see a distinct trend, since the cosine similarity also increases when the attention becomes flatter. We therefore computed the Pearson correlation between the attention weights of each pair of pseudo-anomalies across the normal samples and averaged these correlations over all pairs. Since the Pearson correlation is invariant to shifts and positive rescaling, it is unaffected by uniform flattening of the attention distributions. The effect of the diversity objective becomes clearer with this measure. As shown in the table below, without the diversity objective, the Pearson correlation increases from 0.602 at initialization to 0.682 after optimization, indicating that the attention patterns become more similar during optimization. With the diversity objective, it decreases to 0.592 after optimization, substantially below the value without the diversity objective. These results show that the diversity objective prevents the attention patterns of the pseudo-anomalies from becoming similar during optimization.
[Response] We appreciate this suggestion, which helps verify the reliability of our ablation results. Following your suggestion, we provide the rAUC of each variant in the optimization ablation for each of the 5 seeds on ADBench, as shown in the table below. On average, the optimization improves the rAUC from 0.7531 to 0.7654, and the diversity objective further improves it to 0.7788, with gains larger than their standard deviations across seeds. When comparing the variants within each seed, the optimization improves over the initialization-only variant, and the attention-based diversity objective outperforms the separability-only variant and the input-space and embedding-space objectives on all 5 seeds.