acceptodds
Under review as a conference paper at ICLR 2027

Semi-Supervised Tabular Anomaly Detection with Contamination-Tolerant In-Context Learning

Abstract

In-context learning (ICL) enables pretrained tabular models to adapt to new tasks without task-specific parameter updates by treating training instances as context. However, directly applying ICL to semi-supervised tabular anomaly detection is challenging because unlabeled contextual data may be contaminated by hidden anomalies, while only a few labeled anomalies are available to provide reliable supervision. Moreover, open-world unlabeled data may contain unknown anomaly types that are not represented by the labeled anomalies. To address these challenges, we propose TabPFN-AD, a framework that iteratively constructs a more accurate in-context set under the semi-supervised tabular anomaly detection setting. TabPFN-AD leverages the inter-instance attention of a pretrained tabular foundation model to evaluate the degree of anomalousness at both the instance and group levels. By mining both known and unknown anomalies from unlabeled data, it maximizes the utility of limited labeled supervision to improve detection performance. Experimental results demonstrate that TabPFN-AD achieves state-of-the-art performance on 17 benchmark datasets and exhibits strong robustness under scenarios where unlabeled data is mixed with a high proportion of potential anomalies and labeled anomalies are extremely scarce.

Primary area:
unsupervised, self-supervised, semi-supervised, and supervised representation learning
Keywords:
anomaly detection, semi-supervised learning, in-context learning

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_WTVoOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper studies semi-supervised tabular anomaly detection with a small number of labeled anomalies and contaminated unlabeled data, and proposes TabPFN-AD. The method constructs graphs using inter-instance attention from a frozen TabPFN model, mines potential anomalies through instance-level and group-level scoring, and iteratively updates pseudo-labels in the context set. The authors compare the method against baselines on 17 datasets and report experiments on contamination robustness, component ablations, and different TabPFN versions.

Critical strengths

The problem is practically relevant. Treating all unlabeled instances as normal introduces incorrect supervision into the context of an in-context learning model. Improving context quality to address this issue is a worthwhile research direction. The paper is well written and generally clear.

Critical weaknesses

1.The paper assumes that normal instances have many similar peers and therefore exhibit more dispersed attention, whereas anomalous instances have fewer similar peers and therefore exhibit more concentrated attention. However, this does not hold across all data distributions. In real-world scenarios, anomalies may also form large, recurring patterns or have no similar peers at all. I noticed that Figure 2 primarily presents observations on NB15, but I do not think this is sufficient to establish whether this structural assumption is reliable across other data distributions. Furthermore, the paper selects attention heads based on associations among labeled anomalies and self-similarity, and then treats these heads as capturing same-class associations. My question is whether attention relationships that are useful for a small number of known anomalies are also applicable to unknown anomaly types. 2.Table 3 shows that adding the group-level component improves the average AUPRC from 0.8311 to 0.8469. This shows that the group-level component helps overall detection, but I cannot yet determine whether the gains primarily come from identifying unknown anomaly types. Figure 3(c) illustrates the mining of known and unknown anomalies on NB15. However, the main text does not clearly explain how known and unknown types are partitioned, nor does it report their test performance separately. I therefore also have questions about the component’s generalization to unknown anomaly types. I would therefore like the authors to clarify: How are known and unknown anomaly types partitioned? How much does the group-level component independently contribute to test performance on unknown anomalies? Could rare normal subgroups also receive high anomaly scores, and how is the corresponding false-positive risk evaluated?

Questions for authors

Please see the specific concerns and questions raised in the Weaknesses section.

2 comments

We thank the reviewer for raising important questions about the attention-structure assumption and the detection of unknown anomalies. We clarify these issues below.

1. Attention-structure assumption and applicability to unknown anomalies.

The observation that normal instances often have more similar peers, whereas anomalies have fewer, serves as an empirical motivation for the instance-level score sIs_I, rather than a necessary assumption for all data distributions. We agree that anomalies may form large recurring patterns or have no similar peers. In the former case, strong connections among anomalies may weaken the degree contrast used by sIs_I. We therefore introduce sGs_G to identify internally compact anomaly subgroups that may be under-ranked by sIs_I.

Attention-head selection is not intended to find heads specific only to the few known anomaly types. Some heads exhibit misleading relationships, such as normal instances assigning higher attention to labeled anomalies than anomalous instances do. C1C_1 removes heads without meaningful anomaly-related associations, while C2C_2 retains heads that preserve local similarity. The selected heads therefore provide a more reliable relational space, without requiring every unknown anomaly to share the same attention pattern as the labeled anomalies.

Within this space, sIs_I mainly exploits associations with labeled anomalies, whereas sGs_G is designed to reduce this dependence. Unknown anomalies need not assign high attention to labeled anomalies; they can be identified by sGs_G when instances of the same unknown type form an internally connected local subgroup. Rare but compact normal subgroups may also receive high sGs_G, but the method does not directly label every compact subgroup as anomalous. Group-level mining is restricted to the low-out-degree candidate set, combines instance- and group-level rankings, and adds only the core instances of the highest-scoring subgroup. Across 17 datasets, the mean anomaly fraction increases from 46.3%46.3\% among candidate subgroups to 83.4%83.4\% among selected subgroups, and 89/9689/96 (92.7%92.7\%) of the core instances added to pseudo-label updates are true anomalies.

2. Known/unknown split and the contribution of sGs_G on NB15.

NB15 provides fine-grained attack-type labels, allowing us to evaluate the contribution of sGs_G to unknown-anomaly detection. We treat DoS, Analysis, Fuzzers, and Reconnaissance as known anomaly types and use them to construct the labeled-anomaly set. Backdoor, Exploits, and Generic receive no labeled examples and are treated as unknown types. These type labels are used only to construct the split and perform post-hoc analysis; they are not used in head selection, subgroup construction, or pseudo-label refinement.

On NB15, adding sGs_G improves AUPRC from 0.70560.7056 to 0.83080.8308, a relative gain of approximately 17.7%17.7\%. The subgroup-composition analysis further shows that sGs_G discovers local subgroups containing Backdoor, Exploits, and Generic instances, despite receiving labeled examples only from the known types. This result indicates that the selected heads support not only instance-level mining based on known anomalies but also group-level discovery of previously unseen anomaly types through local similarity structure.

Thank you for the response.After considering it,I’ll keep my original score.

Reviewer_PEfUOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors of the paper tackles an important problem of anomaly detection with contamination. They propose a method that builds on top of the well established TabPFN, termed TabPFN-AD. In essence, TabPFN-AD puts a few labeled anomalies and a contaminated unlabeled pool, treated as normal, into the context of a frozen TabPFN, and then repairs that context. The method works on a sampled subset with the following procedure.

  1. It turns the inter-instance attention of selected heads into a sparse graph.
  2. It ranks low-out-degree candidates by a degree score.
  3. It adds the core of the most compact subgraph to catch "unknown" anomaly types.
  4. It smooths the pseudo-labels over iterations, and the relabeled subsampled dataset becomes the context at test time.

The authors of the paper conduct empirical experiments to demonstrate the effectiveness of the proposed method.

Critical strengths
  • The problem the authors tackle is valid and important.
  • The proposed method is technical sound and achieves good performance.
  • The authors of the paper conduct careful empirical experiments and ablation to demonstrate the effectiveness of the proposed method.
Critical weaknesses
  • The general idea of doing data relabeling using a pretrained model to clean up contamination or label noise is a widely studied topic. I think the proposed idea fall very similar to a lot of the existing methodologies proposed for training machine learning models when dataset contains noisy labels. As such, in my opinion, the proposed method is somewhat incremental in terms of novelty. I also think the authors of the paper should conduct a more careful literature review and describe the similarities and differences of the proposed method against this important line of research on training neural networks with noisy labels in general, and also in the context of robust anomaly detection with contamination.
  • Building on top of the previous point, how does the proposed method compare against these previously proposed methods, if we just naively combined them with TabPFN, or if we compare TabPFN against these noise robust algorithms? I think the paper would be strengthened further if the authors of the paper benchmark against more algorithms explicitly designed against tackling the same problem of AD with contamination. TabPFN seems like a rather weak baseline in my opinion.
Questions for authors

"Noise Immunity in In-Context Tabular Learning: An Empirical Robustness Analysis of TabPFN's Attention Mechanisms" (https://arxiv.org/abs/2604.04868) seems to suggest that TabPFN is already pretty robust against noisy labels. What are you doing differently compared to the experiments done in this paper?

1 comment

We thank the reviewer for recognizing the importance of the problem, the technical soundness of the method, and the thorough empirical evaluation. We address the three main concerns below.

1. Distinction from noisy-label learning and methodological novelty.

Our contribution is not pseudo-labeling or label correction in itself. Conventional noisy-label learning generally assumes corrupted labels in a supervised training set and learns robust model parameters through small-loss selection, sample reweighting, co-training, or robust losses. In our setting, only a few anomalies are labeled, while hidden anomalies in the unlabeled pool are systematically treated as normal context. This produces structured, one-sided contamination under extremely limited positive supervision.

Moreover, the TabPFN backbone remains frozen throughout the procedure. Rather than learning noise-robust parameters, TabPFN-AD extracts inter-instance attention from the pretrained model, constructs sparse relation graphs, and uses instance-level degree statistics and group-level compactness to repair the contaminated context. The former identifies instances structurally related to known anomalies, while the latter captures local anomaly patterns not covered by the limited labeled examples. Thus, the main novelty lies in exploiting the internal attention structure of a frozen tabular foundation model for contamination-tolerant context construction.

2. Contamination-robust baselines and naive iterative refinement.

TabPFN is not our only baseline. We compare TabPFN-AD with unsupervised methods, seven semi-supervised anomaly-detection methods, and two tabular foundation models, TabPFN and LimiX. In particular, RoSAS is explicitly designed to improve robustness against potential anomaly contamination in unlabeled data and is included in our main comparison. TabPFN-AD achieves better average AUPRC and average rank than RoSAS under the same experimental protocol.

We also evaluate naive iterative variants of TabPFN and LimiX to determine whether the gains arise merely from repeated pseudo-label augmentation. At each round, the top 0.1%0.1\% highest-scoring unlabeled instances are labeled as anomalies and added to the next-round context, for a total of ten rounds. This strategy improves performance only on a subset of datasets, and its optimal round varies substantially across datasets. In contrast, TabPFN-AD provides more consistent improvements, showing that its gains primarily come from attention-guided instance- and group-level mining rather than iteration alone.

3. Difference from Noise Immunity in In-Context Tabular Learning.

The referenced work studies the robustness of TabPFN to controlled random label flips and input perturbations in standard binary classification. Our setting involves a different noise mechanism: hidden anomalies are systematically assigned normal contextual labels, producing structured and one-sided contamination. Furthermore, only a few correctly labeled anomalies are available, and the unlabeled data may contain anomaly types absent from the labeled examples. Robustness to random label noise therefore does not imply robustness under this scarce-positive, open-world contamination setting.

Our experiments directly demonstrate this distinction. On NB15, increasing the unlabeled contamination ratio from 1%1\% to 10%10\% reduces the AUPRC of TabPFN from 0.7700.770 to 0.3460.346, a relative decrease of 55.1%55.1\%. Reducing the number of labeled anomalies from 100 to 5 results in a relative decrease of 22.9%22.9\%. TabPFN-AD does not contradict the robustness reported in the referenced work; instead, it exploits the remaining structured attention signals to identify hidden anomalies and repair the context under a different and more asymmetric contamination setting. We will cite this work and clarify the distinction in the revised manuscript.

Reviewer_PyNfOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper studies semi-supervised tabular anomaly detection with a few labeled anomalies and an unlabeled context that may contain hidden anomalies, including types absent from the labeled examples. It proposes TabPFN-AD, a parameter-frozen adaptation of TabPFN that selects attention heads by a labeled-anomaly relevance criterion and a self-similarity criterion, builds a sparse attention graph, and mines potential anomalies with an instance-level score based on in- and out-degree statistics and a group-level score based on subgroup attention compactness, followed by iterative pseudo-label refinement. On 17 datasets, it reports an average AUPRC of 0.847 against unsupervised, semi-supervised, and foundation-model baselines, with particular robustness to context contamination and extreme label scarcity.

Critical strengths

The method is a genuinely interesting reuse of a frozen foundation model's internals. Rather than treating TabPFN as a black-box predictor, the paper reads its inter-instance attention as a similarity graph and derives out-degree, in-degree, and subgroup compactness signals from it, and the distinction between known-type anomalies that attach to labeled examples and unknown-type anomalies that cluster among themselves is well motivated and empirically supported by the attention visualizations.

The problem setting is also the right one: assuming the unlabeled context is clean is unrealistic, and the contamination and label-scarcity experiments show the gap widens exactly where deployment gets hard. The evaluation breadth, including ablations, sensitivity analyses, a naive-iteration control, and runtime accounting, is better than most papers in this area.

Critical weaknesses
  1. The framework's applicability beyond a single backbone family is not established, and this matters more than usual because the whole mechanism is built from architecture-specific parts. The head-selection criteria and the mining scores operate on TabPFN's inter-instance attention, yet the method is only instantiated on TabPFN V2 and V3. TabICL is cited but never used, LimiX appears only as a fixed-configuration baseline, and current checkpoints such as TabPFN V3.5 are not considered.

A practitioner choosing a foundation model today has several families and versions available, and the paper gives no evidence on whether the attention-mining pipeline transfers, or even whether the underlying attention statistics exist in the same form, in other tabular foundation models. As it stands, the contribution reads as a TabPFN-specific technique rather than a framework for tabular foundation models.

  1. The evaluation rests on an aging data basis. Twelve of the seventeen datasets come from ODDS, whose tasks are small and widely regarded as no longer representative, and the paper does not evaluate on broadly used modern benchmarks such as ADBench. For a method whose selling point is practical robustness under contamination, demonstrating this on the standard contemporary testbed would substantially strengthen the claim.

  2. There are also smaller issues that are individually fixable but should be corrected. The main tables report 0.8469 for the default method while Appendix Tables 5, 11, 12, and 14 report 0.858 for what appears to be the same configuration, with different aggregation protocols, and the initialization rule is stated inconsistently across Section 3.3, Appendix A.2, and Table 12. The separation condition in Appendix A.3 assumes i.i.d. Gaussian peripheral attention without testing the assumption, and the unknown-anomaly evidence is a post-hoc audit concentrated on a single dataset, so the open-world claims should be scoped accordingly.

Questions for authors

Can the mining pipeline be instantiated on a non-TabPFN backbone, for example LimiX or TabICL, and do the attention statistics it relies on exist in comparable form there?

Does the method remain compatible with current checkpoints such as TabPFN V3.5?

Can you report results on ADBench or a comparable modern tabular benchmark under the same contamination protocol?

If the authors address the concerns above, I am willing to raise my score.

2 comments

We thank the reviewer for the constructive comments and for recognizing the motivation, methodological design, and empirical evaluation of our work. We address the three main concerns below and will revise the manuscript to clarify the scope of the framework, the dataset selection protocol, and the interpretation of the reported results and theoretical analysis.

1. Applicability to other backbones and TabPFN V3.5.

Although the underlying idea may be extended to other tabular foundation models, the current implementation is not a plug-and-play module. LimiX, TabICL, and TabPFN differ in their attention organization and internal interfaces; applying our method therefore requires model-specific adaptation of attention extraction, head selection, and graph construction. We will clarify that the current evidence supports a TabPFN-based framework, while extension to other backbone families remains future work.

Moreover, TabPFN V3.5 had not been released when we developed the idea and conducted the main experiments. It became available only recently, making reliable adaptation and a full five-seed evaluation over 17 datasets infeasible within the rebuttal period. Our V2/V3 experiments demonstrate compatibility across the TabPFN versions available during method development.

2. ODDS datasets and ADBench.

We agree that evaluation on broader modern benchmarks would further strengthen the paper. However, our protocol requires sufficient anomalies both to construct the labeled-anomaly set and to retain additional hidden anomalies under controlled contamination ratios. Some small ADBench datasets contain too few anomalies to consistently support different labeled-anomaly counts, contamination ratios, and random seeds. The 17 datasets were therefore selected for both comparability with prior semi-supervised anomaly-detection studies and compatibility with a unified contamination protocol.

3. Numerical differences and the scope of the theoretical analysis.

The reported values 0.84690.8469 and 0.8580.858 follow different seed protocols. The main result, 0.84690.8469, is averaged over 17 datasets and five sampling seeds, whereas 0.8580.858 is the 17-dataset average with the sampling seed fixed to 00, as stated at the beginning of the corresponding appendix analysis. We agree that this distinction should be more visible and will repeat it in the relevant table captions. Regarding Appendix A.3, the i.i.d. Gaussian assumption is an analytical approximation used to derive a conditional sufficient condition rather than a universal distributional guarantee. We will revise the wording accordingly. Importantly, the accompanying post-hoc audit does not rely on this assumption: across 17 datasets, the mean anomaly fraction increases from 46.3%46.3\% among candidate subgroups to 83.4%83.4\% among selected subgroups, and 89/9689/96 selected core instances are true anomalies. Fine-grained known/unknown analysis is restricted to NB15 because the remaining datasets provide only binary labels; we will explicitly limit the scope of this claim rather than generalizing it to all datasets.

Thank you for the response; however, as the main experimental and benchmark concerns remain unaddressed, I will keep my current rating.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.