- The score does not measure what the paper claims.
The paper says CoVA-TAD finds rows that break the normal relations between features. To score a row, it hides one column at a time, asks TabICLv2 to predict it from the other columns, and uses how spread out that prediction is as the uncertainty. The problem is that the actual value of the hidden column is never compared to the prediction; it isn't used at all for that column's term. So if a row looks normal everywhere except one column that clearly does not fit the others, the prediction for that column stays confident, and that column adds little to the score. What the score really measures is how unfamiliar the rest of the row looks compared to the 200 normal context rows. That is much closer to a distance from the reference data than to a check of the relations between features.
The theory in the appendix does not fix this. The Gaussian process argument in Appendix B shows that predictive variance grows when a query is far from the context points, but this variance does not depend on the target values at all, so it also describes a distance and not a broken relation. The Gaussian argument in Appendix A has a bigger problem: for Gaussian data the variance of a column given the others is the same for every row, so it can not explain why anomalies would get higher scores. In a Gaussian model it is the prediction error that separates anomalies, which is exactly the part CoVA-TAD leaves out. The authors' own comparison of scoring formulas (Appendix L) includes a version that also uses the prediction error, and it performs worst, which is hard to square with their explanation.
Because of this, it is unclear whether the foundation model adds anything over a simple distance based detector. I would like to see (i) a kNN detector run on the same 200 context rows, (ii) a score based on the prediction error from the same model, (iii) the same pipeline with a simpler regressor in place of TabICLv2, and (iv) results on ADBench's synthetic dependency anomalies [3]. These are rows whose values are each normal on their own but whose combination is not, which is exactly the case the paper claims to handle. The idea of predicting each feature from the others to find anomalies is also not new: FRaC [1] and ALSO [2] do this, scoring the prediction error, and neither is discussed.
- The evidence is weak and some numbers can not be reproduced from the code.
The gain over the strongest baseline, OFA-TAD, is small (+0.66 AUROC, +1.79 AUPRC), and the authors' own tests show it is not significant. It also comes mostly from two datasets: if Hepatitis and fraud are removed, CoVA-TAD is slightly behind OFA-TAD on both metrics. The benchmark uses only 15 of the 47 standard ADBench datasets, all with at most 62 features, and every baseline except Isolation Forest is copied from the OFA-TAD paper rather than rerun.
Code repo: I found two results in Table 4 that do not come from the main experiment.
(a) The table suggests that raising the small constant ϵ in the score from 10−4 to 10−3 improves AUPRC from 78.19 to 81.28. But only the 78.19 row is from the main experiment. The rows for larger ϵ come from a separate script with a different setup: it does not shuffle the context rows, and it keeps only 2,500 normal test rows for fraud, campaign and shuttle. Removing normal rows makes anomalies more common in the test set, which raises AUPRC by itself, as the paper acknowledges in Appendix G. In that script's own run, ϵ=10−4 gives 81.32, the best value of all. So the improvement comes from comparing two different setups, and within one setup ϵ makes no real difference.
(b) The pima results for context size and number of columns are typed in by hand in generate_visualizations_and_ablations.py (lines 163 to 169), and no script in the repo computes them. They also disagree with the repo's own output: the default setting is reported as 75.20 AUROC / 75.60 AUPRC, but full_test_results.csv gives 74.07 / 74.71 for the same setting.
There are other inconsistencies as well. The seed study is described as measuring the effect of sampling different context rows, but lympho has only 71 reference rows, so all of them are used for every seed, and yet its AUPRC still changes by 12.53 points across seeds. Also, the fraud result, one of the two datasets driving the gain, comes from an earlier run that the authors could not repeat.
- Clarity.
The steps of the algorithm are clear enough to reimplement. However, the paper never says that the observed value of a column is not used in that column's uncertainty term, and the introduction and discussion describe the method as if it compares observed values to predictions, so readers will get the wrong idea of what the score does. Some details also contradict each other. Eq. (9) is called "evenly spaced", but for wide tables it never selects the last columns (for campaign, with 62 columns, it only goes up to column 49). Appendix M says breastw and glass have at most 8 features, while Table 5 says 9. Several cross references point to the wrong algorithm, table or appendix. The conclusion claims the method "surpasses all 10 published baselines" and gives "interpretable attributions", which the paper's own results and caveats do not support. Finally, the text in Figures 1, 3 and 4 is too small to read.
[1] Keith Noto et al. FRaC: A Feature-Modeling Approach for Semi-Supervised and Unsupervised Anomaly Detection. Data Mining and Knowledge Discovery, 2012.
[2] Heiko Paulheim et al. A Decomposition of the Outlier Detection Problem into a Set of Supervised Learning Problems. Machine Learning, 2015.
[3] Songqiao Han et al. ADBench: Anomaly Detection Benchmark. NeurIPS, 2022.