Tabular Anomaly Detection via Causal Matryoshka Trajectories

Structural anomalies are samples where individual feature values appear plausible but their combination violates underlying dependencies. Existing anomaly detection (AD) methods often rely on a single fixed latent representation to separate normal and abnormal data, making localized structural violations susceptible to signal dilution in high-dimensional spaces. We propose NestedAno, an unsupervised tabular AD framework that learns a hierarchy of nested representations, inspired by Matryoshka Representation Learning, and progressively organizes them using topologically aligned causal reconstruction objectives. At inference time, it measures local deviation at each nested scale to construct an anomaly trajectory for each sample. The integrated deviation of this trajectory from the normal reference provides the final anomaly score, enabling anomaly evidence to be captured across multiple scales. Extensive experiments on 30 continuous and mixed-type datasets show that NestedAno achieves the highest average AUC-ROC among 17 representative baselines, yielding superior detection accuracy while providing highly interpretable trajectory separation between normal and anomalous data.