Dual-Order Rènyi–Dirichlet Graph Autoencoders for Anomaly Detection on the Simplex

Low- and high-dimensional simplex-valued data arise in microbial communities, spatial transcriptomic profiles, particle-size distributions, and normalized image intensities, where observations may be connected by a graph and corrupted by heterogeneous anomalies. We introduce Dual-Order Rényi–Dirichlet Graph Autoencoders (RD-GAEs), a simplex-geometry preserving graph autoencoder that uses Dirichlet latent variables and Rényi divergences at two distinct orders. A subunit training order emphasizes the dominant, regular structure and reduces the influence of contaminated observations during representation learning; after training, a larger detection order amplifies reconstruction discrepancies in distributional tails, yielding anomaly-sensitive scores without retraining the model. The encoder and decoder preserve compositional support through graph aggregation and gated probability transformations, while the graph can be row-stochastic or doubly stochastic. Across controlled-contamination and held-out detection protocols for image, microbiome, and spatial transcriptomics data, RD-GAE achieves superior anomaly-detection performance while retaining interpretable simplex-valued representations, compared with graph, variational, Dirichlet, and Euclidean-coordinate autoencoder baselines.