Fidelity is Not Enough: Tabular Anomaly Detection under Synthetic Distribution Shift

Real network anomaly data are difficult to collect and label, which makes it difficult to support benchmarking and motivates the growing use of synthetic data for developing and evaluating network anomaly detectors. However, generated data can preserve distributional properties of real data while altering the feature relationships used by a detector. We investigate whether detectors trained exclusively on real data remain effective on generated data. We test this across multiple tabular generators, datasets, and detection models. We find that distributional fidelity does not reliably predict detection performance: detectors can behave differently across generators even when generated data closely match the real-data distributions, and detector rankings on real data can change substantially on generated data. Filtering data by predicted provenance is similarly unreliable, as it can miss attacks from unseen generators while rejecting real benign traffic. Motivated by these findings, we introduce D${}^2$Former, a behavioural anomaly detector that encodes features relative to quantiles of the real training distribution, models cross-feature interactions with self-attention, and aggregates encoder outputs using statistical summaries beyond a single classification token. Across three seeds, D²Former and TabTransformer have comparable overall transfer, while D²Former performs better on BCCC, where the largest transfer losses occur. In these DDoS datasets, generated data provide useful stress tests but do not reliably substitute for held-out real data when evaluating detectors. We therefore argue that original real data should remain the basis for developing and validating network anomaly detectors, with results on generated data treated as supplementary evidence rather than a replacement for evaluation on real data.