acceptodds
Desk rejected at ICLR 2027

Fidelity is Not Enough: Tabular Anomaly Detection under Synthetic Distribution Shift

Desk rejected at ICLR 2027, so left out of the paper lists and the map.

Abstract

Real network anomaly data are difficult to collect and label, which makes it difficult to support benchmarking and motivates the growing use of synthetic data for developing and evaluating network anomaly detectors. However, generated data can preserve distributional properties of real data while altering the feature relationships used by a detector. We investigate whether detectors trained exclusively on real data remain effective on generated data. We test this across multiple tabular generators, datasets, and detection models. We find that distributional fidelity does not reliably predict detection performance: detectors can behave differently across generators even when generated data closely match the real-data distributions, and detector rankings on real data can change substantially on generated data. Filtering data by predicted provenance is similarly unreliable, as it can miss attacks from unseen generators while rejecting real benign traffic. Motivated by these findings, we introduce DFormer, a behavioural anomaly detector that encodes features relative to quantiles of the real training distribution, models cross-feature interactions with self-attention, and aggregates encoder outputs using statistical summaries beyond a single classification token. Across three seeds, D²Former and TabTransformer have comparable overall transfer, while D²Former performs better on BCCC, where the largest transfer losses occur. In these DDoS datasets, generated data provide useful stress tests but do not reliably substitute for held-out real data when evaluating detectors. We therefore argue that original real data should remain the basis for developing and validating network anomaly detectors, with results on generated data treated as supplementary evidence rather than a replacement for evaluation on real data.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
Tabular machine learning, synthetic data, data fidelity, distribution shift, anomaly detection

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_uXx5OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

This paper examines whether synthetic tabular network traffic can serve as a reliable evaluation distribution for anomaly detectors trained on real data. It evaluates five generators across five DDoS datasets, assessing distributional fidelity, real-to-synthetic detector transfer, and cross-generator provenance filtering. The paper also introduces D²Former, which integrates quantile-based numerical embeddings, a Transformer encoder, and statistical aggregation. It argues that synthetic data should supplement, rather than replace, held-out real data in detector evaluation.

Critical strengths

Breadth and relevance of the empirical evaluation. The study addresses a practically relevant question: whether synthetic data preserve the evaluation outcomes of detectors trained on real traffic. By covering five datasets and five generators, while holding detector checkpoints and preprocessing fixed and sharing synthetic test sets across detectors, it provides a useful basis for examining generator-dependent behavior. The additional leave-one-generator-out provenance experiments offer a complementary perspective on the limitations of identifying synthetic origin.

Critical weaknesses
  1. Internal inconsistency in the reported evaluation metrics undermines confidence in the central comparisons. The reported evaluation metrics are internally inconsistent, which undermines confidence in the central comparisons. Appendix A.8 states that balanced accuracy, attack detection rate (ADR), and benign false-positive rate (B-FPR) are computed from the same predictions and test sets. For the binary datasets SDN and CRCDDoS2022, these quantities must satisfy: Balanced Accuracy=ADR+1−B-FPR2.\mathrm{Balanced\ Accuracy}=\frac{\mathrm{ADR}+1-\mathrm{B\text{-}FPR}}{2}. For D²Former on SDN–CTGAN, however, Tables 10 and 11 report ADR = 0.965 and B-FPR = 0.044, which implies a balanced accuracy of approximately 0.9605, whereas Table 3 reports 0.9196. Similarly, for DistilBERT on CRCDDoS2022–CTGAN, the reported ADR = 0.456 and B-FPR = 0.375 imply approximately 0.5405, compared with 0.8076 in Table 3. These discrepancies far exceed rounding error and occur across multiple settings. Because detector rankings and transfer gaps are central to the paper’s claims, reconciling these results is necessary to assess its conclusions.

  2. The fidelity analysis supports a narrower conclusion than the paper’s central framing suggests. Figure 2 shows that generated data are almost perfectly distinguishable from real data on four of the five datasets. According to the paper’s own joint-separability measure, these settings therefore already exhibit substantial distributional mismatch. By contrast, the strongest fidelity cases (APA-DDoS with TabDiff and TabSyn) retain perfect balanced accuracy for D²Former, MLP, and TabTransformer in Table 3. Thus, the evidence more clearly establishes the limitations of marginal and pairwise summary measures than it establishes failure despite high fidelity across the evaluated criteria. There is also an unresolved class-prior issue: Appendix A.2 specifies class-balanced synthetic evaluation sets, whereas the real datasets are imbalanced, in some cases severely. The paper does not clearly state whether fidelity is evaluated conditionally on class or after matching class proportions. Without this control, the measured differences may partly reflect the sampling protocol rather than generator-induced distortion of task-relevant relationships. This limits the explanatory value of the main empirical claim.

Questions for authors
  1. Could the authors reconcile Tables 3, 10, and 11 using the corresponding confusion matrices? Were all metrics computed from identical checkpoints, samples, and predictions, as stated in Appendix A.8? Please indicate which results are correct and whether any corrections would alter the reported detector rankings or transfer conclusions.

  2. Were real and synthetic class proportions matched when computing the fidelity metrics? If not, do the conclusions hold under class-conditional or prior-matched evaluation? In particular, which settings exhibit material transfer degradation despite strong fidelity across the evaluated criteria?

3 comments
AuthorsOpenReview ↗

Response to Reviewer uXx5

Thank you for catching these inconsistencies. You are right that every metric must come from the same predictions, and we have rebuilt the affected tables.

W1 (metric inconsistency). You are correct. In the submission, Tables 9–11 were built from older evaluation exports, while Table 3 came from the final outputs. All 150 balanced-accuracy entries in Table 3 matched the saved confusion matrices, but 5 of 150 entries in Table 9, 25 in Table 10 and 19 in Table 11 did not. In the revision, one script produces Tables 3, 4 and 9–11 directly from the saved confusion matrices. For every binary-dataset entry,

BA=ADR+1−B-FPR2\mathrm{BA}=\frac{\mathrm{ADR}+1-\mathrm{B\text{-}FPR}}{2}

holds to within 5×10−55\times10^{-5}, and the script checks this automatically.

W2 (fidelity supports a narrower conclusion). We agree. The strongest evidence for the original claim was the APA TabDiff/TabSyn pair with discriminator AUROC near 0.55. Those values came from a bug in our generator wrappers: TabDiff and TabSyn copied some feature values from real rows (details in our reply to Reviewer vWfZ). After the fix, their AUROC is 0.82 and 0.75 (TabDDPM 0.77), and Section 5.1 no longer uses them as a high-fidelity counterexample. The claim is now narrower. Discriminator AUROC saturates on four of five datasets and cannot rank generators, and which fidelity metric tracks detector accuracy depends on the dataset (Q2). Figures 2 and 3 are redrawn from the regenerated data. Figure 3 now makes a different point: both pairs overlap closely in PCA, yet the discriminator separates them with AUROC 0.82 and 1.00, so visual overlap says little about fidelity.

Q1 (reconciling the tables). Did the tables use the same predictions? In the submission they did not, since Tables 9–11 came from older exports. Now every metric comes from the same saved predictions. Which values are correct? The balanced accuracies in Table 3. For SDN–CTGAN, D²Former has ADR 0.883 and B-FPR 0.044, which gives BA 0.9196 (seed 42, the seed used in Appendix Tables 9–11; Table 3 now shows three-seed means). The DistilBERT ADR of 0.456 on CRCDDoS2022–CTGAN came from the old export. After retraining with a corrected tokenizer, that entry has ADR 0.998 and B-FPR 0.892, so BA 0.553, as in Table 3. Do the corrections change the conclusions? Fixing Tables 9–11 changes no ranking or transfer gap, because those are based on balanced accuracy, which was already correct. The generator fix changed some cells of Table 3, but the main findings hold: rankings still change across generators, D²Former and TabTransformer are still tied overall, and BCCC still shows the largest loss. Rerunning D²Former, MLP and TabTransformer on the unchanged parts (real test data, CTGAN, TabbyFlow) reproduces the submitted values to within 10−410^{-4}. The conclusion of Appendix A.8 also holds with the corrected values. On CICDDoS2019, TinySEED has a mean TRTS ADR of 0.991 but a B-FPR of 0.242. On CRCDDoS2022, DistilBERT and TinySEED reach ADR ≥ 0.992 with B-FPR of 0.942 and 0.797. A detector can look good on ADR and still be unusable because of false alarms, which is why balanced accuracy is our main metric.

Q2 (class proportions; degradation despite strong fidelity). The synthetic evaluation sets were not class-balanced. They follow the class proportions of the real evaluation data, with the same per-class budget for every generator, and the paper now says so. For this response we also computed class-conditional and prior-matched fidelity. Neither removes the saturation: on SDN, CICDDoS2019, CRCDDoS2022 and BCCC, the discriminator AUROC is at least 0.995 within every class and at least 0.996 after matching class proportions. Class imbalance therefore does not explain the near-perfect separability. As for degradation despite strong fidelity, after the fix there is no such case. The only dataset with fairly high fidelity is APA-DDoS (AUROC 0.75–0.99), and no tabular detector degrades there, so we dropped this claim. A new sweep over 13 training budgets per generator (Appendix A.9) shows that the relation differs by dataset. On SDN all metrics correlate strongly with transfer (|ρ| ≥ 0.69), on CICDDoS2019 only Wasserstein-1 does, and on BCCC better PCD goes with higher accuracy while better Wasserstein-1 goes with lower accuracy. No single fidelity metric is a reliable proxy for transfer, and the Conclusion recommends keeping held-out real data as the basis of detector evaluation.

Thank you for the clarification. The response fails to provide sufficient grounds for me to increase my score. It acknowledges the existence of substantive errors that affected data generation, detector implementation, and the reporting of metrics. While the content of the tables is consistent with the saved confusion matrices, this does not validate the entire pipeline used to generate these predictions. More importantly, the previous claim that "higher fidelity leads to a decline in performance" has been explicitly retracted following corrections to the generator; this fundamentally undermines the basis of the paper’s central argument and its contributions. Although narrowing the scope of the claims is reasonable, the remaining findings must still be evaluated on their own merits; simply noting that some rankings and general trends remain unchanged is insufficient. Given the evidence provided in the response, my core concerns regarding the reliability of the experiments and the strength of the research contributions persist. Consequently, I maintain my original score.

AuthorsOpenReview ↗

Response to Reviewer uXx5

We thank the reviewer for the follow-up and respect the decision on the score. For the record, we briefly address the two concerns raised.

Reliability of the experimental pipeline. Consistency with the saved confusion matrices was one of several checks. In addition: (i) rerunning the detectors on the components unaffected by the corrections (real test data, CTGAN, TabbyFlow) reproduces the submitted values to within 10−410^{-4}; (ii) all generators were audited for memorization using exact-match rates and distance to the closest training record; (iii) all tabular detectors were evaluated over three seeds; and (iv) generated labels were validated against a detector trained exclusively on real data. The generator and tokenizer defects were identified through this audit. We acknowledge that no audit can fully guarantee the absence of errors, but we believe these checks provide a substantially stronger basis than the original submission.

Central argument of the paper. The example in which higher fidelity coincided with lower detector accuracy has been withdrawn, as it resulted from the generator defect. The paper's central question, however, is whether fidelity checks can certify a generated test set for detector evaluation. The corrected results continue to answer this question in the negative:

  • On BCCC, discriminator AUROC is 1.000 for every generator, while D²Former's TRTS balanced accuracy ranges from 0.553 to 0.849.
  • Across generator training budgets, the fidelity metric that tracks transfer differs between datasets; on BCCC, PCD and Wasserstein-1 indicate opposite directions.
  • A detector trained on real data agrees with only 11% of TabSyn's BCCC attack labels, which no fidelity metric reveals.
  • Detector rankings change across generators by more than the seed-level standard deviations.
  • Provenance filtering either rejects real benign traffic or misses most attacks from unseen generators.

We agree that these findings should be assessed on their own merits, and the revised manuscript presents them in this way.

Reviewer_RhWoOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies whether using synthetic data is a reliable approach for developing and evaluating network anomaly detectors. The authors identify a generator-induced distribution shift, in which a classifier that performs well on real data might perform poorly on synthetic data, and argue that distributional fidelity does not predict detector performance and that provenance filters do not reliably catch samples from unseen generators. The authors then introduce D$^2$Former, a transformer-based detector for tabular network traffic that achieves competitive transfer performance under generator-induced distribution shifts.

Critical strengths
  • The paper is well written and easy to follow.
  • The problem is important for practitioners, but also interesting from a scientific perspective. From a practical perspective, building infrastructure monitoring and intrusion detection systems that use machine learning is a non-trivial task, with the main difficulties usually being data collection, labeling, and evaluation. From a scientific standpoint, studying distribution shifts and how models behave under them is interesting, and robustness to such shifts has historically been a weakness of ML methods, especially deep neural networks.
  • The empirical evaluation is broad, with five DDoS datasets and five generative models trained to produce synthetic data. The number of baselines used when benchmarking D^2Former is reasonable, and the hyperparameters of D$^2$Former are reported in the appendix (Table 7).
  • The ranking changes observed in Subsec. 5.3 are good concrete evidence and a cautionary tale for anomaly detection evaluation under synthetic data.
Critical weaknesses
  • I believe that the paper could use better formatting and arrangement. The biggest offenders are Tables 3 and 4 on page 9, which appear within the conclusion; I would recommend that the authors move these tables closer to Subsec. 5.3, where they are discussed.
  • Some of these datasets seem saturated. On APA-DDoS, D$^2$Former, the MLP, and TabTransformer reach a balanced accuracy of 1 on both the real and all generated test sets, and SDN and CRCDDoS2022 are also close to saturation on real data.
  • D$^2$Former is somewhat of a weird addition to the paper. I would suggest that the authors emphasize the model less. In my opinion, the work reads better as an analysis paper than as a methods paper, and the D$^2$Former discussions are somewhat distracting. Nevertheless, if the authors want to make D$^2$Former a more significant contribution, some further analysis should be performed on it. Its gains are relatively small, and it is unclear whether they come from hyperparameter tuning, the quantile tokenization, the statistical aggregation, or some other component of the model. More ablations are needed for the model to be a strong contribution, as well as comparisons to gradient-boosted trees and FT-Transformer, which D$^2$Former closely builds on. My current recommendation is to temper the claims regarding D$^2$Former.
  • Most of the reported results come from a single seed. Table 3, and with it the per-dataset claims in Subsec. 5.3 (e.g., D$^2$Former being best or tied for best on 14 of the 25 combinations), uses only seed 42 for D$^2$Former, the MLP, and TabTransformer, and a single run for DistilBERT and TinySEED. Table 4 reports three seeds, but only as aggregates over datasets and only for three detectors. This matters for the conclusions: the TRTR differences behind some of the ranking changes are small relative to the seed-to-seed variation in Table 4. Multi-seed per-dataset results might therefore weaken D$^2$Former further, which adds to my concern about its position in the paper.
  • Training on synthetic data and testing on real data (TSTR) is discussed in Appendix A.6, but the discussion and the experiments are limited (a single Random Forest configuration). This would also be an interesting contribution if expanded upon.

Overall, I think the paper presents a nice empirical finding and a good discussion that is especially useful to practitioners. The paper has several limitations: D$^2$Former is in a somewhat weird position, some datasets seem to be saturated, and the paper would become significantly more interesting if training on synthetic data and testing on real data were also considered. My main concern, however, is that most of the results are reported for a single seed. I lean towards acceptance, but my recommendation is conditional on the authors reporting multi-seed results for Tables 3 and 4. I will not lower my score if the results make D$^2$Former look worse, but the claims should be tempered if that is the case. I might lower my score if the authors do not provide results on multiple seeds.

Questions for authors
  • Sec. 4.1 and Appendix A.2 state that the generators are "fitted on the complete source datasets". Could the authors clarify what this means? Does it include the real validation and test sets, i.e., the same records that the detectors are trained and evaluated on? If so, the generators may partially memorize training records, which could make TRTS easier if the synthetic test samples are similar to the training dataset, or harder if they are more similar to the validation/test sets.
  • Could the authors further isolate the performance gains of D$^2$Former? Ablations of its components evaluated under TRTS would be interesting.
  • Were the hyperparameters of the baselines (MLP, TabTransformer, DistilBERT, TinySEED) searched, and if so, with which search space and budget compared to D$^2$Former? Their configurations are not reported.
  • Could the authors report Table 3 as mean ±\pm std over multiple seeds for all detectors?
  • Did the authors consider a provenance filter trained on both benign and attack traffic? Since the current filter only sees attacks during training, the high rejection rates on real benign flows (Tables 1 and 2) seem expected by construction.
2 comments
AuthorsOpenReview ↗

Response to Reviewer RhWo

Thank you for the constructive review. We ran the multi-seed experiments and audits you asked for (Section 5.3, Appendix A.9). Answers follow the order of your review.

W1 (table placement). Tables 3 and 4 are now in Section 5.3.

W2 (saturated datasets). Appendix A.9 now says that APA-DDoS is saturated and mainly serves as a sanity check, and that SDN and CRCDDoS2022 leave little room for differences.

W3 (role of D²Former). We agree and have toned down the claims. D²Former is now presented as a detector built around the failure mode the study identifies, not as the best detector on every dataset. We compare it with a tuned GBDT (XGBoost) and with FT-Transformer+PLE, and ablate each component (Q2). It uses no hyperparameter search (Q3).

W4 (single seed). All tabular detectors now have three seeds. Mean TRTS balanced accuracy (bold: best mean and detectors not significantly below it, Welch p ≥ 0.05):

DatasetD²FormerMLPTabTransformerXGBoost
APA1.000±.0001.000±.0001.000±.0001.000±.000
SDN0.934±.0040.933±.0010.950±.0040.930±.004
CICDDoS20190.808±.0360.811±.0510.817±.0220.722±.001
CRCDDoS20220.978±.0100.936±.0170.992±.0040.990±.000
BCCC0.707±.0110.383±.0660.682±.0180.654±.015
Overall0.886±.0070.813±.0200.888±.0040.859±.003

The generator fix barely changed D²Former: 0.886 overall and 0.707 on BCCC, against 0.886 and 0.698 in the submission. We replaced the single-seed claims ("best overall", "14/25") with three-seed means and significance tests. D²Former has the highest mean on BCCC, where the shift is largest, and is significantly better than XGBoost there (p≈0.009); its lead over TabTransformer (p≈0.12) would need more seeds to confirm. Elsewhere it is statistically tied with the best detector, except on SDN (TabTransformer +0.016). Overall it is significantly above XGBoost and the MLP (p≈0.015). The ranking changes in Section 5.3 hold with three seeds and exceed the standard deviations. On BCCC, XGBoost is best on real data (0.985±0.000) but third on generated data, and the MLP matches D²Former on real data (0.978) but drops to 0.383.

W5 (TSTR). We focus on real-trained detectors because that isolates whether generated data can serve as test data. Appendix A.6 uses TSTR only as a sanity check on the generators, and Appendix A.9 notes that one Random Forest per pair is not a full study of training on synthetic data.

Q1 (generator fitting). Yes. The generators used the full source data, including records in the detector splits, with at most 30,000 rows per class on CICDDoS2019; the paper now says so. We also found that our TabDiff and TabSyn wrappers copied low-cardinality features from real rows, so 34% of their APA rows matched a training row. After the fix it is 8.6–11.0% for these two and 0.7–11% across all generators, below the 17.5% of real APA test rows that also appear in training. On the other datasets no generated row matches. On every dataset, generated rows are also further from their closest training record than real test rows are. All results use the regenerated data.

Q2 (isolating the gains). Three-seed ablation (Table 12), BCCC/overall mean TRTS: base Transformer 0.340/0.814; FT-Transformer without per-sample normalization 0.702/0.889; QPL only (FT-Transformer+PLE) 0.694/0.882; D²Former 0.707/0.886. Most of the gain comes from avoiding per-sample normalization, and aggregation adds little. Section 5.3 explains why: generated BCCC flows often contain one extreme feature, and per-sample normalization then pushes all other features towards zero. Without test-time clipping, D²Former drops to 0.652 on BCCC.

Q3 (hyperparameters). XGBoost is chosen from an 8-configuration grid on real validation data. MLP and TabTransformer use fixed settings without search (Appendix A.3). D²Former uses one architecture for all datasets, also without search (Table 7); only the class weights and sampling caps are set per dataset (Eq. 15). To see whether this training recipe explains the BCCC result, we trained TabTransformer and the MLP with the same objective and sampling. They reach 0.657 and 0.481 on BCCC, so it does not.

Q4 (Table 3 as mean±std). Done for every dataset, generator and tabular detector. DistilBERT and TinySEED are single runs, marked as such. Their drops are large compared with seed noise: for example 0.993 to 0.618 on SDN, while the largest standard deviation of any tabular detector's mean TRTS is 0.066.

Q5 (benign-aware provenance). Tables 1–2 now include a filter trained on both benign and attack traffic. It no longer rejects real benign flows (from 10–11% on BCCC and up to 100% on APA down to ≤0.03%), but it still misses 83–98% of attacks from three unseen generators on APA and 59% of TabSyn's attacks on BCCC.

Thank you for your rebuttal. I believe that the paper is in a good shape and can be accepted. Reviewer uXx5 has raised some important inconsistency concerns, but it seems that they were fixed during the rebuttal. Therefore, I will maintain my positive score.

Reviewer_vWfZOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

The paper asks whether synthetic tabular data can stand in for real data when evaluating network attack detectors. Five flow-level DDoS datasets are used, and five tabular generators (CTGAN, TabDDPM, TabDiff, TabSyn, TabbyFlow) are fitted to each one. Every detector is trained only on real data and then tested, unchanged, on class-conditional synthetic samples (TRTS). Three analyses are reported:

  • Distributional fidelity: normalized W1W_1, Spearman correlation difference, and the AUROC of an XGBoost real-vs-synthetic discriminator.
  • Classifier transfer: TRTR vs. TRTS balanced accuracy, and how detector rankings change across generators.
  • Leave-one-generator-out provenance filtering: a real-vs-synthetic-attack classifier is tested on an unseen generator and on real benign flows.

The paper also proposes D²Former. It combines quantile piecewise-linear (QPL) feature embeddings, a Transformer encoder over feature tokens, and a fusion of the CLS token with mean/max/std pooling of the feature tokens. The main findings: fidelity does not predict TRTS performance, rankings shift across generators, and provenance filters either miss unseen-generator attacks or reject real benign traffic. D²Former matches TabTransformer overall across three seeds (0.8859 vs. 0.8860 mean TRTS BA) and does somewhat better on BCCC. The authors conclude that synthetic data should supplement, not replace, real held-out data.

Critical strengths
  1. The question is practical and the protocol is clean in one important respect. Many papers use synthetic traffic for benchmarking without checking whether it preserves what detectors actually rely on. Asking this directly is useful. Here the source data, feature schema and labels are fixed and only the generator varies, so the confounders of cross-dataset evaluation are avoided. The authors are also careful that detectors, preprocessing and thresholds are frozen before any synthetic data is seen.
  2. The paper is honest about its own method. The authors report the three-seed result showing D²Former and TabTransformer are tied, and they state clearly in the abstract and conclusion that neither is consistently best. They do not oversell the single-seed numbers. I appreciate this, and it makes the empirical part easier to trust.
Critical weaknesses
  1. TRTS accuracy is not a well-defined measure of detector quality, and the data do not support the headline claim.
    • For a class-conditional generator, the label of a synthetic sample is just the conditioning input. Nothing checks that the sample is a valid benign flow or attack of that class. A low TRTS score may therefore mean the detector is fragile, or simply that the generator produced mislabeled or implausible records. The paper cannot distinguish the two. For the same reason, "transfers better to generated data" is not obviously a desirable property for a detector, so it is unclear what D²Former's BCCC gain shows.
    • The claim that fidelity "does not reliably predict" detection performance is also not well supported by the data. The only dataset where fidelity varies meaningfully is APA-DDoS (AUROC 0.548–0.999). There, the three reasonable detectors score 1.000 on every generator, so TRTS cannot vary. On the other four datasets the discriminator AUROC is close to 1.0 for all generators, so fidelity is uniformly poor and has no variance to correlate with anything. "High fidelity, yet detectors behave differently" never actually occurs in the results. What the results show is closer to "when fidelity is poor, TRTS is unpredictable".
    • No correlation analysis across the 25 combinations is reported.
    • Generators are fitted on the complete datasets, including the real test partition. Memorization (e.g. distance to closest record) is not checked, which matters for the APA-DDoS 1.000 results in particular.
  2. The method contribution is thin and the comparison is weak.
    • D²Former is essentially FT-Transformer (Gorishniy et al., 2021) with the quantile PLE embeddings of Gorishniy et al. (2022), plus mean/max/std pooling. FT-Transformer with PLE embeddings is therefore the most direct baseline, yet it is missing. There is no ablation of the three design choices, and no analysis of why quantile tokenization should help under generator shift. Clipping out-of-range values to [0,1] may be doing most of the work, but this is not tested.
    • GBDTs (XGBoost/LightGBM/RF) are absent from the main comparison. They are the standard strong baseline for tabular classification, and the paper itself already uses RF (TSTR) and XGBoost (as the fidelity discriminator).
    • Two of the four baselines look undertrained. DistilBERT and TinySEED reach a TRTR balanced accuracy of exactly 0.6667 on APA-DDoS, which other models solve perfectly, and TinySEED's benign FPR there jumps to 0.985 on TabDDPM data. They are evaluated with a single run.
    • With the remaining baseline (TabTransformer) tied across seeds, the method does not add much beyond the empirical study.
    • The title says "anomaly detection", but every model is a supervised multi-class classifier trained on labeled attack classes. That framing should be corrected.
Questions for authors
  1. Validity of synthetic labels. How do you verify that a synthetic sample conditioned on a class actually belongs to that class? For example, you could apply the same validity constraints used in the provenance experiment to the TRTS test sets, or use domain rules or an oracle classifier. Without this, how should a TRTS drop be read: as a detector failure or as a generator failure?
  2. Fidelity vs. transfer. Can you report rank correlations between each fidelity metric and the TRTS gap Δg\Delta_g across the 25 combinations, for each detector? Also, is there any combination where fidelity is high (e.g. AUROC < 0.7) and TRTS varies substantially across detectors or generators? That is the case the headline claim needs.
  3. Missing baselines. How do FT-Transformer with PLE embeddings and a tuned GBDT perform under the same TRTR/TRTS protocol? Please also add an ablation that removes QPL, the pooling, or the clipping. And could you explain why DistilBERT and TinySEED get exactly 0.6667 TRTR on APA-DDoS?
  4. Provenance setup. Which model is the provenance classifier? For TabDiff and TabSyn on APA-DDoS the AUROC is exactly 0.500 with identical catch and RA rates, which looks like a constant-score classifier. Also, the filter is trained only on real attacks vs. synthetic attacks, so rejecting unseen real benign flows is the expected out-of-distribution behavior. Did you try a filter that also sees real benign data during training?
1 comment
AuthorsOpenReview ↗

Response to Reviewer vWfZ

Thank you for the detailed review. Several of your points exposed real problems in our pipeline; we fixed them and reran all experiments. W/Q numbers follow the order of your review.

W1.1 (what TRTS measures). We agree that a higher TRTS score does not by itself mean a better detector, since a generated label can be wrong. We now assess label consistency using a real-trained XGBoost proxy. We explain D²Former's BCCC gain by its mechanism (W2.1): robustness to extreme out-of-range values.

W1.2 (fidelity claim). After the generator fix, no pair has AUROC below 0.7 (lowest 0.747), and on APA all tabular detectors reach about 1.000 whatever the AUROC. The abstract and Section 5.1 now claim only that the common fidelity check saturates and that the metric that tracks accuracy differs by dataset.

W1.3 (correlation analysis). Added to Appendix A.9 (see Q2).

W1.4 (fitting data and memorization). The generators were fitted on the full source data (at most 30,000 rows per class on CIC), as the paper now states. We also found that our TabDiff and TabSyn wrappers copied low-cardinality features from real rows, so 34% of their APA rows matched a training row. After the fix (Appendix A.2) it is 8.6–11.0% for these two and 0.7–11% across all generators, less than the 17.5% of real APA test rows that appear in training, and zero on the other datasets. Generated rows are also further from their nearest training record than real test rows, on every dataset. Memorization therefore does not explain the APA results.

W2.1 (novelty and ablations). We added the FT-Transformer+PLE comparison and ablated tokenization, pooling and clipping (Table 12; see Q3). Quantile tokenization helps because it avoids per-sample normalization. Generated BCCC flows often contain one extreme feature, which under per-sample normalization pushes all others towards zero. Clipping helps, but less.

W2.2 (GBDTs). All tables now include a tuned XGBoost (8-configuration grid). Over three seeds it reaches 0.859±0.003 overall and 0.654±0.015 on BCCC, and D²Former is significantly better on BCCC (0.707±0.011, p≈0.009). We did not add LightGBM or Random Forest.

W2.3 (BERT baselines). The 0.6667 came from a tokenizer bug (Q3). Both models were retrained and are still single runs.

W2.4 (role of D²Former). We agree that the paper is mainly an empirical study. We now present D²Former as a detector built for the failure mode we identify. It ties with TabTransformer overall and has the highest mean on BCCC, where the shift is largest, and the ablation shows why. We no longer claim it is best overall.

W2.5 (title). The text now says supervised attack detection, not anomaly detection; the title will follow once changes are allowed.

Q1 (label validity). Of the options you list, we used an oracle: an XGBoost detector trained only on real data (0.985 on real BCCC) relabels every generated sample. It agrees with the generator on at least 81% of samples on four datasets, but on only 48–80% on BCCC and 11% of TabSyn's BCCC attacks. So both happen: part of the BCCC loss comes from the generator, but detectors tested on the same data still differ widely (MLP 0.383, D²Former 0.707).

Q2 (rank correlations). Across the 25 pairs, worse fidelity goes with larger transfer loss for every metric and detector (Spearman ρ 0.37–0.71), mostly because easy datasets have both high fidelity and small losses. Within a dataset (13 training budgets per generator), it varies. On SDN all metrics track TRTS (|ρ| ≥ 0.69). On CIC only W1 does. On BCCC, better PCD goes with higher accuracy but better W1 with lower accuracy, and AUROC is 1.000 for every version.

Q3 (comparisons, ablations, 0.6667). BCCC/overall mean TRTS (Table 12): base Transformer 0.340/0.814; FT-Transformer without per-sample normalization 0.702/0.889; QPL only (FT-Transformer+PLE) 0.694/0.882; D²Former 0.707/0.886; D²Former without clipping 0.652/0.874. Pooling adds little. The 0.6667 came from our tokenizer: it mapped two-valued features to one token and hid tcp.flags.push, which separates the two APA attack types. After retraining, both reach 1.000 on real APA data, DistilBERT's mean TRTS on APA rises from 0.648 to 0.845, and TinySEED's benign FPR on APA–TabDDPM falls from 0.985 to 0.000.

Q4 (provenance). The filter is XGBoost with 200 trees, depth 5 and learning rate 0.1. The identical 0.500 values were not from a constant-score classifier: both generators had copied the same real rows, so the two rounds saw almost the same inputs. With the fixed generators the values differ (AUROC 0.495 vs 0.589, RA 0.07% vs 0.02%). Rejecting real benign flows is indeed expected for an attack-only filter, but then it cannot be used on mixed traffic. A filter that also sees real benign traffic (now in Tables 1–2) rarely rejects benign flows (≤0.03%) but still misses 83–98% of attacks from three unseen generators on APA and 59% of TabSyn's attacks on BCCC.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response: summary of changes

We thank all three reviewers. Your comments made us audit the whole pipeline, and the audit found real errors. We fixed them, reran every experiment, and added the experiments you asked for. This note summarizes the changes; the individual replies give the details.

Corrections

  • Generator bug. Our TabDiff and TabSyn wrappers copied low-cardinality feature values from the nearest real row, so on APA 34% of their rows matched a training row. Every feature is now generated, all affected test sets were regenerated, and all tables and figures use the new data. Generated rows now match training rows less often than real test rows do.
  • Metric consistency. One script now builds Tables 3, 4 and 9–11 from the saved confusion matrices. This removes the mismatches between balanced accuracy, ADR and B-FPR that Reviewer uXx5 found.
  • BERT tokenizer. A bug hid two-valued features from DistilBERT and TinySEED, including the flag that separates the APA attack types. Both models were retrained.
  • Method details. The paper now states the actual training objective (Eq. 15, Table 7), the CICDDoS2019 fitting cap, how the evaluation subsets are sampled, the baseline settings, and that four generators are our re-implementations.

New experiments

  • Three seeds for all tabular detectors, plus a tuned XGBoost baseline. Tables 3 and 4 now report mean±std.
  • A provenance filter trained with real benign traffic, next to the original attack-only filter (Tables 1–2).
  • A check of generated labels with a detector trained only on real data (Section 5.3, Appendix A.9).
  • A fidelity sweep over 13 training budgets per generator, and class-conditional and prior-matched fidelity (Appendix A.9).
  • A three-seed ablation of D²Former's components, including test-time clipping, and an analysis of why per-sample normalization fails on generated data (Table 12, Section 5.3).

Revised claims

  • We no longer claim that fidelity does not predict detector performance. The claim is now narrower: the most common fidelity check saturates (discriminator AUROC ≥ 0.996 on four of five datasets), and which fidelity metric tracks accuracy differs between datasets.
  • D²Former's numbers hardly changed after the fix (three-seed mean 0.886 overall and 0.707 on BCCC, against 0.886 and 0.698 in the submission). Instead of single-seed rankings we now report significance tests. D²Former has the highest mean on BCCC, where the shift is largest (significantly above XGBoost, p≈0.009), and is statistically tied with the best detector on every other dataset except SDN, where TabTransformer is 0.016 higher. Overall it ties with TabTransformer and is significantly above XGBoost and the MLP. The ablation shows where this comes from: quantile tokenization, which avoids per-sample normalization and clips out-of-range values.
  • Provenance filtering fails in two ways. A filter trained only on attacks rejects real benign traffic, and a filter that also sees benign traffic still misses most attacks from some unseen generators.
  • Section 5.3 now explains why detectors lose accuracy on generated data. Some generated samples do not behave like their label, and some contain extreme feature values. Fidelity metrics show neither.
  • The Conclusion now gives concrete recommendations for evaluating detectors with generated traffic.

What stays the same. The main conclusion holds and is better supported. Generated traffic is useful for stress-testing detectors, but it cannot replace held-out real data for evaluating them. Rankings still change across generators, no fidelity metric consistently identifies faithful test sets, and provenance filtering is no substitute for behavioural detection.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.