- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper examines whether synthetic tabular network traffic can serve as a reliable evaluation distribution for anomaly detectors trained on real data. It evaluates five generators across five DDoS datasets, assessing distributional fidelity, real-to-synthetic detector transfer, and cross-generator provenance filtering. The paper also introduces D²Former, which integrates quantile-based numerical embeddings, a Transformer encoder, and statistical aggregation. It argues that synthetic data should supplement, rather than replace, held-out real data in detector evaluation.
Critical strengths
Breadth and relevance of the empirical evaluation. The study addresses a practically relevant question: whether synthetic data preserve the evaluation outcomes of detectors trained on real traffic. By covering five datasets and five generators, while holding detector checkpoints and preprocessing fixed and sharing synthetic test sets across detectors, it provides a useful basis for examining generator-dependent behavior. The additional leave-one-generator-out provenance experiments offer a complementary perspective on the limitations of identifying synthetic origin.
Critical weaknesses
-
Internal inconsistency in the reported evaluation metrics undermines confidence in the central comparisons. The reported evaluation metrics are internally inconsistent, which undermines confidence in the central comparisons. Appendix A.8 states that balanced accuracy, attack detection rate (ADR), and benign false-positive rate (B-FPR) are computed from the same predictions and test sets. For the binary datasets SDN and CRCDDoS2022, these quantities must satisfy: For D²Former on SDN–CTGAN, however, Tables 10 and 11 report ADR = 0.965 and B-FPR = 0.044, which implies a balanced accuracy of approximately 0.9605, whereas Table 3 reports 0.9196. Similarly, for DistilBERT on CRCDDoS2022–CTGAN, the reported ADR = 0.456 and B-FPR = 0.375 imply approximately 0.5405, compared with 0.8076 in Table 3. These discrepancies far exceed rounding error and occur across multiple settings. Because detector rankings and transfer gaps are central to the paper’s claims, reconciling these results is necessary to assess its conclusions.
-
The fidelity analysis supports a narrower conclusion than the paper’s central framing suggests. Figure 2 shows that generated data are almost perfectly distinguishable from real data on four of the five datasets. According to the paper’s own joint-separability measure, these settings therefore already exhibit substantial distributional mismatch. By contrast, the strongest fidelity cases (APA-DDoS with TabDiff and TabSyn) retain perfect balanced accuracy for D²Former, MLP, and TabTransformer in Table 3. Thus, the evidence more clearly establishes the limitations of marginal and pairwise summary measures than it establishes failure despite high fidelity across the evaluated criteria. There is also an unresolved class-prior issue: Appendix A.2 specifies class-balanced synthetic evaluation sets, whereas the real datasets are imbalanced, in some cases severely. The paper does not clearly state whether fidelity is evaluated conditionally on class or after matching class proportions. Without this control, the measured differences may partly reflect the sampling protocol rather than generator-induced distortion of task-relevant relationships. This limits the explanatory value of the main empirical claim.
Questions for authors
-
Could the authors reconcile Tables 3, 10, and 11 using the corresponding confusion matrices? Were all metrics computed from identical checkpoints, samples, and predictions, as stated in Appendix A.8? Please indicate which results are correct and whether any corrections would alter the reported detector rankings or transfer conclusions.
-
Were real and synthetic class proportions matched when computing the fidelity metrics? If not, do the conclusions hold under class-conditional or prior-matched evaluation? In particular, which settings exhibit material transfer degradation despite strong fidelity across the evaluated criteria?
3 comments
Thank you for the clarification. The response fails to provide sufficient grounds for me to increase my score. It acknowledges the existence of substantive errors that affected data generation, detector implementation, and the reporting of metrics. While the content of the tables is consistent with the saved confusion matrices, this does not validate the entire pipeline used to generate these predictions. More importantly, the previous claim that "higher fidelity leads to a decline in performance" has been explicitly retracted following corrections to the generator; this fundamentally undermines the basis of the paper’s central argument and its contributions. Although narrowing the scope of the claims is reasonable, the remaining findings must still be evaluated on their own merits; simply noting that some rankings and general trends remain unchanged is insufficient. Given the evidence provided in the response, my core concerns regarding the reliability of the experiments and the strength of the research contributions persist. Consequently, I maintain my original score.
Response to Reviewer uXx5
We thank the reviewer for the follow-up and respect the decision on the score. For the record, we briefly address the two concerns raised.
Reliability of the experimental pipeline. Consistency with the saved confusion matrices was one of several checks. In addition: (i) rerunning the detectors on the components unaffected by the corrections (real test data, CTGAN, TabbyFlow) reproduces the submitted values to within ; (ii) all generators were audited for memorization using exact-match rates and distance to the closest training record; (iii) all tabular detectors were evaluated over three seeds; and (iv) generated labels were validated against a detector trained exclusively on real data. The generator and tokenizer defects were identified through this audit. We acknowledge that no audit can fully guarantee the absence of errors, but we believe these checks provide a substantially stronger basis than the original submission.
Central argument of the paper. The example in which higher fidelity coincided with lower detector accuracy has been withdrawn, as it resulted from the generator defect. The paper's central question, however, is whether fidelity checks can certify a generated test set for detector evaluation. The corrected results continue to answer this question in the negative:
- On BCCC, discriminator AUROC is 1.000 for every generator, while D²Former's TRTS balanced accuracy ranges from 0.553 to 0.849.
- Across generator training budgets, the fidelity metric that tracks transfer differs between datasets; on BCCC, PCD and Wasserstein-1 indicate opposite directions.
- A detector trained on real data agrees with only 11% of TabSyn's BCCC attack labels, which no fidelity metric reveals.
- Detector rankings change across generators by more than the seed-level standard deviations.
- Provenance filtering either rejects real benign traffic or misses most attacks from unseen generators.
We agree that these findings should be assessed on their own merits, and the revised manuscript presents them in this way.
Response to Reviewer uXx5
Thank you for catching these inconsistencies. You are right that every metric must come from the same predictions, and we have rebuilt the affected tables.
W1 (metric inconsistency). You are correct. In the submission, Tables 9–11 were built from older evaluation exports, while Table 3 came from the final outputs. All 150 balanced-accuracy entries in Table 3 matched the saved confusion matrices, but 5 of 150 entries in Table 9, 25 in Table 10 and 19 in Table 11 did not. In the revision, one script produces Tables 3, 4 and 9–11 directly from the saved confusion matrices. For every binary-dataset entry,
BA=2ADR+1−B-FPR
holds to within 5×10−5, and the script checks this automatically.
W2 (fidelity supports a narrower conclusion). We agree. The strongest evidence for the original claim was the APA TabDiff/TabSyn pair with discriminator AUROC near 0.55. Those values came from a bug in our generator wrappers: TabDiff and TabSyn copied some feature values from real rows (details in our reply to Reviewer vWfZ). After the fix, their AUROC is 0.82 and 0.75 (TabDDPM 0.77), and Section 5.1 no longer uses them as a high-fidelity counterexample. The claim is now narrower. Discriminator AUROC saturates on four of five datasets and cannot rank generators, and which fidelity metric tracks detector accuracy depends on the dataset (Q2). Figures 2 and 3 are redrawn from the regenerated data. Figure 3 now makes a different point: both pairs overlap closely in PCA, yet the discriminator separates them with AUROC 0.82 and 1.00, so visual overlap says little about fidelity.
Q1 (reconciling the tables). Did the tables use the same predictions? In the submission they did not, since Tables 9–11 came from older exports. Now every metric comes from the same saved predictions. Which values are correct? The balanced accuracies in Table 3. For SDN–CTGAN, D²Former has ADR 0.883 and B-FPR 0.044, which gives BA 0.9196 (seed 42, the seed used in Appendix Tables 9–11; Table 3 now shows three-seed means). The DistilBERT ADR of 0.456 on CRCDDoS2022–CTGAN came from the old export. After retraining with a corrected tokenizer, that entry has ADR 0.998 and B-FPR 0.892, so BA 0.553, as in Table 3. Do the corrections change the conclusions? Fixing Tables 9–11 changes no ranking or transfer gap, because those are based on balanced accuracy, which was already correct. The generator fix changed some cells of Table 3, but the main findings hold: rankings still change across generators, D²Former and TabTransformer are still tied overall, and BCCC still shows the largest loss. Rerunning D²Former, MLP and TabTransformer on the unchanged parts (real test data, CTGAN, TabbyFlow) reproduces the submitted values to within 10−4. The conclusion of Appendix A.8 also holds with the corrected values. On CICDDoS2019, TinySEED has a mean TRTS ADR of 0.991 but a B-FPR of 0.242. On CRCDDoS2022, DistilBERT and TinySEED reach ADR ≥ 0.992 with B-FPR of 0.942 and 0.797. A detector can look good on ADR and still be unusable because of false alarms, which is why balanced accuracy is our main metric.
Q2 (class proportions; degradation despite strong fidelity). The synthetic evaluation sets were not class-balanced. They follow the class proportions of the real evaluation data, with the same per-class budget for every generator, and the paper now says so. For this response we also computed class-conditional and prior-matched fidelity. Neither removes the saturation: on SDN, CICDDoS2019, CRCDDoS2022 and BCCC, the discriminator AUROC is at least 0.995 within every class and at least 0.996 after matching class proportions. Class imbalance therefore does not explain the near-perfect separability. As for degradation despite strong fidelity, after the fix there is no such case. The only dataset with fairly high fidelity is APA-DDoS (AUROC 0.75–0.99), and no tabular detector degrades there, so we dropped this claim. A new sweep over 13 training budgets per generator (Appendix A.9) shows that the relation differs by dataset. On SDN all metrics correlate strongly with transfer (|ρ| ≥ 0.69), on CICDDoS2019 only Wasserstein-1 does, and on BCCC better PCD goes with higher accuracy while better Wasserstein-1 goes with lower accuracy. No single fidelity metric is a reliable proxy for transfer, and the Conclusion recommends keeping held-out real data as the basis of detector evaluation.