- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether a synthetic release of private tabular data keeps the same ranking of detection algorithms you'd get from train/tune/eval on the real data. "Leaderboard fidelity" is measured with Kendall's τ_b, preservation of statistically separable detector pairs and model-selection regret, and bootstrap of the private test period gives an empirical noise scale. Main study: 94,901 interbank transfers w/ bank-assigned suspicion labels, 60 releases from 12 generators, 11 detectors under one tuning protocol. Main findings are that releases of the same generator vary a lot, and the conventional quality criteria they tested don't predict fidelity very well. They also propose a data-holder audit (pick release by measured rank agreement), look at duplicate leakage, and do a public-data replication plus robustness checks (clustered resampling, temporal partitions, held-out detectors).
Critical strengths
-
The target is relevant to how synthetic benchmarks are actually used. A release can give good predictive accuracy and still not support the same model-selection decisions as the private benchmark, and looking at full rankings + separable pairs + selection regret makes this concrete. The bootstrap reference scale makes sense since the private leaderboard is itself noisy. I liked that releases reaching the calibrated band have at most 0.006 private PR-AUC regret, this ties the ranking analysis to an actual practical consequence instead of just a correlation number (Sec 4.3-5.2).
-
Empirical study is substantial and quite transparent. Five releases per generator makes the between-release variation visible, temporal construction and past-only features address real validity concerns for financial data. The account/week resampling, the generation-independent check (App. F.2), held-out detector analysis and public replication all help. Also appreciate that the authors report where things get weaker (low power of within-generator tests, weaker transfer to fully unused detectors). Overall a useful empirical contribution even if the methodological novelty is somewhat modest.
Critical weaknesses
-
The comparison with conventional criteria doesn't really isolate why they fail. Sec 4.2 says train/val/test of each release come from generators fitted separately on the corresponding private splits, but App. B computes the distributional criteria only on the training split. The S→S leaderboard depends on all three synthetic parts, incl. the separately generated val and test data. So if synthetic test quality degrades it can hurt leaderboard fidelity w/o showing up in training-split criteria, and TSTR skips the synthetic test part too. So I think the current results show a limitation of the training-split proxies specifically, less so of distributional screening of a whole release. Since "select via direct audit" is a central practical conclusion, a comparison with criteria computed on val/test (or aggregated over all components) would make it much stronger.
-
The evidence supports auditing a fixed detector panel more than picking a benchmark that stays reliable for new methods. The detector-holdout analysis is nice, but App. F.3 gives only 28% recovery of the available gain when restricted to pairs among the five unused detectors, vs the much stronger fixed-panel number in Sec 5.4. Also most of the repeated private-test half-split checks use synthetic test data generated from the whole private test period; F.2 removes this overlap but only for one split. So the audit looks useful conditionally, but there's more uncertainty for future model development / truly unseen eval data. Since supporting later model comparisons is kind of the main purpose of a public benchmark, this somewhat limits the "audit and publish" recommendation.
Questions for authors
-
Could you redo the within-generator selection analysis with KS, TVD, dependence error, C2ST and label-rate error computed on the val/test components (or a clearly defined aggregate over all 3), keeping the same audit/held-out separation as for direct auditing? Does the advantage of direct auditing still hold when the screening criteria actually see the parts that determine the public leaderboard?
-
For the F.3 setting (only unused detectors), can you report absolute agreement and selection regret of the audit-selected release, next to random selection and leaderboard averaging? The 28% number is informative but these would show whether the remaining ranking errors actually change which new model gets picked. Also, is there any evidence beyond the single generation-independent split in F.2?
-
Do the audit-vs-averaging and cross-release comparisons in Sec 5.4 look similar if the four collapsed generators are excluded? TVAE's τ is described as uninterpretable, but the headline audit recovery averages over all 12 generators. Reporting it on the usable releases only would clarify how much advantage is left among realistic candidate benchmarks.
5 comments
(W1 & Q1, continued) Swapping splits between releases
Within each generator, detectors trained on the training and validation splits of seed i are scored on the test split of seed j. Swapping test splits between the releases of a generator (Appendix F.7) shows that, pooled over generators, the test split accounts for more of the within-generator variation in tau than the other two splits.
| Share of the within-generator sum of squares | test split | training and validation splits | interaction |
|---|---|---|---|
| all twelve generators | 29% | 20% | 52% |
| without the collapsed generators | 40% | 24% | 36% |
We ran every cell a second time with new detector seeds. Run-to-run noise accounts for 11% of the variation (9% without the collapsed generators), and 86% of the interaction remains once it is removed (83%). The interaction is therefore real. How much a test split changes tau depends on the training and validation splits it is paired with. More than half of the pooled test-split share comes from GReaT and TabDiff, whose tau over the 25 combinations varies most among the eight generators that do not collapse. In five of those eight the training and validation splits matter more, and training-split criteria miss that part as well. The published combinations of splits are no better than swapped ones (an average tau of 0.463, compared with 0.455). Among those eight, the largest of the three terms differs by generator.
- Test split: TabDiff, GReaT and TabDDPM.
- Training and validation splits: FinDiff, TabPFGen and its prior-matched variant.
- Interaction: SMOTE and TabSyn, with the training and validation splits ahead of the test split.
The test-split share supports your concern about the synthetic test split, and the training and validation splits matter as well. The split a metric sees changes what it can detect, but as selection tools on the audited detectors the metrics remain far from a direct audit. With all twelve generators, the audit recovers about twice what averaging or the best criterion does. Still, without private data a public user can get about half of the audit's gain, and more once the collapsed generators are set aside, by averaging the releases or by choosing on label-rate error.
As you note, TSTR never sees the synthetic test split. We kept its standard definition and say so in Appendix F.6. The abstract, introduction and contributions now state the training-split result and the every-split result separately.
(W2 & Q2) Unused detectors, and evidence beyond the single generation-independent split
We agree with W2 on detectors outside the audited panel, and ran the checks your Q2 asks about. On 20 generation-disjoint splits, removing the overlap does not measurably lower the audit's share of the gap. On held-out detector families the audit is not consistently ahead of averaging, and inside the top tier, where there is little to choose, no benefit remains.
We extended the generation-disjoint check of Appendix F.2 from the first of the 200 half-splits to the first 20. Each split requires retraining all 55 GPU releases, so we could not repeat it on all 200. We first ran three splits. Before generating the other 17, we fixed the number of splits and the comparison, and we report all 20. Averaged over the 20 splits, selection on A recovers 91.3% of the gap on B without the overlap and 91.5% with the original test splits. Split by split, removing the overlap changes the share by -0.15 points on average (95% t-interval -3.9 to +3.6). The overlap therefore does not measurably inflate the audit's share of the gap, and the interval excludes an inflation of more than four points. The interval treats the splits as independent although their halves share transfers, so it may be somewhat too narrow. On single splits the change ranges from -13 to +19 points. On all 20 splits the share without the overlap stays above 77.6%, the 5th percentile over all 200 half-splits with the original test splits, and the lowest lies just above it.
Within the test period, selection also carries over in time. Chosen on July to September and scored on October to December, it recovers 90% of the gap, and 92% in the opposite direction (Appendix F.3). The synthetic test splits used there were generated from the whole period, and whether a release chosen today holds for a later period is untested.
Thank you for the careful review. We ran the analyses you asked for, and they led us to narrow two of our claims. Changes are blue in the revised PDF, which adds Appendices F.6 to F.8 and keeps the numbering of F.1 to F.5. The code and result files of the new analyses are in the anonymous repository. Comments 1 and 2 answer W1 and Q1, comments 3 and 4 answer W2 and Q2, and comment 4 also answers Q3.
Yes, on the detectors the audit uses. You are right that the submitted evidence covered only criteria computed on the training split, and our training-split analysis had missed what the other splits show. We computed the five criteria on each synthetic split against the private split it imitates, and on the mean of the three (Appendix F.6, Table 11). Throughout, the gap is the difference in held-out rank agreement between a random release and the best release of the same generator, averaged over generators. Half A and half B are the two halves of the private test period used to choose and to score (Section 5.4).
The table breaks the selection result down by criterion and split, as the share of the gap recovered on B (%). Test-split values are computed against half A only, and C2ST on the test split and its mean use the first 20 half-splits.