acceptodds
Under review as a conference paper at ICLR 2027

Leaderboard Fidelity: Auditing Synthetic Releases of Private Financial Data

Abstract

Regulated financial data cannot be shared, so synthetic releases are offered as public stand-ins for private fraud-detection benchmarks. A release is used to choose a model, yet it is judged by distributional fidelity and train-on-synthetic accuracy, which do not say whether the models that win on it also win on the private data. We introduce leaderboard fidelity: the rank agreement between the detector leaderboard a release induces and the leaderboard on the private data, calibrated against what resampling the private test period alone produces. Using transfers from a national interbank clearing network, we build a leakage-free benchmark and evaluate sixty releases, five from each of twelve generators, against eleven detectors under equal budgets. Among generators that do not collapse, rank agreement varies more between releases of one generator than between generators, and the releases that reach the calibrated band come from five different generators. None of the distributional criteria we test, nor a train-on-synthetic score, picks the better release of a generator distinguishably better than chance. Measuring rank agreement against the private data does, and only the data holder can do so. The choice carries over to held-out halves and to the later months of the private test period, so a holder can audit several releases and publish the best with the value it scored. When duplicated records cross a random split, most detectors score near perfectly while their order agrees only weakly with the private one. We publish the sixty releases, the protocol, and the code.

Primary area:
datasets and benchmarks
Keywords:
synthetic data, tabular data, benchmark evaluation, leaderboard fidelity, model selection, rank agreement, fraud detection, financial data, privacy, data leakage

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_fQkvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether a synthetic release of private tabular data keeps the same ranking of detection algorithms you'd get from train/tune/eval on the real data. "Leaderboard fidelity" is measured with Kendall's τ_b, preservation of statistically separable detector pairs and model-selection regret, and bootstrap of the private test period gives an empirical noise scale. Main study: 94,901 interbank transfers w/ bank-assigned suspicion labels, 60 releases from 12 generators, 11 detectors under one tuning protocol. Main findings are that releases of the same generator vary a lot, and the conventional quality criteria they tested don't predict fidelity very well. They also propose a data-holder audit (pick release by measured rank agreement), look at duplicate leakage, and do a public-data replication plus robustness checks (clustered resampling, temporal partitions, held-out detectors).

Critical strengths
  1. The target is relevant to how synthetic benchmarks are actually used. A release can give good predictive accuracy and still not support the same model-selection decisions as the private benchmark, and looking at full rankings + separable pairs + selection regret makes this concrete. The bootstrap reference scale makes sense since the private leaderboard is itself noisy. I liked that releases reaching the calibrated band have at most 0.006 private PR-AUC regret, this ties the ranking analysis to an actual practical consequence instead of just a correlation number (Sec 4.3-5.2).

  2. Empirical study is substantial and quite transparent. Five releases per generator makes the between-release variation visible, temporal construction and past-only features address real validity concerns for financial data. The account/week resampling, the generation-independent check (App. F.2), held-out detector analysis and public replication all help. Also appreciate that the authors report where things get weaker (low power of within-generator tests, weaker transfer to fully unused detectors). Overall a useful empirical contribution even if the methodological novelty is somewhat modest.

Critical weaknesses
  1. The comparison with conventional criteria doesn't really isolate why they fail. Sec 4.2 says train/val/test of each release come from generators fitted separately on the corresponding private splits, but App. B computes the distributional criteria only on the training split. The S→S leaderboard depends on all three synthetic parts, incl. the separately generated val and test data. So if synthetic test quality degrades it can hurt leaderboard fidelity w/o showing up in training-split criteria, and TSTR skips the synthetic test part too. So I think the current results show a limitation of the training-split proxies specifically, less so of distributional screening of a whole release. Since "select via direct audit" is a central practical conclusion, a comparison with criteria computed on val/test (or aggregated over all components) would make it much stronger.

  2. The evidence supports auditing a fixed detector panel more than picking a benchmark that stays reliable for new methods. The detector-holdout analysis is nice, but App. F.3 gives only 28% recovery of the available gain when restricted to pairs among the five unused detectors, vs the much stronger fixed-panel number in Sec 5.4. Also most of the repeated private-test half-split checks use synthetic test data generated from the whole private test period; F.2 removes this overlap but only for one split. So the audit looks useful conditionally, but there's more uncertainty for future model development / truly unseen eval data. Since supporting later model comparisons is kind of the main purpose of a public benchmark, this somewhat limits the "audit and publish" recommendation.

Questions for authors
  1. Could you redo the within-generator selection analysis with KS, TVD, dependence error, C2ST and label-rate error computed on the val/test components (or a clearly defined aggregate over all 3), keeping the same audit/held-out separation as for direct auditing? Does the advantage of direct auditing still hold when the screening criteria actually see the parts that determine the public leaderboard?

  2. For the F.3 setting (only unused detectors), can you report absolute agreement and selection regret of the audit-selected release, next to random selection and leaderboard averaging? The 28% number is informative but these would show whether the remaining ranking errors actually change which new model gets picked. Also, is there any evidence beyond the single generation-independent split in F.2?

  3. Do the audit-vs-averaging and cross-release comparisons in Sec 5.4 look similar if the four collapsed generators are excluded? TVAE's τ is described as uninterpretable, but the headline audit recovery averages over all 12 generators. Reporting it on the usable releases only would clarify how much advantage is left among realistic candidate benchmarks.

5 comments

Thank you for the careful review. We ran the analyses you asked for, and they led us to narrow two of our claims. Changes are blue in the revised PDF, which adds Appendices F.6 to F.8 and keeps the numbering of F.1 to F.5. The code and result files of the new analyses are in the anonymous repository. Comments 1 and 2 answer W1 and Q1, comments 3 and 4 answer W2 and Q2, and comment 4 also answers Q3.

(W1 & Q1) Criteria on the validation and test splits, with the audit's separation

Yes, on the detectors the audit uses. You are right that the submitted evidence covered only criteria computed on the training split, and our training-split analysis had missed what the other splits show. We computed the five criteria on each synthetic split against the private split it imitates, and on the mean of the three (Appendix F.6, Table 11). Throughout, the gap is the difference in held-out rank agreement between a random release and the best release of the same generator, averaged over generators. Half A and half B are the two halves of the private test period used to choose and to score (Section 5.4).

  • Pair reading of Section 5.3 (test split against the whole private test period): C2ST on the test split orders 70% of the pairs correctly, and label-rate error averaged over the three splits orders 68%. For C2ST, p = 0.009 uncorrected and 0.135 with a Holm correction for the fifteen added tests. Without the four collapsed generators its rate rises to 77%, with p = 0.002 and 0.033 after the correction. No variant of KS, TVD or dependence error orders the pairs distinguishably better than chance.
  • Same separation as the audit (criterion on half A, tau on half B, first 20 half-splits): C2ST on the test split orders 61% of the pairs, and the audit 91%. In the pair reading the criterion and tau share the private test rows, which inflates its agreement somewhat.
  • Choosing a release on half A, scored on half B: the best criterion (label-rate error, mean of the three splits) recovers 47% of the gap, and C2ST on the test split recovers 19% on the first 20 half-splits. The audit recovers 92% on both sets of splits, and averaging the five leaderboards recovers 44%. Without the four collapsed generators, TVD on the validation split does best at 58%, although in the pair reading it is not distinguishable from chance. Label-rate error recovers 57% there, the audit 89% and averaging 52%. Label-rate error needs only the split prevalences the paper reports, so a public user can choose by it. The other split-wise criteria need the private data of each split.

The table breaks the selection result down by criterion and split, as the share of the gap recovered on B (%). Test-split values are computed against half A only, and C2ST on the test split and its mean use the first 20 half-splits.

Criterion computed ontrainvaltestmean of three
KS7-18-7
TVD1139-4-12
Dependence error-15-15-1726
C2ST2111911
Label-rate error32164147

(W1 & Q1, continued) Swapping splits between releases

Within each generator, detectors trained on the training and validation splits of seed i are scored on the test split of seed j. Swapping test splits between the releases of a generator (Appendix F.7) shows that, pooled over generators, the test split accounts for more of the within-generator variation in tau than the other two splits.

Share of the within-generator sum of squarestest splittraining and validation splitsinteraction
all twelve generators29%20%52%
without the collapsed generators40%24%36%

We ran every cell a second time with new detector seeds. Run-to-run noise accounts for 11% of the variation (9% without the collapsed generators), and 86% of the interaction remains once it is removed (83%). The interaction is therefore real. How much a test split changes tau depends on the training and validation splits it is paired with. More than half of the pooled test-split share comes from GReaT and TabDiff, whose tau over the 25 combinations varies most among the eight generators that do not collapse. In five of those eight the training and validation splits matter more, and training-split criteria miss that part as well. The published combinations of splits are no better than swapped ones (an average tau of 0.463, compared with 0.455). Among those eight, the largest of the three terms differs by generator.

  • Test split: TabDiff, GReaT and TabDDPM.
  • Training and validation splits: FinDiff, TabPFGen and its prior-matched variant.
  • Interaction: SMOTE and TabSyn, with the training and validation splits ahead of the test split.

The test-split share supports your concern about the synthetic test split, and the training and validation splits matter as well. The split a metric sees changes what it can detect, but as selection tools on the audited detectors the metrics remain far from a direct audit. With all twelve generators, the audit recovers about twice what averaging or the best criterion does. Still, without private data a public user can get about half of the audit's gain, and more once the collapsed generators are set aside, by averaging the releases or by choosing on label-rate error.

As you note, TSTR never sees the synthetic test split. We kept its standard definition and say so in Appendix F.6. The abstract, introduction and contributions now state the training-split result and the every-split result separately.

(W2 & Q2) Unused detectors, and evidence beyond the single generation-independent split

We agree with W2 on detectors outside the audited panel, and ran the checks your Q2 asks about. On 20 generation-disjoint splits, removing the overlap does not measurably lower the audit's share of the gap. On held-out detector families the audit is not consistently ahead of averaging, and inside the top tier, where there is little to choose, no benefit remains.

We extended the generation-disjoint check of Appendix F.2 from the first of the 200 half-splits to the first 20. Each split requires retraining all 55 GPU releases, so we could not repeat it on all 200. We first ran three splits. Before generating the other 17, we fixed the number of splits and the comparison, and we report all 20. Averaged over the 20 splits, selection on A recovers 91.3% of the gap on B without the overlap and 91.5% with the original test splits. Split by split, removing the overlap changes the share by -0.15 points on average (95% t-interval -3.9 to +3.6). The overlap therefore does not measurably inflate the audit's share of the gap, and the interval excludes an inflation of more than four points. The interval treats the splits as independent although their halves share transfers, so it may be somewhat too narrow. On single splits the change ranges from -13 to +19 points. On all 20 splits the share without the overlap stays above 77.6%, the 5th percentile over all 200 half-splits with the original test splits, and the lowest lies just above it.

Within the test period, selection also carries over in time. Chosen on July to September and scored on October to December, it recovers 90% of the gap, and 92% in the opposite direction (Appendix F.3). The synthetic test splits used there were generated from the whole period, and whether a release chosen today holds for a later period is untested.

Reviewer_UqUtOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper evaluates whether synthetic financial benchmarks preserve the ordering of models on their private source data. It uses 60 releases from 12 generator configurations against 11 fraud-detection pipelines. It calibrates rank agreement against private-test resampling variability.

The key finding is that releases from the same generator can differ substantially in fidelity. The authors propose selecting releases through direct private-data auditing and show favorable results within their benchmark. They also demonstrate how duplicated records can produce excellent scores while distorting model rankings.

Critical strengths
  • The evaluation target is practically meaningful. Preserving model-selection decisions matters directly to benchmark users. Reporting selection regret alongside rank agreement makes the consequences clearer.
Critical weaknesses
  • Distributional and label-rate diagnostics uses only the synthetic training data, while rankings also depend on separately generated validation and test splits. Their weak predictive performance may therefore reflect variation they do not measure. Evaluating diagnostics across all three splits is an important missing analysis.
Questions for authors
  • Do diagnostics computed separately on synthetic training, validation, and test splits improve release selection?
  • How much audit benefit remains after an inexpensive public screen removes releases with unusable label counts or degenerate leaderboards?
6 comments

Thank you for the review. The split mismatch you raised led to one of the main changes. In the revised PDF the changed sentences are blue and Appendices F.6 to F.8 are new, while F.1 to F.5 keep their numbers. The code and result files for the new analyses are in the anonymous repository. Comments 1 and 2 answer the weakness and Q1, and comment 3 answers Q2.

(W1 & Q1) Diagnostics computed on the training, validation and test splits

Somewhat, but not to the level of an audit. The best criterion rises from 32% of the gap on the training split to 47% on the mean of the three splits, about what averaging gives, while the audit recovers 92%. You are right that the submitted evidence covered only criteria computed on the training split, and our training-split analysis had missed what the other splits show. We computed the five criteria on each synthetic split against the private split it imitates, and on the mean of the three (Appendix F.6, Table 11). Throughout, the gap is the difference in held-out rank agreement between a random release and the best release of the same generator, averaged over generators. Half A and half B are the two halves of the private test period used to choose and to score (Section 5.4).

  • Pair reading of Section 5.3 (test split against the whole private test period): C2ST on the test split orders 70% of the pairs correctly, and label-rate error averaged over the three splits orders 68%. For C2ST, p = 0.009 uncorrected and 0.135 with a Holm correction for the fifteen added tests. Without the four collapsed generators its rate rises to 77%, with p = 0.002 and 0.033 after the correction. No variant of KS, TVD or dependence error orders the pairs distinguishably better than chance.
  • Same separation as the audit (criterion on half A, tau on half B, first 20 half-splits): C2ST on the test split orders 61% of the pairs, and the audit 91%. In the pair reading the criterion and tau share the private test rows, which inflates its agreement somewhat.
  • Choosing a release on half A, scored on half B: the best criterion (label-rate error, mean of the three splits) recovers 47% of the gap, and C2ST on the test split recovers 19% on the first 20 half-splits. The audit recovers 92% on both sets of splits, and averaging the five leaderboards recovers 44%. Without the four collapsed generators, TVD on the validation split does best at 58%, although in the pair reading it is not distinguishable from chance. Label-rate error recovers 57% there, the audit 89% and averaging 52%. Label-rate error needs only the split prevalences the paper reports, so a public user can choose by it. The other split-wise criteria need the private data of each split.

The table breaks the selection result down by criterion and split, as the share of the gap recovered on B (%). Test-split values are computed against half A only, and C2ST on the test split and its mean use the first 20 half-splits.

Criterion computed ontrainvaltestmean of three
KS7-18-7
TVD1139-4-12
Dependence error-15-15-1726
C2ST2111911
Label-rate error32164147

(W1 & Q1, continued) Swapping splits between releases

Within each generator, detectors trained on the training and validation splits of seed i are scored on the test split of seed j. Swapping test splits between the releases of a generator (Appendix F.7) shows that, pooled over generators, the test split accounts for more of the within-generator variation in tau than the other two splits.

Share of the within-generator sum of squarestest splittraining and validation splitsinteraction
all twelve generators29%20%52%
without the collapsed generators40%24%36%

We ran every cell a second time with new detector seeds. Run-to-run noise accounts for 11% of the variation (9% without the collapsed generators), and 86% of the interaction remains once it is removed (83%). The interaction is therefore real. How much a test split changes tau depends on the training and validation splits it is paired with. More than half of the pooled test-split share comes from GReaT and TabDiff, whose tau over the 25 combinations varies most among the eight generators that do not collapse. In five of those eight the training and validation splits matter more, and training-split criteria miss that part as well. The published combinations of splits are no better than swapped ones (an average tau of 0.463, compared with 0.455). Among those eight, the largest of the three terms differs by generator.

  • Test split: TabDiff, GReaT and TabDDPM.
  • Training and validation splits: FinDiff, TabPFGen and its prior-matched variant.
  • Interaction: SMOTE and TabSyn, with the training and validation splits ahead of the test split.

The test-split share supports your concern about the synthetic test split, and the training and validation splits matter as well. The split a metric sees changes what it can detect, but as selection tools on the audited detectors the metrics remain far from a direct audit. With all twelve generators, the audit recovers about twice what averaging or the best criterion does. Still, without private data a public user can get about half of the audit's gain, and more once the collapsed generators are set aside, by averaging the releases or by choosing on label-rate error.

TSTR never sees the synthetic test split. We kept its standard definition and say so in Appendix F.6. The abstract, introduction and contributions now state the training-split result and the every-split result separately.

(Q2) Audit benefit after an inexpensive public screen

Most of it. After each screen the audit still recovers 88% to 96% of the gap, compared with 38% to 52% for averaging. Following your suggestion, we tested screens that use only the release, on the 200 half-splits of Section 5.4 (Appendix F.8).

  • Degenerate leaderboards: drop a release if fewer than half of the 55 detector pairs are separable on its own synthetic test split, under the paper's bootstrap rule. This keeps 42 of the 60 releases. It removes every CTAB-GAN and CTAB-GAN+ release, three TabDiff releases (one inside the noise band) and four GReaT releases, while four TVAE and all five CTGAN releases pass. We also set the threshold at a quarter and at three quarters of the pairs.
  • Unusable label counts: a floor of 10, 25 or 50 positives in every split removes only TVAE, whose releases have one positive per split.
  • Collapsed generators: a user can recognize the four collapsed generators from the release alone, since every collapsed release has its detectors within 0.021 of one another and every other release spreads by at least 0.029, a cut we chose after seeing the releases (Appendix F.8).
Screenshare of the gap on B, averagingshare of the gap on B, audit
none, all twelve generators44%92%
at least half of the pairs separable (42 releases of ten generators)49%88%
none, the same ten generators47%92%
at least a quarter of the pairs separable47%91%
at least three quarters of the pairs separable38%96%
at least 10, 25 or 50 positives in every split46%92%
collapsed generators set aside52%89%

The screen at half of the pairs helps a public user. After it a random survivor scores 0.563 on half B, and a random release of the same ten generators 0.529.

Outside the audited detectors the audit does about as well as averaging when six of the eleven are audited (Appendix F.3). On pairs involving an unused detector it leads once seven or more are audited. The recommendations now say to audit on as many detectors as possible, to publish the other releases as well and to audit again when a detector is added. We also corrected Section 5.6, which had described the signal of a collapsed generator as "scores near 0.03" instead of detectors all scoring within 0.021 of one another.

We hope these analyses address your concerns, and we would be glad to clarify anything further.

Reviewer_QZa9OpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

The paper evaluates synthetic financial-data releases by their preservation of a private detector leaderboard. It compares sixty releases from twelve generators using eleven detectors, calibrates rank agreement against private-test resampling, and studies whether a data holder can select a better release through direct auditing. The main contribution is an empirical audit protocol and release-level evidence, rather than a new rank statistic.

Critical strengths

The public-user model-selection objective is well motivated and differs from evaluating a single TSTR detector. Multiple releases per generator, separable-pair preservation, selection regret, and a private-reference noise scale make the analysis informative. The appendices contain substantial robustness checks, including account/week resampling, a public-data replication, detector holdouts, and an additional TabM model. The paper is unusually explicit about operational suspicion labels, generator collapse, low statistical power, and the absence of formal privacy guarantees. It also acknowledges and compares the related cross-release evaluation approach of van Breugel et al. (2023).

Critical weaknesses
  1. The comparison against standard quality metrics may be affected by a split mismatch. Appendix B computes distributional criteria on synthetic/private training splits, while each release's train, validation, and test splits are generated separately (Section 4.2), and all three affect the public-user leaderboard. If synthetic-test variation drives rank changes, training-only fidelity metrics would naturally miss them. Split-matched/combined metrics and controls that vary the synthetic test split independently of training/validation are needed to establish whether the negative result concerns the metrics or the part of the release they measure.

  2. Generalization beyond the audit setting remains limited. Appendix F.2 explicitly notes that most holdouts exclude evaluation data from release selection but not from generation; the additional generation-disjoint control is valuable but uses only one split. Appendix F.3 also recovers only 28% of the available gap for pairs entirely among unused detectors. Because public benchmarks should support future data and new methods, repeated generation-disjoint evaluation and family-held-out/top-tier selection regret would materially strengthen the recommendation. Existing results support an audit within the tested period and detector suite more strongly than broad transfer.

Questions for authors
  1. Does the standard-metric result persist when metrics cover all generated splits, and which split contributes most to release-level ranking variance?
  2. How much audit benefit remains under repeated generation-disjoint evaluation and when ranking entirely held-out detector families, especially within the competitive top tier?
5 comments

Thank you for the detailed review. We ran the controls you proposed, and their results are in Appendices F.2, F.3, F.6 and F.7 of the revised PDF. Changes are blue there, F.6 to F.8 are new, and F.1 to F.5 keep their numbers. The code and result files of the new analyses are in the anonymous repository. Comments 1 and 2 answer W1 and Q1, and comments 3 and 4 answer W2 and Q2.

(W1 & Q1) Metrics over all generated splits, and which split contributes most

Partly, as you suspected. In the pair reading of Section 5.3, the negative result persists on every split for KS, TVD and dependence error, while C2ST and label-rate error carry some signal once they see the other splits. You are right that the submitted evidence covered only criteria computed on the training split, and our training-split analysis had missed what the other splits show. We computed the five criteria on each synthetic split against the private split it imitates, and on the mean of the three (Appendix F.6, Table 11). Throughout, the gap is the difference in held-out rank agreement between a random release and the best release of the same generator, averaged over generators. Half A and half B are the two halves of the private test period used to choose and to score (Section 5.4).

  • Pair reading of Section 5.3 (test split against the whole private test period): C2ST on the test split orders 70% of the pairs correctly, and label-rate error averaged over the three splits orders 68%. For C2ST, p = 0.009 uncorrected and 0.135 with a Holm correction for the fifteen added tests. Without the four collapsed generators its rate rises to 77%, with p = 0.002 and 0.033 after the correction. No variant of KS, TVD or dependence error orders the pairs distinguishably better than chance.
  • Same separation as the audit (criterion on half A, tau on half B, first 20 half-splits): C2ST on the test split orders 61% of the pairs, and the audit 91%. In the pair reading the criterion and tau share the private test rows, which inflates its agreement somewhat.
  • Choosing a release on half A, scored on half B: the best criterion (label-rate error, mean of the three splits) recovers 47% of the gap, and C2ST on the test split recovers 19% on the first 20 half-splits. The audit recovers 92% on both sets of splits, and averaging the five leaderboards recovers 44%. Without the four collapsed generators, TVD on the validation split does best at 58%, although in the pair reading it is not distinguishable from chance. Label-rate error recovers 57% there, the audit 89% and averaging 52%. Label-rate error needs only the split prevalences the paper reports, so a public user can choose by it. The other split-wise criteria need the private data of each split.

The table breaks the selection result down by criterion and split, as the share of the gap recovered on B (%). Test-split values are computed against half A only, and C2ST on the test split and its mean use the first 20 half-splits.

Criterion computed ontrainvaltestmean of three
KS7-18-7
TVD1139-4-12
Dependence error-15-15-1726
C2ST2111911
Label-rate error32164147

(W1 & Q1, continued) Swapping splits between releases

No single split accounts for most of the variation, and the answer differs between generators. We found this with the control you described. Within each generator, detectors trained on the training and validation splits of seed i are scored on the test split of seed j. Swapping test splits between the releases of a generator (Appendix F.7) shows that, pooled over generators, the test split accounts for more of the within-generator variation in tau than the other two splits.

Share of the within-generator sum of squarestest splittraining and validation splitsinteraction
all twelve generators29%20%52%
without the collapsed generators40%24%36%

We ran every cell a second time with new detector seeds. Run-to-run noise accounts for 11% of the variation (9% without the collapsed generators), and 86% of the interaction remains once it is removed (83%). The interaction is therefore real. How much a test split changes tau depends on the training and validation splits it is paired with. More than half of the pooled test-split share comes from GReaT and TabDiff, whose tau over the 25 combinations varies most among the eight generators that do not collapse. In five of those eight the training and validation splits matter more, and training-split criteria miss that part as well. The published combinations of splits are no better than swapped ones (an average tau of 0.463, compared with 0.455). Among those eight, the largest of the three terms differs by generator.

  • Test split: TabDiff, GReaT and TabDDPM.
  • Training and validation splits: FinDiff, TabPFGen and its prior-matched variant.
  • Interaction: SMOTE and TabSyn, with the training and validation splits ahead of the test split.

On the closing question of your W1, the negative result concerns both the metrics and the part of the release they measure. The test-split share supports your concern about the synthetic test split, and the training and validation splits matter as well. The split a metric sees changes what it can detect, but as selection tools on the audited detectors the metrics remain far from a direct audit. With all twelve generators, the audit recovers about twice what averaging or the best criterion does. Still, without private data a public user can get about half of the audit's gain, and more once the collapsed generators are set aside, by averaging the releases or by choosing on label-rate error.

TSTR never sees the synthetic test split. We kept its standard definition and say so in Appendix F.6. The abstract, introduction and contributions now state the training-split result and the every-split result separately.

(W2 & Q2) Repeated generation-disjoint evaluation, held-out families, and the top tier

We agree with W2 on detectors outside the audited panel, and ran the checks your Q2 asks about. On 20 generation-disjoint splits, removing the overlap does not measurably lower the audit's share of the gap. On held-out detector families the audit is not consistently ahead of averaging, and inside the top tier, where there is little to choose, no benefit remains.

We extended the generation-disjoint check of Appendix F.2 from the first of the 200 half-splits to the first 20. Each split requires retraining all 55 GPU releases, so we could not repeat it on all 200. We first ran three splits. Before generating the other 17, we fixed the number of splits and the comparison, and we report all 20. Averaged over the 20 splits, selection on A recovers 91.3% of the gap on B without the overlap and 91.5% with the original test splits. Split by split, removing the overlap changes the share by -0.15 points on average (95% t-interval -3.9 to +3.6). The overlap therefore does not measurably inflate the audit's share of the gap, and the interval excludes an inflation of more than four points. The interval treats the splits as independent although their halves share transfers, so it may be somewhat too narrow. On single splits the change ranges from -13 to +19 points. On all 20 splits the share without the overlap stays above 77.6%, the 5th percentile over all 200 half-splits with the original test splits, and the lowest lies just above it.

Within the test period, selection also carries over in time. Chosen on July to September and scored on October to December, it recovers 90% of the gap, and 92% in the opposite direction (Appendix F.3). The synthetic test splits used there were generated from the whole period, and whether a release chosen today holds for a later period is untested.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.