acceptodds
Under review as a conference paper at ICLR 2027

ContinuousBench: Can differentially private synthetic data transfer knowledge?

Abstract

Differentially private (DP) text synthesis promises to unlock sensitive corpora for model training, but it remains unclear whether DP synthetic data transmits genuinely new knowledge and capabilities present only in those corpora. This is because existing evaluations rely on tasks that are nearly solvable without training, so strong benchmark performance does not establish that DP synthesis can substitute original data access. Thus, we introduce ContinuousBench, a continuously and automatically-regenerated benchmark that measures capability gain from DP synthetic text. Each quarter, a new release pairs a never-before-seen training corpus with a derived QA set, constructed to be: (1) unsolvable sans-corpus; and (2) learnable under DP, as the tested knowledge is supported by hundreds of independent records. Researchers produce DP synthetic data from the training corpus and run our standardized training and evaluation harness on their synthetic data to measure gains. We instantiate two tracks: Geminon, a procedurally-generated dataset about fictional creatures; and News, a stream of newly scraped public news articles. Although standard benchmarks are nearly saturated, on ContinuousBench we find that non-private synthesis transfers substantial knowledge from the original corpus, while state-of-the-art DP synthesis methods generally fail to do so, even at ε = 100. ContinuousBench is available at https://hf.co/AnonymousContinuousBench.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, synthetic data, benchmark, contamination

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_7mMvOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies a timely problem (knowledge transfer in PD synthesis data), develops a useful benchmark, and provides some interesting findings. Data is a bottleneck for frontier model training and DP text synthesis is an important method to leverage sensitive corpora currently. However, current evaluations fail to adequately account for knowledge elicitation, superficial matching, and distillation effects. To address these issues, the authors design grounded factual QA tasks to evaluate knowledge transfer by testing model performance on either fictional or post-cutoff knowledge. The generator and downstream model use the same base checkpoint to eliminate distillation effects. Moreover, the authors design an automated framework for continuous data re-generation triggered by new corpus release. The data and evaluation harness are open-sourced.

Critical strengths
  • The paper studies a timely problem, develops a useful benchmark, and has some interesting findings, e.g. DP synthesis methods fail to transfer knowledge from original corpus.
  • The automated re-generation of data and the reusable, scalable framework design of the benchmark seems particularly nice from an architecture perspective.
  • The analysis is rigorous and substantial.
  • The writing is clear.
Critical weaknesses
  • In abstract, "even at ε = 100", would be better to directly explain what does ε mean here (privacy constraint).
  • The major finding is a bit counterintuitive to me. If non-private synthesis transfers knowledge well, why DP synthesis with extremely weak privacy constraint cannot transfer knowledge? Is there any key mechanism in the DP method leads to this outcome, or could there be other underlying factors influencing the results besides privacy? Is it caused by the noise in DP-SGD, as in Appendix E?
Questions for authors

Overall, the problem is timely, the benchmark seems useful, the method seems solid, and the findings are interesting. I appreciate the authors' hard work. The major finding is a bit counterintuitive to me. If non-private synthesis transfers knowledge well, why DP synthesis with extremely weak privacy constraint cannot transfer knowledge? Is there any key mechanism in the DP method leads to this outcome, or could there be other underlying factors influencing the results besides privacy? Is it caused by the noise in DP-SGD, as in Appendix E?

3 comments

W1: Clarifying ϵ\epsilon in the abstract

Thank you for the suggestion. We agree that the meaning of ϵ\epsilon should be made explicit for readers less familiar with differential privacy, and we will revise the abstract accordingly to clarify.

W2/Q: Counterintuitive findings

We thank the reviewer for this question. Indeed, we find that the results are counterintuitive, especially given the substantial knowledge transfer by non-private synthesis. Nevertheless, we believe that uncovering this surprising phenomenon is indeed a contribution of our work. Existing evaluations suggested that DP synthesis methods perform well, yet our results reveal a substantial and previously overlooked gap, even at a weak privacy regime. We believe this finding challenges the current understanding of the DP synthesis methods and deserves attention from the broader research community.

In particular, ϵ=100\epsilon=100 is an extremely weak privacy regime, e.g., randomized response with ϵ=100\epsilon=100 only lies with probability in the order of 10−4410^{-44}. The fact that these well established DP synthesis methods still fail at this level suggests that the problem is not a fundamental limit of DP. Rather, it highlights how far these methods are from achieving comparable utility at meaningful privacy regimes, e.g., ϵ=1\epsilon=1. We hope that understanding the underlying causes and developing methods to overcome this gap will motivate substantial follow-up research.

Our primary goal in this work is to rigorously identify this empirical phenomenon, rather than to provide a complete mechanistic explanation. Nevertheless, we have also taken initial steps toward understanding the failure, and we are happy to engage with your question via our own hypotheses regarding a mechanistic explanation for this phenomenon, please see the response below.

Overall, it seems to us that the main concern in your review is this counterintuitive finding, which, as we argue above, is indeed the paper's central contribution. We also appreciate your positive comments on the timeliness of the work, the usefulness of the benchmark, and the soundness of the methodology. If our responses address your concern, we would be grateful if you would consider raising your score.

An intuitive explanation of why DP synthesis fails to transfer facts

We have taken an initial step toward understanding this gap through empirical investigation. In our ablation experiment, we first found that direct DP fine-tuning on the original corpus performs comparably to the evaluators trained on DP synthetic corpus, so the bottleneck lies in the private training itself. Then, our clipping vs. noising ablation shows that clipping does not hurt much, while noising is the main cause of the degradation.

We thus attempt to explain this phenomenon from a SNR perspective. In GEMINON, a fact supported by ~200 out of 200K records appears in only ~0.5 examples per batch (of 512), and each supporting record contains only a few QA-relevant tokens. Thus, the gradient signal is weak and infrequent, so DP noise can obscure it even at large ϵ\epsilon. However, clipping itself only bounds the magnitude of individual contributions without introducing random signals, so as a result, learning signals that are weak can still accumulate. On the other hand, the style and structure of our corpus contribute signals in every batch, so they can still be learned under DP, as reflected in the MAUVE scores and the qualitative examples. This explains why DP synthetic text can look distributionally similar to the real corpus while transferring few facts.

We emphasize that this is just an intuitive explanation rather than a theoretical characterization. We hope that this counterintuitive findings can motivate further research on DP training dynamics, the learnability of tasks under DP, and improving DP synthesis.

Please let us know if this answers your question or if any clarification you would like before the discussion period ends tomorrow. If your concerns have been addressed, we would greatly appreciate it if you would consider updating your score.

Reviewer_fPQ5OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors present ContinuousBench, a periodically regenerated benchmark for evaluating differentially private (DP) text synthesis methods. Each release ships a private corpus whose facts the base model cannot already know plus a held-out short-answer QA set. A DP synthesis method produces synthetic text, a fixed harness continues pretraining the same Gemma-3 checkpoint on it, and QA accuracy measures knowledge transfer. Real-data training scores high while DP synthetic data recovers little even at epsilon around 10 and 100. Training-free methods (PE, InvisibleInk) do no better than the base. MAUVE stays moderate while QA collapses, and on News DP data only beats the base once a fact appears in ~600 records. These results suggest that existing DP text benchmarks overstate how much private knowledge current DP synthesis methods preserve, and that distribution-level metrics alone are insufficient to detect this.

Critical strengths

Strengths:

  • Problem framing: the authors delineate an important issue arising from being able to evaluate DP synthesis systems. They specifically outline this within fig1 that showcases how the current field is tapped out in regards to benchmark performance for specific benchmark tasks using models trained both with and without DP-synthesis (namely, the imdb and openreview benchmarks). This signifies the importance of this problem and how it would naturally lead to the next point (proposed solution).
  • Proposed solution: the authors propose a system that continuously releases new benchmarks and allows for evaluation of DP-syn methods. As the authors noted that benchmarks are inevitably saturated, pushing for a system that regenerates fresh corpora with facts the base model cannot already know keeps the evaluation from saturating as models improve. Controlling how many records support each fact also lets the benchmark separate knowledge that should survive DP from knowledge that should be hidden (singletons), which existing benchmarks cannot do.
Critical weaknesses

Weaknesses:

  • The benchmark doesn't separate DP-syn methods, and it's unclear why: On Geminon every DP method scores 1 to 9% against a around 1% untrained base, and News spreads them only slightly (around 14 to 26%). Direct DP fine-tuning on the real corpus does about as badly (Table 23), and the ablation shows noise (not clipping) causes the collapse (Table 25). Since only the learning rate was swept (batch size 512), "fails even at epsilon=100" may reflect ablations rather than a fundamental limit. Either way, unsure would want to investigate into these details.
  • Generality across models and releases is not shown: The same gemma-3 checkpoint (1B/4B) generates the synthetic data and is trained and evaluated on it, so a method that does well may partly be matching this model family's tendencies. Only one release per track is shown, so the "continuous" aspect, and whether method rankings stay stable across releases, is untested.
Questions for authors

Questions:

  1. Only the learning rate is swept for DP-SGD (batch 512, 3k steps). Would a sweep over batch size and steps at fixed epsilon change the epsilon=100 result? Is the failure a fundamental limit or specific to this recipe? Relatedly, can a broader set of DP-syn methods be evaluated to show the benchmark can separate them on Geminon and News?
  2. How do results and method rankings change with a non-Gemma synthesizer/evaluator family? Can you report the numbers for the preliminary 12B run mentioned in App. D?
  3. Can you show a second release of either track, and whether method rankings stay stable across releases? What is the intended cadence (annual in the abstract vs quarterly in Fig. 2)? How will releases be frozen and versioned? The HF Geminon card says the current version is untagged and that unpinned loads will pick up future updates.
  4. On the singleton split (Table 5), even non-private Syn scores near the floor (1.2/2.1% vs 0.6/0.9% untrained), so low DP-Syn accuracy says little about privacy. Would an empirical audit (e.g., canaries or membership inference) on synthetic outputs be a more informative privacy check?
10 comments

W2/Q2: Generality to other model families

To investigate whether the observed knowledge transfer gap is specific to Gemma families or training instability, we conducted additional experiments: 1. Extending our evals to Qwen3-1.7B, Qwen3-4B, and the larger Gemma 3 12B, 2. Repeating our main experiments on Gemma 3 1B and 4B across three seeds. The results are consistent with our findings presented in the paper: 1. untrained models perform poorly, while training on the original corpus yields substantial gains; 2. non-private synthesis transfers much of this knowledge, while DP synthesis does not, even at ϵ=100\epsilon=100, although accuracy improves with higher support counts; and 3. direct DP finetuning on the original corpus does no better, with our ablations identifying Gaussian noise as the primary bottleneck. We provide details below.

Experiments across model families and sizes

We extend our training-based synthesis experiments to Qwen3-1.7B and Qwen3-4B on both GEMINON and NEWS, using three seeds per config. We also report a single-seeded run on Gemma 3 12B. For each seed and privacy regime (ϵ=∞,100,10\epsilon=\infty,100,10), we finetune a generator, sample a synthetic corpus, and train a downstream evaluator on the corpus. We also report three reference baselines: 1. No Training, 2. Train on Real, and 3. Direct DP finetuning on Real. All training uses LoRA with rank 128.

We carefully tuned the learning rates for each config above, sweeping over 5×10−5,10−4,2×10−4,5×10−4,10−3{5\times10^{-5}, 10^{-4}, 2\times10^{-4}, 5\times10^{-4}, 10^{-3}} and selecting the learning rate with the lowest val loss.

Exact-match / contains accuracy (%) of No Training on GEMINON-2025-09 and NEWS-2025-09.

ModelGEMINON-2025-09 (support k≈200k\approx200, n=3360)NEWS-2025-09, all (n=1415)NEWS-2025-09, support k≥200k\ge200 (n=560)NEWS-2025-09, support k≥400k\ge400 (n=266)
Qwen3-1.7B-Base0.54 / 2.563.11 / 3.965.36 / 6.616.39 / 7.52
Qwen3-4B-Base1.07 / 3.573.75 / 4.526.43 / 7.326.02 / 6.39
Gemma 3 12B-PT1.49 / 3.574.73 / 4.958.39 / 8.578.27 / 8.27

Exact-match accuracy (%) on GEMINON-2025-09 and contains / LLM-match accuracy (%) on NEWS-2025-09 for Train on Real, Syn (ϵ=∞\epsilon=\infty), DP-Syn (ϵ∈{100,10}\epsilon\in\{100,10\}), and Direct DP-SGD (ϵ∈{100,10}\epsilon\in\{100,10\}). We evaluate on QAs with support k≈200k\approx200 for GEMINON and k≥200k\ge200 for NEWS. Qwen results are reported as mean ±\pm std across three seeds, while Gemma 3 12B results are from a single run. All configs use LoRA with rank 128.

DatasetModelTrain on RealSyn (ϵ=∞\epsilon=\infty)DP-Syn (ϵ=100\epsilon=100)DP-Syn (ϵ=10\epsilon=10)Direct DP-SGD (ϵ=100\epsilon=100)Direct DP-SGD (ϵ=10\epsilon=10)
GEMINON-2025-09Qwen3-1.7B-Base91.56 ±\pm 1.4587.92 ±\pm 2.113.48 ±\pm 0.261.92 ±\pm 0.113.12 ±\pm 0.171.99 ±\pm 0.27
GEMINON-2025-09Qwen3-4B-Base96.08 ±\pm 1.3396.48 ±\pm 0.395.51 ±\pm 0.092.92 ±\pm 0.004.65 ±\pm 0.132.73 ±\pm 0.07
GEMINON-2025-09Gemma 3 12B-PT96.2596.9016.584.3810.833.81
NEWS-2025-09Qwen3-1.7B-Base43.75 ±\pm 1.42 / 56.55 ±\pm 1.3936.73 ±\pm 0.57 / 47.68 ±\pm 0.6411.25 ±\pm 1.12 / 14.88 ±\pm 0.8810.54 ±\pm 0.93 / 14.11 ±\pm 1.5612.20 ±\pm 0.27 / 16.85 ±\pm 0.419.76 ±\pm 0.27 / 13.45 ±\pm 0.45
NEWS-2025-09Qwen3-4B-Base53.39 ±\pm 0.64 / 68.10 ±\pm 0.5249.94 ±\pm 1.22 / 63.21 ±\pm 1.1718.63 ±\pm 0.63 / 24.17 ±\pm 0.2714.05 ±\pm 0.55 / 18.75 ±\pm 1.0919.40 ±\pm 0.55 / 24.23 ±\pm 0.8414.70 ±\pm 0.27 / 19.17 ±\pm 0.74
NEWS-2025-09Gemma 3 12B-PT59.29 / 77.6854.82 / 72.1424.29 / 33.3915.00 / 21.0724.82 / 34.1118.21 / 24.29

The results are consistent with our main findings across model families and size. Training on the real corpus (Train on Real) achieves high accuracy, while the base model performs near chance (No Training). Non private synthesis (Syn) transfers substantial factual knowledge, despite some degradation compared to Train on Real, while DP synthesis (DP-Syn) performs significantly worse than Syn, even at ϵ=100\epsilon=100. Additionally, direct DP finetuning on the original corpus also performs substantially worse than Train on Real or even Syn, with accuracy only comparable to DP-Syn. This suggests that the knowledge transfer gap is not specific to Gemma 3 1B and 4B, but rather more general across model families and sizes.

W2/Q2 (cont.)

We also evaluate NEWS across different support counts and find that accuracy generally increases with support count in all configs, consistent with our findings on Gemma 3 1B and 4B. Additionally, we run a clipping-only ablation on Qwen3-1.7B using GEMINON: the clip-only variant, without Gaussian noise, the model reaches 85.54% test exact-match accuracy after just 1K steps, compared to 3.30% with DP-SGD at ϵ=100\epsilon = 100. This is consistent with Appendix E.1, suggesting that noise, rather than clipping, accounts for most of the degradation.

Experiments across model families and sizes

We further repeat our main experiments in the paper on Gemma 3 1B and 4B using three seeds on both GEMINON and NEWS, using three seeds per config. For each seed and privacy regime (ϵ=∞,100,10\epsilon=\infty,100,10), we finetune a generator, sample a synthetic corpus, and train a downstream evaluator on the corpus. We also report three reference baselines: 1. Train on Real, and 2. Direct DP finetuning on Real. All generator training uses LoRA with rank 128 and all evaluator training uses full parameter finetuning.

Exact-match accuracy (%) on GEMINON-2025-09 and contains / LLM-match accuracy (%) on NEWS-2025-09 for Train on Real, Syn (ϵ=∞\epsilon=\infty), and DP-Syn (ϵ∈{100,10}\epsilon\in\{100,10\}). We evaluate on QAs with support k≈200k\approx200 for GEMINON and k≥200k\ge200 for NEWS. Results are reported as mean ±\pm std across three seeds.

DatasetModelTrain on RealSyn (ϵ=∞\epsilon=\infty)DP-Syn (ϵ=100\epsilon=100)DP-Syn (ϵ=10\epsilon=10)
GEMINON-2025-09Gemma 3 1B-PT87.37 ±\pm 0.8885.56 ±\pm 2.217.05 ±\pm 1.251.88 ±\pm 0.25
GEMINON-2025-09Gemma 3 4B-PT96.99 ±\pm 0.6394.04 ±\pm 1.344.19 ±\pm 0.382.70 ±\pm 0.21
NEWS-2025-09Gemma 3 1B-PT43.16 ±\pm 2.40 / 53.81 ±\pm 2.6943.63 ±\pm 2.06 / 54.64 ±\pm 0.9912.26 ±\pm 0.45 / 16.13 ±\pm 0.1010.54 ±\pm 1.63 / 13.63 ±\pm 1.66
NEWS-2025-09Gemma 3 4B-PT53.09 ±\pm 1.19 / 70.54 ±\pm 0.1851.43 ±\pm 1.46 / 66.25 ±\pm 0.9418.87 ±\pm 1.05 / 25.18 ±\pm 1.7213.63 ±\pm 0.85 / 18.69 ±\pm 1.09

The results are consistent with our main findings. The numbers reported in the paper also fall close to the means over three seeds reported above. Importantly, the variance across seeds is much smaller than the gap between non-private and DP synthesis.

W1/Q1: Separation between DP synthesis methods and hyperparameters sweeps

We thank the reviewer for raising these concerns. We would like to clarify the intended contribution of ContinuousBench and provide additional context on our hyperparameter tuning.

On the limited separation among existing DP synthesis methods

We emphasize that ContinuousBench is deliberately designed to expose limitations of existing DP synthesis methods that have achieved strong results on previous benchmarks. We show that several widely used, state-of-the-art DP synthesis approaches, including both training-based and training-free methods, fail to transfer factual knowledge, despite performing well on previous benchmarks. We acknowledge that the tested DP methods cluster near the accuracy floor on GEMINON, but we believe that this only reflects a shared limitation of the existing approaches and does not mean that our benchmark cannot distinguish them apart. In particular, non-private baselines achieve high accuracy on the same tasks, so there is plenty of headroom for a better DP method to improve. Moreover, the NEWS track provides additional resolution via support-count stratification, and indeed the accuracy increases with support counts

To illustrate this, we compare DP-Syn against Private Evolution and InvisibleInk at ϵ=10\epsilon = 10, on two easier subsets: GEMINON questions about the "classification" attribute and NEWS questions with support k≥600k\ge 600. The differences are more apparent on these subsets. On NEWS with support k≥600k\ge 600t, DP-Syn achieves 22.56% contains accuracy, compared to 9.77% for Private Evolution; on GEMINON classification questions, DP-Syn achieves 10.00%, while Private Evolution remains at 0%. These results show that our benchmark can indeed distinguish between different DP synthesis methods, even though all remain far below non-private training. Hence, we believe their bad performances only reflect a shared limitation of current methods, rather than a limitation of the benchmark itself.

MethodGEMINON QAs on classification (n=255)NEWS QAs with support k≥600k\ge600 (n=133)
Train on Real (ϵ=∞\epsilon=\infty)90.2060.90
No Training0.007.52
DP-Syn (ϵ=10\epsilon=10)10.0022.56
Private Evolution (ϵ=10\epsilon=10)0.009.77
InvisibleInk (ϵ=10\epsilon=10)-2.30

W1/Q1 (cont.)

on the possibility of suboptimal DP training configurations.

We agree that training configurations can substantially affect DP-SGD performance, and we have taken care to adopt well-established and optimized training recipes. For training-free methods, we follow the configurations recommended in the original papers. For the training-based method, we used the largest expected batch size (512) we could afford. This is already 8x larger than the largest swept in a related DP benchmarking paper (ICLR 2026 Oral) [1], and our DP training recipe uses 16x larger batch size than non-private training and about 5x the total FLOPS, even with our optimized code. We also adopt optimized techniques from the literature, including truncated Poisson sampling, PLD privacy accounting, and normalized gradient clipping. Beyond the learning-rate sweeps reported in the paper, our preliminary experiments also explored additional hyperparameters, including the beta’s and weight decay.

[1] Benchmarking Empirical Privacy Protection for Adaptations of Large Language Models

W2/Q3: New release

We thank the reviewer for pointing out this inconsistency. The quarterly release schedule indicated in Figure 2 was incorrect and should be “annually” instead. We will revise Figure 2 and the corresponding text to clarify this. Since each release takes substantial computes and time to produce, we have not yet officially published this year's update. However, we have already constructed the 2026-09 release of both datasets and conducted additional experiments on them.

New release statistics

The new 2026-09 release follows the same curation pipeline as the original 2025-09 release: GEMINON contains an entirely new set of fictional creatures, with no overlap with the 2025-09 entities, while keeping the same number of records (180K training records) and QAs (3360 test QAs). NEWS is constructed from 1000 event clusters collected in September 2026, containing 150K training records and 2891 test QA. We notice that the 2026-09 version of NEWS has fewer supporting articles per question than the original 2025-09 release on average. Full statistics are provided below.

GEMINON 2025-09GEMINON 2026-09NEWS 2025-09NEWS 2026-09
Documents (train / val / test)180,108 / 10,006 / 10,006180,108 / 10,006 / 10,006148,407 / 8,244 / 8,246147,035 / 8,168 / 8,170
Number of Tokens (total; mean per document)23.0M; 11525.3M; 126110.9M; 673112.4M; 688
QA pairs (val / test)2,880 / 3,3602,880 / 3,3602,175 / 2,7292,420 / 2,891
Support per question (mean / median)≈200\approx 200≈200\approx 200226.4 / 14996.8 / 48
Questions with support ≥200\ge 200AllAll1,967 (40.1%)663 (12.5%)

Experimental results on the new 2026-09 release

We repeat the experiments from the paper on the 2026-09 release using Gemma 3 1B and 4B.. For each config, we finetune a generator, sample a synthetic corpus, and train a downstream evaluator on the corpus. We also report three reference baselines: 1. No Training, 2. Train on Real, and 3. Direct DP finetuning on Real. We report exact-match accuracy on GEMINON (with k≈200k\approx 200) and "contains" and LLM-match accuracies on NEWS (with k≥200k\ge 200).

Exact-match and "contains" accuracy (%) of the No Training on GEMINON-2026-09 and NEWS-2026-09.

ModelGeminon-2026-09 (n=3360)News-2026-09, all (n=2891)News-2026-09, support k≥k\ge 100 (n=839)News-2026-09, support k≥k\ge 200 (n=368)News-2026-09, support k≥k\ge 400 (n=123)
Gemma 3 1B-PT0.71 / 1.403.46 / 4.123.34 / 4.532.99 / 5.711.63 / 4.88
Gemma 3 4B-PT1.07 / 2.893.87 / 4.362.98 / 3.812.99 / 4.353.25 / 3.25
Reviewer_9LoLOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces a benchmark (i.e., CONTINUOUSBENCH) to evaluate whether differentially private (DP) text synthesis preserves corpus-specific knowledge. The authors used continuously regenerated, access-dependent corpora with grounded short-answer QA and a fixed downstream training/evaluation harness. Experiments on Gemma 3 1B/4B across a fictional GEMINON track and a post-cutoff NEWS track demonstrate that non-private synthesis transfers substantial knowledge.

Critical strengths

S1: Access dependency is well enforced. Post-cutoff and procedurally regenerated corpora are paired with validation: base models perform near chance, whereas real-corpus training yields large gains.

S2: It is a strong design choice to use support-count stratification, which can separate singleton memorization from population-level knowledge and measure DP utility as a function of repetition.

S3: The fixed evaluation harness reduces confounds. Matched generator–evaluator settings make corpus effects easier to isolate.

S4: The negative result is clear and actionable. MAUVE remains moderate while grounded QA collapses; the ablation identifies noise addition as the primary source of degradation.

Critical weaknesses

W1: Empirical scope is quite limited. The experiments mainly focus on Gemma 3 1B/4B and two tracks, with mostly single-run results and no variance estimates, which weakens the model's generality.

W2: Domain coverage is rather narrow. For example, GEMINON is fictional, and NEWS is factual news QA; the benchmark does not test high-stakes domains, multi-hop reasoning, or procedural knowledge.

W3: The construction of continuousbenchmark relies heavily on generative AI and LLM judges, which introduces potential generation bias, judge bias, and noise. NEWS support counts are conservative lower bounds.

W4: The boundary between DP-learnable population-level knowledge and singleton memorization remains informal, operationalized mainly through support-count buckets and empirical thresholds.

W5: Baseline coverage is incomplete. Several recent DP text synthesis, private prediction, and distillation-style methods are not evaluated.

W6: Short-answer factual QA is quite narrow, and lacks human evaluation or richer task formats. Partial knowledge may not be captured by strict exact-match or LLM-match scoring.

Questions for authors

Please refer to the above comments.

5 comments

W1: limited empirical scope

To investigate whether the observed knowledge transfer gap is specific to Gemma families or training instability, we conducted additional experiments: 1. Extending our evals to Qwen3-1.7B, Qwen3-4B, and the larger Gemma 3 12B, 2. Repeating our main experiments on Gemma 3 1B and 4B across three seeds. The results are consistent with our findings presented in the paper: 1. untrained models perform poorly, while training on the original corpus yields substantial gains; 2. non-private synthesis transfers much of this knowledge, while DP synthesis does not, even at ϵ=100\epsilon=100, although accuracy improves with higher support counts; and 3. direct DP finetuning on the original corpus does no better, with our ablations identifying Gaussian noise as the primary bottleneck. We provide details below.

Experiments across model families and sizes

We extend our training-based synthesis experiments to Qwen3-1.7B and Qwen3-4B on both GEMINON and NEWS, using three seeds per config. We also report a single-seeded run on Gemma 3 12B. For each seed and privacy regime (ϵ=∞,100,10\epsilon=\infty,100,10), we finetune a generator, sample a synthetic corpus, and train a downstream evaluator on the corpus. We also report three reference baselines: 1. No Training, 2. Train on Real, and 3. Direct DP finetuning on Real. All training uses LoRA with rank 128.

We carefully tuned the learning rates for each config above, sweeping over 5×10−5,10−4,2×10−4,5×10−4,10−3{5\times10^{-5}, 10^{-4}, 2\times10^{-4}, 5\times10^{-4}, 10^{-3}} and selecting the learning rate with the lowest val loss.

Exact-match / contains accuracy (%) of No Training on GEMINON-2025-09 and NEWS-2025-09.

ModelGEMINON-2025-09 (support k≈200k\approx200, n=3360)NEWS-2025-09, all (n=1415)NEWS-2025-09, support k≥200k\ge200 (n=560)NEWS-2025-09, support k≥400k\ge400 (n=266)
Qwen3-1.7B-Base0.54 / 2.563.11 / 3.965.36 / 6.616.39 / 7.52
Qwen3-4B-Base1.07 / 3.573.75 / 4.526.43 / 7.326.02 / 6.39
Gemma 3 12B-PT1.49 / 3.574.73 / 4.958.39 / 8.578.27 / 8.27

Exact-match accuracy (%) on GEMINON-2025-09 and contains / LLM-match accuracy (%) on NEWS-2025-09 for Train on Real, Syn (ϵ=∞\epsilon=\infty), DP-Syn (ϵ∈{100,10}\epsilon\in\{100,10\}), and Direct DP-SGD (ϵ∈{100,10}\epsilon\in\{100,10\}). We evaluate on QAs with support k≈200k\approx200 for GEMINON and k≥200k\ge200 for NEWS. Qwen results are reported as mean ±\pm std across three seeds, while Gemma 3 12B results are from a single run. All configs use LoRA with rank 128.

DatasetModelTrain on RealSyn (ϵ=∞\epsilon=\infty)DP-Syn (ϵ=100\epsilon=100)DP-Syn (ϵ=10\epsilon=10)Direct DP-SGD (ϵ=100\epsilon=100)Direct DP-SGD (ϵ=10\epsilon=10)
GEMINON-2025-09Qwen3-1.7B-Base91.56 ±\pm 1.4587.92 ±\pm 2.113.48 ±\pm 0.261.92 ±\pm 0.113.12 ±\pm 0.171.99 ±\pm 0.27
GEMINON-2025-09Qwen3-4B-Base96.08 ±\pm 1.3396.48 ±\pm 0.395.51 ±\pm 0.092.92 ±\pm 0.004.65 ±\pm 0.132.73 ±\pm 0.07
GEMINON-2025-09Gemma 3 12B-PT96.2596.9016.584.3810.833.81
NEWS-2025-09Qwen3-1.7B-Base43.75 ±\pm 1.42 / 56.55 ±\pm 1.3936.73 ±\pm 0.57 / 47.68 ±\pm 0.6411.25 ±\pm 1.12 / 14.88 ±\pm 0.8810.54 ±\pm 0.93 / 14.11 ±\pm 1.5612.20 ±\pm 0.27 / 16.85 ±\pm 0.419.76 ±\pm 0.27 / 13.45 ±\pm 0.45
NEWS-2025-09Qwen3-4B-Base53.39 ±\pm 0.64 / 68.10 ±\pm 0.5249.94 ±\pm 1.22 / 63.21 ±\pm 1.1718.63 ±\pm 0.63 / 24.17 ±\pm 0.2714.05 ±\pm 0.55 / 18.75 ±\pm 1.0919.40 ±\pm 0.55 / 24.23 ±\pm 0.8414.70 ±\pm 0.27 / 19.17 ±\pm 0.74
NEWS-2025-09Gemma 3 12B-PT59.29 / 77.6854.82 / 72.1424.29 / 33.3915.00 / 21.0724.82 / 34.1118.21 / 24.29

The results are consistent with our main findings across model families and size. Training on the real corpus (Train on Real) achieves high accuracy, while the base model performs near chance (No Training). Non private synthesis (Syn) transfers substantial factual knowledge, despite some degradation compared to Train on Real, while DP synthesis (DP-Syn) performs significantly worse than Syn, even at ϵ=100\epsilon=100. Additionally, direct DP finetuning on the original corpus also performs substantially worse than Train on Real or even Syn, with accuracy only comparable to DP-Syn. This suggests that the knowledge transfer gap is not specific to Gemma 3 1B and 4B, but rather more general across model families and sizes.

W1 (cont.)

We also evaluate NEWS across different support counts and find that accuracy generally increases with support count in all configs, consistent with our findings on Gemma 3 1B and 4B. Additionally, we run a clipping-only ablation on Qwen3-1.7B using GEMINON: the clip-only variant, without Gaussian noise, the model reaches 85.54% test exact-match accuracy after just 1K steps, compared to 3.30% with DP-SGD at ϵ=100\epsilon = 100. This is consistent with Appendix E.1, suggesting that noise, rather than clipping, accounts for most of the degradation.

Experiments across model families and sizes

We further repeat our main experiments in the paper on Gemma 3 1B and 4B using three seeds on both GEMINON and NEWS, using three seeds per config. For each seed and privacy regime (ϵ=∞,100,10\epsilon=\infty,100,10), we finetune a generator, sample a synthetic corpus, and train a downstream evaluator on the corpus. We also report three reference baselines: 1. Train on Real, and 2. Direct DP finetuning on Real. All generator training uses LoRA with rank 128 and all evaluator training uses full parameter finetuning.

Exact-match accuracy (%) on GEMINON-2025-09 and contains / LLM-match accuracy (%) on NEWS-2025-09 for Train on Real, Syn (ϵ=∞\epsilon=\infty), and DP-Syn (ϵ∈{100,10}\epsilon\in\{100,10\}). We evaluate on QAs with support k≈200k\approx200 for GEMINON and k≥200k\ge200 for NEWS. Results are reported as mean ±\pm std across three seeds.

DatasetModelTrain on RealSyn (ϵ=∞\epsilon=\infty)DP-Syn (ϵ=100\epsilon=100)DP-Syn (ϵ=10\epsilon=10)
GEMINON-2025-09Gemma 3 1B-PT87.37 ±\pm 0.8885.56 ±\pm 2.217.05 ±\pm 1.251.88 ±\pm 0.25
GEMINON-2025-09Gemma 3 4B-PT96.99 ±\pm 0.6394.04 ±\pm 1.344.19 ±\pm 0.382.70 ±\pm 0.21
NEWS-2025-09Gemma 3 1B-PT43.16 ±\pm 2.40 / 53.81 ±\pm 2.6943.63 ±\pm 2.06 / 54.64 ±\pm 0.9912.26 ±\pm 0.45 / 16.13 ±\pm 0.1010.54 ±\pm 1.63 / 13.63 ±\pm 1.66
NEWS-2025-09Gemma 3 4B-PT53.09 ±\pm 1.19 / 70.54 ±\pm 0.1851.43 ±\pm 1.46 / 66.25 ±\pm 0.9418.87 ±\pm 1.05 / 25.18 ±\pm 1.7213.63 ±\pm 0.85 / 18.69 ±\pm 1.09

The results are consistent with our main findings. The numbers reported in the paper also fall close to the means over three seeds reported above. Importantly, the variance across seeds is much smaller than the gap between non-private and DP synthesis.

W5: Baseline coverage

Thank you for the suggestion. We emphasize that our evaluation already covers three representative approaches to DP text synthesis: (1) DP finetuning generator and sampling, a standard training-based approach; (2) Private Evolution, a training-free method widely regarded as one of the strongest baselines; and (3) InvisibleInk, a recent inference-time private generation method. We agree that evaluating more methods would further strengthen the paper. However, our primary contribution is a benchmark for measuring corpus-specific knowledge transfer, rather than a comprehensive comparison of DP text synthesis methods. Given our focus on identifying the limitations of the existing eval paradigm, we chose one representative method from each of the three families.

W2/W6: Domain and task coverage

We thank the reviewer for this observation and acknowledge that our current benchmark focuses on factual knowledge acquisition rather than a full range of model capabilities.

Domain coverage:

A central challenge in extending the benchmark to high-stakes domains is to ensure that the model cannot answer the questions without learning from the corpus (access dependency). For such domains, suitable data may be hard to obtain and release publicly. For example, SynBench [1] evaluates nine datasets from healthcare, finance, and law. However, it acknowledges potential pretraining contamination, and its datasets are relatively small (620–10K records), making them less suitable for evaluating the knowledge acquisition of facts supported by hundreds of independent records. We therefore introduce two complementary tracks that approximate the knowledge demands of high-stakes domains, while ensuring that the data and knowledge are fresh and can be easily open-sourced. Specifically, GEMINON provides a procedurally generated domain with novel entity-attribute relationships, allowing us to enforce access dependency; NEWS complements this with naturally occurring text about real news events, whose temporal freshness mitigates data contamination.

[1] SynBench: A Benchmark for Differentially Private Text Generation

W2/W6 (cont.)

Task coverage:

We also agree that short-answer QA does not capture all dimensions of downstream utility, such as multi-hop reasoning and procedural knowledge. As discussed above, evaluating these capabilities cleanly is difficult because of the data contamination, and we leave this for future work. Our goal is to take a first step by isolating factual knowledge acquisition: whether models can learn and recall the corpus-specific facts that were fresh to them. This is particularly relevant to sensitive corpora in the high-stakes domains such as clinical/legal records and internal corporate data, where much of the value lies in information the model has not seen before. Short-answer QA therefore provides a natural, automatically verifiable measure of such knowledge transfer, similar to established benchmarks such as MMLU.

W3: LLM usage for benchmark construction and evaluation

Thank you for raising these concerns. We acknowledge the LLM-assisted benchmark construction may introduce potential biases. However, this practice is commonly used in established benchmarks, e.g., TOFU. Importantly, we emphasize that we take several steps to mitigate such biases:

  • For GEMINON: the underlying entity attributes (as well as the QA) are generated through a controlled procedural construction rather than by an LLM freely. For example, numerical attributes are randomly sampled from our prespecified distributions, and article generation is conditioned on these structured records, with carefully designed prompts to encourage linguistic and stylistic diversity.
  • For NEWS: all articles are real news articles rather than LLM generated text. We apply extensive deduplication and event clustering before using LLMs to derive QA pairs. In our preliminary experiments, we explored several QA generation approaches and ultimately adopted the presented two-stage procedure, i.e., first extracting QA relevant facts under our specified rules, then generating QA pairs from these facts and representative source articles. We further apply multiple rounds of filtering to remove ambiguous QAs and those answerable without access to the source corpus. Full construction details are provided in Appendices H and I.

On LLM-based evaluation:

LLM-as-a-judge is widely adopted in benchmarks, e.g., [1, 2]. In particular, [1] uses an LLM to judge the correctness of short factual QAs, closely matching our setup. More broadly, prior studies have empirically justified the reliability of LLM-based evaluation, e.g., [3, 4].

Moreover, we additionally report two fully deterministic metrics, exact-match and “contains”, and our main conclusions remain consistent across all three metrics (reported in Appendix D). Specifically, we use exact-match as the primary metric for GEMINON, where answers have canonical forms, and LLM-match for NEWS, where exact match and contains can be overly restrictive since a correct answer can be phrased in several ways.

To further verify the reliability of LLM-match accuracy, we compared it against “contains” accuracy across all results in Appendices D.1, D.2, D.4, D.5, and E (namely, Tables 10, 11, 12, 13, 23, and 24). The two metrics rank configurations nearly identically, with Spearman's ρ=0.986\rho =0.986 across 36 distinct configurations on GEMINON, 0.994 across 84 distinct configurations on NEWS, and 0.991 across all 120 configurations.

On NEWS support counts:

We agree that our support count estimates are conservative by construction. However, underestimating support counts cannot explain the observed negative results, and it only makes the failure of DP synthesis even more striking. For example, if a fact is actually supported by 2K articles but our procedure identifies only 400, then failing to recover this fact means the knowledge-transfer gap is larger than what our finding suggests. Moreover, we take considerable care to estimate support counts accurately. Specifically, for each QA, we first retrieve the top candidate articles using embedding similarity. Each candidate is then evaluated in an open-book setting, where the article is provided as context, and is counted as supporting the QA only if the model gets the correct answer. We selected this procedure after exploring alternative approaches, including embedding-based matching and direct prompting, and found it to perform best in our preliminary manual inspections.

[1] The FACTS Grounding Leaderboard: Benchmarking LLMs’ Ability to Ground Responses to Long-Form Input

[2] Measuring short-form factuality in large language models

[3] Judging LLM-as-a-Judge with MT-Bench and Chatbot Arena

[4] Long-form factuality in large language models

Comments on the paper

1 comment
Area_Chair_MFh8OpenReview ↗

Reminder: Please review the author rebuttal before the deadline

Dear reviewers,

The rebuttal period is about to close. If you haven’t responded yet, please read the authors’ rebuttal and leave a brief comment before the deadline.

If you’ve already responded, please disregard this reminder. Thanks for your time, and apologies for the extra email.

Best regards,

Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.