acceptodds
Under review as a conference paper at ICLR 2027

Guided Diffusion for Trustworthy Synthetic Text Generation

Abstract

The rapid evolution of Large Language Models is fueled by massive datasets, yet this reliance poses severe privacy risks, as models can inadvertently memorize and even directly output sensitive training data. Synthetic text generation offers a promising solution to this challenge; however, existing methods struggle to reconcile rigorous privacy guarantees with high data utility. In this paper, we propose a new paradigm for private synthetic text generation: Guided Diffusion. Leveraging the unique steerability of Diffusion Language Models, we introduce novel sampling and re-masking procedures that actively guide the iterative denoising steps. These mechanisms steer the generative process to reconstruct fluent text that adheres to a target semantic distribution. We demonstrate that our framework is naturally compatible with Differential Privacy (DP): once the semantic embeddings of the private corpus are protected, the DP guarantee carries over to the generated text by post-processing. Empirically, we show that our approach generates high-quality synthetic text with precise semantic alignment. Furthermore, we validate its effectiveness in DP synthetic text generation: across two datasets and two diffusion backbones, it outperforms Private Evolution and DP-SGD-based fine-tuning in distributional fidelity and downstream utility, while generating significantly faster than Private Evolution.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
differential privacy, synthetic text generation, differential trust, diffusion language model

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_C7F1OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Guided Diffusion, a framework for differentially private (DP) synthetic text generation that privatizes data in semantic embedding space rather than token space. The decoding is steered by two inference-time mechanisms: Semantic-aware sampling that draws many candidate sequences per denoising step and keeps the one whose embedding is closest to the target, and Semantic-awarere-masking that scores each token by how much ablating it changes the distance to the target. Abalation results show the effectiveness of both components. Experimental results show that this method outperforms previous PE and DP-SGD fine-tuning options.

Critical strengths
  1. This paper effectively improves the DP text generation with diffusion model, which is lessly seen in DP text synthesis.
  2. The writing is clear and easy to follow, also the main idea of the paper: embed →\rightarrow privatize →\rightarrow guided generation.
  3. The ablations (Tables 1 and 3) clearly shows that both guidance components matter.
Critical weaknesses
  1. The privacy-utility trade-off is not really tested. PubMed accuracy is 32.66 / 32.65 / 32.48 at ε=1/2/4\varepsilon = 1 / 2 / 4, essentially flat, and likely reflecting a ceiling effect (the non-private bound is 34.60). The paper argues PE is pipeline-limited because it barely improves without DP noise, but the same logic applies to this method. Smaller ε\varepsilon values (or you can use smaller datasets which are more valnerable to noise) are needed to show the method degrades gracefully with increased privacy requirement.
  2. I am afraid that the paper lacks comparison to other previous methods that also claims to "steer DLMs using a reference text as a high-level semantic blueprint". I am happy to see experimental comparisons or reasonable relavent discussions on why it should not be considered as a comparison. Therfore, I am afraid that the claim in line 83 "for the first time, exploits the steering capabilities of DLMs to generate coherent text conditioned on semantic embeddings" is a bit too large.
  3. Experiments are only done on 2 datasets, which seems to be less.

[1] ILRR: Inference-Time Steering Method for Masked Diffusion Language Models

Questions for authors
  1. Re-masking relies on single-token contribution calculated by leave-one-out (LOO) ablation, which ignores redundancy and interactions and uses an out-of-distribution [MASK] input for the encoder. For example, if there appears both “excellent” and “superb” in x^⋆\hat{x}^\star, removing each one of them likely results in near-zero Δi\Delta_{i}. Therefore, it likely that both will be masked-out, even though together they are critical. Please report how LOO rankings compare to a Shapley alternative, or justify the choice.
2 comments

Thank you for the constructive comments. We address the major concerns below.


1. Smaller privacy budgets.

A flat curve is expected when the synthesis method is the main bottleneck. Our goal is to improve synthesis quality and raise utility under privacy constraints. This behavior is visible for both AugPE and Guided-Diffusion on PubMed: increasing ε brings little improvement, but the two methods achieve very different utility levels. The most important comparison is the utility achieved at a given privacy budget.

Following the reviewer's suggestion, we extend the evaluation to ε=0.1 and 0.5. Both methods use LLaDA-8B-Instruct and generate 2,000 texts per configuration. Utility is measured by macro-F1 on ICLR26 and next-token accuracy on PubMed (%, mean ± SD).

εICLR26: AugPEICLR26: Guided-DiffusionPubMed: AugPEPubMed: Guided-Diffusion
0.115.95 ± 2.5021.28 ± 3.0821.98 ± 0.1232.46 ± 0.04
0.514.40 ± 2.2822.23 ± 1.5322.31 ± 0.1232.54 ± 0.06
113.34 ± 1.3824.85 ± 2.8022.15 ± 0.0832.68 ± 0.08
212.94 ± 0.5627.49 ± 3.1922.41 ± 0.0432.65 ± 0.06
413.84 ± 0.8127.66 ± 2.1922.37 ± 0.1232.48 ± 0.06

On PubMed, both curves are nearly flat, yet Guided-Diffusion exceeds AugPE by more than 10 accuracy points at every budget. This demonstrates the value of improving synthesis in a setting where reducing privacy noise alone brings little benefit.

The extended evaluation also shows a clear trade-off on the smaller ICLR26 dataset: Guided-Diffusion's macro-F1 decreases gradually from 27.66% at ε=4 to 21.28% at ε=0.1, while remaining above AugPE at every budget. This pattern is consistent with the smaller aggregation groups: ICLR26 has 8,396 reviews across five classes, with only 145–3,307 examples per class, whereas PubMed has 75,316 abstracts. Smaller groups are more sensitive to privacy noise, and macro-F1 gives equal weight to small and large classes.


2. Comparison with ILRR.

Our contribution is direct generation from semantic embeddings. ILRR [1] extracts representations from a reference text at each denoising step and uses them to steer the DLM's internal activations. As discussed in our introduction, aggregating private texts into a DP reference text is challenging, if not impossible. Extending ILRR directly to DP synthetic text generation is therefore not straightforward.

Our method, in contrast, takes a target in an independent sentence embedding space and guides generation through semantic candidate selection and contribution-based remasking. This is the scope of the contribution in line 83: generation conditioned on external semantic embeddings, instead of reference-text steering in general.

We also added a comparison with SVDD [2], using the same DLM backbones and centroid inputs from the ε=4 setting. Each method generates 2,000 texts. Results are mean ± SD (%) over five random seeds.

BackboneDataset / metricSVDDGuided-Diffusion
LLaDA-8B-InstructPubMed: next-token accuracy28.15 ± 0.1032.48 ± 0.06
LLaDA-8B-InstructICLR26: macro-F121.26 ± 3.0127.66 ± 2.19
Dream-7BPubMed: next-token accuracy27.15 ± 0.0632.94 ± 0.05
Dream-7BICLR26: macro-F126.98 ± 2.8828.41 ± 3.17

Guided-Diffusion achieves higher utility on both datasets with both backbones, providing a direct comparison against an existing DLM steering method under shared semantic guidance.


3. Full-scale Yelp evaluation.

We added Yelp using AugPE's full 1,939,290-review training split and its 5,000-example validation and test splits [3]. Both methods use LLaDA-8B-Instruct and generate 2,000 texts with the same allocation across 50 rating conditions. The rating-prediction macro-F1 results (%, mean ± SD) are:

εAugPEGuided-Diffusion
149.67 ± 3.5052.38 ± 2.06
252.66 ± 3.0052.76 ± 0.77
451.97 ± 1.3654.32 ± 0.72

Consistent with the other two datasets, Guided-Diffusion achieves better downstream utility at different privacy budgets on Yelp, further demonstrating the generalizability of our framework.

4. LOO and Shapley offer different utility–cost trade-offs.

LOO is a computationally efficient choice, and Shapley can account for token interactions within the same remasking framework. LOO measures a token's contribution given the rest of the candidate, whereas Shapley averages that contribution across token subsets. Two redundant tokens, such as “excellent” and “superb,” can both receive small LOO scores. A diagnostic on the initial blocks of 100 generated ICLR26 texts shows a mean Spearman correlation of 0.555 between LOO and Shapley rankings. We also evaluated a Shapley variant during generation.

In these LLaDA experiments, mask tokens are removed before sentence encoding. The encoder therefore sees text with token deletions, not [MASK] symbols. Both attribution rules use this same deletion operation.

We compare LOO with the Shapley variant at ε=1 and 4, using LLaDA-8B-Instruct and 2,000 generated texts per configuration. Shapley-32 averages contributions over 32 orders of the uncommitted tokens in the current block, while keeping the committed context fixed. Values are mean ± SD (%).

Dataset / metricεGuided-Diffusion (LOO, main results)Shapley-32
ICLR26: macro-F1124.85 ± 2.8026.04 ± 2.91
ICLR26: macro-F1427.66 ± 2.1924.06 ± 5.94
PubMed: next-token accuracy132.68 ± 0.0832.54 ± 0.08
PubMed: next-token accuracy432.48 ± 0.0632.48 ± 0.06

The Shapley results show no consistent utility advantage over LOO: the ICLR26 mean is slightly higher at ε=1 but lower at ε=4, while PubMed accuracy remains close.

LOO also reduces generation time. LOO also achieves a speedup of about 4.83–4.95× over Shapley-32 under the same execution environment.

These results support LOO as an efficient choice with comparable or even better utility than Shapley.


[1] Eden Avrahami and Eliya Nachmani. ILRR: Inference-Time Steering Method for Masked Diffusion Language Models. AACL-IJCNLP, 2026.

[2] Tejomay Kishor Padole, Suyash P. Awate, and Pushpak Bhattacharyya. Improving Text Style Transfer Using Masked Diffusion Language Models with Inference-Time Scaling. ECAI, 2025.

[3] Chulin Xie et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text. ICML, 2024.

Reviewer_9XLPOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes synthetic text generation by guiding masked diffusion language models toward target semantic embeddings. Its two components select candidate sequences by semantic alignment and re-mask tokens according to their contribution to that alignment. For private generation, it first releases differentially private embedding centroids, then generates text through post-processing. Experiments on PubMed and ICLR26, using two diffusion backbones and four seeds, report improved distributional fidelity and generally stronger downstream utility than the evaluated baselines.

Critical strengths
  1. Both candidate selection and contribution-based re-masking have measurable effects. On PubMed, removing these components reduces downstream accuracy from approximately 32.7% to 18.0% and 26.7%.

  2. Using the same diffusion backbones across methods helps control generator capability. The method improves every reported fidelity metric across the four dataset–backbone settings and achieves the strongest downstream utility in three settings.

Critical weaknesses

Prior work already explores verifier-guided candidate selection in masked diffusion and embedding inversion with conditional masked diffusion models [1]. The proposed semantic remasking may provide a distinct contribution, but the paper needs to isolate its advantages over these approaches. EPSVec [2] is another related work because both methods encode private dataset information into vector representations, privatize those representations, and use them to guide text generation. Later works like AugPE [3] also claim improvements over the PE baseline. While their work does not use sentence-embeddings, they claim to have considerable improvements over PE. How do you think these works compare? Comparisons primarily against PE and DP-SGD leave unclear whether the proposed approach offers meaningful improvements over these alternatives under comparable privacy and computational budgets.

The evaluation does not adequately substantiate several central claims. Similar utility across \epsilon =1, 2, and 4 does not establish negligible privacy cost without comparing the proposed method using private versus non-private centroids. The reported 34.6% “non-private upper bound” also needs clarification: prior Aug-PE results distinguish 34.6% using 2,000 real examples from 47.6% using the full corpus, which materially changes the interpretation of the remaining utility gap. Finally, MAUVE settings, including the scaling factor, feature encoder, truncation length, and clustering configuration, are unspecified. MAUVE is particularly sensitive to the scaling factor, which can largely sway results. This omission prevents reproducing the fidelity results and assessing their sensitivity.

[1] Padole, Tejomay Kishor, Suyash P. Awate, and Pushpak Bhattacharyya. "Improving text style transfer using masked diffusion language models with inference-time scaling." ECAI 2025: 28th European Conference on Artificial Intelligence 25-30 October 2025, Bologna, Italy-Including 14th Conference on Prestigious Applications of Intelligent Systems (PAIS 2025) Proceedings. 1 Oliver's Yard, 55 City Road, London, EC1Y 1SP: SAGE Publications, 2025.

[2] Banayeeanzade, Amin, et al. "EPSVec: Efficient and private synthetic data generation via dataset vectors." arXiv preprint arXiv:2602.21218 (2026).

[3] Xie, Chulin, et al. "Differentially private synthetic data via foundation model apis 2: Text." arXiv preprint arXiv:2403.01749 (2024).

Questions for authors

See Weaknesses.

2 comments

Thank you for the constructive comments. We address the major concerns below.


1. Semantic remasking is essential beyond existing diffusion steering.

SVDD [1] guides generation by selecting candidate texts. Our semantic remasking also decides which tokens to preserve based on their importance to the target meaning. To evaluate this contribution, we adapt SVDD's decoder to our synthesis task and compare it with Guided-Diffusion using the same DLM backbones and centroid inputs from our ε=4 setting. Each method generates 2,000 texts. Results are reported as mean ± SD (%) over five random seeds.

BackboneDataset / metricSVDDGuided-Diffusion
LLaDA-8B-InstructPubMed: next-token accuracy28.15 ± 0.1032.48 ± 0.06
LLaDA-8B-InstructICLR26: macro-F121.26 ± 3.0127.66 ± 2.19
Dream-7BPubMed: next-token accuracy27.15 ± 0.0632.94 ± 0.05
Dream-7BICLR26: macro-F126.98 ± 2.8828.41 ± 3.17

Guided-Diffusion achieves higher utility on both datasets with both backbones. Table 1 in the paper already shows the contribution of semantic remasking on LLaDA. Here, we also test both backbones on 400 public IMDb/Yahoo texts, removing only semantic remasking while keeping candidate selection unchanged. We measure similarity using an independent MPNet encoder; Table 1 uses sentence-T5 for Yahoo and STSB-RoBERTa for IMDb. Cosine similarity scores have different scales across these embedding spaces, so their absolute values are not directly comparable. Higher is better within each evaluation.

MethodLLaDADream
Without semantic remasking0.17970.1731
Guided-Diffusion0.25900.2903

The drop for both models shows that preserving semantically important tokens adds value beyond candidate selection. Guided-Diffusion is also 2.14–3.89× faster than SVDD across various models and datasets.


2. Relation to EPSVec and AugPE.

EPSVec [2] and our method share the idea of releasing DP vectors and then generating text from them. Our contribution is a method for reconstructing text from explicit semantic targets. EPSVec computes an average steering direction for each attribute and layer in the generator's activation space and injects it during decoding. Our method uses multiple centroids to represent different semantic groups (§4.1), and each generated text is guided toward a chosen centroid. Candidate selection checks whether the text matches that target, and semantic remasking preserves the tokens that contribute to the match. This gives us explicit control over which semantic group each text represents. The centroids also lie in an independent sentence encoder's space, so the same released targets can guide different DLM backbones without further queries to the private data or additional privacy loss.

Table 3 shows why these decoding mechanisms matter. On PubMed, removing semantic candidate selection reduces accuracy from 32.66% to 18.01%, and removing semantic remasking reduces it to 26.73%. These ablations establish the contribution of our decoder. The new SVDD comparison further evaluates it against existing DLM steering method.

Our PE baseline already uses AugPE [3]. It is built on the official AugPE implementation with the same DLM generators as Guided-Diffusion to ensure fair comparison. Table 2 therefore already includes the requested comparison while controlling for backbone differences.


3. Privacy cost.

Guided-Diffusion incurs only a small utility cost from privacy noise. We added the requested comparison with ε=∞, where Guided-Diffusion uses centroids without privacy noise. The table below compares this non-private reference with ε=0.1–4 on PubMed. Each experiment generates 2,000 texts. Values are downstream next-token accuracy (%, mean ± SD).

εLLaDA-8B-InstructDream-7B
0.132.46 ± 0.0432.45 ± 0.09
0.532.54 ± 0.0632.39 ± 0.11
132.68 ± 0.0832.89 ± 0.08
232.65 ± 0.0632.75 ± 0.05
432.48 ± 0.0632.94 ± 0.05
∞32.70 ± 0.0632.46 ± 0.07

The original 34.6% non-private reference in our paper evaluates the downstream task directly on 2,000 real private examples, serving as an empirical upper bound at the same training-set size. This reference bypasses both guided generation and privacy noise. Our new ε=∞ results remove DP noise while retaining the Guided-Diffusion framework, allowing us to assess the utility cost of privacy noise within this framework. These results support our finding that Guided-Diffusion incurs only a small utility cost from privacy noise.

4. The 34.6% reference uses 2,000 private data samples.

34.6% is the result for 2,000 real PubMed examples, while 47.6% uses the full 75,316-example corpus [3]. The 34.6% figure is therefore a real-data reference at the same training-set size as our 2,000 synthetic examples, instead of an upper bound for training on the full corpus. Our main empirical conclusion comes from comparing different synthesis methods using the same backbones: better fidelity in all four dataset–backbone settings and the best downstream utility in three of them.


5. The MAUVE advantage persists across scaling factors.

We use the same MAUVE evaluation settings across methods to ensure fair comparisons. Changing the scale parameter will change the number range, but the ranking among different methods stay the same. Below we show that Guided-Diffusion outperforms AugPE at different scales across the ε=0.1–4 range.

For PubMed, we use all-mpnet-base-v2 features, PCA retaining 90% of variance, and 200 k-means clusters. The reported scaling factor is c=0.8. We also evaluate c=5 using the same features and clustering.

Both methods use LLaDA-8B-Instruct. We also report downstream next-token accuracy (%). Values are mean ± SD.

εAugPE, c=0.8Guided-Diffusion, c=0.8AugPE, c=5Guided-Diffusion, c=5AugPE accuracy (%)Guided-Diffusion accuracy (%)
0.10.7256 ± 0.00460.8448 ± 0.00240.0196 ± 0.00120.0879 ± 0.002721.98 ± 0.1232.46 ± 0.04
0.50.7294 ± 0.00450.8605 ± 0.00330.0206 ± 0.00120.1076 ± 0.004422.31 ± 0.1232.54 ± 0.06
10.7318 ± 0.00400.8637 ± 0.00170.0212 ± 0.00110.1129 ± 0.002722.15 ± 0.0832.68 ± 0.08
20.7363 ± 0.00200.8636 ± 0.00540.0224 ± 0.00060.1126 ± 0.007822.41 ± 0.0432.65 ± 0.06
40.7381 ± 0.00590.8554 ± 0.00270.0229 ± 0.00170.1006 ± 0.003222.37 ± 0.1232.48 ± 0.06

The absolute MAUVE scores change substantially, but the advantage over AugPE holds at both scales across all five budgets. The accuracy advantage exceeds 10 percentage points at every budget, providing separate evidence that does not depend on MAUVE.


[1] Tejomay Kishor Padole, Suyash P. Awate, and Pushpak Bhattacharyya. Improving Text Style Transfer Using Masked Diffusion Language Models with Inference-Time Scaling. ECAI, 2025.

[2] Amin Banayeeanzade et al. EPSVec: Efficient and Private Synthetic Data Generation via Dataset Vectors. ICML, 2026.

[3] Chulin Xie et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text. ICML, 2024.

Reviewer_n7rmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper decomposes the generation of private synthetic text into two steps: preserving semantic embeddings (via DP or DSI to release trusted centroids in the semantic embedding space) and reconstructing text from these embeddings using a diffusion language model. By employing semantic-aware sampling and semantic-aware re-masking—techniques rooted in Guided Diffusion—the method recovers high-quality text from the embeddings. This approach simultaneously improves distribution fidelity, downstream utility, and generation efficiency across the ICLR26 and PubMed datasets.

Critical strengths
  1. This paper proposes "semantic-aware sampling" and "semantic-aware re-masking," which explicitly inject target semantic embeddings into each step of a masked Diffusion Language Model (DLM).
  2. The paper demonstrates that as long as the embedding aggregation satisfies Differential Privacy (DP), the subsequent Guided Diffusion process can inherit this DP guarantee as a form of post-processing.
  3. The same pipeline is extended to DSI (Differentiable Search Index), utilizing the concept of "safe-reference indistinguishability" to formalize the handling of data poisoning and memorization.
Critical weaknesses
  1. The experimental evaluation in this paper is insufficient, as it lacks results on large-scale datasets (e.g., Yelp). I suggest that the authors align their experimental results with those of Aug-PE. Furthermore, the range of ϵ\epsilon (privacy budget) values ​​tested is too limited to demonstrate that Guided Diffusion possesses comprehensive advantages.

  2. It is recommended to explicitly specify the code repositories used for the PE implementations; although both may be "official" implementations, Aug-PE is significantly more efficient than DPSDA.

  3. The primary strength of the PE family of methods lies in leveraging the capabilities of state-of-the-art closed-source models to enhance the utility of differentially private (DP) synthetic text. However, this paper substitutes the original models with DLM (Diffusion Language Model) and fails to provide a comparison between open-source and closed-source LLMs. Consequently, the results only show that Guided Diffusion outperforms PE when applied to DLM; they do not establish superiority over the original Aug-PE framework.

  4. If DP-SGD-FT were implemented using a cutting-edge model like Qwen-3.5, its performance would likely improve substantially. Relying on GPT-2 Large and Llama-3-8B is outdated and fails to represent the current state-of-the-art.

Questions for authors

See Critical Weaknesses.

2 comments

Thank you for the constructive comments. We have completed the requested experiments and address the main concerns below.

1. Our PE baseline already uses AugPE.

It is built on the official AI-secure/aug-pe implementation with our DLM inference adapter. Section 4.2 keeps the generator fixed across AugPE, DP-SGD+LoRA, and Guided-Diffusion to compare the synthesis methods fairly while controlling for backbone differences. Under this setup, the reported ICLR26 generation times are 2.74 hours for Guided-Diffusion versus 12.20 hours for four-round AugPE on eight A100s, a 4.45× speedup. We agree that access to strong proprietary models is an advantage of AugPE's API interface. Our experiments show the benefit of the data synthesis method itself under the same generator backbone. Also, we believe it is important to explore more efficient and effective training-free alternatives using open-source models, especially as the DLM community continues to improve these models' capabilities.


2. Full-scale Yelp evaluation.

We use AugPE's full 1,939,290-review training split and its 5,000-example validation and test splits. Both methods use LLaDA-8B-Instruct and generate 2,000 texts, with the same allocation across 50 rating conditions. The macro-F1 results (%) for rating prediction are:

εAugPEGuided-Diffusion
149.67 ± 3.5052.38 ± 2.06
252.66 ± 3.0052.76 ± 0.77
451.97 ± 1.3654.32 ± 0.72

Guided-Diffusion consistently performs better than AugPE. It is also well suited to larger datasets. At a fixed clustering resolution and privacy budget, more records per cluster improve the accuracy of semantic centroids and reduce the noise required to privatize their means. Once these centroids are released, generating a fixed number of synthetic texts requires no further access to the private corpus, and the cost of this generation step is independent of the corpus size.


3. A wider privacy budget range.

We extend the evaluation to ε=0.1 and 0.5 and report the complete ε=0.1–4 range below. Both methods use LLaDA-8B-Instruct and generate 2,000 texts per configuration. Utility is measured by macro-F1 on ICLR26 and next-token accuracy on PubMed (%, mean ± SD).

εICLR26: AugPEICLR26: Guided-DiffusionPubMed: AugPEPubMed: Guided-Diffusion
0.115.95 ± 2.5021.28 ± 3.0821.98 ± 0.1232.46 ± 0.04
0.514.40 ± 2.2822.23 ± 1.5322.31 ± 0.1232.54 ± 0.06
113.34 ± 1.3824.85 ± 2.8022.15 ± 0.0832.68 ± 0.08
212.94 ± 0.5627.49 ± 3.1922.41 ± 0.0432.65 ± 0.06
413.84 ± 0.8127.66 ± 2.1922.37 ± 0.1232.48 ± 0.06

Guided-Diffusion achieves better utility than AugPE at every privacy budget on both datasets. AugPE benefits little from increasing ε: PubMed accuracy stays within 21.98–22.41%, and ICLR26 shows no systematic improvement. This supports our interpretation that AugPE is mainly limited by its candidate generation and variation process, instead of selection noise. Even with more accurate private selection, performance still depends on the quality and semantic coverage of the generated candidate pool.

Guided-Diffusion responds differently to ε on the two datasets: ICLR26 macro-F1 decreases from 27.66% at ε=4 to 21.28% at ε=0.1, whereas PubMed accuracy remains within 32.46–32.68% across the entire range. This difference is consistent with the group sizes and downstream tasks. ICLR26 contains 8,396 reviews split into five classes, with only 145–3,307 examples per class; PubMed contains 75,316 abstracts. Since the sensitivity of a mean decreases as more records are aggregated, the smaller ICLR26 groups are more affected by privacy noise. Macro-F1 also gives equal weight to every rating class, so drops in performance on small classes affect the overall score. PubMed's larger corpus supports more stable aggregation.

4. The Qwen3.5 baseline.

Following the reviewer's suggestion, we also fine-tuned the official post-trained Qwen3.5-9B checkpoint using DP-SGD+LoRA at ε=1,2,4 on both datasets. The comparison with AugPE, DP-SGD+LoRA, and Guided-Diffusion is shown below. All configurations generate 2,000 texts and use the same downstream task and held-out split within each dataset. Values are mean ± SD (%).

ICLR26: macro-F1

BackboneMethodε=1ε=2ε=4
LLaDA-8B-InstructAugPE13.34 ± 1.3812.94 ± 0.5613.84 ± 0.81
LLaDA-8B-InstructDP-SGD+LoRA20.12 ± 4.1615.89 ± 3.6517.10 ± 6.01
LLaDA-8B-InstructGuided-Diffusion24.85 ± 2.8027.49 ± 3.1927.66 ± 2.19
Dream-7BAugPE12.49 ± 0.5711.84 ± 1.3213.38 ± 1.16
Dream-7BDP-SGD+LoRA29.23 ± 3.3725.86 ± 4.9624.69 ± 1.68
Dream-7BGuided-Diffusion28.68 ± 2.4028.24 ± 2.5928.41 ± 3.17
Qwen3.5-9BDP-SGD+LoRA29.47 ± 2.7729.96 ± 2.7728.44 ± 2.68

PubMed: next-token accuracy

BackboneMethodε=1ε=2ε=4
LLaDA-8B-InstructAugPE22.15 ± 0.0822.41 ± 0.0422.37 ± 0.12
LLaDA-8B-InstructDP-SGD+LoRA25.45 ± 0.1126.05 ± 0.0526.00 ± 0.08
LLaDA-8B-InstructGuided-Diffusion32.68 ± 0.0832.65 ± 0.0632.48 ± 0.06
Dream-7BAugPE20.46 ± 0.1820.60 ± 0.1220.66 ± 0.09
Dream-7BDP-SGD+LoRA25.16 ± 0.1424.86 ± 0.0924.88 ± 0.07
Dream-7BGuided-Diffusion32.89 ± 0.0832.75 ± 0.0532.94 ± 0.05
Qwen3.5-9BDP-SGD+LoRA26.11 ± 0.0625.94 ± 0.0526.13 ± 0.08

Guided-Diffusion with Dream remains close to Qwen on ICLR26 without private fine-tuning: the gaps are only 0.79, 1.72, and 0.03 macro-F1 points at ε=1,2,4, respectively.

Release dates also matter when interpreting the ICLR26 results. Its reviews became public on November 11, 2025, after the release of LLaDA and Dream model checkpoints, but before Qwen3.5-9B's release on March 2, 2026. As discussed in Section 4.2, our DLM backbones were released before these reviews; the same is not true of Qwen. Possible overlap with Qwen's training data could therefore affect the fairness of this comparison.

PubMed provides a complementary comparison: its 2023 corpus was released before all three backbones, removing this difference in release order. Guided-Diffusion outperforms Qwen by more than six accuracy points with both DLM backbones across the tested privacy budgets. In particular, Guided-Diffusion with Dream achieves 32.89/32.75/32.94, compared with Qwen's 26.11/25.94/26.13. These results show a clear PubMed utility advantage for Guided-Diffusion and close ICLR26 performance with Dream, without any model training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.