- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper proposes Guided Diffusion, a framework for differentially private (DP) synthetic text generation that privatizes data in semantic embedding space rather than token space. The decoding is steered by two inference-time mechanisms: Semantic-aware sampling that draws many candidate sequences per denoising step and keeps the one whose embedding is closest to the target, and Semantic-awarere-masking that scores each token by how much ablating it changes the distance to the target. Abalation results show the effectiveness of both components. Experimental results show that this method outperforms previous PE and DP-SGD fine-tuning options.
Critical strengths
- This paper effectively improves the DP text generation with diffusion model, which is lessly seen in DP text synthesis.
- The writing is clear and easy to follow, also the main idea of the paper: embed privatize guided generation.
- The ablations (Tables 1 and 3) clearly shows that both guidance components matter.
Critical weaknesses
- The privacy-utility trade-off is not really tested. PubMed accuracy is 32.66 / 32.65 / 32.48 at , essentially flat, and likely reflecting a ceiling effect (the non-private bound is 34.60). The paper argues PE is pipeline-limited because it barely improves without DP noise, but the same logic applies to this method. Smaller values (or you can use smaller datasets which are more valnerable to noise) are needed to show the method degrades gracefully with increased privacy requirement.
- I am afraid that the paper lacks comparison to other previous methods that also claims to "steer DLMs using a reference text as a high-level semantic blueprint". I am happy to see experimental comparisons or reasonable relavent discussions on why it should not be considered as a comparison. Therfore, I am afraid that the claim in line 83 "for the first time, exploits the steering capabilities of DLMs to generate coherent text conditioned on semantic embeddings" is a bit too large.
- Experiments are only done on 2 datasets, which seems to be less.
[1] ILRR: Inference-Time Steering Method for Masked Diffusion Language Models
Questions for authors
- Re-masking relies on single-token contribution calculated by leave-one-out (LOO) ablation, which ignores redundancy and interactions and uses an out-of-distribution [MASK] input for the encoder. For example, if there appears both “excellent” and “superb” in , removing each one of them likely results in near-zero . Therefore, it likely that both will be masked-out, even though together they are critical. Please report how LOO rankings compare to a Shapley alternative, or justify the choice.
2 comments
4. LOO and Shapley offer different utility–cost trade-offs.
LOO is a computationally efficient choice, and Shapley can account for token interactions within the same remasking framework. LOO measures a token's contribution given the rest of the candidate, whereas Shapley averages that contribution across token subsets. Two redundant tokens, such as “excellent” and “superb,” can both receive small LOO scores. A diagnostic on the initial blocks of 100 generated ICLR26 texts shows a mean Spearman correlation of 0.555 between LOO and Shapley rankings. We also evaluated a Shapley variant during generation.
In these LLaDA experiments, mask tokens are removed before sentence encoding. The encoder therefore sees text with token deletions, not [MASK] symbols. Both attribution rules use this same deletion operation.
We compare LOO with the Shapley variant at ε=1 and 4, using LLaDA-8B-Instruct and 2,000 generated texts per configuration. Shapley-32 averages contributions over 32 orders of the uncommitted tokens in the current block, while keeping the committed context fixed. Values are mean ± SD (%).
| Dataset / metric | ε | Guided-Diffusion (LOO, main results) | Shapley-32 |
|---|---|---|---|
| ICLR26: macro-F1 | 1 | 24.85 ± 2.80 | 26.04 ± 2.91 |
| ICLR26: macro-F1 | 4 | 27.66 ± 2.19 | 24.06 ± 5.94 |
| PubMed: next-token accuracy | 1 | 32.68 ± 0.08 | 32.54 ± 0.08 |
| PubMed: next-token accuracy | 4 | 32.48 ± 0.06 | 32.48 ± 0.06 |
The Shapley results show no consistent utility advantage over LOO: the ICLR26 mean is slightly higher at ε=1 but lower at ε=4, while PubMed accuracy remains close.
LOO also reduces generation time. LOO also achieves a speedup of about 4.83–4.95× over Shapley-32 under the same execution environment.
These results support LOO as an efficient choice with comparable or even better utility than Shapley.
[1] Eden Avrahami and Eliya Nachmani. ILRR: Inference-Time Steering Method for Masked Diffusion Language Models. AACL-IJCNLP, 2026.
[2] Tejomay Kishor Padole, Suyash P. Awate, and Pushpak Bhattacharyya. Improving Text Style Transfer Using Masked Diffusion Language Models with Inference-Time Scaling. ECAI, 2025.
[3] Chulin Xie et al. Differentially Private Synthetic Data via Foundation Model APIs 2: Text. ICML, 2024.
Thank you for the constructive comments. We address the major concerns below.
1. Smaller privacy budgets.
A flat curve is expected when the synthesis method is the main bottleneck. Our goal is to improve synthesis quality and raise utility under privacy constraints. This behavior is visible for both AugPE and Guided-Diffusion on PubMed: increasing ε brings little improvement, but the two methods achieve very different utility levels. The most important comparison is the utility achieved at a given privacy budget.
Following the reviewer's suggestion, we extend the evaluation to ε=0.1 and 0.5. Both methods use LLaDA-8B-Instruct and generate 2,000 texts per configuration. Utility is measured by macro-F1 on ICLR26 and next-token accuracy on PubMed (%, mean ± SD).
On PubMed, both curves are nearly flat, yet Guided-Diffusion exceeds AugPE by more than 10 accuracy points at every budget. This demonstrates the value of improving synthesis in a setting where reducing privacy noise alone brings little benefit.
The extended evaluation also shows a clear trade-off on the smaller ICLR26 dataset: Guided-Diffusion's macro-F1 decreases gradually from 27.66% at ε=4 to 21.28% at ε=0.1, while remaining above AugPE at every budget. This pattern is consistent with the smaller aggregation groups: ICLR26 has 8,396 reviews across five classes, with only 145–3,307 examples per class, whereas PubMed has 75,316 abstracts. Smaller groups are more sensitive to privacy noise, and macro-F1 gives equal weight to small and large classes.
2. Comparison with ILRR.
Our contribution is direct generation from semantic embeddings. ILRR [1] extracts representations from a reference text at each denoising step and uses them to steer the DLM's internal activations. As discussed in our introduction, aggregating private texts into a DP reference text is challenging, if not impossible. Extending ILRR directly to DP synthetic text generation is therefore not straightforward.
Our method, in contrast, takes a target in an independent sentence embedding space and guides generation through semantic candidate selection and contribution-based remasking. This is the scope of the contribution in line 83: generation conditioned on external semantic embeddings, instead of reference-text steering in general.
We also added a comparison with SVDD [2], using the same DLM backbones and centroid inputs from the ε=4 setting. Each method generates 2,000 texts. Results are mean ± SD (%) over five random seeds.
Guided-Diffusion achieves higher utility on both datasets with both backbones, providing a direct comparison against an existing DLM steering method under shared semantic guidance.
3. Full-scale Yelp evaluation.
We added Yelp using AugPE's full 1,939,290-review training split and its 5,000-example validation and test splits [3]. Both methods use LLaDA-8B-Instruct and generate 2,000 texts with the same allocation across 50 rating conditions. The rating-prediction macro-F1 results (%, mean ± SD) are:
Consistent with the other two datasets, Guided-Diffusion achieves better downstream utility at different privacy budgets on Yelp, further demonstrating the generalizability of our framework.