- Rating
- 3(+1)
- Confidence
- 1
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper predicts each token from the other positions of the own noisy input using Leave One Out parameterization. It shows when this preserves the target distribution in jump diffusion, using an analytic correction for continuous updates.
Experiments on a Text8, and ZINC250k show better word-level statistics and more valid, connected molecules in some settings. The benefits depend on the generation process.
Critical strengths
-
A clear theoretical result. The paper shows when LOO predictions can be used for resets without changing the target distributions, and how to recover the full posterior needed for continuous updates. This provides a useful rule for applying LOO correctly.
-
Well-controlled experiments. The main comparisons use matched models and budgets. LOO improves several Text8 metrics and increases valid and connected molecular yield in AJD. These controls make the benefits credible within the studied settings.
Critical weaknesses
-
Unclear connection between the theory and experimental results. I do not understand how the marginal-preservation result explains the observed gains. The theory supports using a full posterior, recovered through Bayesian completion, whereas raw LOO drift need not preserve the prescribed marginals. Yet raw LOO achieves better validity and four-gram agreement in several Text8 settings, including both pure diffusion and repeated-reset diffusion.
-
In my opinion the article has limited experimental scope. The text evaluation uses only short character sequences from Text8, while the molecular evaluation is limited to ZINC250k. This narrow scope leaves it unclear whether the benefits extend to more challenging settings. Evaluation on a larger language benchmark, such as LM1B, would provide stronger evidence.
Questions for authors
Am I correct that, for a fixed corruption process and exact predictors, learning the full posterior directly and recovering it from a LOO predictor through Bayesian completion yield the same posterior? If so, is the benefit of LOO primarily due to how it changes learning under limited model capacity and training budgets? I would appreciate a clearer explanation of this mechanism from theoretical point of view.
2 comments
Official Comment by Reviewer auf5
I thank the authors for their clear and detailed responses to my questions and for the substantial additional experiment. I have correspondingly raised my score. Good luck.
We thank the reviewer for the careful review and constructive comments. We address the three points below, referring to our common response, “Response to all reviewers,” for shared results.
1. Exact posteriors: drift and jumps.
Drift. Yes. Under the fixed, conditionally factorized corruption law, exact LOO prediction with the matching local likelihood recovers the same full posterior as direct prediction (Proposition 3.3), and hence the same drift. Any benefit from learning these drift probabilities through LOO therefore concerns finite models and training budgets: excluding the local noisy state removes a direct denoising shortcut.
Jumps. RJD-BC-D still uses raw LOO predictions for resets, without completion. With exact predictors and full-posterior drift, Proposition 3.2 shows that these LOO resets preserve the same curve of one-time marginals. Thus LOO also changes the jump mechanism even with exact predictors, potentially giving different trajectory laws despite the same marginal curve.
2. Why raw LOO can outperform completion - Connection between theory and experimental results.
We address this in Section 3, “Why can raw LOO outperform completion?”, of our common response, and in Section 4.4 and Appendix G.4 of the revised manuscript.
3. Broader evaluation on LM1B.
We added LM1B, a much larger corpus with an approximately 30,000-token vocabulary. Both raw and completed LOO achieve higher best MAUVE than matched unrestricted predictors (common response, Section 1; paper Section 4.2 and Appendices I.1–I.3). We also added LM1B subject–verb interventions, an external-context diagnostic, and blinded LLM assessments to test contextual dependencies and generation quality (Section 4.4; Appendices I.6–I.7).