acceptodds
Under review as a conference paper at ICLR 2027

Learning from Context: Leave-One-Out Prediction in Jump Diffusions

Abstract

Continuous diffusion models generate discrete data through stochastic dynamics in an embedding space. We consider jump diffusions that additionally reset coordinates to embeddings of categorical values. A coordinate's own corrupted state can make prediction easy, potentially reducing the pressure to learn dependencies between coordinates. We study leave-one-out (LOO) prediction, in which each coordinate is predicted from the corrupted states of the other coordinates. We show that, under stated assumptions, jump diffusions with complete coordinate resets and state-independent clocks preserve the prescribed marginals when reset destinations use exact LOO probabilities and the drift uses the full endpoint posterior. We further show that this drift can be constructed from LOO probabilities by analytic likelihood completion without additional network evaluations. We evaluate unrestricted, LOO, and completed predictions for character-level text generation on Text8 and molecular graph generation on ZINC250k. Compared with unrestricted prediction, LOO improves lexical statistics on Text8 and the fraction of valid, connected molecules for absorbing jump diffusion on ZINC250k. For molecular pure diffusion, completion outperforms using LOO probabilities directly in the drift.

Primary area:
generative models
Keywords:
discrete generative modeling, jump diffusion

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_auf5OpenReview ↗
Rating
3(+1)
Confidence
1
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper predicts each token from the other positions of the own noisy input using Leave One Out parameterization. It shows when this preserves the target distribution in jump diffusion, using an analytic correction for continuous updates.

Experiments on a Text8, and ZINC250k show better word-level statistics and more valid, connected molecules in some settings. The benefits depend on the generation process.

Critical strengths
  1. A clear theoretical result. The paper shows when LOO predictions can be used for resets without changing the target distributions, and how to recover the full posterior needed for continuous updates. This provides a useful rule for applying LOO correctly.

  2. Well-controlled experiments. The main comparisons use matched models and budgets. LOO improves several Text8 metrics and increases valid and connected molecular yield in AJD. These controls make the benefits credible within the studied settings.

Critical weaknesses
  1. Unclear connection between the theory and experimental results. I do not understand how the marginal-preservation result explains the observed gains. The theory supports using a full posterior, recovered through Bayesian completion, whereas raw LOO drift need not preserve the prescribed marginals. Yet raw LOO achieves better validity and four-gram agreement in several Text8 settings, including both pure diffusion and repeated-reset diffusion.

  2. In my opinion the article has limited experimental scope. The text evaluation uses only short character sequences from Text8, while the molecular evaluation is limited to ZINC250k. This narrow scope leaves it unclear whether the benefits extend to more challenging settings. Evaluation on a larger language benchmark, such as LM1B, would provide stronger evidence.

Questions for authors

Am I correct that, for a fixed corruption process and exact predictors, learning the full posterior directly and recovering it from a LOO predictor through Bayesian completion yield the same posterior? If so, is the benefit of LOO primarily due to how it changes learning under limited model capacity and training budgets? I would appreciate a clearer explanation of this mechanism from theoretical point of view.

2 comments

We thank the reviewer for the careful review and constructive comments. We address the three points below, referring to our common response, “Response to all reviewers,” for shared results.

1. Exact posteriors: drift and jumps.

Drift. Yes. Under the fixed, conditionally factorized corruption law, exact LOO prediction with the matching local likelihood recovers the same full posterior as direct prediction (Proposition 3.3), and hence the same drift. Any benefit from learning these drift probabilities through LOO therefore concerns finite models and training budgets: excluding the local noisy state removes a direct denoising shortcut.

Jumps. RJD-BC-D still uses raw LOO predictions for resets, without completion. With exact predictors and full-posterior drift, Proposition 3.2 shows that these LOO resets preserve the same curve of one-time marginals. Thus LOO also changes the jump mechanism even with exact predictors, potentially giving different trajectory laws despite the same marginal curve.

2. Why raw LOO can outperform completion - Connection between theory and experimental results.

We address this in Section 3, “Why can raw LOO outperform completion?”, of our common response, and in Section 4.4 and Appendix G.4 of the revised manuscript.

3. Broader evaluation on LM1B.

We added LM1B, a much larger corpus with an approximately 30,000-token vocabulary. Both raw and completed LOO achieve higher best MAUVE than matched unrestricted predictors (common response, Section 1; paper Section 4.2 and Appendices I.1–I.3). We also added LM1B subject–verb interventions, an external-context diagnostic, and blinded LLM assessments to test contextual dependencies and generation quality (Section 4.4; Appendices I.6–I.7).

Reviewer_auf5OpenReview ↗

Official Comment by Reviewer auf5

I thank the authors for their clear and detailed responses to my questions and for the substantial additional experiment. I have correspondingly raised my score. Good luck.

Reviewer_gTiCOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper investigates leave-one-out (LOO) prediction for continuous and jump diffusion models on discrete data. The main motivation is that continuous denoising models may rely heavily on the local noisy embedding to predict the corresponding clean token, without sufficiently learning dependencies across positions. To address this issue, the authors propose an input-isolated attention architecture that predicts each token using only the corrupted states of other positions.

The paper further studies how LOO predictions can be incorporated into jump diffusion processes. It establishes that using LOO probabilities for complete-reset jumps, while retaining the full posterior for the continuous drift, preserves the prescribed marginal distributions under certain assumptions. To obtain the full posterior from LOO predictions, the authors introduce an analytic Bayesian likelihood completion procedure. Experiments on Text8 and ZINC250k demonstrate improvements in lexical statistics and molecular generation quality.

Critical strengths
  1. Interesting motivation and intuitive idea. The observation that local noisy embeddings may provide an easy shortcut for denoising is interesting and well motivated. Enforcing LOO prediction is a simple but meaningful way to encourage the model to learn dependecies across different positions.

  2. Useful theoretical analysis. I appreciate the theoretical investigation of how LOO predictions interact with jump diffusion dynamics. In particular, the marginal-preservation result for complete-reset jumps is interesting, and the distinction between using LOO predictions for jump destinations and continuous drift provides useful insights.

  3. Promising empirical results. The experiments demonstrate substantial improvements over unrestricted prediction, particularly in Text8 word validity and ZINC250k molecular connectivity. The comparison between raw LOO prediction and Bayesian completion is also interesting, as it reveals different behaviors across generative processes.

Critical weaknesses
  1. Potential limitations of the attention architecture. The proposed input-isolated attention architecture uses fixed source features across layers, which may limit the interactions between contextual tokens. Unlike standard Transformers, contextual representations cannot be progressively exchanged across positions. It is unclear how much this restriction affects the expressive power of the model, especially for more complex dependencies.

  2. Motivation for the hybrid diffusion process. The paper combines continuous Brownian bridge dynamics with repeated discrete resets. However, both pure diffusion and pure jump processes can already serve as valid generative models. The motivation for combining these two mechanisms, beyond enabling the theoretical analysis, could be better explained.

  3. Gap between theoretical guarantees and empirical observations. The marginal-preservation guarantee requires the full posterior for the continuous drift, obtained through Bayesian completion. However, directly using raw LOO predictions in the drift often produces better results, especially on Text8. This is a rather interesting observation, but the current paper does not fully explain why the theoritically justified approach can perform worse.

  4. Limited evaluation of language generation quality. The Text8 experiments mainly focus on lexical validity, word diversity, and character-level distributional statistics. These metrics do not necessarily capture longer-range dependencies or semantic coherence, which are particularly relevant to the motivation of learning contextual dependencies.

Questions for authors
  1. Could the authors discuss whether the fixed-source attention architecture limits the model's expressive power? Is it possible to preserve the LOO property while allowing richer interactions among contextual tokens?

  2. What is the main advantage of combining continuous diffusion with repeated discrete resets, compared with using either mechanism alone? Is there a particular benefit of the hybrid process beyond the theoretical marginal-preservation property?

  3. Why does raw LOO drift outperform Bayesian-completed drift on Text8 despite not satisfying the marginal-preservation guarantee? Could this be related to approximation errors, numerical discretization, or a potentially beneficial bias introduced by LOO dynamics?

  4. Could the authors provide additional evidence that LOO prediction improves the learning of longer-range dependencies, rather than primarily improving local lexical statistics?

3 comments

We thank the reviewer for the careful review and constructive comments. We address the four points below, referring to our common response, “Response to all reviewers,” for shared results.

1. Fixed-source attention and expressive power.

The restriction limits contextual exchange: source keys and values remain token-local, although evolving queries can combine information from all other positions. The new LM1B control with standard self-attention improves the unrestricted baseline but remains strongly below RJD-LOO in best MAUVE and in Figure 2 (Section 4.2; Appendix I.1). A better LOO architecture could thus further improve the LOO advantages but is subject to future work.

2. Motivation for the hybrid process.

Continuous states can retain noisy information about other tokens before they are discretely resolved, but an unrestricted predictor can also exploit its own noisy token as a shortcut. Masked DFM avoids that local shortcut at masked positions, while other still-masked positions provide no token identity. RJD-LOO combines informative noisy context with predictions that exclude the target's own state. Repeated resets additionally allow discrete token hypotheses to be revised as context changes (Sections 1–2).

On Text8, RJD-LOO improves validity over EPJ and diversity over PD-LOO (Section 4.1). On ZINC250k, the absorbing hybrid AJD-LOO achieves the highest fraction of valid and connected molecules and the lowest mean raw and corrected FCD in the main comparison (Section 4.3). On LM1B, RJD-LOO and EPJ reach similar best MAUVE and RJD-LOO has a better PPL-entropy curve, see Figure 2 in the updated manuscript (common response, Section 1; paper Section 4.2).

3. Raw LOO drift and Bayesian completion.

Please see Section 3, “Why can raw LOO outperform completion?”, of our common response, and Section 4.4 and Appendix G.4 of the revised manuscript. The pure-diffusion validity gap also persists at larger sampling budgets (Appendix G.8), so a numerical discretization reason seems unlikely.

4. Longer-range dependencies and generation quality.

We added LM1B subject–verb interventions across intervening words and distracting nouns. For example: “The key near the cabinets is still here.” The subject “key” requires “is”, even with the distracting plural noun “cabinets” and local noisy evidence for “are”.

The table compares correct verb preferences with correct or wrong local evidence.

Correct verb preferred (%)RJD-LOO (t=0.25t=0.25)RJD (t=0.25t=0.25)RJD-LOO (t=0.55t=0.55)RJD (t=0.55t=0.55)
Correct local verb state62.0097.0081.56100.00
Wrong local verb state62.003.2581.560.00

LOO more often prefers the correct verb under misleading local evidence. Figure 3 of the revised manuscript (Section 4.4) reports subject tracking across distances, where LOO clearly outperforms unrestricted predictions. We have more extensive experiments which underline better long range dependency learning for LOO in Appendix I.6 (Table 26) and I.7.

Thank you for the response. I appreciate the clarification about the motivation behind the hybrid process. However, I still believe that the architectural limitation is an important issue worth further investigation. I will maintain my positive assessment.

We thank the reviewer for their feedback and their positive assessment, and we agree that the architectural limitation deserves further investigation. This is in particular the case since already now the LOO architecture produces much better results than the standard transformer:

ModelTop-pMAUVE ↑Generative PPL ↓Token entropy (nats)
RJD-LOO0.80.9039 ± 0.005863.25 ± 0.294.2987 ± 0.0002
RJD-SA0.70.4704 ± 0.0192202.77 ± 0.894.3415 ± 0.0009
These results motivate developing more expressive architectures that preserve LOO invariance, which we leave to future work
Reviewer_w3XYOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper explores the hypothesis that conditioning on a current position’s token reveals too much information, and so ignoring the current position (through leave-one-out prediction) benefits the learning of relationships. It explores this theoretically and with some experimental results.

Critical strengths

Interesting investigation and several theoretical contributions to our understanding of how LOO prediction works. Nice empirical results that corroborate and contextualize their findings.

Critical weaknesses

The authors start with a hypothesis “that easy local prediction can reduce the pressure to learn dependencies between positions”. However, the main experiments establish an empirical benefit but don’t isolate the mechanism- so they don’t directly address the initial hypothesis.

The introduction could use some serious work to make it more clear what the problem is and what the gap that they’re solving with their investigation is. The related works section of the introduction feels a bit like a citation salad without a clear storytelling throughline. This should be cleaned up, and perhaps cut down a bit.

Questions for authors

Can you demonstrate what are the mechanisms that are driving the empirical improvement of LOO drift over analytic completion? Can you show this using experimental results?

1 comment

We thank the reviewer for the careful review and constructive comments. We address the points below, referring to our common response, “Response to all reviewers,” for shared results.

1. Testing the local-prediction hypothesis.

Sections 2 and 4 of our common response address this through LM1B local-state replacements and subject–verb interventions, testing reliance on misleading local input and learned contextual dependencies. The protocols and results are also in Section 4.4 and Appendices I.5–I.6 of the revised paper.

2. Why can raw LOO drift outperform analytic completion?

Please see Section 3, “Why can raw LOO outperform completion?”, of our common response. It explains how completion can reinforce misleading local evidence and hinder recovery. Further details are in Section 4.4 and Appendix G.4 of the revised paper.

3. Introduction and related work.

We rewrote the Introduction to clarify the problem, the gap addressed by LOO in jump diffusions, and the distinction between theoretical guarantees and empirical gains. We shortened the related-work discussion and moved the broader overview to the appendix.

Comments on the paper

2 comments
AuthorsOpenReview ↗

Response to all reviewers

Response to all reviewers

We thank all reviewers for their careful reviews and detailed comments. Since several questions overlap across the reviews, we address them together in this common response.

1. New dataset LM1B

LM1B is a much larger text corpus than Text8, and our experiments use an approximately 30,000-token subword vocabulary.

LOO substantially improves PPL and MAUVE over unrestricted prediction, with or without Bayesian completion. RJD-LOO achieves the highest mean MAUVE. PD-LOO's low entropy indicates reduced token diversity, so its very low PPL should be interpreted cautiously.

ModelTop-ppPPL ↓Entropy (nats)MAUVE ↑
RJD-LOO0.863.25 ± 0.294.2987 ± 0.00020.9039 ± 0.0058
RJD-BC-D0.862.25 ± 0.104.2115 ± 0.00070.7624 ± 0.0126
RJD (unmasked)0.7272.95 ± 1.914.3735 ± 0.00100.2019 ± 0.0032
EPJ (pure jump)0.981.08 ± 0.124.2977 ± 0.00140.8965 ± 0.0103
PD-LOO0.830.63 ± 0.104.1686 ± 0.00150.5970 ± 0.0089
PD-BC1113.77 ± 0.644.1969 ± 0.00140.3576 ± 0.0071
PD (unmasked)0.7159.64 ± 0.144.2318 ± 0.00040.2111 ± 0.0138

Entries show mean ± sample SD across generation seeds at each method's best tested top-pp by mean MAUVE. Bold marks the lowest PPL and highest MAUVE. PD denotes pure diffusion, RJD repeated-reset jump diffusion, and BC Bayesian completion. Details are in Section 4.2 and Appendices I.1–I.3 of the revised paper.

2. Dependence on the token's own noisy input

The reviews ask what drives LOO's gains with learned predictors. We test whether misleading local input can override context.

Test. At t=0.5t=0.5, we replace one noisy LM1B token state with a state for a different token, initially leaving the surroundings unchanged, then continue generation.

Final token (%)RJD-LOORJD
Original token, after replacement30.810.15
Inserted token, after replacement6.0199.71

Interpretation. Unrestricted RJD almost always keeps the inserted token, whereas LOO more often recovers the original. This supports reduced reliance on misleading local input as a possible contributor to LOO's gains. LOO can also change correct inputs; alternative tokens may still be valid. More details and time sweep: Section 4.4 and Appendix I.5.

3. Why can raw LOO outperform completion?

The reviews ask why raw LOO can perform better although the marginal-preservation result uses the full posterior in the drift.

The completed probabilities p(a)p(a) combine the LOO prediction q(a)q(a) with the local likelihood:

$ p(a) \propto q(a)\ell_t(x_i \mid a), \qquad \frac{p(d)}{p(y)} = \frac{q(d)}{q(y)} \cdot \frac{\ell_t(x_i \mid d)}{\ell_t(x_i \mid y)}. $

Here yy is the original character and dd the inserted one. Misleading local evidence can outweigh the contextual preference for yy.

Test. We replace a local Text8 state with a noisy state for a wrong character dd (e.g., “o” instead of “a” in “cat”), then continue generation using raw or completed drift from the same LOO predictor. Completion uses the full vocabulary; RJD resets remain LOO.

Outcome (%)PD-LOOPD-BCRJD-LOORJD-BC-D
Start t=0.3013t=0.3013
Immediate argmax is dd1.95 ± 0.6140.26 ± 1.931.67 ± 0.3842.60 ± 2.12
Final character is dd13.65 ± 0.4031.52 ± 1.0915.05 ± 1.0116.06 ± 0.34
Final word validity86.42 ± 0.8356.32 ± 0.5878.16 ± 0.3169.80 ± 1.10

Mean ± sample SD across training seeds. Bold compares raw and completed drift within each path: lower donor preference or retention, and higher word validity. Immediate argmax uses drift probabilities. Local validity requires the surrounding words to occur in the Text8 corpus.

Interpretation. The immediate argmax shows that completion much more often predicts the inserted wrong character dd, reinforcing misleading local evidence. Completion also lowers final word validity in both families. The smaller drop for RJD-BC-D may reflect its LOO resets, which can override wrong completed drift predictions. Details: Section 4.4 and Appendix G.4.

4. Revisions and additional contextual tests

We revised the Introduction (Section 1) and experiments (Section 4), adding LM1B subject–verb interventions and an external-context diagnostic to probe longer-range dependencies (Section 4.4; Appendices I.6–I.7), complemented by blinded LLM assessments of generated text (Appendix I.7).

Dear reviewers,

The author-reviewer discussion period is ending soon. If you have not done so yet, please read the authors’ rebuttal and reply to it. Tell the authors whether it addresses your concerns, and update your score if your assessment has changed.

The ICLR 2027 Reviewer Guidelines list « discussing with the authors to clarify and improve the paper » as one of the core duties you accepted as a reviewer. With the high number of submissions this year, your input matters more than ever. Recall that reviewers that do not fulfill their duties may also face penalties on their own submissions.

Thank you for your time and help.

Best regards,


Your Area Chair

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.