acceptodds
Under review as a conference paper at ICLR 2027

Slowly Annealed Jump Processes: Theory and Applications to MDM Fine-tuning

Abstract

Annealing is a powerful strategy for sampling from complex distributions and reward-based fine-tuning of diffusion models, using a sequence of intermediate distributions to connect an easily sampled reference distribution to the desired target. However, if the target changes faster than the sampling or learning dynamics can adapt, substantial tracking error can persist. This motivates slow annealing, which gives the dynamics more time to adapt to the evolving target and recent work has established the effectiveness in both continuous-state sampling and guided generation. However, whether slow annealing yields comparable benefits for discrete jump processes and reward-based fine-tuning of masked diffusion models remains insufficiently understood. In this work, we develop a theory of slowly-annealed jump processes with forward KL tracking guarantees of slowly-annealed Glauber dynamics. We further derive conditional KL bounds for slowly-annealed masked diffusion fine-tuning under idealized learning dynamics. Experiments on tractable distributions and LLaDA-8B-Instruct demonstrate the benefits.

Primary area:
probabilistic methods (Bayesian methods, variational inference, sampling, UQ, etc.)
Keywords:
Langevin Dynamics, Theory, Reward Fine-tuning, Generative Modeling

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_Lku8OpenReview ↗
Rating
2(+1)
Confidence
2
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper studies the effect of annealing to sample for discrete-space target distributions, when the target changes faster than the sampling or learning dynamics. A general KL-tracking bound is first provided, with applications to slowly-annealed Glauber dynamics. Then the general bound is applied to the fine-tuning process of MDMs, with emphasis on the slowly-annealed versions of two existing algorithms DTM and DAM. Experiments are performed to validate the effect of slow annealing, to some extent.

Critical strengths

The mathematical analysis is largely sound.

Critical weaknesses
  1. The first and larger weakness is on paper connectedness. Right now Sections 3 and 4 are kind of disconnected: no results in Section 3 is anywhere invoked in the proof of the results in Section 4 (i.e., on MDM fine-tuning). This puts in question how much of the prior mathematical theory can be "applied" to MDM fine-tuning, and, more generally, can be helpful to ML-related applications.
  2. The next weakness is in the MDM fine-tuning result. Not only that the setup and many assumptions are only in the appendix, but also that the interpretation of the theorems can be questioned. To showcase the advantage of the annealing schedule, the authors claim that "applying slowdown to the annealing speed allows improved convergence rate," but that seems to be overclaiming. Theoretically speaking, a lower quantity might not results in a better complexity rate rather than a constant-factor improvement. Second, the result established is merely an upper bound. Third, the theory only characterizes the KL error given a schedule, but it is unclear how such an error reversely informs us for the schedule design.
Questions for authors
  1. Please address my comment on the paper connectedness and on the interpretation of the theory on MDM fine-tuning.
  2. How practical are the assumptions listed for the MDM fine-tuning theorems in the appendix?
  3. For the current experiment, can you clarify how is it related to the case with fast-changing target of primary interest in this paper (with clear mathematical definition for such a fast-changing nature)? Also, is there a principled way to determine the annealing schedule?
  4. There seems to be a new algorithm in Section 3. It might be helpful to check its empirical performance. (This can be a future work.)
Reviewer_dkHgOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

They analyses slow annealing in discrete jump processes and reward based fine-tuning of masked diffusion models. They first establish a bound on the forward-KL and apply it to mean-field Ising and Potts models. They derive a bound as well for discrete tilt and adjoint matching with idealised training dynamics and lastly evaluate slowly annealed fine-tuning on toy experiment and LLaDA-8B-Instruct. Experiments show that a more annealing stages increases the reward, but the reward decreases if the number of steps for each stage is not sufficient.

Critical strengths
  • The problem is important and I believe the paper to provide insights on slowly annealed process, in particular the trade-off between number of stages and steps per stage.
  • The method is tested at a real scale (LLaDA-8B-Instruct) and not only toy problems
  • With the same number of gradient steps a finer annealing schedule gives larger gains, except at N=40 on Countdown. This is an important result since one needs to choose the schedule in DTM.
Critical weaknesses
  • The experiments do not test the slowdown of the theory directly. A slower schedule means a larger total training time T which I understand as number of stages times number of steps for each stage. However it seems that this value is held constant (5120 for the LLM tasks). A larger N gives smaller tilt steps at the same speed and the theory doesn't seem to say anything about N at fixed T.
  • Statistics are not sufficient, experiments (Table 1-3) use one seed without error bars, which is a problem in particular when differences are seemingly small.
  • A baseline with no annealing (N = 1) on the LLM task would also be interesting as a reference point.
Questions for authors
  • Can you test the slowdown of the theory by running the LLM experiments with a constant number of steps for each stage, so that the total number of steps grows with N?
  • Could you add results over several seeds for Table 1?
  • Could you run the N=1 (no annealing) baseline for Table 1?
Reviewer_QWMrOpenReview ↗
Rating
2
Confidence
2
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

The paper develops a theory of slowly annealed jump processes for discrete state spaces and applies it to sampling with annealed Glauber dynamics and to reward fine-tuning of masked diffusion models (MDMs). Theorem 1 bounds forward KL for jump processes under a modified log-Sobolev inequality (mLSI), separating the initial error term from a transport term controlled by a discrete α-action (an exponential-moment analogue of the quadratic action). This yields polynomial-complexity guarantees for exact annealed Glauber sampling of the mean-field Ising and Potts models (Theorems 2–3). The authors further derive conditional-KL Lyapunov bounds for slowly annealed Discrete Tilt Matching (DTM) and Discrete Adjoint Matching (DAM) under idealized learning dynamics, with a tracking term quadratic in the annealing speed (Theorems 4–5). Under a fixed gradient budget, finer annealing schedules lower terminal KL on a small synthetic distribution and generally improve DTM rewards on maze planning, Sudoku and Countdown tasks.

Critical strengths

S1. A nontrivial extension of slow-annealing theory to discrete jump processes.

The paper extends the analysis of slowly annealed Langevin dynamics from continuous spaces to jump processes on discrete state spaces (Theorem 1). This extension is nontrivial, as it requires replacing the log-Sobolev inequality with a modified log-Sobolev inequality and introducing a discrete counterpart of the α-action. The paper further applies this framework to annealed Glauber dynamics, giving forward-KL guarantees that complement He et al. (2026) (Theorems 2 and 3), and to slowly annealed DTM and DAM (Theorems 4 and 5). In addition, the remarks clearly position each result relative to prior work, such as He et al. (2026) and Nitanda et al. (2026), which makes it easy to see how the contribution relates to prior work.

Critical weaknesses

W1. The practical impact of the theoretical analysis is unclear.

Theorems 4 and 5 analyze how the annealing speed affects tracking error under idealized learning dynamics. The experiments, however, fix the total training budget, vary the number of annealing stages, and do not measure the quantities that the theorems bound. The analysis also does not lead to concrete guidance for practice, such as how to choose the annealing schedule or the number of stages, which the paper instead selects empirically. Similarly, it is unclear whether the mean-field Ising and Potts models in Theorems 2 and 3 are practically important targets or mainly serve as examples that admit tractable analysis. This weakness is central to my assessment because it remains unclear what the theoretical results enable beyond the intuition that slower annealing helps, which limits the significance of the paper.

Questions for authors

Q1 (Clarification). What practical impact could the theorems have? I am not deeply familiar with discrete-space sampling, so this question may come from a misunderstanding on my part.

Q2 (Clarification). How realistic is Assumption 1 in the settings of Sections 4.1 and 4.2?

Q3 (Clarification). Why is DAM evaluated only on the toy distribution?

Q4 (Suggestion for future work). Since the bound in Theorem 1 depends on the annealing path through the α-action, it would be interesting to use this bound to design efficient annealing paths.

Comments on the paper

1 comment

Thank you for the additional experiments and clarifications. I find the explanation of the choice of testbeds reasonable. However, the other main concerns in W1, the gap between the theory and the experiments and the lack of practical guidance, are not fully addressed, for three reasons.

1. Only the left-hand sides of the bounds are measured. Tables R1–R3 do not report the right-hand sides, so the tightness of the bounds remains unclear. Moreover, the measured values decrease very differently from the bounds. With a fixed stage length, T∝NT \propto N, and with λa=a0/2\lambda_a = a_0/2 and a˙=A/T\dot a = A/T, the bound in Theorem 4 decays as e−a0T/2e^{-a_0 T/2} and 1/T21/T^2. In Table R2, however, H(T)H(T) decreases only by factors of 8.4, 2.9, and 1.7 from N=1N=1 to 128128. This suggests that the assumptions and analysis of Theorem 4 do not apply well to this setting.

2. The schedule in the experiments is piecewise constant. The tilt increases in steps of h=A/Nh = A/N, while Theorems 4–5 assume a∈C1a \in C^1 (Assumption 2(2)). For such a step schedule, ∫a˙2 dt\int \dot a^2\,dt is infinite and the bounds become vacuous.

3. Rule (i) of the practical guidance does not match Proposition 4. Proposition 4 only covers the adjoint estimator of DAM, so it does not support rule (i) for DTM (Tables R1–R2). Moreover, the condition N≥A⋅osc(r)N \ge A\cdot\mathrm{osc}(r) is only an assumption of Proposition 4 (A4). Its polynomial guarantee is shown only with N=⌈A2osc2(r)/(2ε)⌉N = \lceil A^2\mathrm{osc}^2(r)/(2\varepsilon) \rceil, which is much larger for small ε\varepsilon.

On clarity. I also think both the paper and the rebuttal should be more concise and easier to follow.

While reading the paper more closely, I found several issues in Section 4. It is only briefly mentioned at the end of Section 2.2, before contexts are defined, that Theorem 1 is applied to the evolution of the model output at each context over training time, not to the MDM generation process. This becomes clear only after Theorems 4–5 and with the appendix. The notation is also inconsistent across sections (e.g., tt and QQ change meaning, and the tilted target appears as ρ1,a\rho_{1,a}, ρ∗,a\rho^{*,a}, and ρ⋆\rho^\star). Finally, some expressions are more complex than needed: Theorems 4–5 and Corollary 1 could simply be stated with λ=a0/2\lambda = a_0/2 (i.e., αχ=1\alpha_\chi = 1), since the mLSI holds with this constant for every context and tilt level (Eq. (35)).

The rebuttal also consists of four long comments, which are hard to digest in a short time. As the discussion phase is close to its end, I have responded only to W1 for now.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.