- Rating
- 2(+1)
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
This paper studies the effect of annealing to sample for discrete-space target distributions, when the target changes faster than the sampling or learning dynamics. A general KL-tracking bound is first provided, with applications to slowly-annealed Glauber dynamics. Then the general bound is applied to the fine-tuning process of MDMs, with emphasis on the slowly-annealed versions of two existing algorithms DTM and DAM. Experiments are performed to validate the effect of slow annealing, to some extent.
Critical strengths
The mathematical analysis is largely sound.
Critical weaknesses
- The first and larger weakness is on paper connectedness. Right now Sections 3 and 4 are kind of disconnected: no results in Section 3 is anywhere invoked in the proof of the results in Section 4 (i.e., on MDM fine-tuning). This puts in question how much of the prior mathematical theory can be "applied" to MDM fine-tuning, and, more generally, can be helpful to ML-related applications.
- The next weakness is in the MDM fine-tuning result. Not only that the setup and many assumptions are only in the appendix, but also that the interpretation of the theorems can be questioned. To showcase the advantage of the annealing schedule, the authors claim that "applying slowdown to the annealing speed allows improved convergence rate," but that seems to be overclaiming. Theoretically speaking, a lower quantity might not results in a better complexity rate rather than a constant-factor improvement. Second, the result established is merely an upper bound. Third, the theory only characterizes the KL error given a schedule, but it is unclear how such an error reversely informs us for the schedule design.
Questions for authors
- Please address my comment on the paper connectedness and on the interpretation of the theory on MDM fine-tuning.
- How practical are the assumptions listed for the MDM fine-tuning theorems in the appendix?
- For the current experiment, can you clarify how is it related to the case with fast-changing target of primary interest in this paper (with clear mathematical definition for such a fast-changing nature)? Also, is there a principled way to determine the annealing schedule?
- There seems to be a new algorithm in Section 3. It might be helpful to check its empirical performance. (This can be a future work.)
Thank you for the additional experiments and clarifications. I find the explanation of the choice of testbeds reasonable. However, the other main concerns in W1, the gap between the theory and the experiments and the lack of practical guidance, are not fully addressed, for three reasons.
1. Only the left-hand sides of the bounds are measured. Tables R1–R3 do not report the right-hand sides, so the tightness of the bounds remains unclear. Moreover, the measured values decrease very differently from the bounds. With a fixed stage length, T∝N, and with λa=a0/2 and a˙=A/T, the bound in Theorem 4 decays as e−a0T/2 and 1/T2. In Table R2, however, H(T) decreases only by factors of 8.4, 2.9, and 1.7 from N=1 to 128. This suggests that the assumptions and analysis of Theorem 4 do not apply well to this setting.
2. The schedule in the experiments is piecewise constant. The tilt increases in steps of h=A/N, while Theorems 4–5 assume a∈C1 (Assumption 2(2)). For such a step schedule, ∫a˙2dt is infinite and the bounds become vacuous.
3. Rule (i) of the practical guidance does not match Proposition 4. Proposition 4 only covers the adjoint estimator of DAM, so it does not support rule (i) for DTM (Tables R1–R2). Moreover, the condition N≥A⋅osc(r) is only an assumption of Proposition 4 (A4). Its polynomial guarantee is shown only with N=⌈A2osc2(r)/(2ε)⌉, which is much larger for small ε.
On clarity. I also think both the paper and the rebuttal should be more concise and easier to follow.
While reading the paper more closely, I found several issues in Section 4. It is only briefly mentioned at the end of Section 2.2, before contexts are defined, that Theorem 1 is applied to the evolution of the model output at each context over training time, not to the MDM generation process. This becomes clear only after Theorems 4–5 and with the appendix. The notation is also inconsistent across sections (e.g., t and Q change meaning, and the tilted target appears as ρ1,a, ρ∗,a, and ρ⋆). Finally, some expressions are more complex than needed: Theorems 4–5 and Corollary 1 could simply be stated with λ=a0/2 (i.e., αχ=1), since the mLSI holds with this constant for every context and tilt level (Eq. (35)).
The rebuttal also consists of four long comments, which are hard to digest in a short time. As the discussion phase is close to its end, I have responded only to W1 for now.