- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 1
Read review
Paper summary
This paper proposes a method for optimizing the initial noise distribution, achieving some performance gains on GenEval, aesthetic quality, and rewards for text generation.
Critical strengths
-
The authors propose a method for optimizing a distribution over a discrete codebook, which achieves some performance gains.
-
The proposed method (DDNO) is evaluated fairly thoroughly, and the reported performance improvements are reasonably convincing.
Critical weaknesses
The proposed method appears to optimize the initial noise distribution for individual samples against an existing reward function. This raises several concerns:
-
As the authors themselves point out (line 1129), DDNO is a per-instance optimization method. The value of this method as a standalone contribution is limited, since it requires optimization for each query in the evaluation set against a known reward function. This is effectively equivalent to optimizing a policy on the evaluation set. It does not provide a systematic way to optimize the network, nor does it allow test-time performance to be improved through iterative optimization on the training or validation set. At a minimum, the method should be able to train a policy that initializes the noise distribution such that the resulting sampled images achieve consistently higher scores under, for example, aesthetic evaluation.
-
The authors state that the proposed method requires a differentiable reward function, which substantially limits its applicability. In many domains, reward signals are sparse, making it difficult in practice to train a reward model that provides continuous and reliable scores. Taking GenEval as an example, commonly used reinforcement learning approaches can optimize directly against its verifier, which provides discrete scores between 0 and 1(like 0, 0.25, 0.75, 1), without requiring a separate model to estimate the probability of an affirmative answer. In image-generation, it is also common to use a multimodal model to decompose a prompt into multiple questions and verify each one independently. The resulting scores do not appear to be differentiable either.
Minor Concern
The paper title, Discrete Diffusion Noise Optimization, positions the work within the discrete diffusion literature. However, the method is mostly based on discrete flow matching (DFM), a relatively less common framework in this area, as exemplified by the Fudoki model used by the authors, which identifies itself as such. This framework differs from UDLM-type models (the training and sampling method differs, while concepts alike), which are commonly used in discrete diffusion and employ uniform transition kernels, as well as the more widely used MDLM-type models, which employ absorbing-state transition kernels. The positioning of the work may therefore warrant reconsideration.
Questions for authors
My main concerns are already outlined in the Weaknesses section.
I also have one additional question: Does this method impose any requirements on or depend on classifier-free guidance (CFG) during the iterative optimization process? If so, would a policy optimized with a particular CFG scale remain effective if the CFG scale were changed during sampling?
6 comments
2. Differentiable rewards and VLM verification.
We agree this is a limitation, and state it in Section 5. Rewards available only as rule-based checks, program execution or human judgments are out of scope, and there search-based methods remain the right tool.
However, the limitation is narrower than the review suggests. The question-based VLM verification the reviewer mentions is what our paper already does, and is actually differentiable. Our fidelity reward (App. F.3) asks NVILA-Lite-2B-Verifier, a VLM fine-tuned for verification, a yes/no question about each generated image:
"You are an AI assistant specializing in image analysis and ranking. Your task is to analyze and compare an image based on how well it matches the given prompt. The given prompt is: {prompt}. Please consider the prompt and the image to make a decision and respond directly with 'yes' or 'no'."
The reward is the probability the VLM assigns to the token "yes", which is differentiable with respect to the image. Decomposing a prompt into several questions gives a sum of yes-probabilities that is differentiable in the same way.
Many other widely used rewards are differentiable too: preference models (HPS, ImageReward, PickScore), CLIP scores, NLI classifiers, and LLM reward models such as the QRM we use for text. Moreover, the same requirement holds for every gradient-based method including classifier guidance, gradient-based LLM steering (PPLM[9], MuCoCo[10], ∇-Reasoner) and gradient-based fine-tuning like DRAKES[11]. Against the gradient-free alternatives (Best-of-N, SVDD-PM, FK-Steering), our compute-matched experiments show that gradients keep improving after search plateaus (Figs. 3, 5).
[9] Dathathri et al. "Plug and Play Language Models: A Simple Approach to Controlled Text Generation". ICLR 2020.
[10] Kumar et al. "Controlled Text Generation as Continuous Optimization with Multiple Constraints". NeurIPS 2021.
[11] Wang et al. "Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design". ICLR 2025.
4. CFG. DDNO neither uses nor depends on classifier-free guidance. All reward-based methods run without CFG during both optimization and decoding (App. F.4). To answer the question directly, we ran a small check, noise optimized without CFG is decoded with the original 50-step FUDOKI sampler at different CFG scales and scored with held-out ImageReward (5 prompts from Figure 2 × 2 seeds, 4 samples each, same sampler seeds in both rows):
| ImageReward ↑ | CFG 1.5 | CFG 3 | CFG 5 | CFG 7.5 |
|---|---|---|---|---|
| Random noise | 0.97 | 1.34 | 1.46 | 1.47 |
| DDNO noise (optimized at CFG 0) | 1.12 | 1.62 | 1.75 | 1.65 |
| Gain | +0.16 | +0.28 | +0.29 | +0.18 |
DDNO noise scores higher at every CFG scale, and the two combine: DDNO with CFG 5 gives the best result here (1.75 vs. 1.46 for CFG alone). This adds to the evidence that the optimized noise transfers across sampler changes, alongside step counts (Figure 8) and with the original sampler (0.67 → 0.86).
3. Positioning (minor concern): discrete diffusion vs. discrete flow matching.
We use "discrete diffusion" in the general sense that includes discrete flow matching (DFM). For mixture kernels, which include UDLM-type (uniform) and MDLM-type (absorbing) models, the two frameworks define the same probability path, and the reverse diffusion process is one of the DFM generators of that path. Campbell et al. [12] show that DFMs include discrete diffusion as a special case (see also [13, Remark 6]). Notation follows Sec. 2.1: is noise, is data, and is the rate of jumping from to .
Same path. The diffusion forward process has generator with . Since , with ,
which is exactly the DFM mixture path (Eq. 2). Uniform gives UDLM; gives MDLM. Conversely, every mixture path with increasing arises from this kernel with .
Same reverse process. Take as the DFM conditional rate the time reversal of the conditional forward process, for . Averaging over the posterior, as in DFM, gives
which is exactly the reverse-diffusion rate. The same argument applies coordinatewise to sequences, with the posterior conditioned on the full noisy sequence.
Other samplers for the same path. DFM also admits other rates for this path, e.g. the minimal-jump rate . For masking it coincides with the diffusion rate. For uniform noise, the difference is in detailed balance with and therefore leaves every marginal unchanged; it is the freedom of [12, Prop. 3.3]. Both rates are functionals of the same posterior .
The training and sampling differences the reviewer notes therefore reduce to two choices:
- Sampling: which generator to simulate for one fixed path, analogous to DDPM vs. DDIM or probability-flow sampling of one continuous diffusion model.
- Training: cross-entropy (DFM) vs. ELBO (diffusion). Both are minimized by the true posterior and differ only in how they weight errors.
Neither choice affects DDNO, which needs only a uniform source and a sampler built from categorical transitions.
Our text model is UDLM-type. GIDD-Unif-3B [14] is a uniform-noise discrete diffusion language model, trained with the GIDD ELBO and sampled with its adaptive sampler, so it is a UDLM-type model in the reviewer's sense. FUDOKI, used for images, is a DFM model. DDNO applies to both without any change to the method; only the surrogate of each model's own sampler differs (Algorithms 1–2). Masked (MDLM-type) models start every trajectory from the same all-[MASK] state and so have no initial noise to optimize (Sec. 2.1).
[12] Campbell et al. "Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design". ICML 2024.
[13] Pauline et al. "Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction". arXiv 2025.
[14] von Rütte et al. "Scaling Behavior of Discrete Diffusion Language Models". ICLR 2026.
We hope this addresses the reviewer’s concerns, in particular regarding soundness, and would be grateful if they could let us know whether any remain.
We thank the reviewer for their time and feedback. The reviewer finds our evaluation "fairly thorough" and the improvements "reasonably convincing," identifying no error in our method, theory, or experiments.
However, we disagree with the central weaknesses raised in this review. The
Soundness: 1score rests on two objections: DDNO optimizes per instance against a known reward, and it requires a differentiable reward. Neither is specific to DDNO. The first objects to test-time scaling as a problem setting, and the second applies to every gradient-based method in it.1. Per-instance optimization against a known reward is what test-time scaling is.
At test time, the model is frozen, a reward is given, and extra compute is spent per query to improve the output. This is the standard setting across test-time scaling, including for LLMs verifier-guided search [1], tree search [2,3], and test-time gradient ascent (∇-Reasoner[4]), as well as MCTS in AlphaZero [5] or, most relevant to our setting, in diffusion inference scaling (
Best-of-N,SVDD-PM,FK-Steering,DOODL,D-Flow,ReNO). Every reward-based baseline in Table 2 and Figure 5 shares this exact per-prompt setup. If this disqualifies DDNO, it disqualifies all of test-time scaling.Specifically, "optimizing a policy on the evaluation set" does not describe DDNO:
NVILA-Lite-2B-Verifier+HPSv2.1, whileGenEval's object detector is never seen during optimization or selection (App. F.2-F.3). Yet held-outGenEvalrises from0.65to0.94. On text, DDNO optimizesDeBERTa-v3-Large, while held-outRoBERTa-Large-MNLIreaches0.38vs0.22for BoN (Sec. 3.3).[1] Snell et al. "Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning". ICLR 2025.
[2] Hao et al. "Reasoning with Language Model is Planning with World Model". EMNLP 2023.
[3] Zhou et al. "Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models". ICML 2024.
[4] Wang et al. "∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space". ICLR 2026.
[5] Silver et al. "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play". Science 2018.
[6] Venkatraman et al. "Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models". ICML 2025.
[7] Eyring et al. "Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models". NeurIPS 2025.
[8] Mammadov et al. "Variational Flow Maps: Make Some Noise for One-Step Conditional Generation". ICML 2026.