acceptodds
Under review as a conference paper at ICLR 2027

DDNO: Discrete Diffusion Noise Optimization

Abstract

Test-time reward alignment of discrete diffusion models currently relies on either step-wise guidance, which uses myopic approximations that can distort the generative process, or on search with stochastic generation. We introduce Discrete Diffusion Noise Optimization (DDNO), a training-free method for uniform discrete diffusion models that enables gradient-based reward optimization at test-time despite the non-differentiable categorical sampling that blocks gradient flow through the reverse process. DDNO relaxes the discrete initial noise to continuous logits and propagates terminal reward gradients end-to-end through straight-through estimation and soft mixing. By fixing all intermediate randomness, generation becomes controllable through variation of the initial noise. On text-to-image generation and steering text generation, DDNO outperforms existing test-time methods, achieving 0.94 GenEval overall accuracy, with gains measured under held-out evaluators distinct from the optimization reward. Under matched compute, search-based methods are preferable at smaller budgets but plateau, while DDNO scales consistently with optimization budget across both modalities, establishing gradient-based noise optimization as a new axis for test-time scaling in discrete generative models.

Primary area:
generative models
Keywords:
discrete diffusion, reward alignment, noise optimization, test-time scaling, guidance

Reviews

Among:
Shape:
1234this paper, Top 32%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.50 over 2 reviews, above 68% of the others.
Reviewer_s2bxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes a method for optimizing the initial noise distribution, achieving some performance gains on GenEval, aesthetic quality, and rewards for text generation.

Critical strengths
  • The authors propose a method for optimizing a distribution over a discrete codebook, which achieves some performance gains.

  • The proposed method (DDNO) is evaluated fairly thoroughly, and the reported performance improvements are reasonably convincing.

Critical weaknesses

The proposed method appears to optimize the initial noise distribution for individual samples against an existing reward function. This raises several concerns:

  1. As the authors themselves point out (line 1129), DDNO is a per-instance optimization method. The value of this method as a standalone contribution is limited, since it requires optimization for each query in the evaluation set against a known reward function. This is effectively equivalent to optimizing a policy on the evaluation set. It does not provide a systematic way to optimize the network, nor does it allow test-time performance to be improved through iterative optimization on the training or validation set. At a minimum, the method should be able to train a policy that initializes the noise distribution such that the resulting sampled images achieve consistently higher scores under, for example, aesthetic evaluation.

  2. The authors state that the proposed method requires a differentiable reward function, which substantially limits its applicability. In many domains, reward signals are sparse, making it difficult in practice to train a reward model that provides continuous and reliable scores. Taking GenEval as an example, commonly used reinforcement learning approaches can optimize directly against its verifier, which provides discrete scores between 0 and 1(like 0, 0.25, 0.75, 1), without requiring a separate model to estimate the probability of an affirmative answer. In image-generation, it is also common to use a multimodal model to decompose a prompt into multiple questions and verify each one independently. The resulting scores do not appear to be differentiable either.

Minor Concern

The paper title, Discrete Diffusion Noise Optimization, positions the work within the discrete diffusion literature. However, the method is mostly based on discrete flow matching (DFM), a relatively less common framework in this area, as exemplified by the Fudoki model used by the authors, which identifies itself as such. This framework differs from UDLM-type models (the training and sampling method differs, while concepts alike), which are commonly used in discrete diffusion and employ uniform transition kernels, as well as the more widely used MDLM-type models, which employ absorbing-state transition kernels. The positioning of the work may therefore warrant reconsideration.

Questions for authors

My main concerns are already outlined in the Weaknesses section.

I also have one additional question: Does this method impose any requirements on or depend on classifier-free guidance (CFG) during the iterative optimization process? If so, would a policy optimized with a particular CFG scale remain effective if the CFG scale were changed during sampling?

6 comments

We thank the reviewer for their time and feedback. The reviewer finds our evaluation "fairly thorough" and the improvements "reasonably convincing," identifying no error in our method, theory, or experiments.

However, we disagree with the central weaknesses raised in this review. The Soundness: 1 score rests on two objections: DDNO optimizes per instance against a known reward, and it requires a differentiable reward. Neither is specific to DDNO. The first objects to test-time scaling as a problem setting, and the second applies to every gradient-based method in it.


1. Per-instance optimization against a known reward is what test-time scaling is.

At test time, the model is frozen, a reward is given, and extra compute is spent per query to improve the output. This is the standard setting across test-time scaling, including for LLMs verifier-guided search [1], tree search [2,3], and test-time gradient ascent (∇-Reasoner[4]), as well as MCTS in AlphaZero [5] or, most relevant to our setting, in diffusion inference scaling (Best-of-N, SVDD-PM, FK-Steering, DOODL, D-Flow, ReNO). Every reward-based baseline in Table 2 and Figure 5 shares this exact per-prompt setup. If this disqualifies DDNO, it disqualifies all of test-time scaling.

Specifically, "optimizing a policy on the evaluation set" does not describe DDNO:

  • Nothing is trained or shared across prompts. Model weights fθf_\theta are frozen, and each prompt starts from fresh random logits ℓ0∼N(0,σ2I)\ell_0 \sim \mathcal{N}(0, \sigma^2 \mathbf{I}). With no shared parameters across queries, nothing can fit the evaluation set.
  • The optimization reward is distinct from the held-out evaluator. On images, all methods optimize NVILA-Lite-2B-Verifier + HPSv2.1, while GenEval's object detector is never seen during optimization or selection (App. F.2-F.3). Yet held-out GenEval rises from 0.65 to 0.94. On text, DDNO optimizes DeBERTa-v3-Large, while held-out RoBERTa-Large-MNLI reaches 0.38 vs 0.22 for BoN (Sec. 3.3).
  • Per prompt, DDNO already yields a noise distribution, not a single sample. The optimized logits define Cat(softmax(ℓ0⋆/τ))\mathrm{Cat}(\mathrm{softmax}(\ell_0^\star/\tau)). Sampling initial noise from it gives diverse images without further optimization: at τ=0.5\tau = 0.5 it recovers 82% of the base model’s diversity while still exceeding its reward (Fig. 4).
  • Training a noise policy is amortization: a natural next step, not a prerequisite. Training-free alignment keeps fθf_\theta frozen and needs no training set or specialized checkpoint, so a new reward can be supplied at inference. A trained policy could replace this, as recently proposed in continuous diffusion models through amortized noise models [6,7,8]. Note that these came only after test-time noise optimization had shown that the noise space is worth optimizing, and they build on that insight. No such test-time method existed for discrete diffusion. DDNO provides it, and with it, the foundation that amortization for discrete models can build on. And as AlphaZero and LLM MCTS show, a learned policy does not make per-query search redundant.

[1] Snell et al. "Scaling LLM Test-Time Compute Optimally Can be More Effective than Scaling Parameters for Reasoning". ICLR 2025.

[2] Hao et al. "Reasoning with Language Model is Planning with World Model". EMNLP 2023.

[3] Zhou et al. "Language Agent Tree Search Unifies Reasoning, Acting, and Planning in Language Models". ICML 2024.

[4] Wang et al. "∇-Reasoner: LLM Reasoning via Test-Time Gradient Descent in Latent Space". ICLR 2026.

[5] Silver et al. "A general reinforcement learning algorithm that masters chess, shogi, and Go through self-play". Science 2018.

[6] Venkatraman et al. "Outsourced Diffusion Sampling: Efficient Posterior Inference in Latent Spaces of Generative Models". ICML 2025.

[7] Eyring et al. "Noise Hypernetworks: Amortizing Test-Time Compute in Diffusion Models". NeurIPS 2025.

[8] Mammadov et al. "Variational Flow Maps: Make Some Noise for One-Step Conditional Generation". ICML 2026.

2. Differentiable rewards and VLM verification.

We agree this is a limitation, and state it in Section 5. Rewards available only as rule-based checks, program execution or human judgments are out of scope, and there search-based methods remain the right tool.

However, the limitation is narrower than the review suggests. The question-based VLM verification the reviewer mentions is what our paper already does, and is actually differentiable. Our fidelity reward (App. F.3) asks NVILA-Lite-2B-Verifier, a VLM fine-tuned for verification, a yes/no question about each generated image:

"You are an AI assistant specializing in image analysis and ranking. Your task is to analyze and compare an image based on how well it matches the given prompt. The given prompt is: {prompt}. Please consider the prompt and the image to make a decision and respond directly with 'yes' or 'no'."

The reward is the probability the VLM assigns to the token "yes", which is differentiable with respect to the image. Decomposing a prompt into several questions gives a sum of yes-probabilities that is differentiable in the same way.

Many other widely used rewards are differentiable too: preference models (HPS, ImageReward, PickScore), CLIP scores, NLI classifiers, and LLM reward models such as the QRM we use for text. Moreover, the same requirement holds for every gradient-based method including classifier guidance, gradient-based LLM steering (PPLM[9], MuCoCo[10], ∇-Reasoner) and gradient-based fine-tuning like DRAKES[11]. Against the gradient-free alternatives (Best-of-N, SVDD-PM, FK-Steering), our compute-matched experiments show that gradients keep improving after search plateaus (Figs. 3, 5).

[9] Dathathri et al. "Plug and Play Language Models: A Simple Approach to Controlled Text Generation". ICLR 2020.

[10] Kumar et al. "Controlled Text Generation as Continuous Optimization with Multiple Constraints". NeurIPS 2021.

[11] Wang et al. "Fine-Tuning Discrete Diffusion Models via Reward Optimization with Applications to DNA and Protein Design". ICLR 2025.


4. CFG. DDNO neither uses nor depends on classifier-free guidance. All reward-based methods run without CFG during both optimization and decoding (App. F.4). To answer the question directly, we ran a small check, noise optimized without CFG is decoded with the original 50-step FUDOKI sampler at different CFG scales and scored with held-out ImageReward (5 prompts from Figure 2 × 2 seeds, 4 samples each, same sampler seeds in both rows):

ImageReward ↑CFG 1.5CFG 3CFG 5CFG 7.5
Random noise0.971.341.461.47
DDNO noise (optimized at CFG 0)1.121.621.751.65
Gain+0.16+0.28+0.29+0.18

DDNO noise scores higher at every CFG scale, and the two combine: DDNO with CFG 5 gives the best result here (1.75 vs. 1.46 for CFG alone). This adds to the evidence that the optimized noise transfers across sampler changes, alongside step counts (Figure 8) and with the original sampler (0.67 → 0.86).

3. Positioning (minor concern): discrete diffusion vs. discrete flow matching.

We use "discrete diffusion" in the general sense that includes discrete flow matching (DFM). For mixture kernels, which include UDLM-type (uniform) and MDLM-type (absorbing) models, the two frameworks define the same probability path, and the reverse diffusion process is one of the DFM generators of that path. Campbell et al. [12] show that DFMs include discrete diffusion as a special case (see also [13, Remark 6]). Notation follows Sec. 2.1: t=0t=0 is noise, t=1t=1 is data, and ut(y,x)u_t(y,x) is the rate of jumping from xx to yy.

Same path. The diffusion forward process has generator βtQ\beta_t Q with Q=pnoise1⊤−IQ = p_{\text{noise}}\mathbf{1}^\top - I. Since Q2=−QQ^2 = -Q, with λt=∫t1βsds\lambda_t = \int_t^1 \beta_s ds,

qt(⋅∣x1)=eλtQδx1=αtδx1+(1−αt)pnoise,αt=e−λt,q_t(\cdot\mid x_1) = e^{\lambda_t Q} \delta_{x_1} = \alpha_t \delta_{x_1} + (1-\alpha_t) p_{\text{noise}}, \qquad \alpha_t = e^{-\lambda_t},

which is exactly the DFM mixture path (Eq. 2). Uniform pnoisep_{\text{noise}} gives UDLM; pnoise=δ[MASK]p_{\text{noise}} = \delta_{[\text{MASK}]} gives MDLM. Conversely, every mixture path with increasing αt\alpha_t arises from this kernel with βt=α˙t/αt\beta_t = \dot\alpha_t/\alpha_t.

Same reverse process. Take as the DFM conditional rate the time reversal of the conditional forward process, ut(y,x∣x1)=βtpnoise(x)qt(y∣x1)/qt(x∣x1)u_t(y,x\mid x_1) = \beta_t p_{\text{noise}}(x) q_t(y\mid x_1)/q_t(x\mid x_1) for y≠xy\neq x. Averaging over the posterior, as in DFM, gives

ut(y,x)=Ex1∼q(x1∣xt=x)[ut(y,x∣x1)]=βtpnoise(x)qt(y)qt(x),u_t(y,x) = \mathbb{E}_{x_1\sim q(x_1\mid x_t=x)}\big[u_t(y,x\mid x_1)\big] = \beta_t p_{\text{noise}}(x) \frac{q_t(y)}{q_t(x)},

which is exactly the reverse-diffusion rate. The same argument applies coordinatewise to sequences, with the posterior conditioned on the full noisy sequence.

Other samplers for the same path. DFM also admits other rates for this path, e.g. the minimal-jump rate utDFM(y,x∣x1)=α˙t1−αtδx1(y)u^{\text{DFM}}_t(y,x\mid x_1) = \frac{\dot\alpha_t}{1-\alpha_t} \delta_{x_1}(y). For masking it coincides with the diffusion rate. For uniform noise, the difference udiff−uDFMu^{\text{diff}} - u^{\text{DFM}} is in detailed balance with qt(⋅∣x1)q_t(\cdot\mid x_1) and therefore leaves every marginal unchanged; it is the ηRDB\eta R^{\text{DB}} freedom of [12, Prop. 3.3]. Both rates are functionals of the same posterior q(x1∣xt)q(x_1\mid x_t).

The training and sampling differences the reviewer notes therefore reduce to two choices:

  • Sampling: which generator to simulate for one fixed path, analogous to DDPM vs. DDIM or probability-flow sampling of one continuous diffusion model.
  • Training: cross-entropy (DFM) vs. ELBO (diffusion). Both are minimized by the true posterior q(x1∣xt)q(x_1\mid x_t) and differ only in how they weight errors.

Neither choice affects DDNO, which needs only a uniform source and a sampler built from categorical transitions.

Our text model is UDLM-type. GIDD-Unif-3B [14] is a uniform-noise discrete diffusion language model, trained with the GIDD ELBO and sampled with its adaptive sampler, so it is a UDLM-type model in the reviewer's sense. FUDOKI, used for images, is a DFM model. DDNO applies to both without any change to the method; only the surrogate of each model's own sampler differs (Algorithms 1–2). Masked (MDLM-type) models start every trajectory from the same all-[MASK] state and so have no initial noise to optimize (Sec. 2.1).

[12] Campbell et al. "Generative Flows on Discrete State-Spaces: Enabling Multimodal Flows with Applications to Protein Co-Design". ICML 2024.

[13] Pauline et al. "Foundations of Diffusion Models in General State Spaces: A Self-Contained Introduction". arXiv 2025.

[14] von Rütte et al. "Scaling Behavior of Discrete Diffusion Language Models". ICLR 2026.


We hope this addresses the reviewer’s concerns, in particular regarding soundness, and would be grateful if they could let us know whether any remain.

Reviewer_xajPOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes Discrete Diffusion Noise Optimization (DDNO), a training-free method for test-time reward alignment of uniform-noise discrete diffusion models. It parameterizes the initial discrete noise with continuous logits and uses straight-through estimation, fixed intermediate randomness, and soft mixing to backpropagate reward gradients through a frozen generative model. Experiments on text-to-image and text generation show gains over search-based baselines at larger compute budgets. On GenEval, the method achieves 0.94 with its differentiable surrogate sampler and 0.86 when the optimized noise is decoded with the original sampler.

Critical strengths

Strengths

  • Well-motivated problem. The paper extends initial-noise optimization from continuous to discrete diffusion. Because discrete tokens and intermediate categorical sampling block gradient propagation, this extension is not straightforward.

  • Purposeful surrogate design. Optimizable logits, Gumbel straight-through estimation, fixed intermediate randomness, and soft mixing address distinct obstacles to gradient-based optimization. The ablation study in Table 1 shows that preserving unoptimized sample quality alone is insufficient; these components work together to enable stable optimization.

  • Evaluation beyond the optimization reward. The image experiments report GenEval rather than relying solely on the optimized vision-language reward, while the topic-steering experiments use a separate NLI evaluator. These results provide stronger evidence of meaningful improvements than reward gains alone.

  • Transparent discussion of computational costs. The authors compare methods under matched GPU-hour budgets and discuss the latency advantage of batched Best-of-NN sampling in the appendix. This helps clarify that DDNO is most suitable when output quality matters more than single-request latency.

Critical weaknesses

Weakness

  • A substantial gap between surrogate and original sampling. DDNO achieves its headline GenEval score of 0.94 with the differentiable surrogate sampler. Decoding the optimized noise with the original sampler yields 0.86, below Best-of-NN (BoN) at 0.91. The latter result still demonstrates that noise optimization is effective, but the claim of outperforming BoN depends on using the surrogate for final generation. The paper should distinguish these settings more clearly in its method description and conclusions.

  • A gap between the stated objective and its implementation. The formal objective maximizes expected reward over intermediate randomness, whereas the implementation fixes one set of Gumbel samples and optimizes a deterministic surrogate trajectory. Soft mixing preserves a one-step conditional mean, but does not establish equivalence after subsequent nonlinear transitions. The drop from 0.94 to 0.86 under original decoding warrants a more systematic analysis of this approximation.

  • Matched compute does not imply lower inference latency. DDNO requires backpropagation through each sampling trajectory and sequential optimization steps, with roughly twice the image-generation memory usage. The appendix estimates that batching BoN candidates would move the quality crossover to approximately 30 GPU-hours, near the limit of the measured range. Its efficiency advantage should therefore be framed in terms of output quality at larger total compute budgets, rather than single-request latency.

  • Limited scope and mixed text-generation evidence. The method requires a random uniform-noise prior and does not directly apply to models initialized entirely with [MASK] tokens. Only one base model is tested per modality. In topic steering, improved topic scores accompany an increase in perplexity from approximately 66 to 188. The open-ended quality experiment uses only 10 prompts and evaluates quality with the same reward model used for optimization. These limitations call for a narrower cross-modal generalization claim.

Questions for authors

See the weakness

2 comments

We thank the reviewer for the thoughtful and insightful review. Most of the points raised are already addressed in the paper and we clarify each below.

1. Surrogate vs. original sampling. The paper already distinguishes the two settings: Table 2 reports them as separate rows, Table 1 ablates the surrogate, and Sec. 3.2 and App. F.7 discuss the gap. The surrogate is a sampler of the same frozen fθf_\theta, not a different model, and unoptimized it is nearly as faithful as the original sampler (0.65 vs. 0.67). The original-sampler row answers a different question, namely whether the optimized noise is better on its own, and it is: +19 pp (0.67 → 0.86) under a sampler that was never differentiated through. To avoid any ambiguity, we will name the corresponding sampler whenever we report a result, e.g. the 0.94, throughout the paper.

2. Objective vs. implementation. Fixing ξ\xi is a deliberate design choice stated in Sec. 2.2: it turns the sampler into a deterministic, differentiable map, so DDNO maximizes a single-sample estimate of V(x0)=Eξ[R(Gθ(x0;ξ))]V(x_0) = \mathbb{E}_\xi[R(G_\theta(x_0;\xi))]. We want to note three results that show this is a good proxy for VV itself:

  • Fixing ξ\xi is what makes optimization work. Optimizing VV directly with stochastic gradients, i.e., resampling the Gumbel noise at every step, reaches only 0.85 vs. 0.94 (Table 1). Fixing ξ\xi removes this gradient variance, as with common random numbers.
  • The optimum holds in expectation over ξ\xi. Resampling all intermediate randomness around an optimized x0⋆x_0^\star preserves the reward (−2.26 vs. −2.23, Fig. 4), so the fixed-ξ\xi maximizer also scores high under VV.
  • It transfers beyond the surrogate. The optimized noise carries over across step counts (Fig. 8) and to the original sampler (+19 pp).

Regarding Rao–Blackwellization, the paper claims only mean preservation at the level of a single update (Eq. (12), App. B.1), not equivalence after later nonlinear transitions. That remaining difference is exactly what the 0.94 → 0.86 comparison in Table 2 measures. We will state this scope explicitly in the main text.

3. Latency. As stated in E.3, DDNO targets settings where output quality matters more than single-request latency. At the largest budget, all methods take ~150 s per prompt and DDNO is best (see reply to pDwe). Batched Best-of-N is faster for a single request, while across concurrent prompts DDNO batches equally well. We will move this statement from E.3 into the main text.

4. Scope and text evidence. The abstract and Sec. 2.1 restrict DDNO to uniform noise, and our two backbones already span two model classes and samplers (discrete flow matching with ancestral sampling; uniform discrete diffusion with adaptive sampling). The fluency drop under topic rewards is a property of the reward, which ignores fluency, not of DDNO, as Sec. 5 notes: with a quality-aware reward, DDNO improves fluency. To address the 10-prompt evaluation and the shared reward model, we ran the QRM experiment on 50 new prompts randomly sampled from AlpacaEval [1] (3 seeds per prompt) and added an independent judge, the reasoning reward model RM-R1-DeepSeek-Distilled-Qwen-14B [2]. For each prompt, all 3 × 3 DDNO-vs-BoN pairs are judged in both orders; a pair counts as a win only if both orders agree, otherwise as a tie. We report the mean ± s.e.m. over prompts:

NQRM proxy ↑ (DDNO / BoN)GenPPL ↓ (DDNO / BoN)Judge wins % ↑ (DDNO / BoN)
Base0.242 ± .01427.58 ± 2.89–
100.377 ± .015 / 0.357 ± .01619.65 ± 1.71 / 19.27 ± 2.0732.0 ± 3.9 / 31.3 ± 4.3
300.434 ± .013 / 0.402 ± .01516.76 ± 1.17 / 17.32 ± 2.1038.0 ± 3.6 / 29.3 ± 4.2
500.448 ± .013 / 0.426 ± .01517.23 ± 1.31 / 18.92 ± 3.4836.7 ± 2.7 / 29.3 ± 3.5

Under this conservative protocol, the judge prefers DDNO from N = 30 on, and fluency improves over the base model (GenPPL 17 vs. 28).

[1] Li et al. "AlpacaEval: An Automatic Evaluator of Instruction-following Models". 2023.

[2] Chen et al. "RM-R1: Reward Modeling as Reasoning". ICLR 2026.

Dear Reviewer xajP,

Thank you again for your insightful feedback. As the discussion period draws to a close, we would appreciate hearing whether our rebuttal addresses your concerns and would be happy to clarify any remaining questions. If your concerns have been resolved, we would be grateful if you could consider updating your assessment accordingly.

Best,

The authors

Comments on the paper

3 comments

We thank the reviewer for the careful review and the concrete requests. We ran the requested analyses and address each point below.

1. Scope. DDNO is a test-time method for a fixed discrete backbone: its claims concern how much more a given model delivers at a given budget, not competitiveness across generators. The continuous rows in Table 2 are reference numbers, as its caption states, whereas the FUDOKI-based search and DDNO rows are matched: same checkpoint, same reward, no CFG and equal compute. We will make this distinction explicit in the text. As requested, we added Qwen-Image samples for the Figure 2 prompts at the end of the appendix of the updated PDF (4 seeds each, default settings: 50 steps, guidance scale 4.0). Surrogate and original-sampler decoding are already reported as separate rows in Table 2, and we will name the sampler wherever we report a result.

2. End-to-end latency. One H100 80GB (PyTorch 2.5.1, CUDA 12.4), models resident, one prompt per request, at the largest matched budget: Best-of-N N=64 and DDNO T=64 unbatched, SVDD-PM M=28 and FK-Steering P=28 with particles batched. Timing covers prompt preparation, sampling, reward evaluation, backpropagation, optimizer updates, selection and final decoding. All methods are run without CFG.

MethodRolloutRewardBackpropTotal (s)GenEval
Best-of-N143.66.0–149.70.91
SVDD-PM19.5126.9–146.40.90
FK-Steering20.2133.5–153.70.92
DDNO48.74.195.3148.10.94

At equal end-to-end time (~150 s per prompt), DDNO reaches the highest GenEval. Best-of-N is rollout-dominated, DDNO backprop-dominated, and FK-Steering and SVDD-PM reward-dominated.

3. Budget-annotated per-category results.

MethodSettingOverallSingle objectTwo objectsCountingColorsPositionAttribute binding
DDNOT=160.800.960.890.750.900.690.59
DDNOT=240.840.990.930.740.930.760.67
DDNOT=320.880.980.980.850.950.800.72
DDNOT=500.921.000.980.880.970.870.84
DDNOT=640.941.000.980.890.980.870.91
Best-of-NN=160.850.980.910.740.910.830.75
Best-of-NN=240.880.990.930.780.930.870.78
Best-of-NN=320.890.990.940.800.930.890.82
Best-of-NN=500.900.990.950.820.940.910.82
Best-of-NN=640.910.990.960.830.940.920.83
SVDD-PMM=50.851.000.940.740.900.760.78
SVDD-PMM=100.891.000.970.850.930.790.80
SVDD-PMM=200.910.990.970.850.950.870.81
SVDD-PMM=280.900.990.950.810.930.890.83
FK-SteeringP=50.860.990.940.760.920.840.74
FK-SteeringP=100.900.980.950.840.910.900.80
FK-SteeringP=200.910.990.950.850.940.910.84
FK-SteeringP=280.920.990.960.850.950.910.87

Specifically for the requested categories, the largest budget, DDNO leads on counting (0.89 vs. 0.85 for FK-Steering) and attribute binding (0.91 vs. 0.87), while search remains stronger on position (0.92 for Best-of-N vs. 0.87).

4. Are text rewards computed on decoded text? Topic steering: all reported rewards are recomputed on decoded, re-tokenized text (G.6). DDNO selects its best iterate with the bridge reward, while Best-of-N selects directly on the reported decoded reward, so the protocol, if anything, favors the baseline.

QRM: Table 3 reported bridge rewards. We re-scored every selected output with the official QRM tokenization, chat template and forward pass, without changing selection (10 prompts × 5 seeds, compute-matched):

T / NDDNOBest-of-NΔ95% paired CI
10 / 300.4260.427−0.001[−0.063, +0.058]
30 / 900.4920.485+0.007[−0.029, +0.049]
50 / 1500.5200.514+0.006[−0.015, +0.027]

The bridge closely tracks decoded scoring and overestimates both methods by the same amount (+0.028 for DDNO, +0.025 for Best-of-N at the largest budget), so the approximation does not favor DDNO. On compute-matched QRM reward, the two are on par.

5. Independent validation on more prompts. Since QRM is also the optimization target and Table 3 covers only 10 prompts, we ran the QRM experiment on 50 new prompts randomly sampled from AlpacaEval [1] (3 seeds per prompt) and added an independent judge, the reasoning reward model RM-R1-DeepSeek-Distilled-Qwen-14B [2]. For each prompt, all 3 × 3 DDNO-vs-BoN pairs are judged in both orders; a pair counts as a win only if both orders agree, otherwise as a tie. We report the mean ± s.e.m. over prompts:

NQRM proxy ↑ (DDNO / BoN)GenPPL ↓ (DDNO / BoN)Judge wins % ↑ (DDNO / BoN)
Base0.242 ± .01427.58 ± 2.89–
100.377 ± .015 / 0.357 ± .01619.65 ± 1.71 / 19.27 ± 2.0732.0 ± 3.9 / 31.3 ± 4.3
300.434 ± .013 / 0.402 ± .01516.76 ± 1.17 / 17.32 ± 2.1038.0 ± 3.6 / 29.3 ± 4.2
500.448 ± .013 / 0.426 ± .01517.23 ± 1.31 / 18.92 ± 3.4836.7 ± 2.7 / 29.3 ± 3.5

Under this conservative protocol, the judge prefers DDNO from N = 30 on, and with a quality-aware reward, fluency improves over the base model (GenPPL 17 vs. 28) rather than degrading as with topic-only rewards. However, we do agree that neither QRM nor perplexity measures factual or reasoning correctness, and will present the text results as controllability under quality-aware rewards rather than general response-quality superiority.

[1] Li et al. "AlpacaEval: An Automatic Evaluator of Instruction-following Models". 2023.

[2] Chen et al. "RM-R1: Reward Modeling as Reasoning". ICLR 2026.

Dear Reviewer pDwe,

Thank you again for your insightful feedback. As the discussion period draws to a close, we would appreciate hearing whether our rebuttal addresses your concerns and would be happy to clarify any remaining questions. If your concerns have been resolved, we would be grateful if you could consider updating your assessment accordingly.

Best,

The authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.