acceptodds
Under review as a conference paper at ICLR 2027

Bridging Candidate Selection and Mean-Shift Guidance for Diffusion Reward Alignment via Linear Noise Ranking

Abstract

Training-free reward alignment of diffusion models follows two routes. Candidate selection draws candidate next states at each denoising step and keeps the one with the highest estimated reward, spending neural evaluations per step. Mean-shift guidance instead moves the transition mean along a single reward gradient. We bridge these two regimes by utilizing a linear functional towards a selected direction as a proxy reward model to rank the Gaussian noises at each steps and select according to those scores. Scoring each noises only requires one inner product, bypassing the neural network forward costs. Our analysis locates this rule precisely between the two routes. We propose a single representation that unifies the candidate selection method and the mean shift method, and view their difference only by the distribution of the strength scalar towards the reward optimizing direction. This enables us to derive exact leading order predictions of the relative reward optimization gaps between candidate selection method and mean shift method, which are validated through measurements. We further provide practical methods to compute the reward optimizing direction: a corrected direction whose benefit it predicts, and a least-squares slope, fitted from the same reward queries, that extends the rule to non-differentiable rewards. Our experiments show that our linear noise ranking matches or exceeds per-candidate selection at equal or lower neural cost and ties with strength-matched mean shift method as predicted.

Primary area:
generative models
Keywords:
Diffusion Models, Test-time alignment, Reward optimization

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_pnWMOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

LNR is a training-free inference-time steering method for diffusion models. It ranks Gaussian candidate noises using a shared reward direction, avoiding repeated neural scoring. For the single-winner rule, the paper derives a cubic asymptotic expression for local proxy regret relative to exact scoring of the same candidate pool and characterizes the noise distribution and its bounded density ratio. It also explains the relationship to matched mean-shift guidance. Experiments cover image generation, non-differentiable rewards and D4RL control.

Critical strengths

S1. The core method and local analysis are clearly explained. The linear approximation gives an intuitive reason for inexpensive ranking. The explicit regret coefficient, near-tie argument and noise decomposition offer useful insight, supported by the MNIST experiments.

S2. The 50-step single-winner experiment shows a useful reward–time trade-off. On the same GPU, LNR achieves PickScore 0.2294 in 11.5 seconds, compared with 0.2279 in 19.6 seconds for four-candidate SVDD-PM [1]. Direct winner sampling is a useful consequence of the noise law, and the paired interval and candidate-count sweep support this comparison.

S3. The evaluation includes SDXL, FLUX, black-box rewards, control and direction-refinement ablations. The authors acknowledge mixed held-out metrics, limited task-wise RL advantages and the scope of the single-winner theory.

Critical weaknesses

W1. Baseline validation is my main concern. LAGS [2] gives almost no improvement in Table 2 (0.2167 versus 0.2172 for the base model), and the separate 50-step comparison in C.5 says its original benchmark reproduction remains unverified. DOIT [3] is a JPEG outlier in Table 3: 102.31 KB, compared with 108.50 KB for the base model and 33.59–41.36 KB for the other guided methods. Both results merit validation under the changed reward and sampling protocols; they do not establish implementation errors.

W2. It is unclear how much of the broader practical gain comes from the selection distribution itself. Matched mean shift performs about as well as basic LNR, while stronger results often use aggregation and direction refinement. Some practical details also remain unclear, including signed aggregation and its normalization, the stochastic FLUX transition and the counting configuration.

W3. Uncertainty and inference costs are less well documented outside the primary comparison. Most image results lack uncertainty estimates, making small differences such as two-object accuracy of 0.818 versus 0.808 hard to assess. Table 4 reports RL standard deviations but not run or episode counts. Its averages of 81.2, 81.6 and 82.4 for DOIT, LNR and corrected LNR suggest competitive performance; the efficiency comparison remains unclear because their sampling and gradient budgets differ.

Questions for authors

Q1. Beyond the parameters in C.5, how were baseline configurations transferred and tuned, including reward transformations? Could you validate LAGS against its original SD/PickScore setup [2] and explain DOIT's limited JPEG improvement?

Q2. Could you clarify signed aggregation and normalization, the FLUX stochastic transition, and the RL selection and correction settings? Which counting configuration produced 0.700: M=2 in Section 5.2 or M=3 in Table 6? For Table 5, how do the timings relate to C.5 and the Table 3 results? Please explain the timed reward, hardware and dagger markers, and confirm whether LiDAR's matched budget refers to reward-query counts.

Q3. How does corrected LNR compare with mean shift using a comparable direction-refinement rule, mean displacement and gradient budget?

Q4. Beyond the paired interval in C.5, could you report uncertainty for the other main image comparisons, especially counting and two-object accuracy, and specify the RL run and evaluation-episode counts?

Q5. Could you report inference costs for Table 4 and compare RL returns across sampling budgets, including terminal best-of-N action selection for both LNR variants and representative baselines? DOIT's original evaluation uses this selection [3]. Comparing with unsteered best-of-N at similar total inference time would help show what stepwise steering adds.

Q6. C.3.2 reports that increasing M from 32 to 64 barely changes PickScore but lowers ImageReward from 0.785 to 0.421. How were settings chosen given this reward–quality trade-off, and how consistent is the decline across prompts and seeds?

I may raise my rating if these concerns are convincingly addressed.

References

[1] Xiner Li et al. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding. NeurIPS, 2025.

[2] Heasung Kim et al. Efficient Weighted Sampling via Score-based Generative Models. CVPR, 2026.

[3] Qijie Zhu et al. Training-Free Adaptation of Diffusion Models via Doob's h-Transform. ICML, 2026.

Reviewer_v3JuOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2(+1)
Soundness
2
Read review
Paper summary

The paper introduces Linear Noise Ranking (LNR), a guidance method for training-free reward alignment of diffusion models. LNR uses a first-order approximation of the guidance term to rank candidate noises by their inner products with a shared local gradient. This allows LNR to select high-reward noise without candidate-specific neural evaluations, reducing the computational cost of selection-based guidance. The paper also provides a theoretical analysis of the regret introduced by linear ranking and its difference from mean-shift guidance. Experiments on diffusion alignment tasks, including image generation and RL policies, demonstrate the effectiveness of the proposed method.

Critical strengths
  1. Intuitive motivation and method. Mean-shift guidance has no finite worst-case bound relative to the base transition, while selection-based guidance needs expensive neural evaluation. LNR addresses each method's limitation by using a simple idea: one local reward direction to rank many candidate noises.

  2. Theoretical analysis and proof by experiments. The paper provides a theoretical analysis of the proposed method and effectively validates its predictions in practical experimental settings.

Critical weaknesses
  1. Gap between theoretical analysis and practical implementation. The theoretical analysis focuses on selecting a single noise candidate, whereas the main image experiments use noise aggregation with additional direction correction. The stated guarantees do not directly cover this full procedure, leaving the connection between the theoretical results and the main empirical gains unclear. Without these additional techniques, basic LNR shows no clear performance advantage over mean-shift guidance (Table 1).

  2. Presentation of the paper could be improved. While the proposed method is reasonable, several aspects of the presentation make the paper difficult to follow. For example, the term LNR is used in the Introduction without expanding the acronym, and (w_t) appears in the mean-shift guidance paragraph before being defined. Additionally, the target direction variants and noise aggregation described in Section 5 would be better presented in the method section, given their substantial impact on performance.

Questions for authors
  1. Proposition 4.3 suggests that LNR can outperform mean-shift guidance at matched mean displacement when the local reward proxy has negative curvature along the gradient direction, in the small-noise regime. Is there any way to evaluate whether these conditions hold for rewards used in practical real-world applications? If this cannot be assessed in practice, what justifies choosing LNR over simpler mean-shift guidance?

  2. LNR appears to achieve larger gains over the baselines with non-differentiable rewards than when reward gradients are available. Could the authors explain what drives this difference?

  3. The main image experiments use a noise pool of K=214K=2^{14}. Do the authors provide an ablation showing how performance changes with smaller noise pools? What motivates using such a large pool, and is it necessary to achieve the reported gains?

Reviewer_ypyaOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates the problem of "training-free reward alignment" for diffusion models, aiming to unify two existing approaches: candidate sampling (e.g., SVDD, which generates KK candidate samples per step and scores them individually) and mean-shift guidance (which shifts the mean of the transition distribution along the gradient of the reward function). It proposes the LNR method, which ranks candidate noise vectors using a linear function ⟨wt,ε⟩\langle w_t, \varepsilon \rangle. This approach requires only a single inner-product computation per candidate, thereby avoiding KK forward passes through a neural network. The theoretical analysis provides a precise characterization of this selection mechanism and derives the "regret gap"—relative to exact scoring—between LNR and its corresponding mean-shift counterpart, with leading terms of σ3\sigma^3 and σ2\sigma^2, respectively. The mechanism is validated on the MNIST dataset, with further large-scale experiments covering image generation tasks (using SD1.5, SDXL, and FLUX) as well as control tasks from the D4RL benchmark.

Critical strengths
  1. It provides a clear and accurate theoretical bridge. Proposition 4.2 precisely characterizes the behavior of the chosen noise, enabling a non-trivial comparison of the leading-order terms (involving σ3\sigma^3 versus σ2\sigma^2); the sign of this difference is determined by the directional curvature h∥h_∥. This constitutes a falsifiable, "explanatory" contribution, and the trends observed in MNIST experiments align with the theoretical predictions.

  2. It offers a genuinely practical mechanism for cost adjustment. By employing inner-product-based ranking, the algorithm maintains a nearly constant latency as KK increases (with execution time remaining around 11.5 seconds even as KK grows from 2 to 16,384). Furthermore, by leveraging the least-squares direction, the method can be applied to scenarios where the reward function is non-differentiable, achieving state-of-the-art performance in tasks such as JPEG compression, counting, and multi-objective optimization.

Critical weaknesses
  1. This theory is local and asymptotic in nature; it has been validated solely on the MNIST dataset and fails to encompass configurations that are truly effective in practice. The theorem relies on a single-step expansion as σ→0\sigma \to 0 (the authors themselves acknowledge that it "cannot predict the convergence rate at the trajectory level"), and its empirical validation is restricted to smooth reward functions on MNIST. More importantly, the two best-performing variants—"signed top-M aggregation" and "corrected direction"—fall explicitly outside the scope of the "single-winner" analytical framework. In other words, the configurations accounted for by the theory are of minimal practical significance, while the schemes that actually prove effective are heuristic in nature.

  2. The performance gains observed in the experiments are relatively modest and not consistent across all scenarios; the study’s contribution lies in improved cost-efficiency rather than superior performance outcomes. The LNR (linear method) performs comparably to "matched mean-shift" (0.2294 vs. 0.2297). Meanwhile, the concurrently proposed "NoiseTilt" outperforms this method on the PickScore metric (0.3104 vs. 0.2637). On held-out datasets, the method underperforms relative to SVDD-PM across metrics such as IR and CLIP, while its average performance on the D4RL benchmark is merely on par with that of DOIT/DAS.

Questions for authors
  1. Your first-order predictions have only been validated on the MNIST dataset. Could you discuss whether the distinctions between σ3\sigma^3 and σ2\sigma^2, as well as the sign of the directional curvature h∥h_\parallel, still hold within the high-dimensional latent spaces of models like SD or FLUX? Alternatively, under what conditions does the expansion (as σ→0\sigma \to 0) remain relevant in practical application scenarios—specifically when σ2≈1/T\sigma^2 \approx 1/T?

  2. Methods based on "signed Top-M aggregation" and "corrected directions" yield the most significant performance gains, yet they are not encompassed by the "single-winner" theoretical framework. Is there any—even if informal—bound or theoretical rationale that links these variants to Theorem 4.1 or Proposition 4.2? Or should the performance of these methods be regarded as purely empirical results?

  3. The abstract mentions that you avoid the computational overhead of KK neural network forward passes; however, when dealing with non-differentiable rewards, the "least-squares direction" method introduces PP "scored probes" per step. Could you provide a practical comparison of the neural network computational costs in this setting, and clarify in what respects LNR still offers an advantage?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.