acceptodds
Withdrawn from ICLR 2027

IQ-Diff: Iterative Q-guided Sampling-Time Diffusion Alignment with Black-Box Physics Rewards

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Diffusion models provide generative priors for physical prediction, but statistically plausible outputs need not satisfy downstream physical objectives. We study sampling-time alignment when feedback is available as a scalar terminal reward from a black-box physics simulator. IQ-Diff keeps the diffusion backbone frozen and repeatedly learns action-value functions from trajectories of the current guided sampler. Re-evaluation accounts for changes in the continuation policy, while accumulated Q-scores implement successive KL-regularized updates over local denoising candidates. Deployment uses these learned scores without simulator gradients or further simulator calls. We analyze the finite-candidate selection process directly, establishing a final-iterate bound under controlled Q-estimation error and separating this optimization error from the proposal's coverage limitation. Exact current-policy evaluation yields monotonic reward improvement and an final-iterate guarantee relative to the best selector with the same candidate budget, where is the number of outer iterations. Experiments on fluid dynamics and thermal-process prediction demonstrate improved prediction accuracy and favorable quality–cost tradeoffs against unguided sampling, final-sample search, physics-reference guidance, single-Q guidance, and SMC-style resampling.

Primary area:
applications to physical sciences (physics, chemistry, biology, etc.)
Keywords:
Diffusion Models, Inference-Time Alignment, Black-Box Simulator Guidance, Scientific Machine Learning

Reviews

Among:
Shape:
1234this paper, Bottom 9%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.50 over 4 reviews, above 9% of the others.
Reviewer_GYz6OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IQ-Diff, a method for sampling-time alignment of frozen diffusion models using terminal rewards from a black-box physics simulator. The algorithm repeatedly collects denoising trajectories from the current guided sampler, trains new Q-functions through Monte Carlo return regression, and accumulates their scores with existing guidance for finite-candidate selection at subsequent denoising steps. The authors provide a conditional performance-improvement analysis of the finite-candidate selection process and report comparisons of prediction accuracy and online sampling costs on fluid dynamics and thermal-process prediction tasks.

Critical strengths

The research question is valuable, and the motivation is clear. The paper examines how to improve the outputs of frozen diffusion models when only scalar terminal rewards are available and simulator gradients cannot be used. Changes to the continuation policy can alter the value of earlier denoising actions, motivating re-evaluation of the current sampler. This motivation is consistent with the proposed design of iteratively collecting trajectories and learning new Q-functions.

Critical weaknesses
  1. The experimental evaluation lacks rigor. IQ-Diff performs four training rounds, collecting 1,000 trajectories per round, and accumulates guidance from four Q-networks during sampling. Single-Q, by contrast, uses only one round and one Q-network. The comparison therefore changes the number of policy evaluations, the total amount of reward-labeled data, training compute, and the number of models used at deployment simultaneously. This makes it difficult to determine whether the gains arise from re-evaluating the current policy or from using more data and additional models.

  2. The connection between the theory and the practical Q-learning procedure is insufficient. The theory directly analyzes the finite-candidate selection process, but its main guarantees rely on exact Q-values or a uniform relative-error bound across all relevant states, denoising stages, and candidate sets. The experiments primarily report final prediction metrics without sufficiently demonstrating how current-policy re-evaluation improves candidate-value estimates or whether these improvements explain the observed gains. The theory therefore provides conditional optimization justification, but does not sufficiently establish the reliability of improvements under the practical finite-data learning procedure.

Questions for authors
  1. Does IQ-Diff retain its advantage over Single-Q when the total number of reward-labeled trajectories and the training compute budget are matched? After additionally controlling the number of Q-networks and the effective guidance strength, what are the separate benefits of current-policy re-evaluation and accumulated updates?

  2. Can the authors use additional continuation rollouts at different denoising stages and outer iterations to estimate true expected returns and evaluate relative Q-estimation error, candidate-ranking accuracy, and return calibration? Does re-evaluation actually improve these quantities, and are these changes consistent with improvements in actual rewards and prediction metrics against high-fidelity references?

  3. Can the authors report the full costs of trajectory generation, reward labeling, Q-training, and deployment, and provide an amortization analysis relative to BoN-16 and TDS-Reward? For the fluid task, are simulator reference fields cached, and what are the respective counts of reward-interface calls and actual simulator executions?

Reviewer_H9tWOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a resampling method (IQ-Diff) to guide diffusion model generating the solution of the specific PDE. The proposed method uses a Q-function weighted resampling to attain samples with high reward, which does not gradients of rewards in resampling. IQ-Diff shows good performance on PDE tasks.

Critical strengths

The proposed resampling method does not need gradients of rewards. This gives possibilities for broad applications of guidance sampling.

Critical weaknesses

The proposed guidance method is essentially a resampling method, which would not be very efficient for guidance in my opinion. In the really low probability region, it is supposed to need large amount of samples. To show that pure resampling could be powerful for guidance, the authors need more solid experiments. The current experiments part of this paper is weak.

  1. There are only two PDEs in experiments.

  2. Relative L2 error should be added as a metric in PDE tasks.

  3. Some important related works that also do not need gradients of rewards but add theoretically derived guidance terms to diffusion process, like LiDAR (Lookahead Sample Reward Guidance for Test-Time Scaling of Diffusion Models), should be added to the experiments as baselines.

Questions for authors

The training process and inference process are not very clear in the paper.

  1. In Algorithm 1 Line 7, how is (13) used to generate N samples in the next iteration of training process?

  2. In inference process, does every step choose one sample from the categorical distribution? What if just choose the one with highest Q?

Reviewer_gXHCOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes IQ-Diff, a method for guiding diffusion models using rewards from a black-box physics simulator. It trains Q-functions on sampled trajectories and uses their scores to guide the sampling process. The method keeps the diffusion model frozen and updates the guidance over multiple iterations. Experiments on two physics prediction tasks show improved performance over several baselines.

Critical strengths
  1. The paper studies the setting of black-box physics rewards, which could be useful in practice.
  2. The method achieves good results on both physics prediction tasks, with clear improvements over several baselines.
  3. The paper provides theoretical analysis of the proposed sampling method.
Critical weaknesses
  1. Limited novelty. The main idea is similar to existing reward-guided diffusion methods, especially value-based guidance, such as Soft Value-Based Decoding (SVDD) [1]. Learning value functions from reward signals and using them to guide sampling has been studied before. The main difference is the iterative Q-learning and score accumulation. However, iterative policy evaluation and improvement are standard ideas in RL, and score accumulation is closely related to KL-regularized policy updates. Therefore, I think the novelty is incremental.
  2. Lack of a fair comparison. In Table 3, IQ-Diff uses more reward-labeled trajectories than Single-Q (K=1). This makes it difficult to attribute the improvement to iterative guidance rather than the additional training data. The authors also mention this limitation in Section 5.1 (lines 369–372), but do not provide a matched-budget experiment.
  3. Unclear motivation for black-box guidance. Both benchmarks use explicitly defined physics-based rewards. In particular, the fluid reward is differentiable with respect to the generated prediction. It is unclear whether these experiments sufficiently demonstrate the need for black-box reward guidance.
  4. Missing comparison with reward fine-tuning methods. The paper lacks comparisons with reward-based fine-tuning approaches, such as DDPO [2], which can also handle black-box rewards and avoid reward evaluations during inference. Moreover, the cost analysis excludes offline Q-function training. It is therefore unclear whether IQ-Diff provides a better overall accuracy-cost tradeoff.

[1] X. Li et al. Derivative-Free Guidance in Continuous and Discrete Diffusion Models with Soft Value-based Decoding. NeurIPS, 2025.

Questions for authors
  1. Can the authors provide a matched-budget comparison between Single-Q and IQ-Diff, using the same number of reward-labeled trajectories and a similar training budget?
  2. In the fluid benchmark, the reward is differentiable with respect to the generated prediction. Can the authors explain why black-box guidance is needed in this setting? It would be helpful to include experiments with non-differentiable rewards.
  3. How does IQ-Diff compare with reward-based fine-tuning methods such as DDPO [2]? References

[2] K. Black et al. Training Diffusion Models with Reinforcement Learning. ICLR, 2024.

Reviewer_s216OpenReview ↗
Rating
1
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes IQ-Diff, a sampling-time alignment method for diffusion models with a black-box scalar terminal reward. The diffusion backbone is frozen; at each DDIM step, N candidates are drawn from the transition distribution and one is selected by a softmax over accumulated action-value scores. The Q-functions are trained by Monte Carlo regression on rollouts of the current guided sampler and the simulator reward, and the procedure is iterated K times with the new Q added to the accumulated score. Deployment requires no simulator calls.

Critical strengths
  • The setting is well motivated: a black-box physics reward with no gradients is a realistic constraint, and keeping the backbone frozen is a practical choice.
  • The method is simple to implement, with one U-Net call per step, and the deployment cost is attractive compared to BoN and SMC-style methods.
Critical weaknesses
  • Lack of novelty. I agree with the authors’ own assessment in App. A (L750–755): the candidate-selection mechanism is that of soft value-based decoding (Li et al., 2025), and the accumulation of successively estimated Q-functions is the Politex scheme (Abbasi-Yadkori et al., 2019), with Theorem 1 being the corresponding policy-mirror-descent rate. What remains is the empirical claim that iteration helps, which the ablation does not isolate (see below).

  • The central ablation does not isolate the proposed mechanism. IQ-Diff (K=4) is trained on 4000 labeled rollouts and Single-Q on 1000. The authors themselves write (L371–372) that “a matched-budget comparison is needed to isolate iteration from additional training data”, but no such comparison is provided? In addition, Latest-Q, which uses only the Q trained on the current sampler, is below Single-Q in Table 3 (31.93 vs 32.56 dB). If re-evaluation under the current policy were the source of the gain, I would expect the opposite.

  • The improvement over Single-Q is marginal. In Table 1, MSE goes from 0.0007 ± 0.0005 to 0.0005 ± 0.0003, and in Table 3 the K=1, 2, 4 variants give 0.0007, 0.0007 and 0.0005, all within the test-set std. The same holds for the qualitative comparison: in Fig. 2, I cannot see a visible difference between the Single-Q and IQ-Diff columns, and both show a grainy texture that is absent from the ground truth.

Questions for authors

See Weaknesses.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.