- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes IQ-Diff, a method for sampling-time alignment of frozen diffusion models using terminal rewards from a black-box physics simulator. The algorithm repeatedly collects denoising trajectories from the current guided sampler, trains new Q-functions through Monte Carlo return regression, and accumulates their scores with existing guidance for finite-candidate selection at subsequent denoising steps. The authors provide a conditional performance-improvement analysis of the finite-candidate selection process and report comparisons of prediction accuracy and online sampling costs on fluid dynamics and thermal-process prediction tasks.
Critical strengths
The research question is valuable, and the motivation is clear. The paper examines how to improve the outputs of frozen diffusion models when only scalar terminal rewards are available and simulator gradients cannot be used. Changes to the continuation policy can alter the value of earlier denoising actions, motivating re-evaluation of the current sampler. This motivation is consistent with the proposed design of iteratively collecting trajectories and learning new Q-functions.
Critical weaknesses
-
The experimental evaluation lacks rigor. IQ-Diff performs four training rounds, collecting 1,000 trajectories per round, and accumulates guidance from four Q-networks during sampling. Single-Q, by contrast, uses only one round and one Q-network. The comparison therefore changes the number of policy evaluations, the total amount of reward-labeled data, training compute, and the number of models used at deployment simultaneously. This makes it difficult to determine whether the gains arise from re-evaluating the current policy or from using more data and additional models.
-
The connection between the theory and the practical Q-learning procedure is insufficient. The theory directly analyzes the finite-candidate selection process, but its main guarantees rely on exact Q-values or a uniform relative-error bound across all relevant states, denoising stages, and candidate sets. The experiments primarily report final prediction metrics without sufficiently demonstrating how current-policy re-evaluation improves candidate-value estimates or whether these improvements explain the observed gains. The theory therefore provides conditional optimization justification, but does not sufficiently establish the reliability of improvements under the practical finite-data learning procedure.
Questions for authors
-
Does IQ-Diff retain its advantage over Single-Q when the total number of reward-labeled trajectories and the training compute budget are matched? After additionally controlling the number of Q-networks and the effective guidance strength, what are the separate benefits of current-policy re-evaluation and accumulated updates?
-
Can the authors use additional continuation rollouts at different denoising stages and outer iterations to estimate true expected returns and evaluate relative Q-estimation error, candidate-ranking accuracy, and return calibration? Does re-evaluation actually improve these quantities, and are these changes consistent with improvements in actual rewards and prediction metrics against high-fidelity references?
-
Can the authors report the full costs of trajectory generation, reward labeling, Q-training, and deployment, and provide an amortization analysis relative to BoN-16 and TDS-Reward? For the fluid task, are simulator reference fields cached, and what are the respective counts of reward-interface calls and actual simulator executions?