- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper studies diffusion model alignment from a distributional perspective, focusing on how preference-learning updates affect the model's output distribution and how this effect evolves over generation time. It introduces a directional Fisher information framework for quantifying the impact of terminal potential updates, derives a temporal decomposition linked to intermediate-state predictability and score sensitivity, and uses the resulting terminal Fisher criterion for active preference query selection. The framework is validated analytically and empirically, with experiments demonstrating meaningful temporal structure and improved label efficiency over random selection.
Critical strengths
One strength of the paper is its problem formulation, which shifts the focus from parameter estimation to identifying preference labels that are expected to induce larger changes in the generated output distribution. This provides an interesting and relatively novel perspective on active preference learning for diffusion alignment.
Critical weaknesses
-
Highly confusing presentation: The paper lacks a clear narrative thread, and the relationship between Fisher-based query selection and the temporal decomposition is not established clearly enough. The theoretical sections are particularly difficult to follow and are not sufficiently self-contained. For example, Conditions L, E, S, and N are repeatedly invoked in the main results, while several of them are only fully specified in the appendix, making it difficult to track the assumptions behind each claim. The experimental section further mixes several distinct settings, objectives, and evaluation protocols, which substantially reduces the overall readability of the paper.
-
Limited practical relevance of the proposed criterion: The paper appears to target efficient collection of human comparison labels by selecting pairs whose possible labels are expected to induce large changes in the generated output distribution. However, evaluating this criterion for a general reward model can be prohibitively expensive, since each candidate pair and each possible response requires estimating the corresponding model update. The paper only provides a tractable implementation for a highly simplified linear potential on frozen features, where the criterion reduces to a quadratic form in the feature covariance. It is therefore unclear whether the proposed approach remains practical for realistic neural reward models. Moreover, the temporal decomposition is mainly used as an analysis tool, and the paper provides limited evidence that it offers actionable guidance in realistic alignment settings. The experiments are also restricted to relatively simplified settings, leaving a substantial gap between the demonstrated results and practical preference learning.
Questions for authors
- How can the be computed efficiently when the potential is parameterized by a general nonlinear neural network rather than the linear form? In particular, does the proposed method require a hypothetical model update for every candidate pair and every possible response, or is there a scalable approximation that avoids this cost?
- What is the main motivation for the temporal analysis beyond characterizing when information becomes predictable during generation? More specifically, how is the temporal decomposition expected to guide preference learning in practice, given that the main active learning experiments use only the terminal Fisher information rather than the temporal decomposition?
2 comments
Response to Reviewer Rhw3 (2/2): computation and scope of the experiments
Q1 / W2-a. Is a hypothetical update for every candidate and answer required? How can the criterion be computed for nonlinear reward models?
Retraining for each candidate and answer is not needed. The criterion is defined through the update that each answer would cause, and this update can be evaluated in closed form or by a local approximation (§4, App. D).
- Linear potentials with the DA learner (the setting of the paper). The two hypothetical updates of the coefficients are , where only and depend on the candidate. The cross term cancels after averaging over the answer, and Eq. 7 becomes a constant plus , one quadratic form per candidate. This agrees with explicitly computing every update to a relative difference of and selects the same pair. The submission already computed the criterion in this way. The cost of our original implementation was dominated by forming a covariance matrix. An optimized implementation takes 6.9 ms (feature dimension 128) and 25.4 ms (dimension 256) per acquisition step with 64 candidates, instead of 644 ms and 2,568 ms, with the same values up to floating-point rounding (App. E.5).
- Nonlinear potentials. For a differentiable trained by regularized maximum likelihood, the effect of one more comparison is approximated by one Gauss–Newton step with the generalized Gauss–Newton matrix , where the term of the new comparison enters through the Sherman–Morrison formula. With , the criterion is approximately . It needs gradients at the candidate and reference images and linear solves with , as in influence functions (App. D.6).
- Validation (§5.1, App. F.1). For a multilayer perceptron (MLP) potential with hidden width 64, we compared the approximation with exact retraining for all 64 candidates and both answers. The rank correlation is 0.995 on average, with a minimum of 0.981 over 12 training states. The chosen candidate attains 0.999 of the exact maximum on average, and the computation is 9.2 times faster. With this approximation, Fisher selection improves test CE over random selection (-0.0078 and -0.0054, where the second interval excludes zero; §5.3).
W2-b / W2-d. Only a linear potential on frozen features is used, and the settings are simplified.
The revision extends the evaluation in the following directions, with the method unchanged.
- Learner. The MLP potential above.
- Generator and data. PixArt-α as the reference generator (Fisher minus random in mean CE: -0.0205 and -0.0107) and Pick-a-Pic as a second dataset (App. F.4).
- Seeds and budget. The human-preference comparison now uses 16 seeds per training set and is continued to 512 labels. At 256 labels, Fisher selection is lower than D-optimal in all 12 conditions and for every seed, and the difference remains at 512 labels (-0.0026 [-0.0049, -0.0003] and -0.0030 [-0.0053, -0.0008], §5.3).
- Use in generation. On Stable Diffusion v1.5, the potential learned from human labels is applied during generation by interval-restricted resampling (§5.5).
- Evaluators. Image selection is also evaluated with PickScore, HPS v2.1 and CLIP ViT-H/14, which are not trained on ImageRewardDB (§5.4).
Two limits remain and are stated in §7. The nonlinear experiment uses an MLP on frozen features and does not train all layers of a large reward model. The time-weighted query selection is tested with synthetic preferences.
Soundness. Besides the table of conditions, the appendix now separates the quantities defined in the theory from the approximations used in the image experiments, namely predictions from denoised states in place of exact conditional expectations and a finite number of particles in place of the exact -transform (App. A.6). It also contains the proofs of Corollary 2 and of the bound for finite updates (App. A.3, A.5).
We hope the revision resolves the concerns about clarity and computation, and we would be glad to clarify any remaining point.
Response to Reviewer Rhw3 (1/2): presentation and the role of the temporal analysis
Thank you for the detailed and critical review. We revised the presentation along the lines you indicated, and we added results on computation and on the practical use of the temporal analysis. Section numbers refer to the revised PDF. W1, W2, Q1 and Q2 denote the numbered weaknesses and questions of the review, and W1-a to W1-c and W2-a to W2-d are the points raised within W1 and W2.
W1-a. The relation between Fisher selection and the temporal decomposition is unclear.
Both describe the same object, the update g of the terminal potential that results from one answer.
In the revision, the acquisition criterion is defined in the method section (§4) instead of inside the experiments, and §3.3 states this relation.
W1-b. Conditions L, E, S and N are given only in the appendix.
§3.1 now states which conditions the image experiments rely on, and Table 1 (App. A.1) lists each condition with the results that use it: L (Markov property and square integrability) for the variance decomposition, E (exponential moment) for the score, the Fisher information and the KL expansion, S (smoothness) for the continuous-time statement, N (Gaussian transitions) for the estimator of G, and MC for the unbiased estimator of Ij. The image experiments use the finite-step statements, which do not need S.
W1-c. The experimental settings are mixed.
§5 now starts with a roadmap of the experiments, and App. E starts with an overview table (Table 3: purpose, generator, source of preference labels, learner, metric). The experiments are separated by purpose: numerical validation (§5.1), temporal analysis (§5.2), label efficiency with human labels (§5.3), image selection (§5.4), and use of the decomposition for generation and query selection (§5.5).
Q2 / W2-c. What motivates the temporal analysis, and how does it guide preference learning in practice? It appears to be only an analysis tool.
The motivation is that a potential can be used in a part of generation only to the extent that its information lies there (Corollary 2). We now test two practical uses (§5.5). In both, the prediction is computed from trajectories generated without resampling, before any generation with the potential is run.
Our response continues in the next comment.