acceptodds
Under review as a conference paper at ICLR 2027

Fisher Information and Its Temporal Decomposition for Diffusion Model Alignment

Abstract

Aligning diffusion models with human preferences requires effective use of costly comparison labels. To assess candidate queries by their effects on generation, we study how learning from an answer changes the output distribution and how information about the resulting update is distributed over generation time. We formulate directional Fisher information for updates of a terminal potential, which reweights the model's output distribution. Using Doob's -transform, we connect the sensitivity of the score function to a temporal decomposition of this information. The decomposition characterizes when the terminal value of an update becomes predictable from intermediate states. Terminal Fisher information provides a criterion for query selection, while the decomposition supports analysis of generation. Numerical estimates agree with analytic solutions in a low-dimensional setting. On CIFAR-10, color becomes predictable earlier than car-likeness, and the temporal distribution of update information differs between query selection strategies. In direct preference optimization (DPO) with a fixed diffusion generator, Fisher-based query selection on ImageRewardDB yields lower mean test cross-entropy with 128 labels than random selection with 256 across the evaluated settings. Performance is comparable to D-optimal selection at matched label budgets. When the learned potentials select among shared generated candidates, those learned with Fisher-based query selection yield larger gains than those learned with random selection under the automatic evaluator ImageReward.

Primary area:
generative models
Keywords:
diffusion model, active learning, direct preference optimization, DPO, Fisher information

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_Rhw3OpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper studies diffusion model alignment from a distributional perspective, focusing on how preference-learning updates affect the model's output distribution and how this effect evolves over generation time. It introduces a directional Fisher information framework for quantifying the impact of terminal potential updates, derives a temporal decomposition linked to intermediate-state predictability and score sensitivity, and uses the resulting terminal Fisher criterion for active preference query selection. The framework is validated analytically and empirically, with experiments demonstrating meaningful temporal structure and improved label efficiency over random selection.

Critical strengths

One strength of the paper is its problem formulation, which shifts the focus from parameter estimation to identifying preference labels that are expected to induce larger changes in the generated output distribution. This provides an interesting and relatively novel perspective on active preference learning for diffusion alignment.

Critical weaknesses
  1. Highly confusing presentation: The paper lacks a clear narrative thread, and the relationship between Fisher-based query selection and the temporal decomposition is not established clearly enough. The theoretical sections are particularly difficult to follow and are not sufficiently self-contained. For example, Conditions L, E, S, and N are repeatedly invoked in the main results, while several of them are only fully specified in the appendix, making it difficult to track the assumptions behind each claim. The experimental section further mixes several distinct settings, objectives, and evaluation protocols, which substantially reduces the overall readability of the paper.

  2. Limited practical relevance of the proposed criterion: The paper appears to target efficient collection of human comparison labels by selecting pairs whose possible labels are expected to induce large changes in the generated output distribution. However, evaluating this criterion for a general reward model can be prohibitively expensive, since each candidate pair and each possible response requires estimating the corresponding model update. The paper only provides a tractable implementation for a highly simplified linear potential fθ=θ⊤ϕf_\theta=\theta^\top\phi on frozen features, where the criterion reduces to a quadratic form in the feature covariance. It is therefore unclear whether the proposed approach remains practical for realistic neural reward models. Moreover, the temporal decomposition is mainly used as an analysis tool, and the paper provides limited evidence that it offers actionable guidance in realistic alignment settings. The experiments are also restricted to relatively simplified settings, leaving a substantial gap between the demonstrated results and practical preference learning.

Questions for authors
  1. How can the A(z)A(z) be computed efficiently when the potential ff is parameterized by a general nonlinear neural network rather than the linear form? In particular, does the proposed method require a hypothetical model update for every candidate pair and every possible response, or is there a scalable approximation that avoids this cost?
  2. What is the main motivation for the temporal analysis beyond characterizing when information becomes predictable during generation? More specifically, how is the temporal decomposition expected to guide preference learning in practice, given that the main active learning experiments use only the terminal Fisher information rather than the temporal decomposition?
2 comments
AuthorsOpenReview ↗

Response to Reviewer Rhw3 (1/2): presentation and the role of the temporal analysis

Thank you for the detailed and critical review. We revised the presentation along the lines you indicated, and we added results on computation and on the practical use of the temporal analysis. Section numbers refer to the revised PDF. W1, W2, Q1 and Q2 denote the numbered weaknesses and questions of the review, and W1-a to W1-c and W2-a to W2-d are the points raised within W1 and W2.

W1-a. The relation between Fisher selection and the temporal decomposition is unclear.

Both describe the same object, the update gg of the terminal potential that results from one answer.

  • Terminal Fisher information IT(g)=Varqf(g)I_T(g)=\mathrm{Var}_ {q_f}(g) is the total local change of the output distribution caused by the update (Eq. 3). Fisher selection averages it over the two possible answers (Eq. 7).
  • The decomposition IT=I0+∑jJjI_T=I_0+\sum_j J_j splits this total over generation steps (Proposition 1).
  • New Corollary 2 (§3.3) states what the parts mean. If the update is applied with coefficient ε\varepsilon only at the steps in a set BB, the expected terminal value of gg decreases by ε∑j∈BJj\varepsilon\sum_{j\in B}J_j to first order, and the path-space KL divergence is 12ε2∑j∈BJj\tfrac12\varepsilon^2\sum_{j\in B}J_j to second order. Replacing ITI_T by this sum in Eq. 7 gives the criterion for the case in which the potential is used only in BB (§5.5).

In the revision, the acquisition criterion is defined in the method section (§4) instead of inside the experiments, and §3.3 states this relation.

W1-b. Conditions L, E, S and N are given only in the appendix.

§3.1 now states which conditions the image experiments rely on, and Table 1 (App. A.1) lists each condition with the results that use it: L (Markov property and square integrability) for the variance decomposition, E (exponential moment) for the score, the Fisher information and the KL expansion, S (smoothness) for the continuous-time statement, N (Gaussian transitions) for the estimator of GG, and MC for the unbiased estimator of IjI_j. The image experiments use the finite-step statements, which do not need S.

W1-c. The experimental settings are mixed.

§5 now starts with a roadmap of the experiments, and App. E starts with an overview table (Table 3: purpose, generator, source of preference labels, learner, metric). The experiments are separated by purpose: numerical validation (§5.1), temporal analysis (§5.2), label efficiency with human labels (§5.3), image selection (§5.4), and use of the decomposition for generation and query selection (§5.5).

Q2 / W2-c. What motivates the temporal analysis, and how does it guide preference learning in practice? It appears to be only an analysis tool.

The motivation is that a potential can be used in a part of generation only to the extent that its information lies there (Corollary 2). We now test two practical uses (§5.5). In both, the prediction is computed from trajectories generated without resampling, before any generation with the potential is run.

  1. Choosing where to apply a potential. From trajectories generated without intervention, we compute the first-order response of ideal resampling restricted to an interval, which is the counterpart of the corollary for predictions from denoised states, and use it to predict the gain of resampling with 8 particles. The rank correlation with the measured gain is 0.89–0.94 on CIFAR-10 (12 conditions, independent seeds) and 0.83 on Stable Diffusion v1.5 over four intervals and two resampling scores, the potential learned by Fisher selection from human labels and ImageReward. On CIFAR-10, early resampling raises the color score by 0.52 standard deviations but not the class score, and late resampling raises the class score by 1.22, as the temporal profiles of §5.2 indicate. On Stable Diffusion, resampling with the learned potential improves ImageReward by 0.41 [0.20, 0.60] and HPS v2.1 by 0.19 [0.03, 0.37] within-prompt standard deviations, although neither is used for resampling.
  2. Choosing which labels to request. On CIFAR-10 with three synthetic preferences, we evaluate Eq. 7 with terminal information or with the information of the early or late interval, and we choose among them by the predicted gain. The resulting potentials give larger gains in the true preference of the generated images than D-optimal (+0.057 [0.032, 0.083]), random (+0.042 [0.015, 0.068]), and terminal Fisher selection (+0.025 [0.008, 0.042]), averaged over six conditions and 24 seeds (Table 21 in App. F.8).

Our response continues in the next comment.

AuthorsOpenReview ↗

Response to Reviewer Rhw3 (2/2): computation and scope of the experiments

Q1 / W2-a. Is a hypothetical update for every candidate and answer required? How can the criterion be computed for nonlinear reward models?

Retraining for each candidate and answer is not needed. The criterion is defined through the update that each answer would cause, and this update can be evaluated in closed form or by a local approximation (§4, App. D).

  • Linear potentials with the DA learner (the setting of the paper). The two hypothetical updates of the coefficients are u+cγ(π−y)H−1d~zu+c\gamma(\pi-y)H^{-1}\tilde d_z, where only π\pi and d~z\tilde d_z depend on the candidate. The cross term cancels after averaging over the answer, and Eq. 7 becomes a constant plus c2γ2π(1−π)d~z⊤H−1FθH−1d~zc^2\gamma^2\pi(1-\pi)\tilde d_z^\top H^{-1}F_\theta H^{-1}\tilde d_z, one quadratic form per candidate. This agrees with explicitly computing every update to a relative difference of 4×10−144\times10^{-14} and selects the same pair. The submission already computed the criterion in this way. The cost of our original implementation was dominated by forming a covariance matrix. An optimized implementation takes 6.9 ms (feature dimension 128) and 25.4 ms (dimension 256) per acquisition step with 64 candidates, instead of 644 ms and 2,568 ms, with the same values up to floating-point rounding (App. E.5).
  • Nonlinear potentials. For a differentiable fθf_\theta trained by regularized maximum likelihood, the effect of one more comparison is approximated by one Gauss–Newton step with the generalized Gauss–Newton matrix HH, where the term of the new comparison enters through the Sherman–Morrison formula. With vz=γπ(1−π)∇θ(fθ(a)−fθ(b))v_z=\gamma\sqrt{\pi(1-\pi)}\nabla_\theta(f_\theta(a)-f_\theta(b)), the criterion is approximately Varqf(∇θfθ⊤H−1vz)/(1+vz⊤H−1vz)2\mathrm{Var}_ {q_f}(\nabla_\theta f_\theta^\top H^{-1}v_z)/(1+v_z^\top H^{-1}v_z)^2. It needs gradients at the candidate and reference images and linear solves with HH, as in influence functions (App. D.6).
  • Validation (§5.1, App. F.1). For a multilayer perceptron (MLP) potential with hidden width 64, we compared the approximation with exact retraining for all 64 candidates and both answers. The rank correlation is 0.995 on average, with a minimum of 0.981 over 12 training states. The chosen candidate attains 0.999 of the exact maximum on average, and the computation is 9.2 times faster. With this approximation, Fisher selection improves test CE over random selection (-0.0078 and -0.0054, where the second interval excludes zero; §5.3).

W2-b / W2-d. Only a linear potential on frozen features is used, and the settings are simplified.

The revision extends the evaluation in the following directions, with the method unchanged.

  • Learner. The MLP potential above.
  • Generator and data. PixArt-α as the reference generator (Fisher minus random in mean CE: -0.0205 and -0.0107) and Pick-a-Pic as a second dataset (App. F.4).
  • Seeds and budget. The human-preference comparison now uses 16 seeds per training set and is continued to 512 labels. At 256 labels, Fisher selection is lower than D-optimal in all 12 conditions and for every seed, and the difference remains at 512 labels (-0.0026 [-0.0049, -0.0003] and -0.0030 [-0.0053, -0.0008], §5.3).
  • Use in generation. On Stable Diffusion v1.5, the potential learned from human labels is applied during generation by interval-restricted resampling (§5.5).
  • Evaluators. Image selection is also evaluated with PickScore, HPS v2.1 and CLIP ViT-H/14, which are not trained on ImageRewardDB (§5.4).

Two limits remain and are stated in §7. The nonlinear experiment uses an MLP on frozen features and does not train all layers of a large reward model. The time-weighted query selection is tested with synthetic preferences.

Soundness. Besides the table of conditions, the appendix now separates the quantities defined in the theory from the approximations used in the image experiments, namely predictions from denoised states in place of exact conditional expectations and a finite number of particles in place of the exact hh-transform (App. A.6). It also contains the proofs of Corollary 2 and of the bound for finite updates (App. A.3, A.5).

We hope the revision resolves the concerns about clarity and computation, and we would be glad to clarify any remaining point.

Reviewer_AJE8OpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates Fisher information and its temporal decomposition for preference-based diffusion model alignment. The paper formulates directional Fisher information to quantify how updates to a terminal potential affect the generated output distribution. Using Doob's h-transform, this paper establishes connections between score sensitivity and the temporal distribution of information during generation. Based on this formulation, the paper proposes a Fisher information query selection strategy for active preference learning. Experiments on CIFAR-10 and ImageRewardDB validate the theoretical analysis and demonstrate improved label efficiency compared with random and uncertainty-based selection.

Critical strengths
  1. The paper establishes a connection between directional Fisher information, score sensitivity, and temporal information decomposition using Doob's h-transform. This provides a principled perspective on how preference-induced updates affect the generated output distribution and how their information evolves during generation.

2.On the ImageRewardDB dataset, a query selection strategy based on Fisher information outperforms both random selection and uncertainty-based selection in terms of labeling efficiency. Experiments further demonstrate that the learned potential function enhances the quality of image selection.

Critical weaknesses
  1. Although the selection method based on the Fisher criterion outperforms random sampling and uncertainty sampling, its performance is comparable to that of the D-optimal selection method. In the primary comparisons, no statistically significant improvement was observed (Figure 3(c) compares mean CE over 32–256 labels on TEST_NEW; the differences between the Fisher-based and D-optimal methods are −0.0003 for ALIGNMENT and −0.0017 for OVERALL).

  2. In practical preference learning, the benefits gained from temporal decomposition are quite limited. Consequently, these experiments fail to provide sufficient evidence as to whether the proposed theoretical framework can effectively improve the alignment of diffusion models.

Questions for authors
  1. The analysis in Figure 3 indicates that the selection method based on the Fisher criterion performs comparably to the D-optimal selection method, with no significant differences observed in key performance metrics. Could the authors clarify in which specific scenarios the proposed criterion offers an advantage?

  2. Given that the performance improvement is not significant, could the authors elaborate on what other contributions the proposed framework makes toward the alignment of diffusion models, beyond the filtering of candidate outputs?

2 comments
AuthorsOpenReview ↗

Response to Reviewer AJE

Thank you for the review. Your two main concerns, the comparison with D-optimal selection and the practical benefit of the temporal decomposition, are the focus of the revision. Section numbers refer to the revised PDF. W and Q denote the numbered weaknesses and questions.

W1 / Q1. Performance is comparable to D-optimal (Fig. 3(c)). In which scenarios is the proposed criterion advantageous?

We reran the comparison with 16 seeds per training set instead of 3. The settings and the D-optimal baseline are unchanged.

Difference from D-optimal in test CEALIGNMENTOVERALL
Fisher, 256 labels-0.0050 [-0.0105, 0.0007]-0.0043 [-0.0083, -0.0004]
Fisher, 512 labels-0.0026 [-0.0049, -0.0003]-0.0030 [-0.0053, -0.0008]
Fisher, mean over 32–256 labels+0.0010 [-0.0062, 0.0084]-0.0015 [-0.0048, 0.0018]
Gradient-based variant, mean over 32–256 labels-0.0037 (16 of 16 seeds)-0.0027 (16 of 16 seeds)
  • At 256 labels, Fisher selection is lower than D-optimal in all 12 conditions and for every seed, and the difference remains at 512 labels.
  • The mean over the learning curve, which is the quantity in Fig. 3(c) of the submission, is similar for the two rules because Fisher selection is slower at 32–64 labels in ALIGNMENT. We trace this to density-ratio weights estimated from few labels. A gradient-based variant of the same criterion, with the learner unchanged (§4), is lower than D-optimal at every budget.

Scenarios in which the criterion is advantageous:

  1. The output distribution is the target. With labels generated from a known potential, the KL divergence between the target and learned output distributions is 15–31% lower with Fisher selection than with D-optimal, and 46–48% lower at 256 labels when the preference signal is stronger (§5.3, App. F.4).
  2. Moderate to large budgets (256–512 labels, table above).
  3. The learner's regularization differs from the tuned one. The gradient-based variant is lower than D-optimal with 95% intervals excluding zero in 7 of 8 comparisons, with D-optimal's regularization re-selected as in the paper (App. F.4).
  4. The potential is used in a part of generation (next point).

W2. The benefit of the temporal decomposition is limited, and there is little evidence that the framework improves alignment.

In the revision, the decomposition gives a prediction that we test (§3.3, §5.5).

  • Corollary 2. Applying an update with coefficient ε\varepsilon only at the steps in a set BB lowers the expected terminal value of the update, to first order, by ε∑j∈BJj\varepsilon\sum_{j\in B}J_j. The decomposition therefore tells how much of an update can be realized in each part of generation.
  • Prediction of effective intervals. From trajectories generated without intervention, we predict the gain of resampling with a potential restricted to an interval, and then measure it. The rank correlation between prediction and measurement is 0.89–0.94 on CIFAR-10 (12 conditions, independent seeds) and 0.83 on Stable Diffusion v1.5 (8 conditions: four intervals, with the potential learned by Fisher selection or with ImageReward).
  • Alignment of generated images. On Stable Diffusion, resampling with the learned potential during generation improves ImageReward by 0.41 [0.20, 0.60] within-prompt standard deviations and HPS v2.1 by 0.19 [0.03, 0.37]. Neither evaluator is used for resampling.
  • Query selection. On CIFAR-10, choosing which interval information the acquisition criterion uses by this prediction gives larger gains in the true preference of the generated images than D-optimal (+0.057 [0.032, 0.083]), random (+0.042 [0.015, 0.068]), and terminal Fisher selection (+0.025 [0.008, 0.042]), averaged over six conditions and 24 seeds (Table 21 in App. F.8).

Q2. What does the framework contribute beyond filtering candidate outputs?

  • Use during generation. The learned potential is applied inside the generation process by interval-restricted resampling, with the improvements above. The generator weights stay fixed, as in the rest of the paper.
  • Deciding when to intervene. The decomposition indicates in advance which interval is effective for a given potential. For the potential learned from human labels, the learned preference score increases more under late than under early resampling (0.48 versus 0.22). On CIFAR-10, early resampling raises the color score (+0.52) but not the class score (-0.01), and late resampling raises the class score (+1.22).
  • Label selection. The criterion selects which comparisons to label. It has a closed form for the learner of the paper and an approximation for nonlinear potentials, and neither requires retraining for each candidate (§4).
  • Analysis. The same quantity describes when attributes and learned updates become predictable (§5.2).

The presentation is also revised (§3.1, §4, §5). We hope this clarifies the contribution, and we are happy to discuss further.

Thank you to the authors for the response and the detailed explanation. I have some concerns regarding the general applicability of the proposed method, given that it has only been validated on CIFAR-10 and ImageRewardDB. Furthermore, the paper lacks sufficient qualitative comparisons. I look forward to seeing more such comparisons in a revised version. Consequently, I am maintaining my original score.

Reviewer_1BiaOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper investigates the problem of label-efficient preference alignment for diffusion models from a distributional perspective. It defines directional Fisher information as a metric for the local change in KL divergence of the output distribution—for updating the terminal potential function. By employing a Doob hh-transform, this metric is linked to score sensitivity and derive a temporal decomposition method to analyze the points in the generation process at which updates become predictable. Terminal Fisher information serves as a criterion for active query selection, with the aforementioned decomposition providing a theoretical foundation for such analysis. The approach is validated numerically using low-dimensional Ornstein-Uhlenbeck (OU) processes. Experiments on the CIFAR-10 dataset reveal that color features become predictable earlier than vehicle similarity features. Furthermore, on the ImageRewardDB dataset, a selection strategy based on Fisher information achieves a lower test cross-entropy with only 128 labeled samples compared to 256 samples selected at random, performing on par with the DD-optimal criterion.

Critical strengths
  1. The framework is clear and rigorous. By employing martingale variance decomposition, the hh-transform path rule, and unbiased finite-step estimators, it provides a principled, time-dimension-based analysis of how information is updated. The numerical validation is robust, with an error rate of less than 1%.

  2. It achieves a tangible improvement in labeling efficiency. Across 12 different training/test scenarios, a scientific criterion based on feature covariance (the Fisher selection method) reduced the required labeling budget by half compared to random selection.

Critical weaknesses
  1. The temporal decomposition as the core theoretical contribution of this paper is not the actual driver behind the observed empirical performance gains. The active learning experimental results rely solely on terminal Fisher information; the decomposition itself serves essentially as an analytical tool with limited practical utility (weighting different time segments has a negligible impact on learning performance, and no causal link is established between this weighting and the quality of sample selection). Furthermore, since the theory relies on assumptions of a fixed gg and infinitesimal updates, rather than a scenario involving repeated training, its utility for guiding active learning loops is merely heuristic.

  2. The paper fails to demonstrate that its proposed method outperforms the strongest baseline models. Sample selection based on Fisher information shows no statistically significant difference from the D-optimal criterion (as the confidence intervals include zero); the advantages highlighted by the authors are only relative to random selection. The experiments employ a single fixed generator, linear potential functions, and automated evaluation metrics that are highly correlated with the training signals. Moreover, the lack of human evaluation and the reliance on a single dataset mean that the robustness and generalizability of the method remain to be verified.

Questions for authors

Question 1: Temporal decomposition is your core theoretical contribution; however, the performance gains in active learning are derived solely from the "terminal Fisher information." Could you provide a concrete scenario demonstrating that temporal decomposition genuinely improves the query selection process, rather than serving merely as an analytical tool?

Question 2: From a statistical perspective, selection based on Fisher information is essentially equivalent to the D-optimal criterion. Under what conditions—such as non-linear potential functions or non-Gaussian output distributions—would you expect Fisher information-based selection to outperform the D-optimal criterion? Could you provide a specific application scenario?

Question 3: All experiments utilize linear potential functions and a single, fixed generator, while the automated evaluators employed are highly correlated with the training signals. How robust is this criterion when faced with more powerful learners or non-linear potential functions? Furthermore, could the effectiveness of the image selection process be validated through human evaluation or by using an independent evaluator—one that was not involved in the training process?

2 comments
AuthorsOpenReview ↗

Response to Reviewer 1Bia (1/2): temporal decomposition and theory

Thank you for the detailed review and for the positive assessment of the theory and of the gains over random selection. We respond to each point below. Section and figure numbers refer to the revised PDF. W1, W2 and Q1–Q3 denote the numbered weaknesses and questions of the review, and W1-a/W1-b and W2-a/W2-b are the points raised within W1 and W2.

W1-a / Q1. The temporal decomposition does not drive the AL gains, and time weighting has a negligible effect. Is there a concrete scenario in which it improves query selection?

We agree that the submission used the decomposition only for analysis, and that time weighting changed the selected pairs without a measured benefit. The revision adds a result that states what the interval terms mean operationally, and an experiment in which they improve query selection.

  • Theory (Corollary 2, §3.3). If an update gg is applied with coefficient ε\varepsilon only at the steps in a set BB, the expected terminal value of gg decreases by ε∑j∈BJj(g)\varepsilon\sum_{j\in B}J_j(g) to first order, and the path-space KL divergence is 12ε2∑j∈BJj(g)\tfrac12\varepsilon^2\sum_{j\in B}J_j(g) to second order. For another terminal quantity, the response is the covariance of the corresponding increments. The interval information is the part of an update that can be realized in that part of generation.
  • Scenario. The learned potential is used in a restricted part of generation, for example by resampling particles only in early or only in late steps. The useful labels then depend on where the potential will be used, and the corollary predicts this before any generation with the potential.
  • Experiment (§5.5; Table 21 in App. F.8). CIFAR-10, three synthetic preferences, 128 labels. The learned potential is used by resampling in the early or late interval. We compare the acquisition criterion of the paper (Eq. 7, with the learner refitted for every candidate and answer) measured with terminal information or with the information of the early or late interval, which is estimated from predictions based on denoised states. For each preference and interval, the criterion is chosen by the first-order prediction from separate runs. Averaged over six conditions and 24 seeds, the gain in the true preference of the final images is larger than with D-optimal by 0.057 [0.032, 0.083], than with random selection by 0.042 [0.015, 0.068], and than with terminal Fisher selection by 0.025 [0.008, 0.042]. With late resampling, the gains over D-optimal are 0.110, 0.118 and 0.146 for the three preferences.
  • Causal link. The evaluation is now the outcome of generation instead of test CE. The early-interval criterion selects pairs with larger color differences (1.56 versus 1.17 in standardized units), and the predicted gain correlates with the measured gain at 0.96 (App. F.8). For the mixed preference, the early-interval criterion is better than the late-interval one under early resampling (+0.055) and worse under late resampling (-0.111). The experiment therefore measures each step, from time weighting to the selected pairs to the generated outputs.

The scope is stated in §7: the experiment uses synthetic preferences, and choosing the criterion uses known preference values.

W1-b. The theory assumes a fixed ff and infinitesimal updates, so its use in an iterated loop is heuristic.

  • At each acquisition step the criterion is evaluated at the current potential, so the analysis describes the next update from the current state. Expected-model-change criteria are used in the same way.
  • For finite updates we add a bound (§3.2, App. A.3). For bounded gg with range Δ\Delta, the ratio of the exact KL divergence to 12ε2IT\tfrac12\varepsilon^2 I_T lies between e−∣ε∣Δe^{-|\varepsilon|\Delta} and e∣ε∣Δe^{|\varepsilon|\Delta}.
  • We measured this ratio on the actual updates of the human-preference experiment (§5.1). For 17,280 single-label updates it has median 1.000, with 0.1–99.9% quantiles 0.989–1.004. For the cumulative change after 256 labels the median is 0.962.
  • We do not have a guarantee for the whole loop, and §7 says so. As a check of the outcome of the loop, we generated labels from a known potential. Fisher selection then brings the learned output distribution closer to the target than D-optimal, with a KL divergence that is 15–31% lower (§5.3).

Our response continues in the next comment.

AuthorsOpenReview ↗

Response to Reviewer 1Bia (2/2): comparison with D-optimal and scope of evaluation

W2-a / Q2. Fisher selection does not outperform D-optimal. Are the two essentially equivalent, and under which conditions is Fisher selection better?

The two criteria are not equivalent. §4 now gives the closed form of our criterion next to the D-optimal score. Both multiply the ambiguity π(1−π)\pi(1-\pi) by a quadratic form. D-optimal measures the feature difference with the inverse of the accumulated label information and does not depend on the generator. Our criterion measures the learner's update direction with the feature covariance under the current output distribution. No Gaussian assumption on the outputs is used, and we do not claim that nonlinearity or non-Gaussian outputs alone make Fisher selection better.

Updated comparison (§5.3, App. F.4). Settings and the D-optimal baseline are unchanged. Seeds are extended from 3 to 16 per training set.

TEST_NEW CE3264128256
ALIGNMENT, D-optimal0.64460.64320.64340.6441
ALIGNMENT, Fisher0.64500.64820.64350.6391
ALIGNMENT, Fisher (gradient-based)0.64120.63990.63960.6393
OVERALL, D-optimal0.66960.66710.66790.6685
OVERALL, Fisher0.66870.66760.66570.6641
OVERALL, Fisher (gradient-based)0.66730.66540.66460.6646
  • At 256 labels, Fisher is lower than D-optimal in all 12 conditions and for every seed. Continuing the same runs to 512 labels, the differences on TEST_NEW are -0.0026 [-0.0049, -0.0003] and -0.0030 [-0.0053, -0.0008].
  • Fisher is slower at 32–64 labels in ALIGNMENT. An ablation attributes this to density-ratio weights in the update direction, which are estimated from few labels. The gradient-based variant (§4) keeps the learner and computes the score with a gradient-step direction. It is lower than D-optimal at every budget, and its mean over 32–256 labels is lower for all 16 seeds (-0.0037 and -0.0027).

Conditions in which Fisher selection is better.

  1. The target is the output distribution. With labels from a known potential, the KL divergence to the target distribution is 15–31% lower than with D-optimal, and 46–48% lower at 256 labels when the preference signal is stronger (§5.3, App. F.4).
  2. Moderate to large budgets (256–512 labels, above).
  3. The learner's regularization differs from the tuned one. The gradient-based variant is lower than D-optimal with 95% intervals excluding zero in 7 of 8 comparisons, with D-optimal's regularization re-selected by the validation procedure of the paper.
  4. The potential is used in a restricted part of generation (W1-a).

W2-b / Q3. Single generator, linear potentials, an evaluator correlated with the training signal, no human evaluation, and a single dataset.

  • Generator. With PixArt-α as the reference generator, Fisher selection improves on random selection (-0.0205 and -0.0107 in mean CE), and the gradient-based variant is lower than D-optimal for 7 of 8 seeds in each task (App. F.4). The temporal experiments use a CIFAR-10 DDPM and Stable Diffusion v1.5.
  • Nonlinear learner. For a multilayer perceptron (MLP) potential, the approximation of §4 agrees with exact retraining (rank correlation 0.995), and Fisher selection improves on random selection (-0.0078 and -0.0054, where the second interval excludes zero).
  • Independent evaluators. The AL metric is the prediction of held-out human labels. For image selection we add evaluators that are not trained on ImageRewardDB. Fisher minus random is 0.054 [0.026, 0.082] for PickScore, 0.0019 [0.0010, 0.0030] for HPS v2.1, and 0.0040 [0.0023, 0.0059] for CLIP ViT-H/14, and Fisher with 128 labels exceeds random with 256 labels under all three (§5.4). On Stable Diffusion, resampling with the learned potential during generation improves HPS v2.1, which is not trained on ImageRewardDB, by 0.19 [0.03, 0.37] within-prompt standard deviations, and ImageReward by 0.41 [0.20, 0.60] (§5.5).
  • Dataset. On Pick-a-Pic, the gradient-based variant is lower than D-optimal for 24 of 25 seeds (App. F.4).
  • Human evaluation. We did not run a new human study, and §7 keeps this limitation. We followed the alternative suggested in Q3 and used evaluators independent of the training signal.

We hope these additions address your concerns, and we are glad to answer further questions.

Comments on the paper

1 comment
AuthorsOpenReview ↗

General response and summary of the revision

We thank the reviewers for their careful reading and constructive comments. We have uploaded a revised manuscript in which changes are marked in blue. The abstract is updated to reflect the added results. Below, W and Q denote the numbered weaknesses and questions of each review. The method and the baselines are those of the original submission. The revision clarifies the presentation and adds validation and experiments that respond to the concerns shared by the reviewers. We summarize the main points here and reply to each reviewer separately.

1. Comparison with D-optimal selection (1Bia W2/Q2, AJE8 W1/Q1). The human-preference experiment now uses 16 seeds per training set instead of 3, with the same settings and the same D-optimal baseline. At 256 labels, Fisher selection has lower test CE than D-optimal in all 12 conditions and for every seed (TEST_NEW: -0.0050 for ALIGNMENT and -0.0043 for OVERALL). The difference remains when the same runs are continued to 512 labels (-0.0026 [-0.0049, -0.0003] and -0.0030 [-0.0053, -0.0008]). When labels are generated from a known potential, so that the output distribution can be evaluated directly, Fisher selection reduces the KL divergence to the target distribution by 15–31% relative to D-optimal. Fisher selection is slower at 32–64 labels in one task. We trace this to density-ratio weights estimated from few labels, and a gradient-based variant of the same criterion, with the learner unchanged, is lower than D-optimal at every budget. (§5.3, Fig. 3, App. F.4)

2. What the temporal decomposition is used for (1Bia W1/Q1, AJE8 W2/Q2, Rhw3 W1/Q2). New Corollary 2 shows that the interval information JjJ_j determines the first-order effect of applying an update only at step jj. We test two uses. (i) From trajectories generated without intervention, it predicts in which generation interval resampling with a potential is effective. The rank correlation between predicted and measured gains is 0.89–0.94 on CIFAR-10 and 0.83 on Stable Diffusion v1.5 across four intervals and two resampling scores, the potential learned by Fisher selection and ImageReward. Applying the learned potential during generation improves ImageReward by 0.41 and HPS v2.1 by 0.19 within-prompt standard deviations, although neither evaluator is used in that resampling. (ii) On CIFAR-10, choosing the information used by the acquisition criterion with this prediction gives larger gains in the generated images than D-optimal (+0.057), random (+0.042), and terminal Fisher selection (+0.025). These are means over six conditions and 24 seeds, and all three 95% intervals exclude zero. In practice, the prediction is computed from samples generated without any intervention, so one can tell in advance in which part of generation a given potential can act, and use this to decide which comparisons to label. (§3.3, §5.5; Fig. 9 in App. F.7, Table 21 in App. F.8)

3. Computation and nonlinear potentials (Rhw3 W2/Q1, 1Bia Q3). The criterion does not require retraining for each candidate and answer. For the learner of the paper it has a closed form with one quadratic form per candidate, which agrees with explicit updates to a relative difference of 4×10−144\times10^{-14}. For nonlinear potentials, a one-step Gauss–Newton approximation agrees with exact retraining for a multilayer perceptron (MLP; rank correlation 0.995) and is about nine times faster. (§4, §5.1, App. D)

4. Finite updates (1Bia W1). We add a two-sided bound on the error of the quadratic KL approximation for a finite update. For the 17,280 single-label updates of the human-preference experiment, the ratio of the exact KL divergence to the approximation has median 1.000, with 0.1–99.9% quantiles 0.989–1.004. (§3.2, §5.1, App. A.3)

5. Scope of the evaluation (1Bia W2/Q3, Rhw3 W2). We add a second generator (PixArt-α), a second dataset (Pick-a-Pic), an MLP potential, and evaluators not trained on ImageRewardDB (PickScore, HPS v2.1, CLIP ViT-H/14). (§5.3, §5.4, App. F)

6. Presentation (Rhw3 W1, AJE8). The acquisition criterion is now defined in the method section (§4). §3.1 states which conditions the image experiments rely on, and Table 1 (App. A.1) lists the conditions with the results that use them. App. E starts with an overview table of the experiments (Table 3: purpose, generator, labels, learner, metric), and the new experiments on the temporal decomposition form a separate section (§5.5). Because of the page limit, the main text states the main numbers of the new experiments, and their tables and figures are in App. F.

Limitations that remain are stated in §7. There is no guarantee for the whole active-learning loop, no new human study was run, and the time-weighted query selection is tested with synthetic preferences.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.