acceptodds
Under review as a conference paper at ICLR 2027

Designing diffusion models for tractable likelihood estimation

Abstract

Estimating likelihoods with diffusion models is typically done by expensive approximations of the divergence of a neural network, as exact computation is intractable in general. Learning the time score is a recent idea that may enable more viable and exact computation, but is still a challenging task as previous work has been limited in scale, has suffered with sampling degradation and has mostly focused on the energy setting, which by itself already suffers from training difficulties. In this work, we tackle these issues by proposing a unifying framework to design diffusion models for effective sampling and likelihood estimation. First, we propose a regularization strategy to incorporate the time score without negatively affecting sampling. Then, we introduce a set of design coefficients and learnable terms that generalizes and connects important cases, such as the energy and mean flow formulations. Finally, specific parametrization and learning strategies are proposed in order to avoid singularities and improve training. The resulting algorithms are tested and studied on synthetic examples and on real images for ImageNet on both pixel and latent space.

Primary area:
generative models
Keywords:
Diffusion, flow matching, likelihood estimation, density estimation, energy based models, mean flow

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_9LLvOpenReview ↗
Rating
3(+1)
Confidence
1(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes a diffusion model framework for estimating likelihoods without costly divergence calculations. Its temporal score objective and endpoint stable parameterization aim to preserve generation quality, with results on synthetic data and ImageNet64.

Critical strengths
  1. The framework connects several model formulations and makes clear how each can provide the quantities needed for sampling and likelihood estimation.

  2. The rescaled temporal target and endpoint aware parameterization address identifiable singularities, rather than relying only on training heuristics.

Critical weaknesses
  1. Most systematic likelihood analyses and ablations focus on ImageNet64. Although the paper reports ImageNet256 sampling and likelihood-based filtering, it provides limited quantitative evidence on likelihood accuracy or calibration at that resolution. Evaluation on another image dataset would further strengthen the generalization claim.

  2. The paper reports approximate training times but does not systematically compare its additional training cost with the savings in likelihood evaluation. A comparison under matched hardware and comparable estimation accuracy should report training time, evaluation time, network evaluations, and memory use separately.

  3. The image experiments primarily use an LDiT backbone. Testing a U-Net backbone would help establish whether the stability and quality results transfer to a different architecture. The paper should also clarify which parts of its likelihood-estimation procedure depend on the continuous-time sampling trajectory and whether they extend to other samplers, such as DDIM.

Questions for authors
  1. The paper reports ImageNet256 sampling and likelihood-based filtering, but provides less systematic evidence on the calibration of its likelihood estimates at that resolution. Could the authors provide such an evaluation and, if feasible, test generalization on a dataset outside ImageNet, such as LSUN?

  2. How much additional training time and memory does temporal-score learning require? Could the authors compare likelihood-evaluation time, network evaluations, and accuracy against divergence-based estimation under the same hardware and evaluation budget?

  3. Does the proposed objective and parameterization remain stable with a U-Net backbone or other Transformer variants? Could the authors clarify whether the proposed method can be used with other pipelines, such as LDM or FLUX?

5 comments

We'd like to thank the reviewer for their effort and bringing many relevant practical considerations. Evaluating likelihood accuracy and calibration is indeed a key question, so we focused our efforts on this topic and hope we can incorporate other ideas shortly, at least for a final version, if not in time for discussion.

Whenever possible, we provided a short answer followed by lengthier discussions.

1a. Evidence on likelihood calibration

After reviewer feedback, we used divergence based estimates as a reference to compare the other model estimates. The general trend is that higher λw\lambda_w improved self-consistency, while optimal time scaling was more nuanced and depends on the dataset. The best tested models had likelihoods agreeing within roughly a 0.050.05 bpd scale. We leverage the fact that we focus on efficient learning of the flow vv in order to use its divergence (approximated with Hutchinson's trace estimator) to measure the model's internal belief.

The following two tables compare each model's prediction after 400 Heun steps (each requiring two function evaluations), where these quantities appeared to have converged. We compare the likelihood prediction using the potential function against those computing either the divergence of the flow or the time score, both according to Eq. (1). We use 128 samples from the validation set (held-out) and report the bias that centers the error as well as the residual mean absolute error (rMAE) after the bias is removed to approximately decouple normalization errors and per sample errors. The likelihood scale is in 0.010.01 bpd for visualization purposes.

Latent space models:

Time scalingλw\lambda_wBias (divergence)rMAE (divergence)Bias (time score)rMAE (time score)
100010−210^{-2}-2.323.51-0.09860.0650
100010−310^{-3}-2.794.450.5930.137
100010−410^{-4}-6.755.941.280.246
110−210^{-2}6.083.600.001670.00308
110−310^{-3}15.54.62-0.001580.00402
110−410^{-4}44.24.85-0.003200.00373

Pixel space models:

Time scalingλw\lambda_wBias (divergence)rMAE (divergence)Bias (time score)rMAE (time score)
100010−210^{-2}3.5013.92.981.52
100010−310^{-3}7.1925.13.084.00
100010−410^{-4}35.932.93.074.00
110−210^{-2}-0.7177.770.1960.164
110−310^{-3}12.111.80.1860.160
110−410^{-4}64.314.10.1370.146

The choice of best model is debatable, but λw=10−2\lambda_w=10^{-2} showed consistent improvements over lower values in all settings. Higher time scaling improved consistency in latent space, but had an opposite or dubious effect in pixel space, perhaps reflecting their differences in smoothness and the fact that LDiT was originally developed for latent applications. The agreement between potential and time score based estimates also generally favored higher λw\lambda_w, but benefited from lower time scaling, presumably as it turned ∂tlog⁡p\partial_t \log p smoother and easier to integrate.

1b. Generalization to other datasets

We hadn't considered testing a model trained on ImageNet on other datasets, but, if the reviewer agrees that the likelihood calibration procedure we are proposing is appropriate, we could do so on some version of LSUN, as well as repeating the filtering experiment on samples generated by another model. During the early stages of our work, we used CIFAR-10, but the dataset is small and the main challenge we had was overfitting, which we thought to be less relevant as it is not representative of the issues found at larger scale.

2a. Additional training time and memory for temporal score learning

The short answer is that adding a time score loss as we do has negligible effect on training time compared to training the same formulation with only spatial score losses. This is because it utilizes information that was already being computed (e.g., ∂tlog⁡p\partial_t \log p) and leverages computations that are already being done (e.g., backpropagation through a network) while only adding simple operations.

The main driving factors for compute costs are the formulation and implementation details. In the potential one, the same backward pass (VJP) that is used to compute ∇log⁡p\nabla \log p already produces the information needed for ∂tlog⁡p\partial_t \log p and log⁡p\log p, so the extra effort is that of simple operations and backpropagating from the time score loss into the network outputs, which is negligible compared to the backpropagation through the network itself that was already happening with spatial score. We don't see any meaningful difference (i.e., about 2 MB of extra memory) and don't expect that to be different outside of artificial examples. For similar reasons, in the vector one, the extra cost would be proportional to the effort spent computing the extra scalar (either TT or LL), which we don't expect to require a significant increase in complexity as it is only a somewhat well-behaved scalar. In the experiments of Table 4 we used the simplest strategy of adding a readout head.

The potential one requires a second order derivative to train, so in theory that would double every cost. In practice, that may be closer to 3 times, partially because the software infrastructure is not as mature for that case as it is for the usual single backpropagation. We illustrate that with some values for the larger latent space models (LDiT/B1) on an RTX PRO 6000 Blackwell in the table below. For the pixel space model, the patch size is adjusted to keep the number of tokens the same and so the computational cost is similar.

FormulationAtt. KernelIter/sMemory (GB)
VectorOriginal7.219.7
VectorFused8.516.2
PotentialOriginal2.758.8
PotentialFusedN/AN/A
Potential + VJP⁡sg\operatorname{VJP}_{sg}Original4.627.0
Potential + VJP⁡sg\operatorname{VJP}_{sg}Fused5.519.0

For context, there are heavily optimized fused kernels to compute attention (e.g. FlashAttention) and PyTorch provides access to those and a fallback one, while the implementations in DiT and LDiT use elementary operations. Unfortunately, the most efficient kernels do not yet support double backpropagation, so we use the original one as it was slightly faster than the fallback. We included the potential + VJP⁡sg\operatorname{VJP}_{sg} case to quantify the hypothetical benefits of the trick we identified as the potential analogue to the JVP⁡sg\operatorname{JVP}_{sg} trick used in the mean flow of Geng et al. (2025), one that could mitigate the higher training cost for the potential formulation. We'll only mention that there are also software issues that appear when combining certain features like high order derivatives, compilation and half precision (i.e., potential PyTorch bugs).

2b. Likelihood-evaluation time, network evaluations, and accuracy against divergence-based estimation

Please see our response 1a for accuracy estimates. Timing generally agrees with the theoretical prediction that each slice of the divergence requires an extra derivative, so roughly twice as much computation and memory. Using the potential formulation, though it requires the extra training budget we discussed in our response 2a, has the advantage of being able to evaluate likelihoods with a single network call and no integration is required.

We provide the associated costs observed for likelihood computation for an RTX PRO 6000 Blackwell in the table below. The values seem to follow general rules of thumb based on the type of computation involved. Accounting for the number of evaluations, moving from a single network call to computing its derivative (VJP), increased computation time by a factor of 3.5, which appears explainable by the fact that backward passes require storing intermediate activations and are generally more expensive.

MethodTimeMemory (GB)Computation type
Potential18 ms total0.61 ×\times network evaluation
Time score125 ms/step1.92 ×\times network derivative (1st order)
Divergence239 ms/step4.52 ×\times network derivative (2nd order)

Convergence metrics are reported in the table below for the λw=10−2\lambda_w=10^{-2} case, generally the most effective one in the ablation. We solve the associated ODEs with different total number of steps, starting at 25 and doubling it until 400. We use variations close to 0.010.01 bpd relative to the values achieved at 400 steps as a rough criterion for convergence.

SpaceTime scalingDivergence stepsTime score steps
Latent100050100
Latent15025
Pixel1000200200
Pixel1400400

For the potential case, integrating the time score might be unnecessary given that a single network call already gives a density estimate very efficiently, which is the reason previous work and ours focuses on it. Nevertheless, we consider these results as promising signals for the vector case, besides the reduction in computational requirements. If the model is used to generate a new sample, it is already computing and integrating vv, so including dlog⁡pdt\frac{d\log p}{dt} in the process provides likelihood estimates during sampling, which we believe could be used to steer generation as evidenced by the correlation between sample quality and likelihood displayed in Figures 1 and 4.

Reviewer_btemOpenReview ↗
Rating
3(+1)
Confidence
2(−1)
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies tractable likelihood estimation with diffusion models. Starting from the continuity equation, the authors decompose the log-density time derivative into a spatial score term and a temporal score term, and propose to learn the temporal score as a scalable regularizer. They further introduce a unified coefficient-based design space that covers energy/potential diffusion, vector diffusion, score matching, and mean-flow / few-step integration models, with a focus on non-singular parameterizations that constructively satisfy boundary conditions. Experiments include a 10-D Gaussian mixture sanity check, time-scaling and λw\lambda_w ablations on ImageNet, likelihood-based sample screening, and appendix comparisons of potential vs.\ vector diffusion and NLL/FID numbers.

The main contributions are: (i) a temporal-score regularized training objective L=LSSMv+λwLTSM;L=L_{\mathrm{SSM}}^v+\lambda_w L_{\mathrm{TSM}}; (ii) a unified design space with coefficients (cd,cx,cF,cT,cL)(c_d,c_x,c_F,c_T,c_L) and (ax,aF,aL)(a_x,a_F,a_L); (iii) a non-singular ``ProGaMeR'' parameterization; and (iv) empirical demonstrations that likelihood can be used to rank/filter generated samples.

Critical strengths
  1. Unified coefficient-based design space and constructive parameterization criterion. The paper gives a useful way to compare score-based, energy-based, vector-field, and mean-flow parameterizations through explicit coefficients and boundary conditions. This is valuable independent of the empirical results, because it provides an actionable checklist for designing non-singular diffusion parameterizations.

  2. Likelihood-based sample screening is a practical and promising application. The demonstration that likelihood can rank generated samples and improve FID/IS on selected subsets is a concrete use case for tractable likelihood. This strengthens the paper's potential impact beyond density estimation itself.

Critical weaknesses

1.Clarity of the theoretical presentation. The temporal-score derivation and coefficient relations could be clearer. Explicitly stating the score-to-flow coefficient conversion and providing intermediate steps from Eq. (2) to Eq. (8), and from Eq. (8) to Eq. (9), would make the framework easier to verify.

2.Isolation of empirical contributions. The fields and loss terms used in Tables 3–5 should be specified consistently with their captions. Although Table 2b includes a zero-temporal-weight baseline, a corresponding latent-space baseline would strengthen the assessment of sampling degradation. In Table 3, “ours” performs worse than v-pred, so benefits for likelihood estimation and stability should be distinguished from sampling performance. More direct ablations against the design choices in Guth et al. (2026) would help establish which components account for the reported benefits.

Questions for authors

1.Could you explicitly state the conversion between the score and flow coefficients in Table 1 under the time convention used in Eq. (6)?

2.Please provide the derivation of Eq. (8) from Eq. (2), and clarify its relationship to the temporal-score target in Guth et al. (2026).

3.Please clarify the rescaling from Eq. (8) to Eq. (9), including its effect on the relative weighting of different times.

4.Which fields and loss terms are used in Tables 3, 4, and 5? Please specify the temporal-loss weights and ensure that the descriptions are consistent with the table captions.

5.Please clarify how the latent-space NLL values of 1.04/1.22 bpd in Table 4 should be interpreted relative to the pixel-space NLL of 3.25 bpd in Table 5.

6.In Table 3, “ours” performs worse than v-pred in both FID and IS. How should this result be interpreted in relation to the intended stability and likelihood-estimation benefits of the proposed parameterization?

7.What are the incremental contributions relative to Guth et al. (2026), and which are supported by the reported ablations?

8.Table 2b includes a zero-temporal-weight baseline. Could you provide the corresponding sampling baseline in latent space and clarify whether direct potential-based NLL is meaningful without temporal supervision?

9.Please explicitly identify the ProGaMeR row in Table 1 as the proposed Projected Gaussian Mean Replacement parameterization.

5 comments

We appreciate the reviewer's effort in examining the theoretical and algebraic consistency of our work. Their feedback guided us into uncovering certain nuances and suggested that using a more generic derivation might make the connections between spatial and time scores more evident.

We'll start with question 9 and then follow the numerical order

9. Source for ProGaMeR and relation to our parametrization

ProGaMeR stands for Projected Gaussian Mean Replacement and is a novel parametrization we proposed in this submission. We'll be more explicit in a revision to prevent this type of confusion from arising.

1. Definition of Eq. (6)

It seems there may be some miscommunication or perhaps a parsing issue because what the reviewer claims Eq. (6) should be is correct and is already the definition we presented. We also wrote tv=x+(1−t)stv = x + (1-t)s in line 349, which is the same equation multiplied by tt. We find that AI tools can suffer from parsing errors and imagine that to be the case here.

For instance, sometimes Gemini transforms \frac{t(2t-1) - t^2 - (1-t)^2} into - \frac{t(2t-1) - t^2(1-t)^2} depending on how the source code is copied. Nevertheless, we'll discuss the concerns as they were presented.

The reviewer is correct to claim that Eq. (6) implies both

cx=tax−11−tc_x = \frac{ta_x-1}{1-t} cF=taF1−tc_F = \frac{ta_F}{1-t}

And these relations are fully consistent with the results in Table 1. Starting with vv-pred (ax=0a_x = 0 and aF=1a_F = 1) cx=0−11−t=−11−tc_x = \frac{0-1}{1-t} = \frac{-1}{1-t} cF=t11−t=t1−tc_F = \frac{t1}{1-t} = \frac{t}{1-t}

For ProGaMeR, recall that σ2=t2+(1−t)2\sigma^2 = t^2 + (1-t)^2. Solving for cxc_x is a bit more laborious cx=t2t−1σ2−11−t=t(2t−1)−σ2(1−t)σ2=t(2t−1)−t2−(1−t)2=c_x = \frac{t\frac{2t-1}{\sigma^2}-1}{1-t} = \frac{t(2t-1) - \sigma^2}{(1-t)\sigma^2} = \frac{t(2t-1) - t^2 - (1-t)^2} = =t[(2t−1)−t]−(1−t)2(1−t)σ2=t(t−1)−(1−t)2(1−t)σ2=−t(1−t)−(1−t)2(1−t)σ2== \frac{t\left[(2t-1) - t\right] - (1-t)^2}{(1-t)\sigma^2} = \frac{t(t-1) - (1-t)^2}{(1-t)\sigma^2} = \frac{-t(1-t) - (1-t)^2}{(1-t)\sigma^2} = =−t−(1−t)σ2=−1σ2= \frac{-t - (1-t)}{\sigma^2} = \frac{-1}{\sigma^2}

Finally, cFc_F is straightforward cF=t1−tσ21−t=tσ2c_F = \frac{t \frac{1-t}{\sigma^2}}{1-t} = \frac{t}{\sigma^2}

These are all the entries the reviewer raised concerns for.

To provide an extra layer of confirmation and for the reviewer's convenience, we wrote a Sympy script recreating the content from Table 1.

import sympy as sp

x, t, F = sp.symbols("x t F", real=True, finite=True)
ax, aF, cx, cF = sp.symbols("ax, aF, cx, cF", real=True)
a_vars = (ax, aF)
c_vars = (cx, cF)

v = ax*x + aF*F
s = cx*x + cF*F
v2s_eq = sp.Eq(t*v,x+(1-t)*s)
eqs = [v, s, v2s_eq]

a_from_c = sp.solve_undetermined_coeffs(v2s_eq, (ax, aF), (x, F))
sigma2 = t**2 + (1-t)**2
sigma = sigma2**0.5

# coefficients from Table 1 in the order [cx ,cf]
c_table = {
    "xpred":    [-1/(1-t)**2,  (t/(1-t)**2)],
    "epred":    [0,            -1/(1-t)],
    "vpred":    [-1/(1-t),     t/(1-t)],
    "spred":    [0,            1],
    "EDM":      [-1/sigma2,    t/((1-t)*sigma)],
    "ProGaMeR": [-1/sigma2,    t/sigma2],
}

# parse bool for nicer output
pb = lambda v: "yes" if v else "no"

for name, c_vals in c_table.items():
    # coefficient conversion
    a_vals = [a_from_c[a_var].subs(c_var, c_val).simplify() for a_var, c_var, c_val in zip(a_vars, c_vars, c_vals)]
    print(f"{name}")
    print("-Table coefficients")
    for a_var, a_val in zip(a_vars, a_vals):
        print(f"{a_var}: {a_val}")

    print("-Boundary conditions")
    v_eq = v.subs(zip(a_vars, a_vals))
    s_eq = s.subs(zip(c_vars, c_vals))
    v_at = lambda tv: v_eq.subs(t, tv).simplify()
    s_at = lambda tv: s_eq.subs(t, tv).simplify()
    print(f"v(x,1) BC satisfied: {pb(v_at(1) ==  x):>2} | v(x,1) = {v_at(1)}")
    print(f"s(x,0) BC satisfied: {pb(s_at(0) == -x):>2} | s(x,0) = {s_at(0)}")

    print("-Singularities")
    for tv in [0, 1]:
        # to avoid "None" as a return for is_infinite, we assign concrete value
        vt = v_at(tv).subs(F,1).subs(x,1)
        st = s_at(tv).subs(F,1).subs(x,1)
        print(f"At t={tv}, v: {vt.is_infinite} | s: {st.is_infinite}")

    print("-"*20)

2. Relation between Eq. (8), Eq. (2) and the temporal score target in Guth et al. (2026)

Both our work and that of Guth et al. (2026) build upon the results established by Yu et al. (2025). Briefly, given the conditional probability log⁡p(x,t∣x1)\log p(x,t|x_1) describing how a clean data sample x1x_1 is combined with some noise ϵ\epsilon to produce a noised (and rescaled) sample xx, the same procedure that derives the usual denoising score matching loss for the spatial score ∇log⁡p\nabla \log p can also be applied to time score ∂tlog⁡p\partial_t \log p, resulting in analogous training objectives.

Namely, it holds that

∇log⁡p(x,t)=arg min⁡SxEx1,x[∥Sx(x,t)−∇log⁡p(x,t∣x1)∥2]\nabla \log p(x,t) = \argmin_{S_x} \mathbb{E}_{x_1,x} \left[\Vert{}S_x(x,t) - \nabla \log p(x,t|x_1) \Vert{}^2\right] ∂log⁡p(x,t)=arg min⁡StEx1,x[∥St(x,t)−∂log⁡p(x,t∣x1)∥2]\partial \log p(x,t) = \argmin_{S_t} \mathbb{E}_{x_1,x} \left[\Vert{}S_t(x,t) - \partial \log p(x,t|x_1) \Vert{}^2\right]

Which correspond, respectively, to Eq. (5) and Eq. (8) in our manuscript. The so called "targets" are simply the partial derivatives of the conditional log⁡p(x,t∣x1)\log p(x,t|x_1) as presented in Eq. (2).

Besides the differences in presentation, the most relevant differences are:

  • We define the noising process ∇log⁡p(x,t∣xclean)\nabla \log p(x,t|x_{\text{clean}}) differently. In our work, a conditional sample is obtained by x=txclean+(1−t)ϵx = tx_{\text{clean}} + (1-t)\epsilon, while in theirs it is obtained by x=xclean+tϵx = x_{\text{clean}} + \sqrt{t}\epsilon with t∈[0,+∞[t \in [0, +\infty[. These are well known formulations and we can expand on the advantages of our choice if desired, but it should be obvious that using t→+∞t \to +\infty complicates our goal of studying and tackling numerical issues.

  • Their results are expressed in terms of what they call the energy Uθ(x,t)=−log⁡p(x,t)U_{\theta}(x,t) = -\log p(x,t), which is not mathematically problematic and is not uncommon, but we avoid it as it obfuscates the connection to the diffusion literature that is most relevant to our work and can be confusing since the energy is usually not normalized.

  • We apply an extra algebraic manipulation to further improve conditioning by reducing the order of a singularity in both spatial and temporal losses. Specifically, we use x−tx11−t=ϵ\frac{x-tx_1}{1-t} = \epsilon to modify a denominator from (1−t)2(1-t)^2 to (1−t)(1-t). This simple type of modification is commonplace for the spatial score and was already used by Yu et al. (2025) for the time score.

Perhaps an interesting observation that could explain why the reviewer had concerns about these equations can be made by considering a generic transformation x=α(t)x1+β(t)ϵx = \alpha(t)x_1 + \beta(t)\epsilon. Applying the procedures outlined above yields the targets

∇xlog⁡p(x∣x1,t)=−ϵβ(t)\nabla_x \log p(x \mid x_1, t) = -\frac{\epsilon}{\beta(t)}

∂tlog⁡p(x∣x1,t)=[α˙(t)x1+β˙(t)ϵ]Tϵ−dβ˙(t)β(t)=vtgtTϵ−dβ˙(t)β(t)\partial_t \log p(x \mid x_1, t) = \frac{\left[\dot\alpha(t) x_1 + \dot \beta(t)\epsilon\right]^T\epsilon -d\dot\beta(t)}{\beta(t)} = \frac{v_{tgt}^T\epsilon -d\dot\beta(t)}{\beta(t)}

One might recognize the term vtgt=α˙(t)x1+β˙(t)ϵv_{tgt} = \dot\alpha(t) x_1 + \dot \beta(t)\epsilon as the conditional velocity (and also the flow target) and this substitution leaves the data dependent term vtgtTϵv^T_{tgt}\epsilon consistent among schedules up to the same scaling factor β−1(t)\beta^{-1}(t) that is present in the spatial target.

We didn't initially notice this generality across schedules, but it seems to further support the chosen structure and "cross flow" name. In our formulation, β˙(t)=−1\dot \beta(t) = -1, so the expression is not problematic and even makes the entire numerator independent of time. The choice taken by Guth et al. (2026) leads to β˙(t)=12t\dot \beta(t) = \frac{1}{2\sqrt{t}}, which is singular at t=0t=0 and that could also be tackled using a different rescaling strategy, albeit arguably a less generic one.

Removing the β(t)\beta(t) factor in the denominator is the next step in the paper as well as the reviewer's subsequent question, one for which we provide a self-contained answer.

3. Step-by-step relation between Eq. (8) and Eq. (9)

It is a simple time dependent rescaling proportional to (1−t)2(1-t)^2. Multiplying both sides of Eq. (8) and abbreviating the expectation for conciseness:

(1−t)2LTSMu=(1−t)2E[∣∣∂tlog⁡p(x,t)−d+(x1−ϵ)Tϵ1−t∣∣2](1-t)^2\mathcal{L}^{u}_{TSM} = (1-t)^2\mathbb{E}\left[|| \partial_t \log p(x,t) - \frac{d + (x_1 - \epsilon)^T\epsilon}{1-t} ||^2 \right]

This is already the first half of Eq. (9). We can then move the (1−t)2(1-t)^2 term inside the expectation and then inside the square term (1−t)2LTSMu=E[∣∣(1−t)(∂tlog⁡p(x,t)−d+(x1−ϵ)Tϵ1−t)∣∣2](1-t)^2\mathcal{L}^{u}_{TSM} = \mathbb{E}\left[|| (1-t) \left(\partial_t \log p(x,t) - \frac{d + (x_1 - \epsilon)^T\epsilon}{1-t} \right)||^2\right]

Distributing the (1−t)(1-t) term and canceling the denominator

(1−t)2LTSMu=E[∣∣(1−t)∂tlog⁡p(x,t)−(d+(x1−ϵ)Tϵ)∣∣2](1-t)^2\mathcal{L}^{u}_{TSM} = \mathbb{E}\left[|| (1-t) \partial_t \log p(x,t) - (d + (x_1 - \epsilon)^T\epsilon) ||^2\right]

and using the flow target vtgt=x1−ϵv_{\text{tgt}} = x_1 - \epsilon as labeled in line 172, we get the second half of Eq. (9)

(1−t)2LTSMu=E[∣∣(1−t)∂tlog⁡p(x,t)−(d+vtgtTϵ)∣∣2](1-t)^2\mathcal{L}^{u}_{TSM} = \mathbb{E}\left[|| (1-t) \partial_t \log p(x,t) - (d + v_{\text{tgt}}^T\epsilon) ||^2\right]

Note that this modification reweights the importance of each time tt without changing the actual minimizer at each time tt, thus preserving the global minimizer. This is analogous to the procedure of transforming the score target −ϵ1−t\frac{-\epsilon}{1-t} into the ϵ\epsilon or denoising target ϵ\epsilon.

4. Details on certain experiments

  • Table 3 uses λw=0\lambda_{w} = 0 as it ablates different spatial parametrizations for sampling purposes only. Since it is the potential version, all experiments use the ϕ\phi field.
  • Table 4 uses λw=10−3\lambda_{w} = 10^{-3} following the corresponding reference parameters in Table 2. Note how the FID and IS values are the same. The potential version uses ϕ\phi (by definition), while the vector one uses FT.
  • Table 5 uses λw=10−3\lambda_{w} = 10^{-3} following the same reference parameters in Table 2-b. The values are different because this is not an ablation, so we use a patch size of 4 instead of 8.

5. Discrepancy in NLL values between Table 3 and Table 5

We assume the reviewer meant to ask about the discrepancy between Tables 4 and 5 because Table 3 has no NLL. These experiments are in different spaces: Table 4 is in a latent space and Table 5 is in pixel space. Intuitively, one would expect the latent space to be more compressed and indeed that's why it is in the rough scale of 11 bpd instead of 33 bpd in pixel space.

We are not aware of any effective ways to compare the densities in both spaces without specialized encoders or decoders as one would need to compute the log⁡det⁡∣J∣\log \det |J|, though one could pursue an ELBO estimate. This makes direct comparison between models defined over different spaces harder, but does not prevent other interesting applications and we illustrate that with the experiments displayed in Figures 1 and 4, all of which are on the latent space and correlate to sample quality on pixel space. Due to the existence of this correlation, we believe that future work could leverage the density as a guidance signal to improve generative performance, which in our view is a more interesting application than its usage as a metric for model fit.

6. In Table 4 our parametrization is worse than vv-pred

Based on question 9, we first reiterate that our parametrization is the ProGaMeR one as displayed in Table 1. Both vv-pred and ProGaMeR are singularity free in the axa_x and aFa_F terms needed for sampling, which is the scope of the experiments in Table 4. The singularities of vv-pred only appear when computing ss or log⁡p\log p at t=1t=1, which is arguably the most important time for log⁡p\log p as it is the one corresponding to the clean data space.

Since both parametrizations are already singularity-free in the setting of Table 4, we would expect them to perform comparably. The difference is small and in a similar range to what was observed in Table 2-b. Additionally, most of our training parameters as well as our backbone network and VAE come from the LDiT paper, which used the vv-pred formulation and thus these choices might be better tuned for vv-pred.

Reviewer_mhS7OpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies diffusion model design for efficient likelihood estimation while preserving sampling quality. Building on time-score matching and dual score matching, it introduces a rescaled temporal objective, a modular framework covering vector fields, scalar potentials, and mean-field formulations, and the boundary-aware ProGaMeR parameterization. Experiments include synthetic density estimation and ImageNet generation in pixel and latent spaces.

Critical strengths

The paper addresses an important practical tension between density estimation and generation. Explicitly considering endpoint behavior, temporal-loss weighting, and alternative representations provides a useful perspective on training diffusion models for both tasks. Treating temporal supervision as an adjustable auxiliary objective is a sensible approach, and the reported ablations suggest that it need not substantially degrade sampling quality in the evaluated settings.

Critical weaknesses

Likelihood accuracy and computational benefits are insufficiently validated at image scale. Table 2 measures agreement with another trained model, rather than an independently validated likelihood reference. High correlation and agreement across configurations cannot rule out shared normalization errors. Likewise, sample ranking remains unchanged under additive shifts in log density, so the filtering experiment cannot validate absolute likelihood accuracy. The paper acknowledges normalization concerns, but does not resolve them for its reported image likelihoods.

Questions for authors

For the same trained image model, how closely do the direct or time-score-based likelihood estimates agree with a converged divergence-based probability-flow calculation on held-out images, and what wall-clock speedup is achieved at a comparable accuracy level?

3 comments

We'd like to thank the reviewer for their analysis of our work and for providing actionable directions on how those could be addressed. Following feedback, we implemented an analysis that we believe greatly elevates the likelihood accuracy validation.

Divergence-based likelihood estimates (using Hutchinson's trace estimator) and time-score-based ones are computed following Eq. (1) on models from Table 2. We consider 128 sample images taken from the (held-out) validation set and use a Heun integrator with 400 steps in all cases as a reasonable reference for convergence based on observed trends.

1a. Model's likelihood estimate agreement with divergence-based calculation

The best tested models had likelihoods agreeing with divergence based estimates within roughly a 0.050.05 bpd scale, indicating a reasonable degree of agreement. The general trend is that higher λw\lambda_w improved self-consistency, while optimal time scaling was more nuanced and depends on the dataset. Besides the other models presented in Table 2, additional ones are being trained to further characterize these patterns.

The following two tables compare each model's likelihood prediction using the potential function against those computing either the divergence of the flow or the time score. We report the bias that centers the error as well as the residual mean absolute error (rMAE) after the bias is removed to approximately decouple normalization errors and per sample errors. The likelihood scale is in 0.010.01 bpd for visualization purposes.

Latent space models:

Time scalingλw\lambda_wBias (divergence)rMAE (divergence)Bias (time score)rMAE (time score)
100010−210^{-2}-2.323.51-0.09860.0650
100010−310^{-3}-2.794.450.5930.137
100010−410^{-4}-6.755.941.280.246
110−210^{-2}6.083.600.001670.00308
110−310^{-3}15.54.62-0.001580.00402
110−410^{-4}44.24.85-0.003200.00373

Pixel space models:

Time scalingλw\lambda_wBias (divergence)rMAE (divergence)Bias (time score)rMAE (time score)
100010−210^{-2}3.5013.92.981.52
100010−310^{-3}7.1925.13.084.00
100010−410^{-4}35.932.93.074.00
110−210^{-2}-0.7177.770.1960.164
110−310^{-3}12.111.80.1860.160
110−410^{-4}64.314.10.1370.146

The choice of best model is debatable, but λw=10−2\lambda_w=10^{-2} showed consistent improvements over lower values in all settings. Higher time scaling improved consistency in latent space, but had an opposite or dubious effect in pixel space, perhaps reflecting their differences in smoothness and the fact that LDiT was originally developed for latent applications. The agreement between potential and time score based estimates also generally favored higher λw\lambda_w, but benefited from lower time scaling, presumably as it turned ∂tlog⁡p\partial_t \log p smoother and easier to integrate.

1b. Wall-clock speedup at comparable accuracy level

For presentation purposes, the wall-clock times are reported per iteration alongside memory for completeness, followed by an estimate on the number of iterations required until convergence. Timing generally agrees with the theoretical prediction that each slice of the divergence requires an extra derivative, so roughly twice as much computation and memory.

We provide the associated costs observed for likelihood computation for an RTX PRO 6000 Blackwell in the table below. The values seem to follow general rules of thumb based on the type of computation involved. Accounting for the number of evaluations, moving from a single network call to computing its derivative (VJP), increased computation time by a factor of 3.5, which appears explainable by the fact that backward passes require storing intermediate activations and are generally more expensive.

MethodTimeMemory (GB)Computation type
Potential18 ms total0.61 ×\times network evaluation
Time score125 ms/step1.92 ×\times network derivative (1st order)
Divergence239 ms/step4.52 ×\times network derivative (2nd order)

Convergence metrics are reported in the table below for the λw=10−2\lambda_w=10^{-2} case, generally the most effective one in the ablation. We solve the associated ODEs with different total number of steps, starting at 25 and doubling it until 400. We use variations close to 0.010.01 bpd relative to the values achieved at 400 steps as a rough criterion for convergence.

SpaceTime scalingDivergence stepsTime score steps
Latent100050100
Latent15025
Pixel1000200200
Pixel1400400

For the potential case, integrating the time score might be unnecessary given that a single network call already gives a density estimate very efficiently, which is the reason previous work and ours focuses on it. Nevertheless, we consider these results as promising signals for the vector case, besides the reduction in computational requirements. If the model is used to generate a new sample, it is already computing and integrating vv, so including dlog⁡pdt\frac{d\log p}{dt} in the process provides likelihood estimates during sampling, which we believe could be used to steer generation as evidenced by the correlation between sample quality and likelihood displayed in Figures 1 and 4.

Thanks for the author response. I have raised the score.

Comments on the paper

2 comments

We appreciate the reviewer for their comment, but it seems to be one aimed at another submission.

Thanks for the author response. I apologize for misreading the equations in my original review. I have raised the score.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.