- Rating
- 3(+1)
- Confidence
- 1(−1)
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper proposes a diffusion model framework for estimating likelihoods without costly divergence calculations. Its temporal score objective and endpoint stable parameterization aim to preserve generation quality, with results on synthetic data and ImageNet64.
Critical strengths
-
The framework connects several model formulations and makes clear how each can provide the quantities needed for sampling and likelihood estimation.
-
The rescaled temporal target and endpoint aware parameterization address identifiable singularities, rather than relying only on training heuristics.
Critical weaknesses
-
Most systematic likelihood analyses and ablations focus on ImageNet64. Although the paper reports ImageNet256 sampling and likelihood-based filtering, it provides limited quantitative evidence on likelihood accuracy or calibration at that resolution. Evaluation on another image dataset would further strengthen the generalization claim.
-
The paper reports approximate training times but does not systematically compare its additional training cost with the savings in likelihood evaluation. A comparison under matched hardware and comparable estimation accuracy should report training time, evaluation time, network evaluations, and memory use separately.
-
The image experiments primarily use an LDiT backbone. Testing a U-Net backbone would help establish whether the stability and quality results transfer to a different architecture. The paper should also clarify which parts of its likelihood-estimation procedure depend on the continuous-time sampling trajectory and whether they extend to other samplers, such as DDIM.
Questions for authors
-
The paper reports ImageNet256 sampling and likelihood-based filtering, but provides less systematic evidence on the calibration of its likelihood estimates at that resolution. Could the authors provide such an evaluation and, if feasible, test generalization on a dataset outside ImageNet, such as LSUN?
-
How much additional training time and memory does temporal-score learning require? Could the authors compare likelihood-evaluation time, network evaluations, and accuracy against divergence-based estimation under the same hardware and evaluation budget?
-
Does the proposed objective and parameterization remain stable with a U-Net backbone or other Transformer variants? Could the authors clarify whether the proposed method can be used with other pipelines, such as LDM or FLUX?
5 comments
2a. Additional training time and memory for temporal score learning
The short answer is that adding a time score loss as we do has negligible effect on training time compared to training the same formulation with only spatial score losses. This is because it utilizes information that was already being computed (e.g., ) and leverages computations that are already being done (e.g., backpropagation through a network) while only adding simple operations.
The main driving factors for compute costs are the formulation and implementation details. In the potential one, the same backward pass (VJP) that is used to compute already produces the information needed for and , so the extra effort is that of simple operations and backpropagating from the time score loss into the network outputs, which is negligible compared to the backpropagation through the network itself that was already happening with spatial score. We don't see any meaningful difference (i.e., about 2 MB of extra memory) and don't expect that to be different outside of artificial examples. For similar reasons, in the vector one, the extra cost would be proportional to the effort spent computing the extra scalar (either or ), which we don't expect to require a significant increase in complexity as it is only a somewhat well-behaved scalar. In the experiments of Table 4 we used the simplest strategy of adding a readout head.
The potential one requires a second order derivative to train, so in theory that would double every cost. In practice, that may be closer to 3 times, partially because the software infrastructure is not as mature for that case as it is for the usual single backpropagation. We illustrate that with some values for the larger latent space models (LDiT/B1) on an RTX PRO 6000 Blackwell in the table below. For the pixel space model, the patch size is adjusted to keep the number of tokens the same and so the computational cost is similar.
| Formulation | Att. Kernel | Iter/s | Memory (GB) |
|---|---|---|---|
| Vector | Original | 7.2 | 19.7 |
| Vector | Fused | 8.5 | 16.2 |
| Potential | Original | 2.7 | 58.8 |
| Potential | Fused | N/A | N/A |
| Potential + | Original | 4.6 | 27.0 |
| Potential + | Fused | 5.5 | 19.0 |
For context, there are heavily optimized fused kernels to compute attention (e.g. FlashAttention) and PyTorch provides access to those and a fallback one, while the implementations in DiT and LDiT use elementary operations. Unfortunately, the most efficient kernels do not yet support double backpropagation, so we use the original one as it was slightly faster than the fallback. We included the potential + case to quantify the hypothetical benefits of the trick we identified as the potential analogue to the trick used in the mean flow of Geng et al. (2025), one that could mitigate the higher training cost for the potential formulation. We'll only mention that there are also software issues that appear when combining certain features like high order derivatives, compilation and half precision (i.e., potential PyTorch bugs).
2b. Likelihood-evaluation time, network evaluations, and accuracy against divergence-based estimation
Please see our response 1a for accuracy estimates. Timing generally agrees with the theoretical prediction that each slice of the divergence requires an extra derivative, so roughly twice as much computation and memory. Using the potential formulation, though it requires the extra training budget we discussed in our response 2a, has the advantage of being able to evaluate likelihoods with a single network call and no integration is required.
We provide the associated costs observed for likelihood computation for an RTX PRO 6000 Blackwell in the table below. The values seem to follow general rules of thumb based on the type of computation involved. Accounting for the number of evaluations, moving from a single network call to computing its derivative (VJP), increased computation time by a factor of 3.5, which appears explainable by the fact that backward passes require storing intermediate activations and are generally more expensive.
| Method | Time | Memory (GB) | Computation type |
|---|---|---|---|
| Potential | 18 ms total | 0.6 | 1 network evaluation |
| Time score | 125 ms/step | 1.9 | 2 network derivative (1st order) |
| Divergence | 239 ms/step | 4.5 | 2 network derivative (2nd order) |
Convergence metrics are reported in the table below for the case, generally the most effective one in the ablation. We solve the associated ODEs with different total number of steps, starting at 25 and doubling it until 400. We use variations close to bpd relative to the values achieved at 400 steps as a rough criterion for convergence.
| Space | Time scaling | Divergence steps | Time score steps |
|---|---|---|---|
| Latent | 1000 | 50 | 100 |
| Latent | 1 | 50 | 25 |
| Pixel | 1000 | 200 | 200 |
| Pixel | 1 | 400 | 400 |
For the potential case, integrating the time score might be unnecessary given that a single network call already gives a density estimate very efficiently, which is the reason previous work and ours focuses on it. Nevertheless, we consider these results as promising signals for the vector case, besides the reduction in computational requirements. If the model is used to generate a new sample, it is already computing and integrating , so including in the process provides likelihood estimates during sampling, which we believe could be used to steer generation as evidenced by the correlation between sample quality and likelihood displayed in Figures 1 and 4.
We'd like to thank the reviewer for their effort and bringing many relevant practical considerations. Evaluating likelihood accuracy and calibration is indeed a key question, so we focused our efforts on this topic and hope we can incorporate other ideas shortly, at least for a final version, if not in time for discussion.
Whenever possible, we provided a short answer followed by lengthier discussions.
1a. Evidence on likelihood calibration
After reviewer feedback, we used divergence based estimates as a reference to compare the other model estimates. The general trend is that higher λw improved self-consistency, while optimal time scaling was more nuanced and depends on the dataset. The best tested models had likelihoods agreeing within roughly a 0.05 bpd scale. We leverage the fact that we focus on efficient learning of the flow v in order to use its divergence (approximated with Hutchinson's trace estimator) to measure the model's internal belief.
The following two tables compare each model's prediction after 400 Heun steps (each requiring two function evaluations), where these quantities appeared to have converged. We compare the likelihood prediction using the potential function against those computing either the divergence of the flow or the time score, both according to Eq. (1). We use 128 samples from the validation set (held-out) and report the bias that centers the error as well as the residual mean absolute error (rMAE) after the bias is removed to approximately decouple normalization errors and per sample errors. The likelihood scale is in 0.01 bpd for visualization purposes.
Latent space models:
Pixel space models:
The choice of best model is debatable, but λw=10−2 showed consistent improvements over lower values in all settings. Higher time scaling improved consistency in latent space, but had an opposite or dubious effect in pixel space, perhaps reflecting their differences in smoothness and the fact that LDiT was originally developed for latent applications. The agreement between potential and time score based estimates also generally favored higher λw, but benefited from lower time scaling, presumably as it turned ∂tlogp smoother and easier to integrate.
1b. Generalization to other datasets
We hadn't considered testing a model trained on ImageNet on other datasets, but, if the reviewer agrees that the likelihood calibration procedure we are proposing is appropriate, we could do so on some version of LSUN, as well as repeating the filtering experiment on samples generated by another model. During the early stages of our work, we used CIFAR-10, but the dataset is small and the main challenge we had was overfitting, which we thought to be less relevant as it is not representative of the issues found at larger scale.