- Rating
- 2
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The authors propose a simple idea, similar to REPA, to provide regularization during diffusion model training. Specifically, they propose to learn features which are both useful for conditional score prediction as well as the gradient of the classifier, which the authors call "likelihood score". Consequently, for negligible extra cost, the models appear to learn faster on multiple tasks and compared to multiple baselines.
Critical strengths
Main strengths
- Simplicity of the method - an extra head learning to predict "likelihood score" (which to me is the gradient of the noisy classifier)
- Lack of computational overhead. The method introduces negligible overhead in terms of number of model parameters, training time or inference cost
Critical weaknesses
These are the main weaknesses that I identify
- I feel like the empirical validation is mixed. While indeed in the tables we see quite a few bold numbers in the row corresponding to the proposed method, I read it more as a tradeoff. In multiple examples in table 1, we see that some metrics (FID, CLIP) deteriorate with longer training while no limitations are discussed in the paper. We also only see matches comparisons at what seems like arbitrarily selected training steps (e.g. 10k vs 10k and then 30k vs 25k). This lack of consistency in how the method is compared with baselines and no limitations discussion makes me wonder if these were cherrypicked (because it is not discussed why methods were compared at these specific checkpoints). For example, authors say: "For example, in depth-conditioned image generation, LISA trained for only 4K iterations obtains better FID, CLIP, and RMSE than ControlNet trained for 10K iterations. " However, it also seems like LISA at 4k steps is worse than at 2k steps on both FID and CLIP. So the question is whether this is some sort of reward hacking and the training/fine-tuning length should also be carefully selected.
- The writing could be somewhat streamlined. I feel like section 3 spends almost two pages essentially explaining the consequence of Bayes rule, which I would assume is very clear to people familiar with diffusion literature. The extra space could be spent by more in depth analysis of the empirical results
Questions for authors
- missing citation: the denoising score matching objective (eq 4) was initially proposed by P. Vincent "A Connection Between Score Matching and Denoising Autoencoders", Neural computation 2011
- Lines 151-196 is just explaining conditional denoising score matching - I think this doesn't need that much space. Similarly, equation 11 is the same as eq 8 just slightly rearranged
- Line 249 - "During training, the parameters of the main network are frozen." - I believe the authors mean only the stop gradient operator, right? Or is this procedure done after the main model has been trained? That's what the next sentence suggests: "The standard loss L_main supervises the final conditional prediction". Based on the number of training steps, it seems like this is finetuning only. The setup should be clarified
- I don't think that this is explained anywhere, but do the authors also do the standard thing with diffusion training that s(xt, t) is trained by doing dropout on the conditioning input c some percentage of the time? Otherwise, based on what is written, s(xt, t) has no training signal, because the only place it appears in the loss is after the stop_gradient operation. On the other hand, line 209 says that the unconditional score model has been pretrained. So is the model only learning the conditional score in what the authors are proposing?
- For the results, can the authors explain why and to what degree the results (such as FID or CLIP) deteriorate with longer training? Does it also happen for the baselines? Given that FID is an expensive metric, and I guess this eval was not run at every step, can the authors explain how where the checkpoints selected for matched comparison? It feels arbitrary. Also, why in some settings, we have +REPA and in some we do not (Table 1)? Also I think the choice of using bold numbers is a bit unclear. Is this meant to represent the best result for this selected checkpoint pair? It seems likely that baselines are better at different checkpoints.
- Can the authors discuss limitations of the proposed method? Current draft seems to suggest that this is always beneficial with no downsides
- Can the authors explain the reasoning behind "we directly composed the two conditions by summing their injected features at each corresponding layer"? Why would adding features correspond to the AND operator? And were the numbers if left panel of Figure 5 generated based just on these two examples?
8 comments
Response to Reviewer KrKd (2/2)
[Q6] Discussion of Limitation
Thank you for pointing out this omission.
Our likelihood-score interpretation assumes a sufficiently accurate pretrained main network. With a large downstream domain gap, the residual also includes a domain correction, which is not fully covered by this interpretation and may make alignment harder to optimize. Training stability under extreme domain shifts therefore needs further study. Our experiments are also limited to conditional fine-tuning due to computational constraints. Extending LISA to large-scale pretraining, including interactive video models, is left for future work. We have added this discussion of limitations and future work to the revised manuscript.
[Q7] Clarification of Feature Summation and Evaluation in Compositional-Condition Generation (Section 4.4)
-
Feature summation follows the original ControlNet setup. In Section 3.4 of ControlNet[2], the authors state: “we can directly add the outputs of the corresponding ControlNets to the Stable Diffusion model.” This choice is based on their empirical observation that summing features can provide joint control in practice. We apply the same rule to both ControlNet and +LISA, without assuming that feature addition strictly implements a logical AND operator.
-
The quantitative results in Figure 5 are computed on a test set of 1,154 samples. The two examples on the right are qualitative illustrations, not the basis of the quantitative evaluation.
Thank you for raising these questions. We have clarified the composition rule and evaluation setup in Section 4.4 and the caption of Figure 5 in the revised manuscript.
References:
[1] Vincent, Pascal. "A connection between score matching and denoising autoencoders." Neural computation 23.7 (2011): 1661-1674.
[2] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.
I thank the authors for the response and updated revision.
I am still not convinced by the empirical results. They seem fragmented and not systematic. For example
-
LISA consistently improves the condition-following ability across all four image tasks [...] Overall, these results indicate that LISA not only enhances the fidelity of synthesis but also accelerates convergence.
but in table 1 most of the final values in last columns seem very similar to the baselines, probably within statistical noise. I do not see convincing evidence for "enhancing the fidelity of synthesis".
-
LISA can achieve 2.78× faster convergence
This is claim from Figure 1 and the timings in Table 3 are based on one example (I believe Pose Map)? Also, I asked about REPA not being shown in all comparisons, because REPA also is shown to accelerate convergence. Therefore - in the early stages of training, where LISA shows considerable improvement over the baseline - how does LISA compare to REPA in that range? What I mean is it would be helpful to see REPA's results in the middle rows of Table 1.
- The controllability/quality tradeoff. Since we agree that there is a tradeoff between metrics like FID or CLIP and e.g. PCK, I think that the claim that LISA continues to improve beyond the selected 18K checkpoint means that PCK kept improving. What about the other metrics? How much have they deteriorated? What about other tasks beyond Pose map?
I think in summary, the results show that LISA accelerates the early phase of training, but
- It's unclear how the early acceleration compares to REPA, and
- it's unclear how much the early gains carry over to longer training
Response to Reviewer KrKd (1/2)
Dear Reviewer KrKd,
Thank you very much for taking the time to review our manuscript and for offering many valuable comments that helped us improve the work.
We sincerely appreciate your recognition that our idea is simple and that our method learns faster on multiple tasks than multiple baselines.
We will address your concerns point by point.
[W1 & Q5] Clarification of the Experimental Protocol and Results in Table 1
We apologize for the confusion. Below, we explain (1) why FID and CLIP can deteriorate with longer training, (2) how checkpoints were selected and results were reported, and (3) why +REPA appears only in some comparisons.
Learning stronger visual control can lead to a trade-off with image quality and text alignment later in training. FID measures the distributional distance between generated and real images, while CLIP measures alignment with the text prompt. In visual-condition controllable generation, training aims to incorporate additional controls such as pose and depth, so control accuracy, measured by PCK, mIoU, or RMSE, is a primary target. This trade-off also appears in vanilla ControlNet. For example, on the pose task of Table 1, training from 10K to 30K iterations improves PCK from 19.38% to 89.82%, while FID increases from 56.37 to 58.54 and CLIP decreases from 31.47 to 30.87. We therefore view these modest losses in image quality and text alignment as an expected trade-off for the substantial gain in controllability.
Within each task, we report an early equal-step comparison and the first LISA checkpoint that surpasses a well-trained baseline. Different tasks require different training durations. We save and evaluate checkpoints every 1K training iterations. Taking pose as an example, at 10K iterations, ControlNet achieves only 19.38% PCK, while LISA already reaches 83.02%, showing substantially faster learning of pose control. We then compare against a well-trained ControlNet checkpoint at 30K iterations. We selected 18K because it is the first evaluated LISA checkpoint that outperforms this baseline across all three reported metrics. To address your concern, we provide the PCK results at additional checkpoints below:
These results, consistent with the convergence curves in Figure 1, show that LISA continues to improve beyond the selected 18K checkpoint. Bold numbers in Table 1 indicate the best value within each comparison group.
Our main comparison is between the original baselines and their +LISA versions. REPA serves as an additional regularization baseline, so we include it in one comparison per task in Table 1, using the same number of training iterations as LISA, rather than at every reported checkpoint.
In summary, LISA consistently accelerates convergence and improves controllability. The additional checkpoint results, together with the convergence curves in Figure 1, show that these gains persist at later training stages and are not limited to the selected comparisons in Table 1.
[W2 & Q2] Streamlining the Writing in Section 3
Thank you for this helpful suggestion. In the revised manuscript, we have shortened the discussion of conditional denoising score matching and Bayes’ rule in Section 3, removed redundant explanations, and reorganized the equations to avoid repetition.
[Q1] Missing Citation
Thank you for pointing this out. We have added the citation P. Vincent[1] alongside Eq. (4) in the revised manuscript.
[Q3 & Q4] Clarification of the Training Setup and the Frozen Main Network
We start from a pretrained main network and keep its parameters frozen throughout training, only the side network and auxiliary decoder are optimized. The unconditional score comes from this pretrained network without visual-condition injection. We therefore do not need conditioning dropout to learn the unconditional score during LISA training.
We call the loss L_main because it is the primary loss relative to the auxiliary LISA loss, not because it trains the main network. It supervises the final conditional prediction and backpropagates through the frozen main network to update the side network. The stop-gradient operation is applied to the output of the additional main-network forward pass used to construct the auxiliary-loss target.