acceptodds
Under review as a conference paper at ICLR 2027

LISA: Likelihood Score Alignment for Visual-condition Controllable Generation

Abstract

The prevalent dual-branch paradigm, i.e., training a side network to encode visual conditions and fusing its intermediate-layer features to a frozen pretrained main network, has shown remarkable success in visual-condition controllable generation. Despite its widespread adoption, the role of the side branch and its training efficiency remain underexplored. In this paper, we first revisit this mainstream paradigm through the lens of score-based generative modeling: 1) The main network preserves visual perceptual quality by providing a prior unconditional score. 2) The side network steers conditional control by implicitly contributing a likelihood score. Guided by this perspective, we propose LIkelihood Score Alignment (LISA), an effective regularization method that explicitly aligns the intermediate feature of the side network with an approximated likelihood score. Specifically, we first hook features from a designated layer of the side network and project them into the score latent space by a lightweight decoder. Then, we construct an approximated likelihood score target and calculate the distance between the decoder's output and this target as an additional regularization loss. Finally, we jointly optimize the side network and decoder with both standard diffusion loss and our regularization loss. Experiments across various image/video tasks, architectures, and diffusion/flow models demonstrated that LISA can not only consistently accelerate the training convergence and improve final synthetic results, but also encourage the side network's features to be more disentangled for conditional modeling with negligible additional training cost and zero extra inference cost.

Primary area:
generative models
Keywords:
Controllable Visual Generation, Diffusion Models, Score Matching

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_KrKdOpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a simple idea, similar to REPA, to provide regularization during diffusion model training. Specifically, they propose to learn features which are both useful for conditional score prediction as well as the gradient of the classifier, which the authors call "likelihood score". Consequently, for negligible extra cost, the models appear to learn faster on multiple tasks and compared to multiple baselines.

Critical strengths

Main strengths

  1. Simplicity of the method - an extra head learning to predict "likelihood score" (which to me is the gradient of the noisy classifier)
  2. Lack of computational overhead. The method introduces negligible overhead in terms of number of model parameters, training time or inference cost
Critical weaknesses

These are the main weaknesses that I identify

  1. I feel like the empirical validation is mixed. While indeed in the tables we see quite a few bold numbers in the row corresponding to the proposed method, I read it more as a tradeoff. In multiple examples in table 1, we see that some metrics (FID, CLIP) deteriorate with longer training while no limitations are discussed in the paper. We also only see matches comparisons at what seems like arbitrarily selected training steps (e.g. 10k vs 10k and then 30k vs 25k). This lack of consistency in how the method is compared with baselines and no limitations discussion makes me wonder if these were cherrypicked (because it is not discussed why methods were compared at these specific checkpoints). For example, authors say: "For example, in depth-conditioned image generation, LISA trained for only 4K iterations obtains better FID, CLIP, and RMSE than ControlNet trained for 10K iterations. " However, it also seems like LISA at 4k steps is worse than at 2k steps on both FID and CLIP. So the question is whether this is some sort of reward hacking and the training/fine-tuning length should also be carefully selected.
  2. The writing could be somewhat streamlined. I feel like section 3 spends almost two pages essentially explaining the consequence of Bayes rule, which I would assume is very clear to people familiar with diffusion literature. The extra space could be spent by more in depth analysis of the empirical results
Questions for authors
  1. missing citation: the denoising score matching objective (eq 4) was initially proposed by P. Vincent "A Connection Between Score Matching and Denoising Autoencoders", Neural computation 2011
  2. Lines 151-196 is just explaining conditional denoising score matching - I think this doesn't need that much space. Similarly, equation 11 is the same as eq 8 just slightly rearranged
  3. Line 249 - "During training, the parameters of the main network are frozen." - I believe the authors mean only the stop gradient operator, right? Or is this procedure done after the main model has been trained? That's what the next sentence suggests: "The standard loss L_main supervises the final conditional prediction". Based on the number of training steps, it seems like this is finetuning only. The setup should be clarified
  4. I don't think that this is explained anywhere, but do the authors also do the standard thing with diffusion training that s(xt, t) is trained by doing dropout on the conditioning input c some percentage of the time? Otherwise, based on what is written, s(xt, t) has no training signal, because the only place it appears in the loss is after the stop_gradient operation. On the other hand, line 209 says that the unconditional score model has been pretrained. So is the model only learning the conditional score in what the authors are proposing?
  5. For the results, can the authors explain why and to what degree the results (such as FID or CLIP) deteriorate with longer training? Does it also happen for the baselines? Given that FID is an expensive metric, and I guess this eval was not run at every step, can the authors explain how where the checkpoints selected for matched comparison? It feels arbitrary. Also, why in some settings, we have +REPA and in some we do not (Table 1)? Also I think the choice of using bold numbers is a bit unclear. Is this meant to represent the best result for this selected checkpoint pair? It seems likely that baselines are better at different checkpoints.
  6. Can the authors discuss limitations of the proposed method? Current draft seems to suggest that this is always beneficial with no downsides
  7. Can the authors explain the reasoning behind "we directly composed the two conditions by summing their injected features at each corresponding layer"? Why would adding features correspond to the AND operator? And were the numbers if left panel of Figure 5 generated based just on these two examples?
8 comments
AuthorsOpenReview ↗

Response to Reviewer KrKd (1/2)

Dear Reviewer KrKd,

Thank you very much for taking the time to review our manuscript and for offering many valuable comments that helped us improve the work.

We sincerely appreciate your recognition that our idea is simple and that our method learns faster on multiple tasks than multiple baselines.

We will address your concerns point by point.


[W1 & Q5] Clarification of the Experimental Protocol and Results in Table 1

We apologize for the confusion. Below, we explain (1) why FID and CLIP can deteriorate with longer training, (2) how checkpoints were selected and results were reported, and (3) why +REPA appears only in some comparisons.

  1. Learning stronger visual control can lead to a trade-off with image quality and text alignment later in training. FID measures the distributional distance between generated and real images, while CLIP measures alignment with the text prompt. In visual-condition controllable generation, training aims to incorporate additional controls such as pose and depth, so control accuracy, measured by PCK, mIoU, or RMSE, is a primary target. This trade-off also appears in vanilla ControlNet. For example, on the pose task of Table 1, training from 10K to 30K iterations improves PCK from 19.38% to 89.82%, while FID increases from 56.37 to 58.54 and CLIP decreases from 31.47 to 30.87. We therefore view these modest losses in image quality and text alignment as an expected trade-off for the substantial gain in controllability.

  2. Within each task, we report an early equal-step comparison and the first LISA checkpoint that surpasses a well-trained baseline. Different tasks require different training durations. We save and evaluate checkpoints every 1K training iterations. Taking pose as an example, at 10K iterations, ControlNet achieves only 19.38% PCK, while LISA already reaches 83.02%, showing substantially faster learning of pose control. We then compare against a well-trained ControlNet checkpoint at 30K iterations. We selected 18K because it is the first evaluated LISA checkpoint that outperforms this baseline across all three reported metrics. To address your concern, we provide the PCK results at additional checkpoints below:

    MethodTraining iterations[email protected] (%) ↑
    ControlNet30K89.82
    +LISA14K88.92
    +LISA18K89.90
    +LISA22K90.20
    +LISA26K90.64
    +LISA30K91.02

    These results, consistent with the convergence curves in Figure 1, show that LISA continues to improve beyond the selected 18K checkpoint. Bold numbers in Table 1 indicate the best value within each comparison group.

  3. Our main comparison is between the original baselines and their +LISA versions. REPA serves as an additional regularization baseline, so we include it in one comparison per task in Table 1, using the same number of training iterations as LISA, rather than at every reported checkpoint.

In summary, LISA consistently accelerates convergence and improves controllability. The additional checkpoint results, together with the convergence curves in Figure 1, show that these gains persist at later training stages and are not limited to the selected comparisons in Table 1.


[W2 & Q2] Streamlining the Writing in Section 3

Thank you for this helpful suggestion. In the revised manuscript, we have shortened the discussion of conditional denoising score matching and Bayes’ rule in Section 3, removed redundant explanations, and reorganized the equations to avoid repetition.


[Q1] Missing Citation

Thank you for pointing this out. We have added the citation P. Vincent[1] alongside Eq. (4) in the revised manuscript.


[Q3 & Q4] Clarification of the Training Setup and the Frozen Main Network

We start from a pretrained main network and keep its parameters frozen throughout training, only the side network and auxiliary decoder are optimized. The unconditional score comes from this pretrained network without visual-condition injection. We therefore do not need conditioning dropout to learn the unconditional score during LISA training.

We call the loss L_main because it is the primary loss relative to the auxiliary LISA loss, not because it trains the main network. It supervises the final conditional prediction and backpropagates through the frozen main network to update the side network. The stop-gradient operation is applied to the output of the additional main-network forward pass used to construct the auxiliary-loss target.

AuthorsOpenReview ↗

Response to Reviewer KrKd (2/2)

[Q6] Discussion of Limitation

Thank you for pointing out this omission.

Our likelihood-score interpretation assumes a sufficiently accurate pretrained main network. With a large downstream domain gap, the residual also includes a domain correction, which is not fully covered by this interpretation and may make alignment harder to optimize. Training stability under extreme domain shifts therefore needs further study. Our experiments are also limited to conditional fine-tuning due to computational constraints. Extending LISA to large-scale pretraining, including interactive video models, is left for future work. We have added this discussion of limitations and future work to the revised manuscript.


[Q7] Clarification of Feature Summation and Evaluation in Compositional-Condition Generation (Section 4.4)

  1. Feature summation follows the original ControlNet setup. In Section 3.4 of ControlNet[2], the authors state: “we can directly add the outputs of the corresponding ControlNets to the Stable Diffusion model.” This choice is based on their empirical observation that summing features can provide joint control in practice. We apply the same rule to both ControlNet and +LISA, without assuming that feature addition strictly implements a logical AND operator.

  2. The quantitative results in Figure 5 are computed on a test set of 1,154 samples. The two examples on the right are qualitative illustrations, not the basis of the quantitative evaluation.

Thank you for raising these questions. We have clarified the composition rule and evaluation setup in Section 4.4 and the caption of Figure 5 in the revised manuscript.


References:

[1] Vincent, Pascal. "A connection between score matching and denoising autoencoders." Neural computation 23.7 (2011): 1661-1674.

[2] Zhang, Lvmin, Anyi Rao, and Maneesh Agrawala. "Adding conditional control to text-to-image diffusion models." 2023 IEEE/CVF International Conference on Computer Vision (ICCV). IEEE, 2023.

I thank the authors for the response and updated revision.

I am still not convinced by the empirical results. They seem fragmented and not systematic. For example

  1. LISA consistently improves the condition-following ability across all four image tasks [...] Overall, these results indicate that LISA not only enhances the fidelity of synthesis but also accelerates convergence.

but in table 1 most of the final values in last columns seem very similar to the baselines, probably within statistical noise. I do not see convincing evidence for "enhancing the fidelity of synthesis".

  1. LISA can achieve 2.78× faster convergence

This is claim from Figure 1 and the timings in Table 3 are based on one example (I believe Pose Map)? Also, I asked about REPA not being shown in all comparisons, because REPA also is shown to accelerate convergence. Therefore - in the early stages of training, where LISA shows considerable improvement over the baseline - how does LISA compare to REPA in that range? What I mean is it would be helpful to see REPA's results in the middle rows of Table 1.

  1. The controllability/quality tradeoff. Since we agree that there is a tradeoff between metrics like FID or CLIP and e.g. PCK, I think that the claim that LISA continues to improve beyond the selected 18K checkpoint means that PCK kept improving. What about the other metrics? How much have they deteriorated? What about other tasks beyond Pose map?

I think in summary, the results show that LISA accelerates the early phase of training, but

  • It's unclear how the early acceleration compares to REPA, and
  • it's unclear how much the early gains carry over to longer training
Reviewer_zpZ1OpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper proposes LISA, a training regularizer for visual-condition control branches attached to frozen diffusion or flow backbones. A lightweight decoder maps an intermediate side-network feature to the residual between the standard denoising target and the backbone prediction without visual-condition injection. The decoder and side network are trained jointly with the usual generation objective, and the decoder is removed at inference. Experiments cover four image-control tasks, SD3 segmentation control, SVD pose-guided video generation, and composition of independently trained pose and segmentation branches.

Critical strengths

The residual target is simple, requires no additional semantic encoder, and can be constructed from quantities already available in generative training plus one uncontrolled backbone forward. Removing the auxiliary decoder preserves the original inference architecture.

The reported early-training gains are substantial.

Critical weaknesses

The experiments do not isolate the benefit of likelihood-residual alignment from ordinary auxiliary supervision. Equations (6), (12), and A19 support the target's conditional-expectation interpretation when the frozen predictor approximates the appropriate data score. They do not imply that a nonlinear feature-injected backbone uses its intermediate side features as an identifiable likelihood-score component. The independently trained decoder is not constrained to reproduce the actual controlled-minus-uncontrolled output correction (Eqs. 9–15). Table 2 only varies depth and weight. Matched-capacity auxiliary heads predicting the ordinary denoising target or the control, removal of the frozen prediction from the target, and removal of the side feature from the decoder would help establish what causes the gain. Figure 5 supports improved composition in one setting, but does not establish representation disentanglement. The paper should also compare with or clearly position itself against InnerControl, which already studies intermediate-feature auxiliary feedback for ControlNet; its control-reconstruction target differs from LISA's residual target

Questions for authors

What are the exact training/test datasets, splits, image and generated-sample counts, condition extractors, and metric implementations for each task? What scaling is used for video LPIPS?

Under the same decoder capacity and training budget, how does LISA compare with an ordinary denoising-target auxiliary head, a control-reconstruction head, and a decoder without the side feature? Does the decoded residual track the actual controlled-minus-uncontrolled output correction?

5 comments
AuthorsOpenReview ↗

Response to Reviewer zpZ1 (1/2)

Dear Reviewer zpZ1,

Thank you very much for taking the time to review our manuscript and for offering many valuable comments that helped us improve the work.

We sincerely appreciate your recognition of the clarity of our presentation, the simplicity of the method, and the substantial early-training gains.

We will address your concerns point by point.


[W1 & Q2] Comparison with Ordinary Auxiliary Supervision

To isolate the benefit of LISA’s residual supervision, we compare it with three auxiliary variants on human pose under the same decoder capacity and training budget, i.e., predicting the ordinary denoising target, reconstructing the input control, and removing the side-network feature from LISA’s decoder.

Auxiliary setting[email protected] (%) ↑FID ↓
Ordinary denoising target73.8563.01
Control reconstruction89.2561.53
Without side-network feature85.5556.79
LISA89.9056.50

These results show that LISA’s gains are not simply due to adding an auxiliary head. Both alternative targets yield lower PCK and higher FID, and removing the side-network feature also hurts performance. These comparisons support both key choices in LISA: the likelihood-score-inspired residual target and its use to directly supervise side-network features.

Thank you for this helpful suggestion. We have added these results to the Ablation Study section of the revised manuscript.


[W2] Interpretation of Side-Network Features as the Likelihood Score

Thank you for raising this point. We qualify our claim below to distinguish the output-level likelihood-score interpretation from the auxiliary supervision of intermediate side-network features.

  1. The likelihood-score interpretation applies to the side network’s overall correction at the output level. As expressed in Eq. (10), the difference between the final conditional prediction and the frozen main-network prediction approximates the likelihood score when both predictions approximate their corresponding data scores. Under this assumption, Eqs. (6), (12), and A19 support the alignment target’s interpretation through its conditional expectation.

  2. This interpretation does not establish a strict correspondence between an individual side feature and the likelihood score. We agree that ControlNet combines feature injections from multiple layers through nonlinear transformations before producing the final prediction. Our trainable decoder provides auxiliary supervision to these features, it does not establish an identifiable likelihood-score component within the main network. The alignment should therefore be understood as a regularizer inspired by the output-level interpretation.

We have revised the Introduction sections to make this distinction clear and state our claims more carefully. Thank you for this helpful suggestion.


[W3 & Q3] Decoded Residual and the Actual Output Correction

We compared the decoder output l~\tilde{l} with Δ=scontrolled−suncontrolled\Delta=s_{\mathrm{controlled}}-s_{\mathrm{uncontrolled}} during 1,000 LISA training steps. Without changing the training objective, we computed the average relative MSE using training samples from each 200-step interval:

$ \mathrm{Relative\ MSE}=\frac{\sum_i|\tilde{l}_i-\Delta_i|_2^2}{\sum_i|\Delta_i|_2^2}. $

Training stepsRelative MSE ↓
1–2000.9824
201–4000.9629
401–6000.9557
601–8000.9510
801–1,0000.9492

The relative MSE decreases steadily from 0.9824 to 0.9492. This small test suggests that the decoded residual can better track the actual output correction as training progresses.


[W4] Clarification of Better Composition and Representation Disentanglement

Our experiment supports better composability under feature summation. We directly compose independently trained pose and segmentation checkpoints with comparable single-condition performance, without additional training. LISA achieves better results after composition, which we report as an additional benefit. We considered representation disentanglement a possible explanation, but agree that the compositional results alone do not establish it. Thank you for pointing this out. We have revised the manuscript to focus our claim on improved composability and present disentanglement only as a hypothesis.

AuthorsOpenReview ↗

Response to Reviewer zpZ1 (2/2)

[W5] Additional Discussion of InnerControl

Thank you for highlighting InnerControl[1]. It demonstrates the value of intermediate-feature feedback for spatial control, and we agree that this closely related work should have been discussed.

Both works aim to improve controllability through auxiliary regularization, with different supervision targets. InnerControl reconstructs the input control from intermediate diffusion features using a pretrained probe that remains frozen during alignment training. LISA jointly trains the side network and an auxiliary decoder to predict the residual between the denoising target and the frozen main-network prediction without control. Our contribution is the likelihood-score perspective and the residual supervision it motivates, which we show accelerates convergence.

We have added InnerControl to the Introduction and Related Work and clarified this shared goal and the differences between the two methods in the revised manuscript.


[Q1] Clarification of Implementation Details

Thank you for pointing out these missing details. We summarize the datasets and evaluation settings below.

TaskDatasetTraining samplesEvaluation samplesMetrics
PoseCustom-built13,2031,154FID, CLIP, [email protected] (OpenPose; bounding-box diagonal)
SegmentationCaptioned ADE20K20,2102,000FID, CLIP, mIoU (Mask2Former-Swin-Large, 150 classes)
DepthCustom-built16,0001,550FID, CLIP, RMSE (MiDaS; each depth map normalized to [0, 1])
Low-resolutionCustom-built40,0002,000FID, PSNR (RGB, [0, 1]), LPIPS (AlexNet)
Pose + segmentationCustom-builtNo additional training1,154FID, CLIP, PCK, mIoU; same evaluators as above
Pose-guided videoUBC Fashion500 videos100 videos, 14 frames eachFVD (I3D-ResNet50), SSIM, LPIPS (AlexNet), [email protected] (OpenPose; bounding-box longest side)

Image conditions are provided with the datasets; video poses are extracted with OpenPose. FID uses CleanFID, and CLIP score uses TorchMetrics with CLIP ViT-B/16. Video LPIPS is averaged over frames and videos and multiplied by 100 for reporting.

We have added these details to the revised manuscript and will release all code and data.


References:

[1] Konovalova, Nina, et al. "InnerControl: Aligning ControlNet with Internal Feature Feedback." Proceedings of the IEEE/CVF Winter Conference on Applications of Computer Vision (WACV), 2027.

AuthorsOpenReview ↗

Follow-up on Author Responses and Revised Manuscript

Dear Reviewer zpZ1,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper.

As the discussion period approaches its end, we would be grateful for the opportunity to address any further concerns you may have before the discussion phase concludes.

We would greatly appreciate any further feedback you may have, as it would help us further improve our work. Thank you again for your time and valuable feedback.

The Authors

Reviewer_6UVmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies the widely adopted dual-branch paradigm for visual-condition controllable generation, where a pretrained diffusion/flow backbone is frozen and a trainable side network encodes visual conditions and injects intermediate features into the main network.

The paper interprets this architecture from a score-based perspective. Specifically, the authors argue that the frozen main network mainly provides an unconditional score, while the side network is implicitly responsible for the residual between the conditional and unconditional scores, which corresponds to a likelihood score under Bayes' rule.

Motivated by this interpretation, the paper proposes LISA (Likelihood Score Alignment), an auxiliary training regularization that explicitly encourages an intermediate feature of the side network to predict an approximated likelihood score. The target is constructed by subtracting the frozen main-network prediction from the standard sample-wise denoising target. A lightweight decoder maps the selected side-network feature into the corresponding prediction space, and the resulting alignment loss is jointly optimized with the standard conditional diffusion objective. The auxiliary decoder is discarded at inference time.

The experiments cover multiple visual conditions, including pose, segmentation, depth, low-resolution images, and pose-conditioned videos, as well as different architectures and formulations including U-Net-based diffusion models, Stable Diffusion 3 with flow matching, and Stable Video Diffusion. The results show that LISA generally improves early-stage convergence and condition-following performance, while introducing no additional inference cost. The paper further reports improved compositional control when independently trained condition branches are combined.

Critical strengths
  1. The paper provides a useful perspective on the dual-branch controllable-generation paradigm. Interpreting conditional control as the residual between a pretrained base score and a conditional score is intuitive and provides a reasonable motivation for directly regularizing the side branch rather than supervising only the final generative prediction.

  2. LISA is simple, lightweight, and practically easy to integrate. The method introduces only a lightweight auxiliary decoder during training and does not modify the inference architecture. This makes the proposed regularization applicable to existing dual-branch methods with little engineering overhead.

  3. The empirical improvements are reasonably consistent across tasks. The paper evaluates pose-, segmentation-, depth-, and low-resolution-conditioned image generation. In particular, LISA provides acceleration during early training, suggesting that the auxiliary supervision facilitates optimization of the condition branch.

  4. The generalization experiments are a strong aspect of the work. The method is evaluated not only with U-Net-based diffusion models but also with Stable Diffusion 3 / flow matching and controllable video generation.

  5. The method incurs zero additional inference cost. Since the auxiliary decoder is removed after training, the final conditional generation architecture remains unchanged, which is appealing for practical deployment.

Critical weaknesses
  1. The theoretical interpretation that the side-network features represent the likelihood score is stronger than what the derivation directly establishes. Equation (10) establishes that the difference between the final conditional prediction and the frozen main-network prediction approximates the likelihood-score term. However, this does not imply that an individual intermediate feature of the side network itself has a uniquely identifiable likelihood-score representation. In common dual-branch architectures such as ControlNet, side-network features are injected at multiple layers and subsequently transformed through nonlinear layers of the frozen backbone. Many different intermediate representations may result in the same final residual prediction. Therefore, aligning one intermediate feature, after an additional trainable decoder, to a likelihood-score target is a reasonable regularizer, but it does not by itself establish that the side branch inherently represents the likelihood score. I therefore find the method well motivated, but the paper should distinguish more carefully between a likelihood-score-inspired auxiliary supervision and a stronger claim that the side network explicitly corresponds to the likelihood score.

  2. The claimed training-efficiency improvement should be reported more clearly in wall-clock terms. Constructing the LISA target requires an additional forward pass through the frozen main network without condition injection. However, Table 3 reports only a relatively small increase in per-iteration training time, from 2.1s to 2.3s. Since the paper emphasizes substantially faster convergence, e.g., the reported 2.78× improvement, it would be important to clarify whether this factor is measured in optimization iterations or actual wall-clock training time. The latter is more relevant because LISA performs additional computation in each iteration. The implementation details of the additional backbone forward pass, including whether gradients are disabled and whether intermediate computation can be reused between the two passes, should also be explained more explicitly.

  3. The evidence for “more disentangled” condition representations is limited. The paper argues that likelihood-score alignment encourages more disentangled side-network representations and supports this claim primarily through the pose + segmentation compositional-generation experiment. While the compositional results are interesting, improved performance after summing two condition branches does not necessarily establish representation disentanglement. It may also result from better individual condition modeling or better-scaled injected features. I suggest weakening the disentanglement claim unless more direct evidence is provided, for example feature-level analyses, cross-condition interference measurements, or controlled comparisons where the individual-condition performance is carefully matched.

  4. The final-quality improvements are sometimes considerably smaller than the early-convergence improvements. One of the most convincing observations is that LISA strongly improves controllability during early training. At later stages, however, some improvements become relatively modest. This raises an interesting question about whether LISA mainly acts as an optimization accelerator or whether it consistently changes the quality of the final converged solution. A clearer separation between faster convergence and better asymptotic performance would make the contribution easier to assess.

Questions for authors
  1. On the likelihood-score interpretation: Equation (10) concerns the residual of the final conditional prediction relative to the frozen main-network prediction. Given that side-network features are injected into multiple nonlinear layers of the main network, why should a particular intermediate feature itself admit a likelihood-score interpretation? Is the intended claim instead that LISA encourages the selected feature to become predictive of the likelihood-score residual through the auxiliary decoder? If so, I think this weaker formulation would be theoretically more precise.

  2. On approximation error of the target: The proposed likelihood target relies on sθ(xt,t)≈∇xtlog⁡pt(xt)s_\theta(x_t,t)\approx\nabla_{x_t}\log p_t(x_t). How sensitive is LISA to inaccuracies in this pretrained score estimate? In particular, if the downstream conditional dataset differs substantially from the distribution used to pretrain the backbone, the residual may contain both condition-related information and model/domain error. Have the authors investigated this setting?

  3. On training-time efficiency: Could the authors clarify how the additional unconditional/base forward pass is implemented? In particular: Is this pass executed under no_grad? Is any computation reused between the two backbone passes? Does Table 3 include the complete additional forward pass? Is the reported “2.78× faster convergence” based on iterations or wall-clock training time? Reporting wall-clock time required to reach the same PCK/mIoU threshold would make the efficiency claim substantially clearer.

  4. On disentanglement versus better condition fitting: In the compositional experiment, how can the authors distinguish improved representation disentanglement from the simpler explanation that LISA learns stronger or better-calibrated individual condition features? A comparison where the vanilla ControlNet and LISA branches are matched at nearly identical single-condition performance before composition would help isolate this effect.

  5. On convergence versus final performance: My main concern of the paper is that does LISA mainly accelerate optimization, or does it consistently improve the converged optimum? The results from the paper seems to support that LISA mainly accelerates optimization while it improves the final performance with a minimal gap shown by the quantitative results from the paper.

5 comments
AuthorsOpenReview ↗

Response to Reviewer 6UVm (1/2)

Dear Reviewer 6UVm,

Thank you very much for taking the time to review our manuscript and for offering many valuable comments that helped us improve the work.

We sincerely appreciate your recognition of the clarity of our presentation, the useful perspective, and our generalized improvement across tasks and backbones with a simple method.

We will address your concerns point by point.


[W1 & Q1] Interpretation of Side-Network Features as the Likelihood Score

Thank you for raising this important point. We qualify our claim below to distinguish the output-level likelihood-score interpretation from the auxiliary supervision of intermediate side-network features.

  1. The side network supplements the frozen main network with a likelihood-score correction at the output level. Its intended role is to bridge the gap between the unconditional and conditional predictions. When these predictions approximate their corresponding scores, the resulting output-level correction approximates the likelihood score, as expressed in Eq. (10). This provides a theoretical interpretation of the side network’s overall contribution.

  2. The side-network features serve as a proxy for auxiliary supervision inspired by this interpretation, without assuming a strict likelihood-score correspondence. We agree that, in dual-branch architectures such as ControlNet, the final correction results from aggregated multiple feature injections. An individual intermediate feature therefore cannot be directly identified with the likelihood score. Our alignment through a trainable decoder is a regularizer inspired by the output-level interpretation, rather than a claim that the intermediate features themselves inherently represent the likelihood score.

We agree that our claim needed to be stated more carefully, and we have revised the Introduction sections to clarify the scope. Thank you for this helpful suggestion, which has made the manuscript clearer and more accurate.


[W2 & Q3] Clarification of Training-time Efficiency

We apologize for the confusion about training time.

  1. Figure 1 reports convergence in training iterations, while Table 3 reports the time per iteration. The 2.78× speedup in Figure 1 comes from reaching the same PCK level in 18K rather than 50K iterations. Accounting for the per-iteration cost, the wall-clock speedup is 2.78 / (2.3 / 2.1) ≈ 2.54×.

    MethodIterations to the same PCKTime per iterationWall-clock training time
    ControlNet50K2.1s29.2h
    + LISA18K2.3s11.5h
  2. LISA applies stop-gradient to the unconditional prediction from the additional main-network forward pass. The main network remains frozen, and this output is used only to construct the alignment target. The 2.3s per iteration reported in both the table above and Table 3 includes this extra forward pass. We do not reuse intermediate computations, as the overhead over the 2.1s baseline is already small. No backward computation is needed for this extra pass, and its overhead is outweighed by the reduction in training iterations.

Thank you for this helpful suggestion. We have clarified the iteration-based and wall-clock speedups in Figure 1 and Section 4.2 (Computational Overhead Analysis), and added details of the extra forward pass to the revised manuscript.

AuthorsOpenReview ↗

Response to Reviewer 6UVm (2/2)

[W3 & Q4] Clarification of the Compositional Experiment Setup

For a fair comparison, we used checkpoints with comparable single-condition performance before composition, as stated in Section 4.4 of the original manuscript. Specifically, the checkpoints and results are listed below.

SettingMethodCheckpoint[email protected] (%) ↑mIoU (%) ↑
Pose onlyControlNet30K89.82—
Pose only+ LISA16K89.40—
Segmentation onlyControlNet30K—34.52
Segmentation only+ LISA15K—34.28
Pose + SegmentationControlNet[30K,30K]89.512.4
Pose + Segmentation+ LISA[16K,15K]90.414.7

We composed these same checkpoints by summing their injected features at each corresponding layer, without additional training. Despite comparable single-condition performance, LISA achieves better results after composition on a test set of 1,154 samples.

Besides, we considered representation disentanglement a possible explanation for this composition improvement, but agree that the results alone do not establish it. Thank you for pointing this out. We have revised the manuscript to focus our claim on improved composability and present disentanglement only as a hypothesis.


[W4 & Q5] Clarification of LISA’s Contributions to Convergence Speed and Final Performance

We trained both ControlNet and LISA until convergence on the human pose task. ControlNet converges at around 50K training iterations, while LISA converges at around 30K. We take their converged results as the final performance, as reported below.

MethodFID ↓CLIP ↑[email protected] (%) ↑
ControlNet60.034330.634789.88
+LISA56.858131.031691.02

These results show that LISA substantially accelerates convergence and can also lead to a better final model.


[Q2] Clarification of Target Approximation Error under Domain Shift

  1. Our likelihood-score interpretation assumes a sufficiently accurate backbone and neglects the domain gap with downstream data. Under this assumption, the side network mainly provides the likelihood-score correction. We agree that, more generally, the residual also includes a correction for the domain gap. We will add this as a limitation in the revised manuscript.

  2. To test our effectiveness under domain shift, we further evaluate LISA on medical images, i.e., Kvasir-SEG[1] for segmentation-conditioned generation. Both methods use the same frozen SD2.1 backbone without medical-domain adaptation.

    MethodTraining iterationsFID ↓mIoU (%) ↑
    ControlNet5K72.6331.79
    +LISA3K70.9233.22
    +LISA5K69.1833.45

These results show that LISA still provides gains on this medical-image task, both with fewer training iterations and at the same training budget. The residual target can still provide useful supervision, as it includes the correction needed for conditional control and compensation for the domain gap.


References:

[1] Jha, Debesh, et al. "Kvasir-seg: A segmented polyp dataset." International conference on multimedia modeling. Cham: Springer International Publishing, 2019.

AuthorsOpenReview ↗

Follow-up on Author Responses and Revised Manuscript

Dear Reviewer 6UVm,

We sincerely appreciate the time and effort you have dedicated to reviewing our paper.

As the discussion period approaches its end, if you have any additional questions or concerns, we will do our best to address them before the discussion phase concludes.

We would greatly appreciate any further feedback you may have, as it would help us further improve our work. Thank you again for your time and valuable feedback.

The Authors

Comments on the paper

2 comments

Dear Reviewers,

Thank you for your thoughtful reviews. The authors have provided detailed responses to the concerns raised.

I would appreciate it if you could revisit the author responses and clarify whether your main concerns have been adequately addressed, particularly which issues remain critical to your overall assessment.

Given the differences in the initial recommendations, I also encourage you to consider and discuss one another's key arguments to help clarify the remaining disagreements and develop a more consistent understanding of the paper's merits and limitations.

Please feel free to update your ratings if your assessments have changed following the discussion.

Thank you for your continued contributions.

AuthorsOpenReview ↗

Author Final Remarks

We sincerely thank all reviewers for their constructive feedback. We appreciate their recognition of our clear presentation [6UVm, zpZ1], simple yet effective faster convergence [6UVm, zpZ1, KrKd], useful perspective [6UVm], zero additional inference cost [6UVm, zpZ1, KrKd], and applicability across tasks and backbones [6UVm].

We have revised our paper according to the comments. The major changes are summarized as follows:

  • According to Reviewer 6UVm’s comments:

    • Clarification.
      • We distinguish the output-level likelihood-score interpretation from the auxiliary supervision of intermediate features.
      • We distinguish iteration-based acceleration from wall-clock savings: the 2.78× iteration speedup on pose corresponds to approximately 2.54× wall-clock speedup.
      • We clarify the composition setup using checkpoints with comparable single-condition performance and limit the claim to improved composability and present disentanglement as a hypothesis.
      • We clarify that LISA’s main contribution is faster convergence, with task-dependent improvements in final performance.
    • Experiment.
      • We provide medical-image results under domain shift.
      • We provide a multi-layer fusion ablation supporting the simpler single-layer alignment design.
  • According to Reviewer zpZ1’s comments:

    • Experiment.
      • We add equal-budget auxiliary-supervision ablations, supporting the benefits of the residual target and direct side-feature supervision.
      • We provide a diagnostic comparison between the decoded residual and the actual output correction, which suggests that the decoded residual can better track the actual output correction as training progresses.
    • Related Work.
      • We discuss InnerControl and clarify the distinction between its control-reconstruction supervision and LISA’s likelihood-score-inspired residual supervision.
    • Reproducibility.
      • We specify datasets, splits, sample counts, condition extractors, metric implementations, and video LPIPS scaling, and commit to releasing code and data.
  • According to Reviewer KrKd’s comments:

    • Clarification.
      • We clarify checkpoint selection and discuss the trade-off between controllability, synthesis fidelity, and text alignment.
      • We clarify the frozen-backbone training setup and loss notation.
      • We clarify the feature summation and evaluation protocol in compositional-condition generation experiments.
    • Presentation.
      • We streamline Section 3, add one missing citation, and discuss limitations under extreme domain shifts.
    • Experiment.
      • We add early equal-step comparisons with REPA and detailed training trajectories across all four image tasks, providing a more comprehensive evaluation beyond selected checkpoints.
    • Analysis.
      • We report an average iteration speedup of 2.06× over ControlNet across five image settings, with a maximum of 2.78×.
      • Final-performance gains are more evident on pose and depth and modest on segmentation and low-resolution generation.
    • Comparison.
      • We highlight that LISA offers competitive early-training performance with REPA using supervision derived from the diffusion task and an existing backbone, without an external pretrained encoder, which is a core paradigm contribution.

Following these responses, Reviewer zpZ1 stated that most concerns had been addressed and retained weak acceptance. Reviewer 6UVm explicitly acknowledged meaningful training-efficiency gains and raised further questions, which we addressed by clarifying our claims and providing additional design ablations. Reviewer KrKd also provided follow-up feedback, and we responded to the remaining concerns point by point.

Many thanks,

The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.