- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents UniOwl, a unified framework for multimodal understanding and image generation with adaptive dual-channel conditioning. The proposed method extracts semantic condition tokens from LLM hidden states and dynamically refines them at each diffusion step using the current denoising features and timestep information. In parallel, it introduces generation-driven visual conditioning, which extracts visual features from intermediate denoising results through a shared visual encoder. These two types of conditions are fused to guide image generation. Additionally, a cycle consistency loss is introduced to encourage semantic alignment between understanding and generation. Experiments show that UniOwl improves image generation performance while maintaining competitive visual understanding capabilities compared with existing unified multimodal models.
Critical strengths
- Effective adaptive semantic conditioning. The proposed method dynamically refines LLM-derived semantic conditions using the current denoising features and timestep information, without repeatedly running the LLM. The ablation results show a substantial improvement in GenEval from 0.67 to 0.81, demonstrating the effectiveness of the proposed conditioning mechanism.
- Strong image generation performance. UniOwl achieves competitive results among compact unified multimodal models, improving GenEval from 0.74 (Mobile-O) to 0.83 while maintaining comparable visual understanding performance (62.3 vs. 62.1). These results demonstrate the practical value of the proposed approach.
Critical weaknesses
-
Limited Novelty. Prior works such as ELLA [1] and DATE [2] have explored timestep-aware and generation-state-dependent conditioning during diffusion. However, the paper neither discusses nor compares with these closely related methods, leaving the novelty and advantages of its adaptive conditioning mechanism unclear.
-
High Computational Cost of Visual Conditioning. As shown in Table 3, visual conditioning only improves GenEval from 0.81 to 0.83, while Table 6 shows a substantial increase in inference latency from 771.9 ms to 2725.0 ms, raising concerns about its cost-effectiveness.
[1] Hu et al., ELLA: Equip Diffusion Models with LLM for Enhanced Semantic Alignment, 2024. https://arxiv.org/abs/2403.05135
[2] Na et al., Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models, 2025. https://arxiv.org/abs/2510.23974
Questions for authors
Please clarify the concerns raised in the weaknesses.
3 comments
Thanks for the authors' response, which solved my main concerns. I still think running VAE decode multiple times at inference time is quite heavy, but the trade-off in Table 7 makes the analysis complete in my opinion, and the discussion with ELLA and DATE help clarify the methodological differences. Regarding the approach is reasonable and empirically effective, I will keep my score of 3.
Dear Reviewer fyfg,
Thank you for your constructive feedback and for acknowledging that our response addressed your main concerns. We agree that repeated VAE decoding remains costly and have discussed this limitation in the revised manuscript.
Response to Reviewer fyfg
Dear Reviewer fyfg,
We thank you for your constructive assessment and helpful suggestions. We appreciate your recognition of the effectiveness of our adaptive semantic conditioning mechanism and its generation performance. We have added experiments and analyses, revised the manuscript, and provided point-by-point responses below.
W1: Novelty of Adaptive Dual-Channel Conditioning
Thank you for suggesting these relevant comparisons. ELLA [1], DATE [2], and our UniOwl adapt diffusion conditioning through different information sources and update mechanisms. We clarify these distinctions and the specific contributions of UniOwl below.
Differences from ELLA and DATE. ELLA extracts timestep-dependent conditions from frozen LLM features through a trained semantic connector. DATE uses the current generation state to optimize text embeddings through gradients of a text-conditioned objective during inference. UniOwl’s Diffusion-Adaptive Semantic Conditioning instead learns residual cross-attention from cached semantic conditions to the current noisy latent and timestep. It therefore uses sample-specific state feedback beyond timestep information and updates conditions by forward computation, without test-time embedding optimization or an external reward evaluator. Our timestep-only control achieves 0.70 on GenEval, compared with 0.81 for semantic-only UniOwl. This supports the value of state feedback within our mechanism; it is not a direct experimental comparison with ELLA or DATE.
Contributions of UniOwl. Our Adaptive Dual-Channel Conditioning combines state-dependent semantic refinement with Generation-Driven Visual Conditioning. The former refines semantic conditions using latent-state feedback. The latter extracts visual feedback from intermediate generations through the shared visual encoder. Cycle Consistency Learning further encourages agreement between understanding outputs for original and generated images. These mechanisms connect dynamic generation conditioning with shared visual representations and understanding–generation consistency. We have discussed ELLA and DATE in the revised manuscript.
W2: Generation Performance and Inference Efficiency
We thank the reviewer for pointing out the computational cost of visual conditioning. We have added GenEval evaluations of UniOwl-Latent and UniOwl-Lite alongside the timing measurements below. Timing measurements cover the instrumented generation-stage GPU operations, including final decoding, and exclude the initial LLM prompt-encoding pass. Latent achieves 0.81 without intermediate-image visual feedback. Lite refreshes visual conditions every four steps and achieves 0.82, compared with 0.83 for Full. Lite reduces the measured generation time by 54.8% relative to Full, with a 0.01 lower GenEval score. Full provides a further gain at substantially higher computational cost because it decodes and visually encodes intermediate latents at every step. These results quantify the performance–efficiency trade-off and show that dense visual feedback is optional. We have included these results and their analysis in the revised manuscript, and clarified that the low-overhead claim applies to semantic refinement. We have also discussed the computational cost of visual feedback in the Limitations section.
Reference
[1] Hu X, Wang R, Fang Y, Fu B, Cheng P, Yu G. Ella: Equip diffusion models with llm for enhanced semantic alignment. arXiv preprint arXiv:2403.05135, 2024.
[2] Na B, Park M, Sim G, Shin D, Bae H, Kang M, Kwon SJ, Kang W, Moon IC. Diffusion Adaptive Text Embedding for Text-to-Image Diffusion Models. Advances in Neural Information Processing Systems. 2025.