- Rating
- 1
- Confidence
- 3
- Informativeness
- 1
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper studies text-image-to-video (TI2V) generation, which aims to generate realistic videos conditioned on both text and a reference image. Existing approaches either rely on large-scale training or manipulate pretrained T2V models at inference time without additional training. The former requires expensive optimization, while the latter is limited by the lack of learnable adaptation to the reference image.
To address these limitations, the authors propose TI2V-Adapter, an efficient adaptation framework that introduces lightweight trainable modules to incorporate reference-image information into a pretrained T2V model. Specifically, the method contains two components: (1) a Visual Reference Adapter (VRA), which injects reference-image cues into both the input and feature spaces of the pretrained T2V pipeline, and (2) a Video Temporal Controller (VTC), which reinforces reference-image information during denoising through gated cross-attention.
Compared with training-free methods, TI2V-Adapter improves FVD by 15.3% on the UCF101 and Kinetics-600 datasets, while using only 0.01% of the training data and 0.83% of the trainable parameters required by training-intensive approaches. The paper argues that effective T2V-to-TI2V adaptation requires not only injecting reference information into the backbone, but also preventing this information from fading during denoising.
Critical strengths
None
Critical weaknesses
- The paper does not sufficiently demonstrate the importance of the proposed adaptation strategy. Given that open-source TI2V models already exist, such as Step-Video-TI2V, which is trained from Step-Video-T2V, I believe an important missing experiment is to apply TI2V-Adapter to Step-Video-T2V and compare it directly with Step-Video-TI2V. If the proposed lightweight adaptation can achieve better or comparable quality with substantially lower training cost, this would provide much stronger evidence for the practical value of the method.
- The supplementary material lacks video visualizations. Since temporal consistency and motion quality are central to video generation, static frames are insufficient to assess the actual quality of the generated videos.
- The technical novelty is relatively limited. VRA appears to be largely based on LoRA-style adaptation. For VTC, the motivation is that attention from noisy latents to the reference image decreases during denoising. However, I am not convinced that this decrease itself indicates a problem. It is reasonable that early denoising stages rely more on conditioning information to establish global structure, while later stages focus more on refinement and may naturally require less reference attention. Therefore, the decreasing trend in Figure 3 is not necessarily undesirable. More importantly, the paper does not clearly demonstrate that VTC actually resolves the claimed issue.
- The dataset appears somewhat outdated and limited in resolution. Given the recent availability of larger and higher-resolution video datasets, it would be useful to evaluate whether the proposed adaptation strategy remains effective when trained on more modern datasets such as OpenVid. This would also help assess the scalability of the method to higher-quality video generation.
Questions for authors
- Can the authors apply TI2V-Adapter to Step-Video-T2V and compare the resulting model with Step-Video-TI2V in both generation quality and training cost?
- Can the authors provide video results in the supplementary material so that temporal consistency, motion quality, and reference preservation can be properly evaluated?
- For Figure 3, what does the attention curve look like after adding VTC? Does VTC actually lead to stronger or more persistent attention to the reference image during denoising?
- Why should the decreasing attention to the reference image during denoising be regarded as undesirable, rather than a natural behavior of the denoising process? What experimental evidence shows that maintaining stronger reference attention at later timesteps improves generation quality?
- Have the authors considered training or evaluating on newer, higher-resolution datasets such as OpenVid? It would be helpful to know whether the proposed lightweight adaptation remains effective in a more modern, high-quality data regime.