acceptodds
Withdrawn from ICLR 2027

TI2V-Adapter: Efficient Adaptation from Text-to-video Models to Text-image-to-video Generation

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Text-image-to-video (TI2V) generation aims to synthesize realistic videos from text prompts and reference images. Existing deep-learning-based TI2V methods either require costly training with large-scale video data or rely on training-free inference-time manipulation of pretrained Text-to-video (T2V) models. While training-intensive methods require time-consuming optimization and substantial computational resources, training-free methods often exhibit limited performance because of the lack of learnable reference-image adaptation. These limitations motivate us to develop a TI2V method with low training cost and improved use of reference-image cues. We therefore propose TI2V-Adapter, an efficient adaptation method that introduces lightweight learnable modules to incorporate reference-image information into pretrained T2V models, enabling T2V-to-TI2V adaptation with minimal training cost. Specifically, TI2V-Adapter consists of a Visual Reference Adapter (VRA) that injects reference-image cues at both the input and feature levels of the pretrained T2V pipeline, and a Video Temporal Controller (VTC) that reinforces image information during denoising. Compared with training-free methods, TI2V-Adapter achieves a 15.3% FVD improvement with only 0.01% of the training samples and 0.83% of the trainable parameters required by training-intensive methods. Code is available in the supplementary material.

Primary area:
generative models
Keywords:
Video generation, Diffusion models, Parameter-efficient learning

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_AmhaOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
3
Soundness
3
Read review
Paper summary

This paper studies text-image-to-video (TI2V) generation, which aims to generate realistic videos conditioned on both text and a reference image. Existing approaches either rely on large-scale training or manipulate pretrained T2V models at inference time without additional training. The former requires expensive optimization, while the latter is limited by the lack of learnable adaptation to the reference image.

To address these limitations, the authors propose TI2V-Adapter, an efficient adaptation framework that introduces lightweight trainable modules to incorporate reference-image information into a pretrained T2V model. Specifically, the method contains two components: (1) a Visual Reference Adapter (VRA), which injects reference-image cues into both the input and feature spaces of the pretrained T2V pipeline, and (2) a Video Temporal Controller (VTC), which reinforces reference-image information during denoising through gated cross-attention.

Compared with training-free methods, TI2V-Adapter improves FVD by 15.3% on the UCF101 and Kinetics-600 datasets, while using only 0.01% of the training data and 0.83% of the trainable parameters required by training-intensive approaches. The paper argues that effective T2V-to-TI2V adaptation requires not only injecting reference information into the backbone, but also preventing this information from fading during denoising.

Critical strengths

None

Critical weaknesses
  • The paper does not sufficiently demonstrate the importance of the proposed adaptation strategy. Given that open-source TI2V models already exist, such as Step-Video-TI2V, which is trained from Step-Video-T2V, I believe an important missing experiment is to apply TI2V-Adapter to Step-Video-T2V and compare it directly with Step-Video-TI2V. If the proposed lightweight adaptation can achieve better or comparable quality with substantially lower training cost, this would provide much stronger evidence for the practical value of the method.
  • The supplementary material lacks video visualizations. Since temporal consistency and motion quality are central to video generation, static frames are insufficient to assess the actual quality of the generated videos.
  • The technical novelty is relatively limited. VRA appears to be largely based on LoRA-style adaptation. For VTC, the motivation is that attention from noisy latents to the reference image decreases during denoising. However, I am not convinced that this decrease itself indicates a problem. It is reasonable that early denoising stages rely more on conditioning information to establish global structure, while later stages focus more on refinement and may naturally require less reference attention. Therefore, the decreasing trend in Figure 3 is not necessarily undesirable. More importantly, the paper does not clearly demonstrate that VTC actually resolves the claimed issue.
  • The dataset appears somewhat outdated and limited in resolution. Given the recent availability of larger and higher-resolution video datasets, it would be useful to evaluate whether the proposed adaptation strategy remains effective when trained on more modern datasets such as OpenVid. This would also help assess the scalability of the method to higher-quality video generation.
Questions for authors
  • Can the authors apply TI2V-Adapter to Step-Video-T2V and compare the resulting model with Step-Video-TI2V in both generation quality and training cost?
  • Can the authors provide video results in the supplementary material so that temporal consistency, motion quality, and reference preservation can be properly evaluated?
  • For Figure 3, what does the attention curve look like after adding VTC? Does VTC actually lead to stronger or more persistent attention to the reference image during denoising?
  • Why should the decreasing attention to the reference image during denoising be regarded as undesirable, rather than a natural behavior of the denoising process? What experimental evidence shows that maintaining stronger reference attention at later timesteps improves generation quality?
  • Have the authors considered training or evaluating on newer, higher-resolution datasets such as OpenVid? It would be helpful to know whether the proposed lightweight adaptation remains effective in a more modern, high-quality data regime.
Reviewer_6U3AOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TI2V-Adapter adapts frozen text-to-video models using a Visual Reference Adapter and a gated Video Temporal Controller. It combines reference-image latent initialization with learned image tokens and low-rank weight updates. Experiments on selected UCF101 and Kinetics-600 classes show FVD improvements with 935 training clips and relatively few trainable parameters.

Critical strengths
  1. The small training-set result is of practical interest. On UCF101, the adapter achieves 404.16 FVD versus 477.19 for TI2V-Zero, using the same ModelScopeT2V backbone.

  2. Testing both a UNet backbone and Vchitect's transformer architecture provides useful evidence that the adaptation approach is not limited to a single implementation.

Critical weaknesses
  1. The evaluation scope is substantially narrower than the general TI2V claim. Evaluation uses nine UCF101 and twelve Kinetics classes, with the first video frame as reference and templated action labels as text. Table 1 also reports substantially weaker Vchitect results than the ModelScope-based variant. I would recommend evaluation on held-out action categories and richer prompts.

  2. FVD/tFVD alone do not directly establish reference preservation or detailed text control. Direct image-consistency/text-alignment measures and uncertainty estimates across training and generation seeds would strengthen the evaluation.

  3. Table 2 includes VRA without VTC, but does not isolate the benefit of adaptive gating from additional capacity. A parameter-matched VRA-only control or a fixed-gate VTC would strengthen this comparison. Reporting parameter counts for each ablation would help assess this. I would recommend comparing these controls at equal data and compute.

  4. The manuscript leaves gate/fusion settings and backbone-specific insertion points underspecified. Sample and parameter counts do not quantify actual training cost. I would recommend reporting GPU-hours and memory, and providing the missing implementation settings. Comparisons with systems trained on ten million samples do not establish the best use of 935 clips.

Questions for authors

Equation 3 mixes image latents into inference noise, while Equations 13-14 use standard forward noising. Is the same fusion applied during training, and at which timesteps? Please also specify the fusion coefficient, gate schedule, frame count, and sampling settings per backbone.

Reviewer_wMSsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

The paper addressed the problem of converting a pre-trained T2V model into a TI2V model. It proposes an adaptation framework that injects the image as a CLIP embedding and VAE noise as extra conditions along with the text, where CLIP and VAE are pre-trained, and there is a trainable projection. Besides the trainable projection, a Video Temporal Controller is also introduced, which is a learnable cross-attention module that takes the image CLIP embedding as the query to condition each of the T2V denoising blocks during decoding. The evaluation is on UCF101 and Kinetics-600 using different pretrained T2V backbones and shows improvements in FVD and tFVD over the corresponding pretrained models. The paper also provides ablation studies on the individual components, encoder configurations, training data size, and adaptation rank.

Critical strengths
  • The paper addresses a practically relevant problem: efficiently adapting pretrained T2V models to TI2V generation without expensive full-model retraining. This is a meaningful direction given the computational cost of training video generation models.

  • The proposed method is sound and simple

Critical weaknesses
  • Main concern 1 - Empirical contribution: The paper would be a sound contribution if it provided a systematic report of how to tackle efficient adaptation from T2V to TI2V, but the current layout of experiments feels more like selling the problem itself of the T2V-to-Ti2V adaptation task. The experiments compare only the proposed architecture to fully trained TI2V or training-free models, while many straightforward adaptation techniques, such as LoRA and full fine-tuning, are missing from the comparison. The efficiency-performance trade-off of the proposed method is hard to judge.

  • Main concern 2 - Technical contribution: The overall technical contribution is limited. The image conditioning with CLIP and VAE are common and does not have much novelty or insight. The finding of temporal degradation on image conditioning and VTC might add some novelty, but not much insight into this aspect, as it's unclear whether VTC solves the claimed temporal degradation problem or is it simply adding more image-conditioning capacity.

Questions for authors

In the pretrained T2V backbone, how frequently are text-conditioning features injected, and how does this differ from the image-conditioning mechanism introduced by VRA? Since VRA already concatenates image and text tokens, why is additional image-specific cross-attention through VTC necessary? Does VTC address actual information degradation across denoising timesteps, or does it simply increase image-conditioning capacity?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.