- Rating
- 4(+1)
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper proposes TempAct, an RL based framework to jointly learn how to decompose a long global instruction into chunk-level input prompts for executable temporal steps, and how to make the modern ar distilled video diffusion (e.g. longlive, sf) act correctly with those temporal instructions. The authors design careful ablation study to verify the effectiveness of each components in the joint RL framework. Finally, quantitative and qualitative experiments across self forcing and longlive codebase demonstrate the effectiveness of the approach.
Critical strengths
- The problem and motivation is good and reasonable. Most modern bidirectional to ar model either use global prompts or directly combining chunk-wise prompt without tailored design improvement, making one of the important purpose of ar-diffusion (interactive rolling) less explored.
- The design is neat and the validation is clean. The planner and executor design is a natural and comfortable extension of the motivation. And the effectiveness of each component is validated by ablation by the authors.
Critical weaknesses
- "Distinction between TempAct and previous work might need further discussion". KVPO [1] shares similar idea on the executor side whereas VideoDirectorGPT [2] shares the idea of planner. Longlive also create a baseline for the current setting, which makes the distinction of TempAct to previous work relatively weak.
[1] KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
[2] VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
Questions for authors
-
Is the planner reward limited effectiveness (seen through Table 2 and figs on the right) caused by the small planner model (1.3B qwen) or the method itself, since I observe that most of the gain is from the executor.
-
Could the author test on the setting with extended rollout duration, since current 141 frames is not deviated much from the training horizon, and might be far from what a user want during inference, that could also test the framework's generalization capacity to OOD domain.
-
Could the author provide more discussion regarding previous work like KVPO [1] and VideoDirectorGPT [2] ?
[1] KVPO: ODE-Native GRPO for Autoregressive Video Alignment via KV Semantic Exploration
[2] VideoDirectorGPT: Consistent Multi-scene Video Generation via LLM-Guided Planning
3 comments
Thank you for response, since llm model scale does not influence much, the bottleneck is the DiT. I learn something useful and will raise the score.
Dear Reviewer,
Thank you again for your timely and constructive review. Following your recommendation, we have incorporated the explanations from our rebuttal into the main text of the revised manuscript. We are grateful for the time and care you devoted to our paper.
Response to Reviewer wept
Thank you for the constructive feedback. We address each of your suggestions below.
W1 / Q3. Distinction from previous work.
A1: We are sorry that the introduction did not end with a summary of our contributions, which made the novelty unclear. Our method is fundamentally different from VideoDirectorGPT and KVPO. The contributions are:
KVPO extends the policy decision to the KV-cache router and proposes a reward-weighted contrastive flow-matching objective, using the VideoAlign reward to improve general video-generation properties such as motion quality, visual quality, and text alignment. Both its planning and execution rely on the diffusion model; it does not explore hierarchical sampling or multi-granularity reward credit assignment, nor does it focus on temporal consistency.
We add a comparison with KVPO. Since it uses VideoAlign's general video-generation reward, for a fair comparison on temporal generation, we instead use Qwen3-VL-8B for the Local Step-Following Score and the Temporal-Following Score, together with the same aesthetic reward and weights as TempAct. The planner is kept at initialization and frozen. The experiment is conducted on LongLive, and only the first chunk is trained. The results are shown in the table below:
As expected, KVPO improves temporal plausibility by enhancing the executor and the cache-router scheduling. However, the gain is limited. On one hand, the router's language scheduling ability is very weak and cannot match planner-level semantic scheduling, resulting in a lack of collaboration between the planner and the executor. On the other hand, both the global reward (semantic planning consistency) and the local reward (action execution consistency) are optimized through diffusion, mixing the two credits and reducing optimization efficiency. We will add this result to the final version.
VideoDirectorGPT only switches the prompt at scene timestamps, with a frozen GPT-4 writing the scene text and layout for a fixed generator. T2I-R1 is joint RL for one autoregressive image. We introduce time into streaming video, so planning and execution are coupled differently. LongLive's KV-recache is an inference method for smoother prompt switches. It is orthogonal to our RL. Table 1 shows that TempAct still improves LongLive.
Q1. Is the limited planner gain caused by the small planner or by the method?
A2: Our planner is Qwen3-1.7B. Replacing it with a frozen GPT planner, while keeping Self-Forcing as the executor, only moves the Qwen scores from 0.414 / 0.269 / 0.342 to 0.421 / 0.286 / 0.354:
The gain is limited. A strong planner can produce a good plan, but the bottleneck is the diffusion model, which cannot effectively execute that plan. We will add this result to Table 2.
Q2. Extended rollout duration.
A3: Following your suggestion, we provide a 30s evaluation on LongLive, using the 100 Hard-Set samples and Gemini. Baseline: 0.529. TempAct: 0.562. As the video gets longer, the VLM's frame sampling rate and accuracy drop, and real-time models are still weak at complex temporal logic. The main evaluation in this paper is therefore at 15s.