- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper studies instruction-controlled streaming avatar generation. The authors identify two failure modes in this topic: stagnation (the action never starts) & repetition (a completed action is redone). To address these problems: the proposed NextAvatar adds Next-State Planning to a Helios-style Wan2.1-1.3B generator. Then, a frozen Qwen2.5-VL-based planner reads the preceding chunk and the active instruction and predicts ViT-space state tokens.
To evaluate the model, the authors introduce MIRA-Bench (50 reference images & 200 ten-instruction sequences), which is scored by VBench-2.0 / VisionReward / human annotation of stagnation and repetition. Based on the reported numbers, NextAvatar has the best Instruct Follow & VisionReward. Additionally, the ablation studies show that planning raises Instruct Follow but lowers identity.
Critical strengths
[S1]
It is insightful to read the failure modes that the authors point out. More importantly, they are measurable. This weighs heavily for me because the human axis captures what standard metrics miss. Specifically, SoulX-LiveAct has the second-best Instruct Follow yet a 76% combined error rate. Also, SoulX-FlashTalk and Live Avatar have opposite failure types (71% stagnation vs. 53% repetition), which Table 1 cannot reveal. MIRA-Bench is a reusable contribution however the method questions resolve.
[S2]
I like the proposed method which is cheap to train and is modular fix with a large measured effect. NextAvatar trains only small modules for 5k steps on a 1.3B generator. It cuts combined error from 52.5% to 30.5% against the best external system. It also raises Instruct Follow over its own base by about three binomial standard errors. If the attribution questions below are solved, this is a practical recipe for other streaming generators.
Critical weaknesses
[W1]
My major concern is that the evidence does not isolate next-state planning as the source of the gains. This is from multiple perspectives:
- First, stagnation and repetition are measured only in the human study (Table 2), which omits NextAvatar-Base and all ablation variants. The ablations rely instead on Instruct Follow, which can diverge sharply from progression judgments (refer to [S1] above).
- Also, the joint stage adds 5k steps of cross-attention LoRA training, and those key/value projections are shared with the text branch. So the paper cannot rule out that extra fine-tuning or extra visual context, rather than planning, explains the gain.
- I notice that several baselines are audio-driven by design (e.g., SoulX-FlashTalk, Live Avatar, AptAvatar) and received silent audio. Moreover, the benchmark instructions were generated in the style of NextAvatar’s own training captions.
This concerns especially matters to me because the main claim is causal. The current results are equally consistent with gains from base training data, extra fine-tuning, or the evaluation protocol.
[W2]
My other major concern is that the mechanism is asserted rather than shown, and the planner looks structurally unable to resolve repetition after completion.
Specifically, nothing shows that the frozen planner’s tokens differ between not-started and completed histories, or that they match the true next chunk. The released Bernini planner is a Qwen2.5-VL-7B model trained to predict a target clip’s ViT-space semantics, and its public task interface lists only generation and editing tasks. So “next state” is an interpretation of its output, not its training objective.
Also, it seems the inputs are too ambiguous. Once an action has finished and the avatar has rested for a full chunk, the history can look identical to a not-started one as far as the action is concerned. This is exactly the Fig. 1(a) case, and it leaves the planner no reliable basis to choose “remain still” over "initiate".
This concern is also significant to me because the stated mechanism can explain reduced stagnation but not reduced repetition after completion. That is half of the problem the paper targets.
Questions for authors
[Q1]
My first question is about planner's inputs and outputs. Could you please specify the frames and prompt given to the MLLM, the Bernini task mode used, N_p, and the planner’s size? What evidence shows that s_k encodes a future state rather than an instruction-edited version of the input chunk? An edited-input plan would push the generator toward re-executing the action, which bears directly on [W2].
[Q2]
Regarding the human evaluation. How many annotators rated each interval? Were methods blinded and presented in random order? How were stagnation and repetition defined, and are they mutually exclusive? I notice that total Error excludes wrong actions, so motion that is incorrect but not repeated currently scores well.
[Q3]
For each baseline, how were instruction switches delivered every ~6 s? Which runs required adaptation?
[Q4]
A minor question but good to have in the paper: what are the total parameters (the planner and visual prediction module)? What is NextAvatar’s per-chunk latency (50 steps with CFG plus one MLLM call) compared with the real-time baselines? The 1.3B figure in Table 1 omits a planner that, if it is the released Bernini checkpoint, is the 7B MLLM noted in [W2], and it runs on every chunk.
5 comments
[2/3] Baseline protocols, computational cost, and training controls
[Q3 & W1-3]. Instruction switches, audio adaptation, and baseline protocols.
To make the comparison reproducible, we clarified instruction switching and the input adaptations used for each type of baseline.
Instruction switching. At each instruction change, we update the prompt and recompute its text embeddings while retaining visual history and continuing generation. All methods use the same 50 reference images and 200 ten-instruction sequences.
Baseline adaptation. Language-conditioned methods receive the reference image and instruction through their supported interfaces. Audio-dependent methods receive silent audio matching the target duration because MIRA-Bench has no speech input. Each method retains its released or documented chunking and sampling configuration.
Manuscript revisions (blue). Appendix E clarifies instruction updates, history retention, silent-audio inputs, and native chunking and sampling configurations.
[Q4]. Total parameters and per-chunk latency.
To clarify system cost, we added detailed parameter accounting and profiled Base and Full under matched conditions.
Profiling setup. Both variants use eight NVIDIA A800-80GB GPUs, resolution, batch size 1, 50 sampling steps, CFG 5.0, bf16 precision, and seed 42, with compilation disabled. Each video contains 99 frames across three chunks. Wall-clock timings use CUDA synchronization and exclude model loading. State conditioning adds 0.43 s/chunk to denoising, while planning adds 7.97 s per subsequent chunk.
| Component | Steady-state latency |
|---|---|
| Base denoising, 50 steps | 29.35 s/chunk |
| Full denoising, 50 steps | 29.78 s/chunk |
| Full next-state planning | 7.97 s/chunk |
| Full denoising + planning | 37.75 s/chunk |
Denoising throughput is 1.71 sampling iterations/s for Base and 1.68 for Full, while peak memory increases from 25.2 to 44.2 GiB per GPU.
Parameter accounting. The 1.3B value in Table 1 refers to the Wan2.1 generator backbone. The detailed accounting below includes frozen and trainable weights for the listed components, with LoRA weights unmerged.
| Component | Base | Full |
|---|---|---|
| DiT backbone and history modules | 1.4262B | 1.4262B |
| LoRA weights | 0.1785B | 0.1785B |
| State-conditioning modules | — | 0.1046B |
| Planner | — | 9.8149B |
| Total listed components | 1.6047B | 11.5242B |
Manuscript revisions (blue). Appendix C.4 and Tables A2–A3 report matched latency, denoising throughput, memory, and parameter accounting; Table 1 retains the backbone-only convention.
[W1-2]. Additional fine-tuning, shared projections, and extra visual context.
To test whether additional training explains the gains, we added Base (+5K), which continues Base training for 5,000 steps without planning. Instruct Follow decreases from 0.6035 to 0.5945, and the combined stagnation and repetition rate remains 55.5%. Additional Base training alone therefore does not reproduce the improvement.
TextPlan reduces the combined rate to 42.0%, while visual NSP reduces it further to 32.5% without MLLM reference input and 29.0% with reference input. These controls support the usefulness of visual-state conditioning.
The joint stage also updates cross-attention LoRA parameters, including key/value projections shared by the text and state branches.
Manuscript revisions (blue). Section 4.3 reports the additional-training and planning controls in Tables 2 and 4.
[3/3] Evaluation scope, planner training, and mechanism limitations
[W1-4]. Similarity between benchmark instructions and training captions.
To clarify the evaluation scope, we made the instruction-construction procedure explicit. Candidate instructions follow the style of the training descriptions and are checked by humans for plausibility, duplication, duration feasibility, and conflicts.
This supports a controlled evaluation of sequential instruction following. Our conclusions therefore concern the evaluated instruction distribution.
[W1-5]. Strength of the causal claim.
To strengthen attribution, we added the Base (+5K), TextPlan, and human-progression comparisons. Additional Base training does not reproduce the gains, while visual-state conditioning improves progression over both Base and the evaluated text-planning control.
[W2-2]. “Next state” versus the planner’s pretraining objective.
We clarify that ``frozen'' refers to the MLLM's optimization status during joint training. We primarily use its ability to output visual representations, while planning draws on the MLLM's existing capabilities. NextAvatar uses that capability to produce visual-state conditions for the upcoming chunk. Figure 5 shows action-relevant responses associated with these conditions, and Table 4 demonstrates improved action progression. Together, these results support the use of the planner, while finite-history limitations remain, as discussed below.
Manuscript revisions (blue). Section 3.2 describes the planner's role; Appendix C.3 specifies its subsequent freezing, Bernini checkpoint, task modes, and system prompts.
[W2-3]. Ambiguity after completed actions leave the history window.
We expanded the limitations analysis to address this case. Repetition still occurs in our five-chunk instruction setting, as illustrated in Figure A3. Once earlier execution falls outside the one-chunk planner window, a natural resting pose may correspond to either a pending or a completed action. Under an unchanged instruction, this ambiguity can lead the planner to initiate the action again.
Despite this limitation, NextAvatar achieves the lowest combined stagnation and repetition rate among the evaluated external methods. Longer or adaptive history and compact execution memory could help retain completion information beyond the current window.
Manuscript revisions (blue). Appendix F and Figure A3 document residual repetition, the history-window ambiguity, and possible extensions.
[W2-4]. Whether the mechanism also addresses repetition.
To test repetition directly, we added the human-progression ablations in Table 4. Repetition decreases from 29.5% for Base to 24.5% for NSP without reference input, 21.5% for NSP with reference input, and 24.5% for Full.
The proposed explanation applies to both failure modes: insufficient use of execution history can leave a pending action unexecuted or cause a completed action to be repeated. Visual-state conditioning provides explicit guidance for how to continue from the observed history. The measured repetition reductions support this role.
[1/3] Planner implementation and human evaluation
We thank Reviewer dWkr for the detailed questions. We have clarified the planner implementation and evaluation protocols, added controlled ablations, and reported matched Base/Full computational costs. Below, we address the questions and the related concerns.
[Q1 & W2-1]. Planner inputs, outputs, and predicted-state evidence.
To clarify the planner interface and the information represented by its output, we expanded the implementation description and added a predicted-state visualization.
Planner interface. We initialize the planner from Bernini’s Qwen2.5-VL-based MLLM and keep it frozen during NextAvatar joint training. From the second chunk onward, it receives the immediately preceding generated 33-frame chunk and the active instruction. The first chunk is generated by Base without planner conditioning.
Predicted-state evidence. Figure 5 visualizes the absolute change in similarity responses to the predicted-state tokens between the current and history chunks in the MLLM visual-encoder space. Changes concentrate on the hands and arms in hand-raising and thumbs-up examples, with smaller changes on the face and background. This correspondence supports the presence of action-relevant information in the predicted state. Together with the progression ablations, it supports the usefulness of the state condition.
Manuscript revisions (blue). Section 3.2 describes the planner's role and state interface; Appendix C.3 gives the Bernini checkpoint, task modes, and system prompts, while Figure 5 adds the predicted-state analysis.
[Q2 & W1-1]. Human-evaluation protocol and attribution.
To clarify the evaluation and assess gains beyond Base, we expanded the annotation protocol and evaluated all ablation variants on the same 200 selected intervals.
Evaluation protocol. We randomly select one of ten equal-frame intervals from each of the 200 videos, using the same interval across methods. Two annotators independently review each interval, its preceding history, and the assigned instruction. Samples are randomized and method identities are hidden. Each interval receives one of four mutually exclusive labels:
We report stagnation + repetition for the two targeted progression failures and wrong action separately.
Full reduces stagnation from 26.0% to 6.0% and repetition from 29.5% to 24.5% relative to Base, lowering their combined rate from 55.5% to 30.5%. Wrong-action rates remain comparable (27.5% vs. 25.0%). Full has the lowest stagnation rate among the ablations, while NSP with reference input has the lowest repetition and combined rates. These results support a contribution from the proposed modules beyond Base.
Manuscript revisions (blue). Appendix D specifies the annotation protocol, and Table 4 reports the human-progression ablations.