acceptodds
Under review as a conference paper at ICLR 2027

NextAvatar: Next-State Planning for Streaming Avatar Video Generation

Abstract

Streaming avatar generation synthesizes video chunk by chunk, conditioned on visual history and text instructions. Each chunk typically covers about one second, often too short to complete an action, so a single instruction spans multiple chunks during training and inference. Coherent action progression therefore requires the generator to account for what the avatar has already done. However, we observe that streaming generators often stagnate before executing an action or repeat it after completion. To investigate these failures, we analyze the attention mechanisms that incorporate visual history and text instructions into generation. We identify history-state confusion: attention to both conditions remains largely unchanged across distinct execution states under the same instruction, failing to adequately distinguish action progress and leading to stagnation and repetition. To address this problem, we propose NextAvatar. We use a frozen pretrained multimodal large language model to interpret the history state and active instruction, then provide an explicit next-state visual plan through state-conditioned attention. To reduce appearance deviations in the planned state, we introduce Reference-Grounded State Anchoring, which refines its visual features through asymmetric attention to fixed initial-reference features. We further introduce the Multi-Instruction Responsive Avatar Benchmark (MIRA-Bench), comprising 2,000 expression–action intervals to evaluate Human Fidelity, Instruction Responsiveness, Video Quality, and Action Progression. With a 1.3B generator backbone, NextAvatar achieves the strongest Instruction Responsiveness among the evaluated methods and reduces combined stagnation and repetition from the best external baseline's 52.5% to 30.5%, while maintaining competitive Human Fidelity and Video Quality. A demo page is available at https://anonymous.4open.science/w/nextavatar_site/.

Primary area:
generative models
Keywords:
Streaming avatar video generation, video diffusion transformers

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_dWkrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies instruction-controlled streaming avatar generation. The authors identify two failure modes in this topic: stagnation (the action never starts) & repetition (a completed action is redone). To address these problems: the proposed NextAvatar adds Next-State Planning to a Helios-style Wan2.1-1.3B generator. Then, a frozen Qwen2.5-VL-based planner reads the preceding chunk and the active instruction and predicts ViT-space state tokens.

To evaluate the model, the authors introduce MIRA-Bench (50 reference images & 200 ten-instruction sequences), which is scored by VBench-2.0 / VisionReward / human annotation of stagnation and repetition. Based on the reported numbers, NextAvatar has the best Instruct Follow & VisionReward. Additionally, the ablation studies show that planning raises Instruct Follow but lowers identity.

Critical strengths

[S1]

It is insightful to read the failure modes that the authors point out. More importantly, they are measurable. This weighs heavily for me because the human axis captures what standard metrics miss. Specifically, SoulX-LiveAct has the second-best Instruct Follow yet a 76% combined error rate. Also, SoulX-FlashTalk and Live Avatar have opposite failure types (71% stagnation vs. 53% repetition), which Table 1 cannot reveal. MIRA-Bench is a reusable contribution however the method questions resolve.

[S2]

I like the proposed method which is cheap to train and is modular fix with a large measured effect. NextAvatar trains only small modules for 5k steps on a 1.3B generator. It cuts combined error from 52.5% to 30.5% against the best external system. It also raises Instruct Follow over its own base by about three binomial standard errors. If the attribution questions below are solved, this is a practical recipe for other streaming generators.

Critical weaknesses

[W1]

My major concern is that the evidence does not isolate next-state planning as the source of the gains. This is from multiple perspectives:

  • First, stagnation and repetition are measured only in the human study (Table 2), which omits NextAvatar-Base and all ablation variants. The ablations rely instead on Instruct Follow, which can diverge sharply from progression judgments (refer to [S1] above).
  • Also, the joint stage adds 5k steps of cross-attention LoRA training, and those key/value projections are shared with the text branch. So the paper cannot rule out that extra fine-tuning or extra visual context, rather than planning, explains the gain.
  • I notice that several baselines are audio-driven by design (e.g., SoulX-FlashTalk, Live Avatar, AptAvatar) and received silent audio. Moreover, the benchmark instructions were generated in the style of NextAvatar’s own training captions.

This concerns especially matters to me because the main claim is causal. The current results are equally consistent with gains from base training data, extra fine-tuning, or the evaluation protocol.

[W2]

My other major concern is that the mechanism is asserted rather than shown, and the planner looks structurally unable to resolve repetition after completion.

Specifically, nothing shows that the frozen planner’s tokens differ between not-started and completed histories, or that they match the true next chunk. The released Bernini planner is a Qwen2.5-VL-7B model trained to predict a target clip’s ViT-space semantics, and its public task interface lists only generation and editing tasks. So “next state” is an interpretation of its output, not its training objective.

Also, it seems the inputs are too ambiguous. Once an action has finished and the avatar has rested for a full chunk, the history can look identical to a not-started one as far as the action is concerned. This is exactly the Fig. 1(a) case, and it leaves the planner no reliable basis to choose “remain still” over "initiate".

This concern is also significant to me because the stated mechanism can explain reduced stagnation but not reduced repetition after completion. That is half of the problem the paper targets.

Questions for authors

[Q1]

My first question is about planner's inputs and outputs. Could you please specify the frames and prompt given to the MLLM, the Bernini task mode used, N_p, and the planner’s size? What evidence shows that s_k encodes a future state rather than an instruction-edited version of the input chunk? An edited-input plan would push the generator toward re-executing the action, which bears directly on [W2].

[Q2]

Regarding the human evaluation. How many annotators rated each interval? Were methods blinded and presented in random order? How were stagnation and repetition defined, and are they mutually exclusive? I notice that total Error excludes wrong actions, so motion that is incorrect but not repeated currently scores well.

[Q3]

For each baseline, how were instruction switches delivered every ~6 s? Which runs required adaptation?

[Q4]

A minor question but good to have in the paper: what are the total parameters (the planner and visual prediction module)? What is NextAvatar’s per-chunk latency (50 steps with CFG plus one MLLM call) compared with the real-time baselines? The 1.3B figure in Table 1 omits a planner that, if it is the released Bernini checkpoint, is the 7B MLLM noted in [W2], and it runs on every chunk.

5 comments
AuthorsOpenReview ↗

[1/3] Planner implementation and human evaluation

We thank Reviewer dWkr for the detailed questions. We have clarified the planner implementation and evaluation protocols, added controlled ablations, and reported matched Base/Full computational costs. Below, we address the questions and the related concerns.

[Q1 & W2-1]. Planner inputs, outputs, and predicted-state evidence.

To clarify the planner interface and the information represented by its output, we expanded the implementation description and added a predicted-state visualization.

Planner interface. We initialize the planner from Bernini’s Qwen2.5-VL-based MLLM and keep it frozen during NextAvatar joint training. From the second chunk onward, it receives the immediately preceding generated 33-frame chunk and the active instruction. The first chunk is generated by Base without planner conditioning.

Implementation detailSpecification
History inputImmediately preceding generated 33-frame chunk
Text inputCurrent active instruction
Task modesV2V
System prompts"You are a video continuation planning assistant. Based on the previous video clip, together with the text instruction provided afterward, plan the next video clip. Keep the characters and the scene continuous, and prioritize following the text instruction. Output only the continuation plan."
Planner trainingFrozen during joint training
Plan representationUp to 256 visual tokens.

Predicted-state evidence. Figure 5 visualizes the absolute change in similarity responses to the predicted-state tokens between the current and history chunks in the MLLM visual-encoder space. Changes concentrate on the hands and arms in hand-raising and thumbs-up examples, with smaller changes on the face and background. This correspondence supports the presence of action-relevant information in the predicted state. Together with the progression ablations, it supports the usefulness of the state condition.

Manuscript revisions (blue). Section 3.2 describes the planner's role and state interface; Appendix C.3 gives the Bernini checkpoint, task modes, and system prompts, while Figure 5 adds the predicted-state analysis.

[Q2 & W1-1]. Human-evaluation protocol and attribution.

To clarify the evaluation and assess gains beyond Base, we expanded the annotation protocol and evaluated all ablation variants on the same 200 selected intervals.

Evaluation protocol. We randomly select one of ten equal-frame intervals from each of the 200 videos, using the same interval across methods. Two annotators independently review each interval, its preceding history, and the assigned instruction. Samples are randomized and method identities are hidden. Each interval receives one of four mutually exclusive labels:

CategoryDefinition
CorrectAppropriate execution or continuation without the failures below.
StagnationFailure to initiate or advance the requested action.
RepetitionUnnecessary repetition of an already completed action.
Wrong actionAn action inconsistent with the instruction.

We report stagnation + repetition for the two targeted progression failures and wrong action separately.

VariantRepetitionStagnationWrong actionStagnation + repetition
Base29.5%26.0%27.5%55.5%
Base (+5K)29.0%26.5%23.0%55.5%
Base + TextPlan25.5%16.5%23.0%42.0%
Base + NSP (w/o R)24.5%8.0%28.0%32.5%
Base + NSP (w/ R)21.5%7.5%26.0%29.0%
NextAvatar (Full)24.5%6.0%25.0%30.5%

Full reduces stagnation from 26.0% to 6.0% and repetition from 29.5% to 24.5% relative to Base, lowering their combined rate from 55.5% to 30.5%. Wrong-action rates remain comparable (27.5% vs. 25.0%). Full has the lowest stagnation rate among the ablations, while NSP with reference input has the lowest repetition and combined rates. These results support a contribution from the proposed modules beyond Base.

Manuscript revisions (blue). Appendix D specifies the annotation protocol, and Table 4 reports the human-progression ablations.

AuthorsOpenReview ↗

[2/3] Baseline protocols, computational cost, and training controls

[Q3 & W1-3]. Instruction switches, audio adaptation, and baseline protocols.

To make the comparison reproducible, we clarified instruction switching and the input adaptations used for each type of baseline.

Instruction switching. At each instruction change, we update the prompt and recompute its text embeddings while retaining visual history and continuing generation. All methods use the same 50 reference images and 200 ten-instruction sequences.

Baseline adaptation. Language-conditioned methods receive the reference image and instruction through their supported interfaces. Audio-dependent methods receive silent audio matching the target duration because MIRA-Bench has no speech input. Each method retains its released or documented chunking and sampling configuration.

Manuscript revisions (blue). Appendix E clarifies instruction updates, history retention, silent-audio inputs, and native chunking and sampling configurations.

[Q4]. Total parameters and per-chunk latency.

To clarify system cost, we added detailed parameter accounting and profiled Base and Full under matched conditions.

Profiling setup. Both variants use eight NVIDIA A800-80GB GPUs, 960×576960\times576 resolution, batch size 1, 50 sampling steps, CFG 5.0, bf16 precision, and seed 42, with compilation disabled. Each video contains 99 frames across three chunks. Wall-clock timings use CUDA synchronization and exclude model loading. State conditioning adds 0.43 s/chunk to denoising, while planning adds 7.97 s per subsequent chunk.

ComponentSteady-state latency
Base denoising, 50 steps29.35 s/chunk
Full denoising, 50 steps29.78 s/chunk
Full next-state planning7.97 s/chunk
Full denoising + planning37.75 s/chunk

Denoising throughput is 1.71 sampling iterations/s for Base and 1.68 for Full, while peak memory increases from 25.2 to 44.2 GiB per GPU.

Parameter accounting. The 1.3B value in Table 1 refers to the Wan2.1 generator backbone. The detailed accounting below includes frozen and trainable weights for the listed components, with LoRA weights unmerged.

ComponentBaseFull
DiT backbone and history modules1.4262B1.4262B
LoRA weights0.1785B0.1785B
State-conditioning modules—0.1046B
Planner—9.8149B
Total listed components1.6047B11.5242B

Manuscript revisions (blue). Appendix C.4 and Tables A2–A3 report matched latency, denoising throughput, memory, and parameter accounting; Table 1 retains the backbone-only convention.

[W1-2]. Additional fine-tuning, shared projections, and extra visual context.

To test whether additional training explains the gains, we added Base (+5K), which continues Base training for 5,000 steps without planning. Instruct Follow decreases from 0.6035 to 0.5945, and the combined stagnation and repetition rate remains 55.5%. Additional Base training alone therefore does not reproduce the improvement.

TextPlan reduces the combined rate to 42.0%, while visual NSP reduces it further to 32.5% without MLLM reference input and 29.0% with reference input. These controls support the usefulness of visual-state conditioning.

The joint stage also updates cross-attention LoRA parameters, including key/value projections shared by the text and state branches.

Manuscript revisions (blue). Section 4.3 reports the additional-training and planning controls in Tables 2 and 4.

AuthorsOpenReview ↗

[3/3] Evaluation scope, planner training, and mechanism limitations

[W1-4]. Similarity between benchmark instructions and training captions.

To clarify the evaluation scope, we made the instruction-construction procedure explicit. Candidate instructions follow the style of the training descriptions and are checked by humans for plausibility, duplication, duration feasibility, and conflicts.

This supports a controlled evaluation of sequential instruction following. Our conclusions therefore concern the evaluated instruction distribution.

[W1-5]. Strength of the causal claim.

To strengthen attribution, we added the Base (+5K), TextPlan, and human-progression comparisons. Additional Base training does not reproduce the gains, while visual-state conditioning improves progression over both Base and the evaluated text-planning control.

[W2-2]. “Next state” versus the planner’s pretraining objective.

We clarify that ``frozen'' refers to the MLLM's optimization status during joint training. We primarily use its ability to output visual representations, while planning draws on the MLLM's existing capabilities. NextAvatar uses that capability to produce visual-state conditions for the upcoming chunk. Figure 5 shows action-relevant responses associated with these conditions, and Table 4 demonstrates improved action progression. Together, these results support the use of the planner, while finite-history limitations remain, as discussed below.

Manuscript revisions (blue). Section 3.2 describes the planner's role; Appendix C.3 specifies its subsequent freezing, Bernini checkpoint, task modes, and system prompts.

[W2-3]. Ambiguity after completed actions leave the history window.

We expanded the limitations analysis to address this case. Repetition still occurs in our five-chunk instruction setting, as illustrated in Figure A3. Once earlier execution falls outside the one-chunk planner window, a natural resting pose may correspond to either a pending or a completed action. Under an unchanged instruction, this ambiguity can lead the planner to initiate the action again.

Despite this limitation, NextAvatar achieves the lowest combined stagnation and repetition rate among the evaluated external methods. Longer or adaptive history and compact execution memory could help retain completion information beyond the current window.

Manuscript revisions (blue). Appendix F and Figure A3 document residual repetition, the history-window ambiguity, and possible extensions.

[W2-4]. Whether the mechanism also addresses repetition.

To test repetition directly, we added the human-progression ablations in Table 4. Repetition decreases from 29.5% for Base to 24.5% for NSP without reference input, 21.5% for NSP with reference input, and 24.5% for Full.

The proposed explanation applies to both failure modes: insufficient use of execution history can leave a pending action unexecuted or cause a completed action to be repeated. Visual-state conditioning provides explicit guidance for how to continue from the observed history. The measured repetition reductions support this role.

Reviewer_k1eROpenReview ↗
Rating
2
Confidence
3
Informativeness
1
Presentation
2
Soundness
2
Read review
Paper summary

NextAvatar addresses action stagnation and repetition in streaming avatar generation. It uses a frozen multimodal language model to predict visual state features from recent video history and the current instruction. These features are refined using the initial reference image to preserve identity and injected into the video generator as additional conditions. The paper also introduces MIRA-Bench and reports improvements in instruction following and action progression.

Critical strengths

None

Critical weaknesses

None

Questions for authors
  1. Could the authors clarify how NextAvatar differs from, and what advantages it offers over, a simpler alternative that uses an external VLM to inspect the generated video history and dynamically update the text prompt for each subsequent chunk? Such an approach could, in principle, also track action progress and guide the continuation at the chunk level. A controlled comparison using the same video generator and history context, with comparable fine-tuning budgets, would help establish the additional benefits of the proposed visual state representation and conditioning mechanism beyond those of explicit history-based planning.
  2. Could the authors provide examples of more complex interactions with objects and the environment, such as walking to a table, picking up a cup, taking a sip, and putting it back? How reliably can NextAvatar track action progress and determine completion when earlier steps fall outside the planner’s history window? It remains unclear whether the NSP module can fully understand complex task requirements and the dependencies between subtasks.
  3. How much inference overhead does the MLLM planner introduce? Please report the full model size, including the planner, and compare end-to-end latency and throughput with NextAvatar-Base under matched settings. Without these measurements, the practical cost of improved action control in streaming applications remains unclear.
4 comments
AuthorsOpenReview ↗

[1/2] Text-planning comparison

We thank Reviewer k1eR for the questions about text planning, long-horizon scope, and inference cost. We have added a text-planning control, clarified the history-window limitations and evaluation scope, and measured matched Base/Full computational costs.

[Q1]. External VLM that dynamically updates the text prompt.

To evaluate this alternative, we added a Base + TextPlan control. It uses the original Qwen2.5-VL-7B checkpoint to generate a textual action plan for each chunk and appends it to the active instruction, while retaining the base generator’s reference and history conditions. We use the original checkpoint because the MLLM used in NSP no longer directly generates textual plans after visual-state training. We also included Base (+5K) to test whether additional training alone explains the gains.

VariantHuman IdentityInstruct FollowVision RewardAesthetic QualityImaging Quality
Base0.93880.60350.93050.56340.7399
Base (+5K)0.95050.59450.93770.54040.7299
Base + TextPlan0.88370.63750.93290.54320.7308
NextAvatar (Full)0.93230.64950.93970.57000.7427

Additional training alone does not improve instruction following. TextPlan improves Instruct Follow over Base but falls below NextAvatar (0.6375 vs. 0.6495). This may be because it generates similar action descriptions across chunks without specifying action progress. TextPlan also substantially reduces Human Identity (0.9388 → 0.8837). NextAvatar preserves identity better than TextPlan and achieves higher Vision Reward, aesthetic quality, and imaging quality, although its identity score remains slightly below Base.

We also evaluated these variants under the same human action-progression protocol.

VariantRepetitionStagnationWrong actionStagnation + repetition
Base29.5%26.0%27.5%55.5%
Base (+5K)29.0%26.5%23.0%55.5%
Base + TextPlan25.5%16.5%23.0%42.0%
NextAvatar (Full)24.5%6.0%25.0%30.5%

Compared with TextPlan, NextAvatar reduces stagnation from 16.5% to 6.0% and repetition from 25.5% to 24.5%, lowering their combined rate from 42.0% to 30.5%. The wrong-action rate is slightly higher (23.0% → 25.0%). These results show that our visual-planning framework improves action progression and preserves identity better than the evaluated text-planning alternative.

Manuscript revisions (blue). Section 4.3 includes the TextPlan implementation, checkpoint rationale, and comparisons in Tables 2 and 4.

AuthorsOpenReview ↗

[2/2] History limitations, task scope, and computational cost

[Q2]. Complex object/environment interactions and history beyond the planner window.

History-window choice and limitations. To address this concern, we expanded the limitations analysis to explain the history-window choice, its failure cases, and the evaluation scope. The planner uses the most recent generated chunk to balance history information against inference cost. In our avatar setting, most expressions and body actions span two to three chunks (approximately two to four seconds), so recent history often provides sufficient cues for local continuation.

We discussed this limitation in Appendix F of the original submission. NextAvatar still exhibits repetition when an instruction spans five chunks (6.875 seconds), as illustrated in Figure A3. Once earlier execution falls outside the planner’s history window, a natural resting pose may correspond to either a pending or a completed action, leading to repeated execution. Despite this limitation, NextAvatar achieves the lowest combined stagnation and repetition rate among the evaluated state-of-the-art methods under the same setting.

Environment interaction. NextAvatar focuses on local facial and body actions in streaming avatar generation. MIRA-Bench excludes object interactions and lower-body actions. Tasks such as walking to a table, picking up a cup, drinking, and replacing it additionally require tracking the environment, object states, and subject–object interactions. These capabilities are beyond our current evaluation.

Complex task planning. Our focus is how an avatar continues a given action across chunks, particularly whether it initiates the action or repeats it after completion. Task decomposition and dependency management are complementary problems that could be handled by an upper-level planner. For example, as shown in Appendix B, TempAct converts a complex prompt into textual subgoals with temporal spans, whereas NextAvatar predicts a history-conditioned visual state for each chunk under a persistent action instruction. Combining these planning levels, together with longer or adaptive history and compact execution memory, is a promising extension.

Manuscript revisions (blue). Appendix F and Figure A3 clarify the history limitation and evaluation scope; Appendix B distinguishes TempAct’s task-level textual planning from NextAvatar’s chunk-level visual-state planning.

[Q3]. Planner overhead, full model size, latency, and throughput.

To clarify system cost, we added matched Base/Full profiling and detailed parameter accounting. The 1.3B value in the main comparison table denotes the Wan2.1 generator backbone. NextAvatar additionally includes a frozen Qwen2.5-VL-based planner and visual-state prediction and conditioning modules. Planning starts from the second chunk; the first chunk is generated by Base.

Matched profiling. We evaluate both variants on eight NVIDIA A800-80GB GPUs at 960×576960\times576 resolution, batch size 1, 50 sampling steps, CFG 5.0, bf16 precision, and seed 42, with compilation disabled. Each video contains 99 frames across three chunks. Wall-clock timings use CUDA synchronization and exclude model loading. We average two videos after one warm-up video and exclude the first chunk from chunk-level timings. State conditioning adds only 0.43 s/chunk to denoising, while planning adds 7.97 s per subsequent chunk.

ComponentSteady-state latency
Base denoising, 50 steps29.35 s/chunk
Full denoising, 50 steps29.78 s/chunk
Full next-state planning7.97 s/chunk
Full denoising + planning37.75 s/chunk

Denoising throughput is 1.71 sampling iterations/s for Base and 1.68 for Full. Peak memory increases from 25.2 to 44.2 GiB per GPU.

Parameter accounting. The following counts include frozen and trainable weights for the listed components, with LoRA weights unmerged.

ComponentBaseFull
DiT backbone and history modules1.4262B1.4262B
LoRA weights0.1785B0.1785B
State-conditioning modules—0.1046B
Planner—9.8149B
Total listed components1.6047B11.5242B

The full model therefore has substantially greater parameter and memory costs than the generator backbone alone. Our external comparisons assess generation quality and action progression.

Manuscript revisions (blue). Appendix C.4 and Tables A2–A3 report matched latency, memory, denoising throughput, and parameter accounting; the Table 1 caption states the backbone-only convention in the main comparison.

Dear Reviewer, We have provided a detailed response to your concerns and updated the manuscript with additional experiments and clarifications. We would appreciate it if you could review these updates and let us know whether any concerns remain. Thank you for your time and consideration.

Dear Reviewer,

As the discussion period draws to a close, we would appreciate it if you could review our rebuttal and manuscript revisions and let us know whether they address your concerns. We would be happy to clarify any remaining questions. Thank you for your time and consideration.

Reviewer_YMckOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces NextAvatar, a streaming avatar generation framework that addresses action stagnation and repetition when instructions span multiple video chunks. It proposes Next-State Planning (NSP), which uses a frozen MLLM to predict the next visual state from video history and instructions, and Reference-Grounded State Anchoring to mitigate identity drift. Experiments on the newly introduced MIRA-Bench demonstrate improvements in instruction responsiveness and action progression over existing avatar generation systems.

Critical strengths
  • The paper identifies a concrete and under-examined failure mode in streaming generation. Stagnation and repetition across chunk boundaries are clearly defined and directly measured by human evaluation, motivation is clear.
  • The paper is clearly written and easy to follow. The experiments demonstrate substantial improvements in action progression over existing baselines.
Critical weaknesses
  • Several external baselines use few-step distilled inference, whereas NextAvatar uses 50 sampling steps. Given these differences, a comparison with NextAvatar-Base under the same inference settings is particularly important. However, the human evaluation in Table 2 does not include NextAvatar-Base, making it difficult to how directly the method contributes to reducing stagnation and repetition.
  • The benefit of visual next-state planning over simpler text-based planning with the same MLLM remains unclear. Given that visual state conditioning introduces identity degradation, which requires an additional anchoring module to mitigate, it is unclear whether its advantages justify the added complexity and remaining identity trade-off.
Questions for authors
  • Could the authors provide stagnation and repetition rates for NextAvatar-Base under the same inference settings as NextAvatar, to better isolate the contribution of next-state planning?

  • Could the authors discuss whether similar improvements could be achieved by using the frozen MLLM to generate next-action text instructions rather than visual state tokens, without introducing additional identity degradation?

4 comments
AuthorsOpenReview ↗

Human evaluation of ablations and Comparison with Text Planning

We thank Reviewer YMck for highlighting the questions of attribution. We have added human evaluation of ablations and comparison with Text Planning to address both concerns.

[Q1 & W1]. Human evaluation of ablations.

To isolate the contribution of our proposed modules beyond the base generator, we added NextAvatar-Base to the same MIRA-Bench human action-progression evaluation as the full model. Both variants are evaluated on the same 200 selected intervals using four mutually exclusive labels, with randomized presentation and method identities hidden from the two annotators.

Compared with Base, the full model reduces stagnation from 26.0% to 6.0% and repetition from 29.5% to 24.5%, lowering their combined rate from 55.5% to 30.5%. The wrong-action rate remains comparable (27.5% vs. 25.0%). This comparison shows that our proposed modules contribute to the progression gains beyond what the base generator achieves alone.

VariantRepetitionStagnationWrong actionStagnation + repetition
NextAvatar-Base29.5%26.0%27.5%55.5%
NextAvatar (Full)24.5%6.0%25.0%30.5%

Manuscript revisions (blue). We added the progression results in Section 4.3 (Table 4) and clarified the human-evaluation and baseline protocols in Appendices D–E.

[Q2 & W2]. Visual next-state planning versus text planning.

To test whether visual next-state planning offers benefits over textual planning, we added a TextPlan control and compared it with NextAvatar. TextPlan uses the original Qwen2.5-VL-7B checkpoint to generate a textual action plan for each chunk and appends it to the active instruction before conditioning the generator. We use the original checkpoint because the MLLM used in NSP no longer directly generates textual plans after visual-state training. We also included Base (+5K) to assess whether additional training alone explains the gains.

The automatic evaluation results are reported below.

VariantHuman IdentityInstruct FollowVision RewardAesthetic QualityImaging Quality
Base0.93880.60350.93050.56340.7399
Base (+5K)0.95050.59450.93770.54040.7299
Base + TextPlan0.88370.63750.93290.54320.7308
NextAvatar (Full)0.93230.64950.93970.57000.7427

Additional training alone does not improve instruction following. TextPlan improves Instruct Follow over Base but falls below NextAvatar (0.6375 vs. 0.6495). This may be because it generates similar action descriptions across chunks without specifying action progress. Adding these textual plans also substantially reduces Human Identity (0.9388 → 0.8837). NextAvatar achieves higher identity consistency (0.9323) and better Vision Reward, aesthetic quality, and imaging quality than TextPlan.

To assess action progression directly, we also evaluated these variants under the same human-evaluation protocol.

VariantRepetitionStagnationWrong actionStagnation + repetition
Base29.5%26.0%27.5%55.5%
Base (+5K)29.0%26.5%23.0%55.5%
Base + TextPlan25.5%16.5%23.0%42.0%
NextAvatar (Full)24.5%6.0%25.0%30.5%

Compared with TextPlan, NextAvatar reduces stagnation from 16.5% to 6.0% and repetition from 25.5% to 24.5%, lowering their combined rate from 42.0% to 30.5%. The wrong-action rate is slightly higher (23.0% → 25.0%). Together, these results show that our visual-planning framework improves action progression and preserves identity better than the evaluated text-planning control.

Manuscript revisions (blue). We added the TextPlan implementation and these comparisons in Section 4.3 (Tables 2 and 4), with supporting analyses in Figures 4–6.

Dear Reviewer, We have provided a detailed response to your concerns and updated the manuscript with additional experiments and clarifications. We would appreciate it if you could review these updates and let us know whether any concerns remain. Thank you for your time and consideration.

Thank you for the detailed rebuttal and additional experiments. The comparison with NextAvatar-Base clearly demonstrates improvements in both stagnation and repetition, and the additional TextPlan comparison and ablations help address my original concerns.

However, the newly added ablations raise some remaining concerns about repetition. The full model achieves only a marginal improvement over TextPlan (25.5% to 24.5%) and performs worse than Base + NSP (w/ R) (21.5%), suggesting that the full framework does not consistently improve repetition.

While I appreciate the substantial improvements to the experimental evaluation, I agree with Reviewer dWkr's concern that the evidence for repetition reduction remains less convincing. I will therefore maintain my original score.

Thank you for acknowledging the additional experiments. We would like to clarify that Base + NSP (w/ R) is a variant of our proposed visual planning framework without state anchoring. Its lower repetition rate therefore supports NSP’s effectiveness, while showing that adding anchoring does not further improve repetition. Anchoring primarily preserves identity, increasing Human Identity from 0.9126 to 0.9323.

Both TextPlan and the NSP variants reduce repetition relative to Base (29.5%), supporting the usefulness of MLLM-based guidance. The clearer advantage of visual NSP is in reducing stagnation: from 16.5% with TextPlan to 6.0% with Full. We acknowledge that Full’s repetition improvement over TextPlan is small; its benefit is the combination of substantially lower stagnation and better identity preservation.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Revisions and Responses to Reviewer Concerns

We thank all reviewers for their time and constructive feedback. We have provided detailed responses to each review and highlighted manuscript revisions in blue. The main updates are summarized below.

1. Module contribution and additional training (YMck, dWkr). We added Base (+5K) and human evaluations of all ablation variants. Additional training does not improve action progression, while Full reduces combined stagnation and repetition from 55.5% to 30.5%. Section 4.3 presents these controls, with automatic metrics in Table 2 and human progression results in Table 4.

2. Visual versus text planning (YMck, k1eR). We added a TextPlan control, showing that NextAvatar improves action progression and preserves identity better than the evaluated textual alternative. Section 4.3 reports the quantitative comparisons in Tables 2 and 4 and illustrates their behavioral and appearance differences in Figure 4.

3. Planner implementation and state evidence (dWkr). We clarified the planner inputs, task modes, system prompts, and frozen status during joint training. Appendix C.3 provides these implementation details, while Figure 5 and its analysis in Section 4.3 show predicted-state response changes concentrated in action-relevant regions.

4. Evaluation and baseline protocols (dWkr). We clarified the two-annotator protocol, randomized presentation, hidden method identities, and four mutually exclusive labels, with wrong-action rates reported separately. Appendix D.5 details human evaluation, Tables 3–4 report the results, and Appendix E explains instruction switching and baseline adaptations.

5. Model size and inference cost (k1eR, dWkr). We added matched Base/Full measurements of parameter counts, latency, and memory. Appendix C.4 describes the profiling setup, Tables A2–A3 report computational costs and parameter breakdowns, and the Table 1 caption clarifies that its parameter counts cover only the generator backbone.

6. History limitations and task scope (k1eR, dWkr). We expanded the rationale for the recent-chunk history window and explained how losing earlier completion evidence can cause repetition. Appendix F discusses this limitation using Figure A3 and outlines possible extensions; Appendix D.1 specifies the benchmark’s scope, while Appendix B distinguishes chunk-level state planning from task-level planning.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.