acceptodds
Under review as a conference paper at ICLR 2027

Not All Regions Need Equal Tokens: Motion-Adaptive Representation Allocation for Complex Human Image Animation

Abstract

Complex human motion creates spatially non-uniform representation demands. Fast-changing body regions require finer representations, while slowly varying regions can be represented with fewer tokens. However, existing video Diffusion Transformers typically use a uniform patch grid, assigning the same token density to regions with substantially different representation needs. We rethink complex human image animation from this representation-allocation perspective and propose Motion-Adaptive Token Allocation (MATA), which reallocates spatial representation capacity according to motion demand. MATA assigns denser representations to motion-intensive regions while compressing redundant tokens in low-motion regions, allowing additional capacity to be directed where complex motion requires it without uniformly increasing the token budget. On HyperMotionX, MATA improves generation under complex human motion, reducing FVD by 7.97% and increasing by 6.54 percentage points over UniAnimate-DiT 5B, with only a 1.29% increase in estimated main-DiT FLOPs. These results highlight the importance of spatially non-uniform representation allocation for complex human animation.

Primary area:
generative models
Keywords:
Diffusion Models

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_FsEuOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
2
Soundness
3
Read review
Paper summary

This paper proposes Motion-Adaptive Token Allocation, which reallocates spatial representation capacity according to motion demand, assigning denser representations to motion-intensive regions while compressing redundant tokens in low-motion regions. The paper also designs a method for restoring compressed regions and a refinement strategy that disables compression late in denoising. The paper presents quantitative and qualitative comparisons on HyperMotionX using 5B and 14B models. It also analyzes performance across motion difficulty levels, conducts module ablations, and reports computational costs.

Critical strengths

The method solves a meaningful problem and has a complete design. Regions with complex human motion and static regions may need different representation densities. This paper combines local enrichment, compression of other regions, and output restoration. This provides a concrete approach to improving human animation under a limited computational budget.

Critical weaknesses

The added value of the allocation strategy is not fully established. There has been prior work that explored content-adaptive or motion-adaptive token allocation, but this paper seems to provide no direct comparison with these methods. Also, as the paper illustrated, motion-guided allocation achieves similar PCK to random allocation. It improves FVD and VFID but shows no additional benefit in pose accuracy.

Questions for authors

For the random-region baseline in Table 2, did you repeat the evaluation with different randomly selected regions?

2 comments

Reviewer FsEu

We thank the reviewer for asking how MATA compares with ADAPTOR and how sensitive the random-allocation baseline is to the choice of regions.

Response to W1: Comparison with ADAPTOR

MATA’s main contribution is the redistribution of a constrained token budget according to motion demand. ADAPTOR targets acceleration and retains the original token resolution in selected regions. MATA uses pose cues to sample regions with complex articulation more finely than the original patch grid. Compression in low-motion regions offsets the cost of this enrichment, keeping main-DiT FLOPs within 1.29% of the original 5B backbone.

A direct comparison may require adapting ADAPTOR to keep its token selection aligned with the pose-conditioning features in our animation pipeline. We will aim to complete the evaluation before the discussion deadline and share the quality and compute results in a follow-up if they are ready in time.

Response to Q1: Sensitivity to random region allocation

In Table 2, we applied random placement only at inference to a checkpoint trained with motion guidance. For the new comparison, we trained two enrichment-only models from the same Stage-1 checkpoint. One used random placement throughout training and inference, and the other used motion-guided placement throughout. Both received 9,794 Stage-2 updates with identical trainable modules and matched per-frame token counts.

To isolate the effect of which regions are selected, we fixed the generation seed at 0 for all runs on the same 100 videos. For the trained random baseline, we varied only the inference-time allocation seed (0–4) and report the mean ± sample standard deviation across those five runs. Motion-guided placement determines the regions from the input pose sequence, so its row reports a single result at the same generation seed.

Enrichment policyMean per-video FVD ↓Global FVD ↓[email protected] (%) ↑
Random (5 allocation seeds, mean ± SD)892.04 ± 5.28138.51 ± 0.7364.69 ± 0.08
Motion-guided (pose-determined regions)863.15132.4465.30

At this generation seed, mean per-video FVD ranged from 886.76 to 900.44 across the five random allocations, compared with 863.15 for motion-guided placement.

We separately tested sensitivity to generation sampling by fixing the random allocation seed at 0 and varying the generation seed (0–2) for both methods. Motion guidance reduced Global FVD by 7.09, 5.40, and 3.04, respectively. PCK increased by 0.72 and 0.36 percentage points at the first two seeds and was essentially unchanged at the third.

Thank you for the additional experiments. The matched-training comparison and repeated random allocations substantially address my concern. The new results also show modest gains in pose accuracy. I lean toward accepting this paper.

Reviewer_5ZpAOpenReview ↗
Rating
2
Confidence
3
Informativeness
3
Presentation
2
Soundness
2
Read review
Paper summary

This paper addresses the limitation of Human Image Animation video generation's uniformly allocating tokens across all spatial regions problems. This paper proposes, in animation tasks, fast-moving regions require more tokens while static regions need fewer.Therefore, the paper proposes a Motion-Adaptive Token Allocation (MATA) strategy. It first uses the difference between adjacent frame masks as a motion measure, where regions with larger differences are considered to have stronger motion and are allocated more tokens. It then introduces local reconstruction error to identify regions that do not require as many tokens. A smaller reconstruction error indicates that the region has a lower demand for tokens, allowing fewer tokens to be allocated to these areas. The method increases the number of tokens by applying Patch Embedding with a smaller stride and reduces redundant tokens through token merging. During the restoration stage, to adapt to the original VAE decoder, the paper introduces overlap-aware aggregation to merge enriched tokens from overlapping regions. For compressed tokens, the authors observe that directly copying tokens in high-velocity regions introduces artifacts. Therefore, they propose x0-consistent restoration, which first returns to the x0 space, upsamples the meaningful image content in x0, and then recalculates the velocity representation.

Critical strengths

The motivation of this paper is well justified. It addresses a common limitation of DiT-based methods, where all image patches are represented with the same number of tokens. This is a meaningful and worthwhile problem to study.

Critical weaknesses
  1. In fact, in Fig. 4, besides the highlighted regions, I can still find cases where other methods, such as SCAIL-2 and HyperMotionX, achieve better results in certain areas of the images compared with ours. For example, in the third row, although the paper highlights the foot region in the air, the face orientation generated by ours is not entirely accurate, while HyperMotionX produces a more consistent face orientation. Therefore, it is difficult to clearly evaluate the superiority of the proposed method based only on these highlighted regions.
  2. The supplementary material lacks dynamic video results. Since this is a human image animation task, static frames alone are insufficient to demonstrate the significant advantages of the proposed method in handling high-motion regions. Providing video comparisons would be necessary to better evaluate the temporal consistency and motion quality.
  3. The same issue exists in the ablation studies. The paper claims that Velocity Restoration can reduce artifacts, and both LoRA-only and Enrichment-only also serve related purposes. However, it is difficult to evaluate the effectiveness of this component based only on static images. In my opinion, the visual differences among the selected frames are not particularly significant, making it challenging to verify the actual contribution of Velocity Restoration.
Questions for authors

Since this is a human image animation task, the absence of dynamic video results makes it difficult to assess the effectiveness and advantages of the proposed method. Could the authors provide additional video comparisons to better demonstrate the improvements in motion quality and temporal consistency?

2 comments

Reviewer 5ZpA

Thank you for pointing out the differences outside the highlighted regions in Figure 4.

Response to W1 and W2: Video comparisons and head orientation

We have uploaded video comparisons to the Supplementary Material so that foot placement and head orientation can be compared throughout each sequence. As you observed, HyperMotion follows the head orientation more accurately in the third row of Figure 4. One possible reason is ambiguity in the driving pose: different 3D head orientations can produce similar 2D keypoint configurations.

Response to W3: Restoration artifacts in compressed regions

The colored block artifacts in Figure 5 arise when velocity predictions on the compressed grid are interpolated back to the original grid. Both LoRA-only and enrichment-only retain the original parent grid throughout denoising, so neither variant performs this restoration step.

Table 3 isolates the source of the artifacts by keeping compression fixed and changing only what is restored. Reconstructing the clean latent x0x_0 on the original grid before recovering the velocity suppresses the visible blocks. LPIPS decreases from 0.196 to 0.176 and mean per-video FVD from 1066.42 to 882.73, while PCK changes by only 0.02 percentage points.

Dear Reviewer 5ZpA,

We would appreciate your feedback on our response and the video comparisons uploaded to the Supplementary Material. We have also clarified the role of 2D pose ambiguity and how the restoration mechanism addresses compression-induced artifacts. Could you let us know whether these explanations and videos address your concerns, or whether further clarification would be helpful?

Thank you for your time.

Reviewer_nwCeOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper studies spatial representation allocation for pose-driven human image animation. MATA adds fine-grained tokens to regions with substantial motion and compresses low-motion regions to offset the additional sequence length. The resulting representations are mapped back to the original grid through overlap-aware aggregation and restoration in the predicted clean-latent domain. Experiments on HyperMotionX show improvements in pose accuracy and several visual quality metrics over the original 5B backbone.

Critical strengths

1、The method connects representation allocation with an explicit token budget. MATA uses the available pose sequence to identify regions requiring finer representations and recovers tokens from low-motion regions. During compressed steps, the sequence contains at most two more tokens than the dense baseline. Compared with Motion LoRA, the complete method improves PCK from 60.60% to 65.89% (Table 1). The enrichment-only model reaches a PCK of 65.20% and an FVD of 859.95. Adding compression raises PCK to 65.89%, while FVD increases to 882.73 and LPIPS increases from 0.169 to 0.176. These results quantify the visual-quality cost of recovering the token budget.

2、The restoration mechanism has clear analytical and experimental support. Restoring compressed predictions in the clean-latent domain addresses the resampling inconsistency introduced by directly interpolating velocity predictions. This change reduces FVD from 1066.42 to 882.73 while leaving pose accuracy almost unchanged (Table 3). The same change reduces VFID from 17.61 to 12.57 and FID from 100.47 to 85.41. Figure 5 shows the corresponding reduction in colored block artifacts.

Critical weaknesses

1、The independent contribution of motion-guided placement is not sufficiently isolated. In Table 2, random and motion-guided placement achieve PCK values of 65.38% and 65.20%, respectively, with identical LPIPS of 0.169. Motion guidance reduces FVD from 881.50 to 859.95 and VFID from 13.43 to 12.81. Moreover, the random baseline changes placement only at inference using a checkpoint trained with motion-guided allocation. This comparison shows that placement affects FVD and VFID in the trained model. It does not show whether motion-guided allocation outperforms random allocation when both are used throughout training. Training the random-allocation model with the same PatchEmbed, sub-position embeddings, training steps, and token budget would isolate the contribution of the placement policy. Appendix B.4 states that the 5B Stage-2 model starts from a two-epoch pose/Motion-LoRA checkpoint and receives two additional epochs, with LoRA and the pose module continuing to update. The total training budget of the Motion-LoRA-only baseline is unclear. Continuing that baseline for the same number of steps would show how much of the Table 1 gain can be obtained through additional fine-tuning.

2、The visual quality conclusion depends on how FVD is aggregated. The main evaluation fits a feature distribution separately for each video, although each video supplies only approximately two to five clip features. With at most five clips per video, the sample covariance has rank at most four. The sensitivity of the resulting distances to individual clips and video length needs to be assessed. In Table 6, MATA 14B outperforms HyperMotion under mean per-video FVD, 733.81 versus 808.92, but performs worse under pooled global FVD, 131.48 versus 119.82. These metrics measure different distributional scopes, as discussed in Appendix B.6. At 14B, per-video averaging favors MATA, whereas pooled global FVD favors HyperMotion. The claimed distributional-quality advantage over HyperMotion therefore depends on the aggregation scheme. For 5B, both aggregations favor MATA over its original backbone, including a reduction in global FVD from 151.26 to 135.93. Video-level confidence intervals and results across generation seeds are also missing. The reported generation protocol uses seed 0 (Appendix B.4). Paired video-level bootstrap intervals would quantify uncertainty in the benchmark comparison, and repeated generation seeds would measure variation from sampling. The small number of clips used to estimate each video’s covariance requires a separate sensitivity check. The complete method improves pose accuracy, and the restoration ablation supports the clean-latent formulation. I recommend weak rejection because the training controls leave the contribution of motion-guided allocation unresolved, while the FVD evaluation needs stronger evidence of statistical reliability. These concerns affect the paper’s explanation of why the method works and the strength of its quality comparisons.

Questions for authors

1、Under matched training steps, trainable parameters, and token counts, how does motion-guided allocation compare with a model trained using random allocation throughout? Does the Motion-LoRA-only baseline receive the same total number of training steps, including the additional Stage-2 training? Please keep the enrichment PatchEmbed and sub-position embeddings identical in the random-versus-motion-guided comparison.

2、Are the per-video and global FVD differences stable under video-level resampling and multiple generation seeds? Please provide paired video-level uncertainty intervals and discuss how the 14B ranking reversal affects the claimed quality advantage over HyperMotion. How sensitive are the per-video distances to estimating covariances from only two to five clip features?

3、The appendix reports MATA’s latency and memory usage. What are the corresponding end-to-end measurements for the original backbone and Motion LoRA on the same hardware? The reported 1.29% increase in main-DiT FLOPs does not establish the actual runtime overhead.

3 comments

Reviewer nwCe — Part 1/2

We thank the reviewer for the questions about training controls, FVD reliability, and computational cost. This response covers the matched training controls and same-hardware measurements. Part 2 reports the FVD uncertainty and sensitivity analyses.

Response to W1 and Q1: Training budget and motion-guided placement

The Motion-LoRA-only baseline in Table 1 was trained for two epochs and achieved 60.60% PCK. We trained the new Motion-LoRA-only control (A) for four full epochs, matching the total number of training epochs of B, C, and D.

The random baseline in Table 2 changed placement only at inference. For the new comparison, B used random placement throughout Stage-2 training and inference, and C used motion-guided placement throughout. Both started from the same Stage-1 checkpoint and received 9,794 updates with the same trainable modules, including the enrichment PatchEmbed and sub-position embeddings, and identical per-frame token counts.

The table reports mean ± sample SD across three generation seeds (0–2) on the same 100 videos, with B’s allocation seed fixed at 0. In the original single-seed evaluation, C and D achieved mean per-video FVDs of 859.95 and 882.73 and PCK scores of 65.20% and 65.89%, respectively. The new evaluation uses C retrained alongside B and fresh inference runs for D.

5B configurationMean per-video FVD ↓[email protected] (%) ↑
A. Motion LoRA only, 4 epochs992.89 ± 8.1060.65 ± 0.42
B. Random enrichment901.52 ± 25.2064.84 ± 0.29
C. Motion-guided enrichment873.50 ± 11.0065.21 ± 0.37
D. Full MATA881.80 ± 17.1966.08 ± 0.45

With training epochs matched, full MATA exceeded Motion LoRA by 5.42 PCK percentage points on average. Random enrichment alone yielded a 4.19-point gain over Motion LoRA. Under the matched B/C comparison, motion guidance added 0.36 points on average. Its clearest benefit was in FVD: both aggregation schemes favored motion guidance at all three seeds, although the difference was smaller at seed 2.

Response to Q3: End-to-end runtime and peak GPU memory

We timed all five configurations on the same 100 videos using one A100 PCIe 40 GB GPU, bfloat16, FlashAttention 2, and identical sampling settings. Each run generated 81 frames. End-to-end time includes input encoding, denoising, and VAE decoding. Videos were trimmed to their target lengths after timing.

5B configurationEnd-to-end (s/video)Denoising (s/video)Time change vs. backbonePeak allocated / reserved (GiB)
O. Original backbone141.58108.71—34.280 / 38.049
A. Motion LoRA only141.38108.64−0.14%34.280 / 38.049
B. Random enrichment149.41116.63+5.53%34.282 / 38.053
C. Motion-guided enrichment149.71116.78+5.74%34.282 / 38.053
D. Full MATA147.24113.78+4.00%34.282 / 38.053

Times are means over 100 videos. We reset the memory counters before each video and report the maximum peak over the dataset. The mean per-video peak and the maximum agreed to the precision shown for every configuration.

The paper’s 1.29% figure measures the increase in main-DiT FLOPs. On this hardware, full MATA added 5.66 s/video (+4.00%) over the original backbone and 5.86 s/video over Motion LoRA. Compression recovered 2.47 s/video relative to motion-guided enrichment alone. Peak allocated memory was 34.282 GiB for full MATA and 34.280 GiB for both the backbone and Motion LoRA.

Reviewer nwCe — Part 2/2

Response to W2 and Q2: FVD uncertainty, clip sensitivity, and the 14B comparison

Video-level uncertainty and generation seeds. We tested two 5B comparisons: motion-guided versus random enrichment (C − B), and full MATA versus four-epoch Motion LoRA (D − A). For each generation seed, we performed 2,000 paired bootstrap resamples of the 100 videos, using the same sampled video IDs for both methods. Within each resample, we averaged the video scores for mean per-video FVD and recomputed Global FVD from the pooled clip features. Brackets give the 95% percentile interval for each difference.

ComparisonSeedΔ mean per-video FVD [95% CI]Δ Global FVD [95% CI]
C − B0−37.29 [−59.55, −15.48]−7.09 [−17.34, −1.39]
C − B1−42.20 [−73.96, −13.94]−5.40 [−15.12, −0.50]
C − B2−4.57 [−23.29, 12.86]−3.04 [−7.33, 1.04]
D − A0−109.63 [−145.78, −74.49]−21.69 [−41.12, −15.21]
D − A1−101.71 [−143.71, −65.37]−13.87 [−31.19, −7.36]
D − A2−121.94 [−180.03, −63.44]−22.88 [−59.66, −11.55]

For full MATA versus four-epoch Motion LoRA, both intervals were below zero at every seed. For motion-guided versus random enrichment, both intervals were below zero at seeds 0 and 1 and included zero at seed 2.

Sensitivity to clip count and position. We used the 53 of 100 test videos that each contained five complete clips. This kept the video set fixed as we varied the number of clips. For each count from two to five, we evaluated every combination of clip positions, using the same positions for both methods. With two clips, changing their positions reversed the D/A ranking for 39–41 of the 53 individual videos, depending on the seed. After averaging over clip choices and videos, full MATA had lower mean per-video FVD at every clip count and seed, by 87.09–136.02. The smaller placement effect was more sensitive: on this subset at seed 2, C − B changed from −0.10 with two clips to +2.51 with five. These results support the reviewer’s concern about interpreting individual-video rankings from very few clips.

14B ranking reversal. Across all 100 test videos in Table 6, MATA had lower mean per-video FVD than HyperMotion (733.81 vs. 808.92) but higher Global FVD (131.48 vs. 119.82). Mean per-video FVD compares each generated video with its corresponding ground-truth video and weights those distances equally. Global FVD pools the generated clips and the ground-truth clips before computing a single distance. The two measures can therefore rank the same methods differently.

On the 53-video subset described above, MATA also had lower mean per-video FVD than HyperMotion at all four clip counts. For Global FVD, the MATA-minus-HyperMotion difference was +11.66 in Table 6 and +17.44 in the supplementary re-evaluation (paired 95% CI [−11.49, 31.68]). We therefore limit our 14B FVD advantage to the within-video comparison. The pooled FVD point estimate favors HyperMotion in both evaluations, and the interval in the supplementary comparison includes zero.

Dear Reviewer nwCe,

We would appreciate your feedback on our two-part response, which includes matched training controls, multi-seed results with paired confidence intervals, clip-sensitivity analysis, and same-hardware runtime and memory measurements. Could you let us know whether these results address your main concerns, or whether any points need further clarification?

Thank you for your time.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.