acceptodds
Under review as a conference paper at ICLR 2027

Stable Self-Evolving World Action Models via Self-Perplexity Aware Learning

Abstract

While self-evolution shows great promise for improving model performance without human demonstrations, how to self-evolving World Action Models (WAMs) that jointly model actions and videos remains unexplored. However, directly fine-tuning WAMs on self-generated trajectories induces a pronounced V-shaped curve where performance drops sharply during early training and then gradually recovers. This may stems from the distribution shift between self-generated trajectories and pre-training human demonstrations. To address this challenge, we propose SPASE, a multi-round self-evolution framework for WAMs. SPASE first introduces an Execution-Grounded Dual Verification module to filter out low-quality data with incorrect actions or inconsistent videos. To alleviate the adverse effects of distribution shift, SPASE introduces Self-Perplexity to measure data divergence from the model’s parametric knowledge. Using Self-Perplexity, SPASE adaptively downweights highly divergent data to stabilize training and imposes a deterioration penalty on previously learned data to mitigate catastrophic forgetting. Extensive evaluations across robotic manipulation benchmarks demonstrate the superiority of SPASE. Our code is available at https://anonymous.4open.science/r/stable_wam_evolution/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
world action models, self-evolution, embodied intelligence

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 2 reviews, above 27% of the others.
Reviewer_dVNsOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

This paper introduces SPASE, a multi round self-evolution method for world action models. Self-evolution describes the process where a generative model is trained on its own outputs to improve itself. For world action models, this means collecting evaluation rollouts, filtering out failure and inconsistent data, and training the video and action heads on the model's own outputs. However, a vanilla application of self-evolution reveals that model performance drops initially before rising again, creating a v-shaped curve. To address the dip, the paper proposes weighting the per-sample updates by a measure of self-perplexity. The paper shows that this self-perplexity weighting leads to improved multi-round self-evolution results, achieving higher success and generalization compared to baselines across multiple WAM backbones. The main results are supplemented with ablations of design choices and hyperparameters.

Critical strengths
  1. The paper proposes a novel self-evolution method for world action models. The main idea is similar to iterative behavior cloning: filter for success and then train on your own data. Concretely, each round of SPASE consists of the following steps:
    1. Collect a number of rollouts in the environment.
    2. Filter trajectories for success and consistency. This step has three substeps: (1) segmenting trajectories into subgoals, (2) evaluating the completion of each subgoal, discarding failed segments and their suffixes, (3) verifying the consistency between real and generated frames, discaring inconsistent segments.
    3. Train on generated data using self-perplexity as weighting. In particular, self-perplexity is defined as the flow-matching loss on a set of queries. Data with high self-perplexity are downweighted to avoid drastic weight updates.
  2. I like that the main algorithmic contribution is motivated by a grounded empirical observation (the v-shaped curve).
  3. The proposed method is substantiated by ample simulation results and ablations. In particular, Figure 2 visualizes the model perplexity of human data and self data before and after self-evolution, showing that the positions are swapped.
Critical weaknesses
  1. The proposed method requires collecting environment rollouts. However, the real frames from the environment are discarded, and instead the method only trains on generated frames. The paper doesn't compare to training on real frames.
  2. Perplexity is a well-defined term in machine learning (exponentiated negative log likelihood of data under the model). This paper's proposed measure does not correpond to that definition. A term like "flow discrepancy" would be more appropriate.
  3. The theoretical results in the appendix are redundant. They do not make predictions that the experiments could test.
  4. The real robot results are limited (only appendix H, one task). The implications of efficient self-evolution are huge for deploying models on real robots. However, real data can have different characteristics from sim data.
Questions for authors
  1. Why train on the model's generated frames rather than the real frames, given you've already collected these rollouts?
  2. Is self-perplexity connected to the actual negative log likelihood of the data under the model?
  3. How does self-evolving WAMs compare to self-evolving VLAs or VLAs + world models (e.g. VLAW)?
  4. Can you plot the equivalent of Figure 2 (self-perplexity before/after evolution) for real robot data?
Reviewer_LEEiOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors studies how world-action-model can learn from its own generated experience as action-video trajectories. They focused on limiting the training instability and forgetting. Their method (SPASE) uses simulator to check trajectories and uses flow matching error score to down-weight difficult examples. They show improvement on task success and better stability in training across three models and three robotic manipulation benchmarks. They found that ordinary self training initially affects the performance negatively, and experiments across three robotic benchmarks show improved task success over the tested baselines by reporting only mean values.

Critical strengths

1- The method shows improvements across three world-action-models and three benchmarks. Their evaluation supports the practical value of the method beyond single model or task setting.

2- The comparison with action only training provides useful information about why video supervision also important aspect. While action only training achieve higher success in two benchmarks, adding video supervision transfers better to LIBERO-Goal. This finding explains clearly video supervision is important for generalisation which is important for robotics.

Critical weaknesses

1- The core methodological advance is unclear. In Equation 1, Self-Perplexity is the standard flow matching error averaged over fixed queries used by the main method (SPASE) to reduce the weight of difficult samples and penalise deterioration on previously learned samples. Methods such as GEM (Gradient Episodic Memory for Continual Learning, Lopez-Paz et al, 2017) is closely related to these combinations. Although authors' proposed combination shows promising results on experiments, the paper needs to explain more clearly what is specific weighting and protection rules contribute beyond GEM, or other continual learning methods.

2-The choice to train on generated videos is not sufficiently justified. The authors already execute the actions in simulator and record the videos with these actions, so the observations they have is simply what actually happened. I didn't find any comparison training on these recorded videos with trining on the model's generated videos. Keeping the retained action windows and training budget the same would would make direct comparison of the two supervision sources. Its clear that without this comparison, the advantage of the learning from generated video target remains unclear.

Questions for authors

1 -Can you provide mean +- standard deviation for Table1 and individual run curves for Figures 2 and 4 ?

2- Do these three seeds in section 4.1 correspond to independent training and data collection runs, or repeated evaluations the same trained model?

3- Have authors compared training on generated videos with training on the recorded execution videos using the same retained action windows and training budget?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.