- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 1
Read review
Paper summary
This paper introduces SPASE, a multi round self-evolution method for world action models. Self-evolution describes the process where a generative model is trained on its own outputs to improve itself. For world action models, this means collecting evaluation rollouts, filtering out failure and inconsistent data, and training the video and action heads on the model's own outputs. However, a vanilla application of self-evolution reveals that model performance drops initially before rising again, creating a v-shaped curve. To address the dip, the paper proposes weighting the per-sample updates by a measure of self-perplexity. The paper shows that this self-perplexity weighting leads to improved multi-round self-evolution results, achieving higher success and generalization compared to baselines across multiple WAM backbones. The main results are supplemented with ablations of design choices and hyperparameters.
Critical strengths
- The paper proposes a novel self-evolution method for world action models. The main idea is similar to iterative behavior cloning: filter for success and then train on your own data. Concretely, each round of SPASE consists of the following steps:
- Collect a number of rollouts in the environment.
- Filter trajectories for success and consistency. This step has three substeps: (1) segmenting trajectories into subgoals, (2) evaluating the completion of each subgoal, discarding failed segments and their suffixes, (3) verifying the consistency between real and generated frames, discaring inconsistent segments.
- Train on generated data using self-perplexity as weighting. In particular, self-perplexity is defined as the flow-matching loss on a set of queries. Data with high self-perplexity are downweighted to avoid drastic weight updates.
- I like that the main algorithmic contribution is motivated by a grounded empirical observation (the v-shaped curve).
- The proposed method is substantiated by ample simulation results and ablations. In particular, Figure 2 visualizes the model perplexity of human data and self data before and after self-evolution, showing that the positions are swapped.
Critical weaknesses
- The proposed method requires collecting environment rollouts. However, the real frames from the environment are discarded, and instead the method only trains on generated frames. The paper doesn't compare to training on real frames.
- Perplexity is a well-defined term in machine learning (exponentiated negative log likelihood of data under the model). This paper's proposed measure does not correpond to that definition. A term like "flow discrepancy" would be more appropriate.
- The theoretical results in the appendix are redundant. They do not make predictions that the experiments could test.
- The real robot results are limited (only appendix H, one task). The implications of efficient self-evolution are huge for deploying models on real robots. However, real data can have different characteristics from sim data.
Questions for authors
- Why train on the model's generated frames rather than the real frames, given you've already collected these rollouts?
- Is self-perplexity connected to the actual negative log likelihood of the data under the model?
- How does self-evolving WAMs compare to self-evolving VLAs or VLAs + world models (e.g. VLAW)?
- Can you plot the equivalent of Figure 2 (self-perplexity before/after evolution) for real robot data?