- Rating
- 2
- Confidence
- 2
- Informativeness
- 1
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper applies a pose-conditioned video generation model to an interactive game world model. It introduces a framework, Marionette, that decouples explicit dynamics prediction (character motion and terrain) from appearance generation. Specifically, it first generates character poses with two models: ActionGPT predicts an action for each character, such as attacking, dodging or walking, and PoseGPT generates poses conditioned on the action and the terrain. A deterministic graphics bridge then rasterizes the character poses together with the terrain into a pose-control video. Finally, a video diffusion model renders the motion into the final frames.
Contribution: The paper proposes to autoregress interaction-relevant dynamics in an explicit representation (3D skeletal animation) rather than in pixel or latent space.
Validation: The method is evaluated on recordings from a commercial action game. The experiments test whether overriding action tokens changes the generated pose (RA-MPJPE), whether state constraints reduce ground penetration and character separation in long rollouts, and compare FVD against an end-to-end pixel-autoregressive baseline.
The explanation of action and
Critical strengths
The paper proposes to autoregress interaction-relevant dynamics in an explicit representation (3D skeletal animation) rather than in pixel or latent space. It builds a framework that combining pose generation and video generation.
Critical weaknesses
The motion generation module falls far behind existing motion generation methods. Generating poses that conform to the given condition is a basic requirement of motion generation, but in this paper it becomes a question that needs to be validated. Moreover, in some cases the ground-truth action stream performs worse than the shuffled one.
The experiment on state constraints in long-horizon rollouts seems meaningless. Adding a constraint to motion generation will certainly bring the results closer to that constraint, for any motion generation method.
The FVD evaluation does not support the proposed framework. FVD evaluates the video generation model. In this paper, the FVD scores are all very close, which only demonstrates the robustness of the video generation model and has little to do with the proposed framework.
This method is strongly limited by specific game scenes.
Questions for authors
Since each action ID indexes a specific animation in the game's animation bank, and ActionGPT also predicts the animation progress, the pose appears to be largely determined by the ID and progress. Why is a learned PoseGPT needed instead of retrieving the corresponding animation? For example, use way like motion matching?