acceptodds
Withdrawn from ICLR 2027

Marionette: Explicit World State for Interactive World Model

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Interactive game world models typically generate video autoregressively in pixel or latent space. As a model repeatedly conditions on its own predictions, its inputs can drift away from the training distribution, causing errors to accumulate over long horizons. Correcting this drift is difficult when the dynamics remain implicit in visual latents. Our key insight is to identify the dynamics relevant to interaction and autoregress them in an explicit representation where geometric constraints can correct implausible predictions. In games with articulated characters, these dynamics take the form of 3D skeletal animation, which makes constraints on bone lengths, joint angles, and terrain contact directly expressible. We introduce Marionette, a world model that decouples explicit dynamics prediction from appearance generation. An action-conditioned model predicts character poses and trajectories; a deterministic graphics bridge applies state constraints and rasterizes the resulting animation into pose-control videos. A video diffusion model then renders the motion into the final frames. Experiments show that the predicted motion responds to action inputs and that state constraints reduce ground penetration while preserving skeletal structure during long rollouts, without changing the video model.

Primary area:
generative models
Keywords:
world models, interactive video generation, controllable video generation, 3D motion generation, video diffusion, skeletal animation

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_sgDkOpenReview ↗
Rating
2
Confidence
2
Informativeness
1
Presentation
2
Soundness
1
Read review
Paper summary

This paper applies a pose-conditioned video generation model to an interactive game world model. It introduces a framework, Marionette, that decouples explicit dynamics prediction (character motion and terrain) from appearance generation. Specifically, it first generates character poses with two models: ActionGPT predicts an action for each character, such as attacking, dodging or walking, and PoseGPT generates poses conditioned on the action and the terrain. A deterministic graphics bridge then rasterizes the character poses together with the terrain into a pose-control video. Finally, a video diffusion model renders the motion into the final frames.

Contribution: The paper proposes to autoregress interaction-relevant dynamics in an explicit representation (3D skeletal animation) rather than in pixel or latent space.

Validation: The method is evaluated on recordings from a commercial action game. The experiments test whether overriding action tokens changes the generated pose (RA-MPJPE), whether state constraints reduce ground penetration and character separation in long rollouts, and compare FVD against an end-to-end pixel-autoregressive baseline.

The explanation of action and

Critical strengths

The paper proposes to autoregress interaction-relevant dynamics in an explicit representation (3D skeletal animation) rather than in pixel or latent space. It builds a framework that combining pose generation and video generation.

Critical weaknesses

The motion generation module falls far behind existing motion generation methods. Generating poses that conform to the given condition is a basic requirement of motion generation, but in this paper it becomes a question that needs to be validated. Moreover, in some cases the ground-truth action stream performs worse than the shuffled one.

The experiment on state constraints in long-horizon rollouts seems meaningless. Adding a constraint to motion generation will certainly bring the results closer to that constraint, for any motion generation method.

The FVD evaluation does not support the proposed framework. FVD evaluates the video generation model. In this paper, the FVD scores are all very close, which only demonstrates the robustness of the video generation model and has little to do with the proposed framework.

This method is strongly limited by specific game scenes.

Questions for authors

Since each action ID indexes a specific animation in the game's animation bank, and ActionGPT also predicts the animation progress, the pose appears to be largely determined by the ID and progress. Why is a learned PoseGPT needed instead of retrieving the corresponding animation? For example, use way like motion matching?

Reviewer_h9B6OpenReview ↗
Rating
2
Confidence
4
Informativeness
1
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes to separate explicit character dynamics from appearance generation for interactive game video generation. Two modules are introduced, the ActionGPT predicts action tokens and PoseGPT predicts the next 3D skeletal state. A deterministic graphics bridge uses the predicted state and scanned terrain, applies optional trajectory constraints, and renders pose-control videos for a Wan-based observation model. The model generates long videos by chunk relay. The experiments study video quality, action-token interventions, and the effects of terrain and separation rules on generated trajectories.

Critical strengths
  1. The paper studies an important problem of reliable skeletal motion in long video generation where errors accumulate over time, the idea and method could inspire future research in this direction.

  2. I like that the intermediate state can be inspected and edited directly. Some specific failures, such as ground penetration and characters drifting apart, are easier to measure than in the RGB output alone.

  3. Overall, the paper is nicely written and clearly presented. Video demos are presented in the supplemental materials.

Critical weaknesses
  1. Scope. The concept of world state presented in this paper is very limited. The dynamics evaluation is restricted to one monster type, a fixed weapon, and one-on-one combat. Rather than the more general idea of consistent spatial memory or game state.

  2. contribution. The experiments do little to justify particular design choices. For instance, would a single state predictor with the same actions, terrain, and state supervision work similarly? How much do the geometry-buffer channels add over ordinary pose control with the same motion and observation backbone? The current rule and token ablations do not answer these questions.

  3. Baseline. Table 1 is a useful comparison of the complete pipelines, but it does not isolate the benefit of explicit dynamics. The systems share RGB training data and a backbone family, but Marionette additionally uses state supervision and pose controls, which the baseline does not. The lower FVD could therefore reflect several differences at once. I also could not find a clear description of the Pixel-AR baseline's model and training setup in the paper.

  4. Experiments:

    • Overall, the experiments presented in this paper are very shallow and limited.
    • FVD on twelve 16.2-second rollouts can hardly reflect video quality and cannot measure whether RGB follows the driving skeleton. I expect broader video-quality evaluation, including action execution and pose following.
Questions for authors

Please respond to the concerns I listed in the critical Weaknesses above.

  1. Could the authors specify the exact Pixel-AR baseline?

  2. I wonder how reliably does the final RGB video execute user-specified action schedules instead of relying on recorded future-action goals, and how accurately does it follow the driving skeleton over time?

Reviewer_b6fsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

Task. From a seed of recorded game state, a terrain height map and per-step action ids, generate a continuing stream of explicit two-character skeletal state and render it into RGB video, learned from recordings of one commercial action game.

Problem it tries to solve. Pixel- or latent-space interactive world models accumulate error over long rollouts and offer no explicit variable on which physical rules can be stated or enforced.

Key insight. Predict an explicit metric state with two small transformers (action choice, then pose), convert it to a pose-control video with a parameter-free graphics bridge, render RGB with a control-conditioned video diffusion model, and clamp the state with hand-written terrain and separation rules before rendering.

Result. Rendered video scores slightly better FVD than a self-trained pixel-autoregressive baseline with overlapping intervals, overriding the action token changes the generated pose, and the clamps remove ground penetration and bound the fighters' separation, which otherwise drift far apart over long rollouts.

What I learned (actually not too much). Every reported rollout reads both characters' upcoming actions from the recording, including its future, so the model only synthesises poses for a scripted action and renders them; nothing predicts how the world evolves, and the central claim that explicit state beats implicit latents is never tested.

Critical strengths
  1. Reproducible, candid system description. The state layout, the terrain encoding, the collider rules, the probe protocol and the two-layer evaluation are all specified; the paper openly reports overlapping intervals, the recorded-action goal conditioning and its failure modes.
  2. The control probe is the right test. Overriding the action token and measuring the resulting change in pose directly checks that the explicit action interface steers the generator, which pixel- or latent-space world models cannot offer.
Critical weaknesses
  1. It does not predict state transitions, so I don't feel it proper to call it a "world model". At every step both human's and the monster's next actions (including monster reaction), come from the recorded sequence's future (sec 3.2: "the released generator obtains upcoming-action goals from the recorded action sequence, including its future portion"; Appendix D: "a goal is the action id at the next transition in the recorded sequence"). ActionGPT receives them as input and PoseGPT only places the skeleton for an action it was told. If so, then there is no experiment that removes the recorded goals, so the drift curves, the constraint ablation and the controllability claim are all measured on a scripted pipeline. What is shown is action-conditioned motion synthesis feeding pose-guided video generation, the setting the paper itself separates from world modelling in sec 2. *

  2. The central claim has no supporting comparison. The only quantitative baseline is a self-trained text-conditioned pixel model that differs in conditioning and supervision, with overlapping FVD intervals; no published interactive world model and no explicit-state alternative (e.g. ARDY as the dynamics stage) is evaluated. Twelve evaluation windows, one monster, one weapon, two different data configurations, no seeds, and no ablation of any architectural choice.

  3. The "state constraints" are render-time clamps. The terrain collider and separation cap are applied after the neural rollout and never fed back (Appendix C), and the separation result is bounded "by construction" per its own caption. The abstract's claim that explicit state lets constraints correct long-rollout predictions describes post-processing of the renderer input, not a property of the learned dynamics.

Questions for authors
  1. Remove the recorded upcoming-action goals and let ActionGPT choose both characters' actions from state history alone; report the drift curves, state metrics and control probe under this setting. Without it the paper has no world-model result.
  2. Run at least one published interactive world model from the same first frames under the same chunk-relay protocol and report FVD; different control interfaces do not prevent an observation-layer comparison.
  3. Evaluate on more windows with per-window seeds so the intervals become readable, and report the per-chunk FVD behind Fig. 5B.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.