acceptodds
Withdrawn from ICLR 2027

ActionSplice: In-Flight Action Editing for Interactive World Models

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Chunk-autoregressive video world models typically generate each chunk under one action. When an action changes during sampling, waiting until the next chunk delays the response, while directly switching the conditioning leaves the intermediate solver state shaped by the previous action. Restarting sampling under the revised action avoids this mismatch but repeats completed computation. We introduce ActionSplice, which edits the interrupted state through Counterfactual State Transport (CST). A lightweight corrector moves the interrupted solver state toward the matched counterfactual solver state induced by the revised action at the same solver step. The world model and sampler remain frozen, and sampling resumes without replaying completed evaluations. CST-R retargets the entire active chunk, while CST-T preserves the temporal prefix at the intervention step and corrects only the suffix. On minWM–Wan Action2V and HY-WM1.5, we measure fidelity against matched rollback references. Relative to condition swapping, CST-R reduces LPIPS by 61.5% and 75.9%, and CST-T reduces suffix LPIPS by 56.1% and 77.5%, respectively.

Primary area:
generative models
Keywords:
interactive world models, controllable video generation, autoregressive video generation, diffusion models, action editing, counterfactual state transport

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 4 reviews, above 27% of the others.
Reviewer_PzGkOpenReview ↗
Rating
1
Confidence
4
Informativeness
1
Presentation
1
Soundness
1
Read review
Paper summary

This paper tackles the problem of how to denoise and generate the observation in interactive world models when a new action is received during the rollout within an ongoing chunk. First, I question the fundamental motivation of this paper, as it appears to formulate a problem that lacks necessity. In interactive world models, generation naturally proceeds sequentially. The model receives an action, completes the rollout conditioned on it, and subsequently awaits the next input. I fail to see any practical justification for interrupting or altering the action during the rollout within an ongoing chunk.

Furthermore, the manuscript suffers from severe presentation issues. To point a few (the paper has many other issues):

  1. The proposed method is inadequately described in the main text. Specifically, both the training pipeline and the inference usage of the "CORRECTOR" module remain largely incomprehensible.

  2. The model (HY-World 1.5) is referred to interchangeably and inconsistently throughout the text (e.g., using terms like "WorldPlay" and "HY-WorldPlay" without clarification).

  3. The paper repeatedly mentions the 'HY-WorldPlay benchmark', yet fails to clearly define what it is or provide citations.

  4. The authors discuss and compare methods in "Inference-time reuse and acceleration." This entire line of literature is completely orthogonal to the problem the paper claims to solve.

In conclusion, this submission falls far below the standards of peer review. I recommend an unequivocal Reject, and I strongly recommend the Program Committee to strictly punish such irresponsible, unpolished submissions to preserve the integrity of the review process.

Critical strengths

None

Critical weaknesses

None

Questions for authors

None

Reviewer_FRTLOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActionSplice to handle action updates that arrive while an interactive video world model is still sampling a chunk. The method uses Counterfactual State Transport (CST), which trains a lightweight corrector to move the interrupted state toward the counterfactual state under the revised action, avoiding the need to repeat completed denoising steps. Two variants are introduced: CST-R modifies the entire chunk, while CST-T preserves the prefix and only corrects the suffix. Experiments on minWM and HY-WM1.5 show improved fidelity and action following compared to directly swapping action conditions.

Critical strengths
  1. The paper addresses action updates during sampling rather than only at chunk boundaries. This is a relevant problem for improving the responsiveness of interactive world models.

  2. The method does not require retraining the original world model and only adds a lightweight corrector. The two variants also cover both whole-chunk updates and within-chunk action transitions.

  3. The authors evaluate the method on two different backbones and compare against several baselines. The results show that CST produces outputs closer to full rollback while maintaining relatively low latency.

Critical weaknesses
  1. Both backbones support much richer camera controls: minWM accepts continuous camera poses, while HY-WM1.5 supports keyboard and mouse inputs. However, the training data only covers six transitions among three actions (forward, backward, and yaw-left). The quantitative evaluation in Table 2 and other experiments also uses only these six transitions. Action could involve yaw-right, lateral movement, looking up or down, moving while turning, or different turning speeds. None of these are quantitatively evaluated. Figures 4 and 5 show yaw-right transitions that are absent from the stated training set, but the paper does not clarify whether these are out-of-distribution examples or provide quantitative results. If each new action requires collecting additional rollback pairs and retraining the corrector, the practical applicability of the method would be limited.

  2. Some experimental results are difficult to interpret. In Figure 2, the frames across different action segments look very similar, making it hard to verify whether the camera follows the requested actions. In the castle example in Figure 4, the pre-update frames of Wait and CST-R appear different, although they should be identical if both methods share the same initial state and history. Table 4 also lacks details on how CST-R is applied to the HY-WorldPlay benchmark. Without in-flight action updates, CST-R should not affect the original generation. If such updates are introduced, the comparison with existing inference acceleration methods may not be directly comparable.

Questions for authors
  1. I would like to see an ablation that completely removes the interrupted state zr−z_r^-. The counterfactual target zr⋆z_r^\star is obtained by running the backbone from the initial state under the revised action, and the corrector already receives the initial state, previous chunk, and action conditions. It is therefore possible that the corrector simply learns to approximate the backbone's first few denoising steps rather than making use of zr−z_r^-. A baseline trained from scratch to directly predict zr⋆z_r^\star without using zr−z_r^- at any stage of training or inference would help establish whether the interrupted state is actually necessary.

  2. How does CST generalize to unseen actions, such as yaw-right, lateral movement, combined actions, or different action magnitudes? Does handling new actions require additional training?

  3. Could the authors provide the videos corresponding to Figure 2, with action update timestamps and camera trajectories, to better demonstrate action following?

  4. What is the evaluation protocol for Table 4?

Reviewer_XtdROpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

ActionSplice handles action changes that arrive while a video world model is already sampling a chunk. The main contributioon is to learn a correction to the intermediate state, allowing sampling to continue under the revised action, crucially without restarting. The work considers whole-chunk updates and suffix editing. Experiments on two backbones show improvements in rollback fidelity and action following, with lower latency than restarting.

Critical strengths
  • Setup is relevant and idea is neat. Simply switching the conditioning can leave stale motion, while restarting repeats computation - neat. Learning a correction from matched counterfactual states is a sensible solution, especially since the world model stays frozen. I assume this generalizes to other backbones; then it would be particular useful, though it is not shown here.
  • Main results are very decent on both backbones. CST improves rollback fidelity while remaining faster than restarting, and the whole-chunk comparison against partial rollback on HY-WM1.5 is encouraging. I also appreciate the human and action-following evaluations: these show that the correction actually improves the requested behavior.
Critical weaknesses
  • Not quite convinced how broadly the quality-latency advantage holds. On HY_WM1.5, LPIPS increases from 0.0831 at (r=2) --> 0.2694 at (r=3), and rotation error 0.048 --> 0.180. Late updates are clearly relevant here, so why are the baselines missing from this sweep? Only one partial-rollback depth is evaluated, too - I think these are missing
  • Prefix handling in CST_T is still unclear. Target construction uses prefix clamping, but HY_WM1.5 continues without repeated prefix replacement. Now, how does this relate to the rollback reference and the baselines? If I understand correctly this handling can affect the results, so it is hard to judge how much of the improvement comes from your correction. Please clarify or add.
Questions for authors
  • Eval: Does CST retain its advantage over condition swapping and partial rollback(r=3) in both action following and rollback fidelity? And how sensitive is the quality-latency comparison to rollback depth?
  • Clarification: Which prefix quantities are clamped during CST-T target construction and continuation? Does the HY-WM1.5 rollback reference also continue without repeated prefix replacement? I also wonder if the baselines receive equivalent prefix and cache handling?
  • Based on what assumptions can we assume that your results generalize to other backbones - different architectures and model sizes?
Reviewer_6X94OpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes ActionSplice, a framework for in-flight action editing in chunk-autoregressive video world models. To avoid the state misalignment caused by directly switching action conditions and the computational overhead of restarting sampling, the authors introduce Counterfactual State Transport (CST), which trains a lightweight corrector to map interrupted solver states toward matched counterfactual states. Two variants are developed for whole-chunk retargeting (CSTR) and within-chunk action transitions (CSTT). Experiments on minWM and HY-WM1.5 demonstrate improved fidelity and action responsiveness compared with existing alternatives.

Critical strengths
  1. The paper targets on a well-motivated and practically relevant problem, identifying an important challenge in interactive video world models: responding to action updates during the generation of an active chunk. The proposed problem formulation addresses a practical limitation of existing chunk-autoregressive generation pipelines.
  2. The experiments cover two different world model backbones and compare against several reasonable alternatives, achieving remarkable improvements in rollback-relative fidelity and action-following responsiveness.
Critical weaknesses
  1. Limited methodological novelty. The core method largely consists of training a lightweight residual network to regress between matched intermediate solver states. While the problem formulation and counterfactual data construction are meaningful, the underlying learning objective and correction mechanism are relatively straightforward. It remains unclear whether this approach offers substantial methodological novelty beyond existing diffusion editing and state correction techniques.
  2. Limited evaluation of generalization to more complex action spaces and settings. The experiments primarily consider transitions among three camera motion commands, with a separate corrector trained for each backbone and editing variant. This leaves open questions about generalization to more diverse or continuous camera actions, robot EEF/joint actions, and more complex interactive environments. Demonstrating broader applicability as a general solver-state transport framework would strengthen the methodological contribution; otherwise, the approach appears closer to an effective inference engineering solution for a relatively narrow class of interactive world models.
  3. Insufficient justification of the counterfactual state correction design. While the results demonstrate that learned state correction is effective, the paper provides limited justification for its specific design choices. In particular, the method directly regresses intermediate solver-state residuals, without comparing against alternative formulations, such as correcting the flow/velocity field, or predicting the counterfactual clean latent. Moreover, the unmatched-supervision ablation uses targets from different prompts, which is a relatively weak baseline for demonstrating the benefits of the proposed counterfactual supervision.
Questions for authors
  1. What information is essential for predicting the counterfactual solver state? The corrector takes the interrupted state, initial state, previous chunk, and both action conditions as inputs. Why are the initial state and previous chunk necessary if the interrupted solver state already contains information accumulated from earlier denoising steps? Is the counterfactual state generally recoverable from the interrupted state and action change alone, or is additional historical information fundamentally required?
  2. What causes the substantial degradation when the action update arrives at later solver steps? For HY-WM1.5, rollback-relative LPIPS increases significantly at r=3 (0.0831->0.2694).

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.