acceptodds
Under review as a conference paper at ICLR 2027

CAST: Control-Aware Sparse Transport for Efficient Interactive World Models

Abstract

Interactive world models must respond quickly to controls while preserving scene consistency. Existing acceleration methods can miss heterogeneous control responses and spatial transport when recovering skipped features. We observe that interaction-induced feature changes correlate with approximation error, while low-frequency interpolation errors are phase-sensitive and show more predictable phase progression. These findings motivate CAST, a reconstruction-coupled inference framework. CAST selects anchors by interaction sensitivity and cross-layer coverage, reconstructs skipped residuals with frequency- and confidence-aware Phase-Aware Reconstruction (PAR), and coordinates historical KV routing according to downstream reconstruction responsibility. On Matrix-Game 3.0 and HY-World 1.5, CAST achieves and speedups, respectively, with the highest VBench scores among compared methods. It also leads non-native baselines on seven and six of thirteen WorldMark dimensions, balancing speed, quality, and interactive responsiveness under real-time control.

Primary area:
generative models
Keywords:
world models, video diffusion models, inference acceleration, temporal sparsity, sparse attention, feature approximation

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_smSvOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
1
Soundness
3
Read review
Paper summary

This paper investigates training-free inference acceleration for interactive world models. The authors propose CAST, which selects anchors by interaction sensitivity and cross-layer coverage, reconstructs skipped residuals with frequency, and coordinates historical KV routing. On the Matrix-Game 3.0 and HY-World 1.5, the method achieves speedups of 2.15× and 3.48× respectively.

Critical strengths
  1. The method coherently combines control-aware frame selection, phase-aware reconstruction, and reconstruction-coupled sparse attention.
  2. Results on WorldMark and VBench demonstrate a favorable trade-off between visual quality and inference efficiency relative to the evaluated baselines.
Critical weaknesses
  1. The dense layout and unclear notation make parts of the paper difficult to follow. For example, Section 3.1 would benefit from a reminder of what RR denotes, while notation such as RaR_a around line 181 omits the layer superscript without clearly explaining the change.
  2. The benefits for actual interaction responsiveness require further validation. The delay from a user's control input to the first visible response depends on control handling, chunk scheduling, and decoding. End-to-end latency measurements that include VAE decoding, together with response measurements for control changes during generation, would better establish the method's impact on real-time interaction.
Questions for authors
  1. Could you provide more detailed experimental configurations, particularly the exact settings used for the uniform/interleaved configuration in Table 3?
  2. Could you provide measurements of actual interaction-response latency?
Reviewer_JjCgOpenReview ↗
Rating
3
Confidence
2
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces CAST. It combines control-aware anchor selection, phase-aware reconstruction of skipped residuals, and reconstruction-coupled historical KV routing. Its main conceptual contribution is to model the dependency between anchor approximation and downstream frame reconstruction, rather than treating frame sparsity and historical attention independently. Experiments on Matrix-Game 3.0 and HY-World 1.5 report 2.15× and 3.48× speedups, with competitive VBench and WorldMark results.

Critical strengths
  1. CAST recognizes that an anchor’s approximation error also affects the skipped frames reconstructed from it. The resulting KV-routing objective is supported by a controlled experiment: with identical anchors and KV budgets, coupling reduces anchor and interval errors by 6.9% and 10.1%, respectively.

  2. The paper evaluates two interactive world-model backbones using control, memory, visual-quality, and runtime measurements. On Matrix-Game, the controlled ablations show consistent benefits from CAFS and PAR, while the runtime analysis includes routing and reconstruction overhead. These results make the reported speedups more credible than operation-count estimates alone.

Critical weaknesses

1.CAST achieves the highest speedup and VBench score on both backbones, but the broader claim of the best overall quality–efficiency balance is not fully established by single operating-point comparisons. Some slower baselines provide better native-reference fidelity, such as TeaCache on Matrix-Game and Light Interaction on HY-World. This concern does not negate CAST’s runtime advantage, but it limits the strength of the overall comparative claim.

2.Much of the novelty lies at the framework level rather than in the individual operators. CAFS extends sensitivity-based allocation with control responses and coverage debt, PAR adapts phase-based interpolation through frequency- and confidence-aware gating, and the pooled KV routing builds on Light Interaction.

Questions for authors

1.Could you provide quality–latency curves or matched-speed comparisons for TeaCache and Light Interaction?

2.Could you report multi-seed results or bootstrap confidence intervals for the main quality comparisons? Several highlighted margins, particularly the HY self-comparison differences over TeaCache, are quite small.

Reviewer_zK5mOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

CAST accelerates interactive world models by coordinating frame selection, residual reconstruction, and historical attention. It selects frame anchors using control sensitivity and refresh age, reconstructs skipped residuals through phase-aware interpolation or extrapolation, and jointly refines neighboring anchors' historical KV selections. Experiments on Matrix-Game 3.0 and HY-World 1.5 report speedups of 2.15× and 3.48×, respectively.

Critical strengths

A Coherent Approach to Few-Step World Models.** The paper addresses a useful setting. With only three or four denoising evaluations, these models leave limited room for caching across steps. Reducing computation within each evaluation is a practical direction. The three components fit together well. Anchor selection determines which frames support reconstruction, while historical routing considers how errors at those anchors affect neighboring frames. This gives the method a clear motivation beyond combining separate acceleration techniques.

Useful Controlled Experiments and Measured Speedups.** The evaluation includes control and memory metrics, matched-budget ablations, and a runtime breakdown. On Matrix, coupled routing reduces interval reconstruction error by 10.1% over independent routing at the same KV budget, with 31 ms of additional latency per chunk. The controlled selector and reconstruction comparisons in Table 7 also help explain the gains.

Critical weaknesses

PAR Needs Clearer Evaluation Under Difficult Motion and Boundary Extrapolation.PAR estimates transport from endpoint phases and uses agreement across channels to control its strength. As discussed in Appendix A.2, strong agreement can still occur when the phase is ambiguous. Fast motion, occlusion, and direction changes are therefore important cases for this method. Boundary reconstruction deserves particular attention because the selector does not require the first or last frame to be an anchor. HY retains only two of four latent frames, so the selected pair may need to extrapolate to the remaining positions. The aggregate reconstruction ablations are encouraging, but do not show where PAR improves recovery and where it becomes unreliable.

The Benefit of the Coupling Objective Needs Further Isolation.The fixed-budget comparison with independent routing supports coordinating neighboring masks. The remaining question is how much the proposed reconstruction-risk objective adds over a simpler coordination rule. The derivation freezes the phase transport and replaces full residual perturbations with pooled omission scores. Its resulting coefficients depend on the anchor spacing, while the realized phase and confidence no longer appear in the routing objective. This makes a simpler diversity-based rule a relevant comparison, for example penalizing neighboring anchors for omitting the same important historical blocks.

Questions for authors

Q1. Could the authors separate interpolation and extrapolation results, and report reconstruction errors for large displacements, action reversals, and occlusions? It would be useful to know whether the confidence score identifies cases where PAR performs worse than linear reconstruction.

Q2. At the same frame and KV budgets, how does the proposed objective compare with a simple shared-omission penalty that ignores reconstruction distance? Testing both rules with linear reconstruction and PAR would help clarify which gains come from general mask coordination and which depend on the reconstruction design.

1 comment

Dear Authors,

Thank you for the detailed response and the additional experiments. The comparison with linear reconstruction addresses my question about when PAR helps. The breakdown is especially useful in showing smaller gains for boundary extrapolation and a slight increase in error under severe occlusion.

The comparison with the tuned distance-free rule also addresses my concern about the coupling objective. It shows how much simpler coordination can achieve and what the proposed reconstruction weighting adds under the same budget. I would welcome both comparisons in the revised paper.

I am keeping my rating at 3 (Weak acceptance) and my confidence at 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.