acceptodds
Under review as a conference paper at ICLR 2027

InWorld: An Instant-Interactive Multimodal World Model for Autonomous Driving

Abstract

World models have become a scalable alternative to real-world interaction for policy learning and evaluation in autonomous driving. A high-performance driving world model must satisfy three core requirements simultaneously: cross-modal consistency, real-time generation, and pose-controllable interaction. However, existing methods struggle to fulfill all these criteria, frequently requiring trade-offs among them. We present InWorld, an interactive driving world model that synchronously generates cross-modally consistent images and LiDAR point clouds in real time. To faithfully capture intrinsic LiDAR sensor characteristics, we design a dedicated LiDAR VAE optimized for range image representation, which integrates horizontal circular convolution and a differentiable ray-drop module. For cross-modally consistent generation, we present a layout architecture equipped with epipolar masked attention, alongside image-LiDAR consistent Pl\"ucker coordinates encoding. To enable controllable interactive modeling, we propose temporal decay KV caching and embed it into the teacher-student pipeline via distribution matching distillation. Extensive experiments on the nuScenes dataset demonstrate that InWorld achieves state-of-the-art performance in both generation efficiency and cross-modal alignment, while the student enables interactive generation at a 10 Hz streaming inference rate.

Primary area:
applications to robotics, autonomy, planning
Keywords:
World Model, Multimodal Generation, LiDAR Generation

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_skwxOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The authors propose a multimodal generative model with a clear structure, but the novelty of each modality is insufficient. For example, the use of circular convolution and ray-drop for LiDAR range images are existing and mature solutions.

Critical strengths
  1. It can generate LiDAR and RGB simultaneously and reaches 10 fps, which is relatively fast.
  2. Using cross-modal masks with 3D geometric constraints to suppress spurious alignment is intuitive.
Critical weaknesses
  1. The camera setup uses only 3 views and does not achieve full coverage.
  2. The authors claim closed-loop evaluation, but there are no true closed-loop evaluation experiments.
  3. The title is “xxx for Autonomous Driving,” but the work ultimately does not extend to trajectory planning; it is only data generation for AD scenes.
  4. Whether the use of independent short clips affects practical utility, e.g., cumulative error and related information cannot be analyzed.
  5. Insufficient comparison methods.
  6. The ablations are conducted on a 20-epoch checkpoint that has not fully converged, so the conclusions can only indicate relative trends.
  7. The details of the MoE are unclear.
  8. How exactly is the temporal decay applied?
  9. The mask construction is unclear: if objects move, is the mask still correct? What are the details of the Plücker for LiDAR method, and how is it specifically implemented?
Questions for authors

Please refer to weaknesses.

Reviewer_c5YkOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

In this paper, the authors introduce a real-time, interactive multimodal world model named InWorld that generates both images and LiDAR points. To align LiDAR and image latents within a shared latent space, the authors design a new LiDAR VAE. They also introduce a temporal decay KV caching mechanism for the AR student model to enable real-time interactive generation. And the experimental results show that InWorld achieves competitive performance in multimodal generation on the nuScenes dataset.

Critical strengths

The authors use a teacher–student distillation pipeline to obtain an efficient autoregressive student model for real-time, interactive world modeling. I think real-time interaction is a valuable research direction for world models. In addition, the authors use a LiDAR VAE to align LiDAR and visual latents in a shared latent space, which I consider an intuitive design for driving world models. The experiments also show that InWorld achieves higher generation speeds while maintaining comparable visual quality in generated videos and geometric quality in generated LiDAR point clouds in Table 1.

Critical weaknesses

I suggest improving the clarity of Figure 2. The images of the teacher model and discriminator are of low quality, affecting the readability of the overall figure.

In addition, the authors mention that the model generates videos and LiDAR points autoregressively. Does the model support long-video generation? How long can it continue generating videos while maintaining generation quality and temporal consistency? I noticed that the current experiments only evaluate shot videos of up to 20s, so I am curious about the generation quality over longer durations.

Questions for authors

Do the authors plan to support the generation of additional modalities, such as occupancy or semantic map? I think these modalities are important for using InWorld in practical real-time interactive applications.

In addition, I noticed that the model uses future ego poses as control conditions. During long-video or LiDAR pointcloud generation, is the appearance and geometry of other vehicles determined solely by the first frame? Can their positions and motion trajectories also be controlled?

Reviewer_vmRDOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper presents InWorld, a driving world model that jointly generates multi-view camera video and LiDAR point clouds under ego-pose conditioning, targeting real-time interaction. It is trained in two stages. The backbone is Wan2.1-1.3B with high-noise and low-noise MoE experts. LiDAR is converted to a range image with each row repeated 4×. The image VAE is reused with a frozen encoder; the decoder is fine-tuned with horizontal circular convolution and a differentiable ray-drop head. Image and LiDAR latents share one concatenated layout, cross-view/cross-modal geometry is enforced by epipolar-masked attention, and pose is injected via Plücker-ray encodings. The student is distilled with Self-Forcing + DMD, using a KV cache that keeps the first frame permanently plus a sliding window of recent frames. On nuScenes, the teacher reaches FID 3.07 and the best VSC/DAS among compared methods; the student runs at 10.2 FPS on a single H800 with some quality loss.

Critical strengths
  1. Few works jointly address cross-modal consistency, real-time generation and pose control for driving simulation. Real-time joint image–LiDAR generation with an autoregressive student is, to my knowledge, rarely explored.
  2. Freezing the image VAE encoder and decoding LiDAR from the same latent space is simple and well motivated, and Appendix E.2 supports it with negative results from heterogeneous VAEs. Circular convolution and ray-drop target real LiDAR properties.
  3. Strong cross-modal consistency. VSC 0.852 and DAS 1.54 clearly improve on X-Drive; Fig. 4 projections are convincing.
Critical weaknesses
  1. All metrics come from independently generated short clips without autoregressive rollout, so long-horizon drift of the streaming student is never measured. There are no closed-loop or downstream experiments, and pose control is tested only on ground-truth trajectories. "Real-time" is reported as average FPS (10.2, below the ~12 Hz camera rate of nuScenes), with no per-chunk latency.
  2. All baselines are quoted from papers with different settings (3 vs. 6 cameras, resolution, clip length, metric protocols), and FPS is not measured on the same hardware. The teacher trails X-Drive and RangeLDM on LiDAR MMD/JSD and Genesis/OmniNWM on FVD, the student is mid-to-low, and UniDriveDreamer, whose pipeline InWorld follows, is not compared.
  3. Most components are adopted from prior work (range-image VAE tricks, UniDriveDreamer's layout, Wan2.2-style MoE, Plücker conditioning, Self-Forcing + DMD). The epipolar attention resembles X-Drive's cross-modal epipolar module, and "temporal decay KV caching" is essentially attention sink + sliding window, with no decay.
  4. The KV cache and distillation design (contribution 3) is not ablated, and the other ablations use an unconverged 20-epoch checkpoint with small differences and no seeds. There is no image-only variant to show that joint generation helps. In Table 3 the student has the worst R-Err and T-Err of all methods, and its architecture is never described.
  5. The main text and appendix disagree on where epipolar masking is applied. The MoE is not described in Sec. 3, Eq. 3 mixes ε-prediction with a flow-matching teacher and calls the fake-score network a "discriminator", and camera–LiDAR time synchronization is not discussed.
Questions for authors
  1. What clip length and how many samples are used for FID/FVD/MMD? Are the Fig. 3 frames (40–80) student autoregressive rollouts? How do quality and consistency degrade with rollout length?
  2. What are first-frame and per-chunk latency, and the runtime breakdown (DiT, VAE decode, ray-drop, unprojection)? Does one "frame" in 10.2 FPS mean 3 views + 1 LiDAR sweep? At what frame rate are generated sequences played back, given nuScenes cameras run at about 12 Hz?
  3. What is the student architecture and where does the "50% reduction in model size" come from? How many denoising steps and what chunk size? What is the teacher's actual FPS?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.