acceptodds
Under review as a conference paper at ICLR 2027

Into the LidarVerse: A Real-Time Unified LiDAR World Model Across Tasks and Sensors

Abstract

We present LidarVerse, a real-time unified LiDAR world model for sensor simulation across tasks and sensors. Existing LiDAR simulators largely follow one of two complementary paradigms: reconstruction-based methods preserve the geometry of a captured scene but require lengthy per-scene optimization, whereas generative methods amortize learning across data but lack the consistency needed for long-horizon roll-outs. LidarVerse brings both paradigms into a single, unified conditional generative framework. A universal, fully attention-based tokenizer maps LiDAR range sequences with different scanning patterns into a shared latent space, where a bidirectional diffusion teacher learns a common interface for conditioning on a LiDAR prefix, an HD-Map, a persistent spatial cache, and optionally, text. Varying the available scene evidence turns the same model into a generator of new worlds, a point-cloud forecaster, or a generalizable resimulator of observed scenes without per-scene optimization. We distill the teacher into a four-step autoregressive causal student and maintain both a rolling temporal K/V cache and a world-aligned spatial cache, enabling persistent long-horizon roll-outs with real-time inference at 11 FPS on a single H100 GPU. LidarVerse achieves state-of-the-art performance on most reported metrics across all three evaluation protocols, while its tokenizer supports 32-, 64-, and 128-beam LiDAR and generalizes to an unseen scanning pattern. These results demonstrate that generative and reconstruction-grounded LiDAR simulation can be realized within one model, providing the core capabilities for future closed-loop sensor simulation. Anonymous project page: https://anonymous11693.github.io/website/.

Primary area:
generative models
Keywords:
LiDAR, World Models, 4D Generation, Generalizable Reconstruction

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_33zCOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents LidarVerse, a conditional LiDAR world model intended to unify LiDAR generation, forecasting, and reconstruction-grounded resimulation. A LiDAR-native range-image tokenizer based on neighborhood attention maps LiDAR sequences into a shared latent space without temporal downsampling. A 2B-parameter bidirectional flow-matching teacher is trained with independently sampled conditions including a LiDAR prefix, an HD-map and actor layout, and a persistent spatial cache (PSC). The teacher is distilled into a four-step block-causal student using self-forcing distribution matching. During causal rollout, a temporal K/V cache retains recent dynamics, while a rolling world-aligned voxel cache stores geometry generated in previous blocks. The paper evaluates tokenizer reconstruction, 129-frame autoregressive generation, ego-action-conditioned generation, point-cloud forecasting, and sparse-input resimulation. Separately fine-tuned bidirectional variants are used for cross-sensor generation and intensity modeling.

Critical strengths
  1. Well-motivated LiDAR-specific design. The attention-based range-image tokenizer handles depth discontinuities, while the temporal K/V cache and world-aligned PSC provide complementary short- and long-term memory.

  2. Broad and strong empirical evaluation. The paper covers long-horizon generation, forecasting, resimulation, multiple sensors, intensity, and efficiency. The four-step causal model achieves a strong quality-throughput trade-off against the included baselines.

Critical weaknesses
  1. The unified-model claim is overstated and closely related work is missing. Cross-sensor and intensity results use separately fine-tuned bidirectional teachers rather than the four-step causal student, while unseen PandaSet transfer is only qualitative. More importantly, LiDAR-GS++ already combines LiDAR reconstruction with diffusion priors for extrapolated resimulation. LiSTAR covers reconstruction, prediction, and conditional generation. OmniLiDAR study cross-sensor/domain LiDAR generation or transfer. LiDAR-GS, GS-LiDAR, and XSIM are also directly relevant real-time resimulation baselines. These works substantially narrow the claimed novelty.

  2. The resimulation evaluation uses privileged annotations and does not test genuine novel trajectories. The frozen-dynamic PSC requires annotated object poses during inference, but it is unclear whether competing methods receive the same information. Quantitative evaluation only predicts held-out frames along the recorded Waymo trajectory, rather than testing the large off-trajectory viewpoint changes advertised by the paper.

Questions for authors
  1. Which exact checkpoint supports each task/sensor combination? Can the same four-step causal student perform generation, forecasting, resimulation, 32/64/128-beam generation, unseen-pattern transfer, and intensity modeling without further fine-tuning?
  2. Please add discussion or comparison with the closely related baseline works LiDAR-GS++ (arXiv:2511.12304), LiSTAR (arXiv:2511.16049), OmniLiDAR (arXiv:2605.13815), Towards Foundational LiDAR World Models (arXiv:2506.23434), LiDAR-GS (arXiv:2410.05111), GS-LiDAR (arXiv:2501.13971), and XSIM (arXiv:2602.05617).
  3. During resimulation, are ground-truth future 3D actor boxes or poses supplied at inference time? Are LiDAR4D, LiDAR-RT, and SplatAD provided with the same object tracks?
  4. Could you evaluate resimulation quality under genuinely off-trajectory viewpoints, such as lateral translations or trajectories that revisit previously observed regions?
  5. Regarding the efficiency problem, can a new ego action be injected every frame, or only once per four-frame block? Does the 21-frame forecasting configuration remain above 10 FPS?

The lack of discussion and comparison on the baseline prevents me from giving a higher rating at this stage. I will consider improving my rating after the complete experiment.

6 comments

Thank you for the thoughtful review and valuable feedback. We are pleased that our design is “well-motivated” and that our experimental designs are “broad” and “strong”.

Q1: Overstated Unified Model Claims and Discussion with SOTA models

A1: We would like to clarify that our unified model is targeting multiple different tasks being supported within a single checkpoint and a single inference interface. To our understanding, we are the first unified model that can achieve all 3 tasks of long-horizon 4D LiDAR generation, LiDAR forecasting, and generalizable LiDAR resimulation. We provide the discussion with the requested baseline in the table below.

Method4D Generation¹ForecastingGeneralizable resimulation
LiDAR-GS++ [1]✘ (single-frame only)✘✘ (per-scene optimization)
LiSTAR [2]✓✓✘
OmniLiDAR [3]✘ (single-frame only)✘✘
Towards Foundational LiDAR World Models [4]✓✓✘
LiDAR-GS [5]✘✘✘ (per-scene optimization)
GS-LiDAR [6]✘✘✘ (per-scene optimization)
XSIM [7]✘✘✘ (per-scene optimization)
LidarVerse✓✓✓

Notably, LiDAR-GS++ can only generate single-frame and cannot generate long-horizon LiDAR from scratch. LiSTAR’s reconstruction evaluation describes tokenizer’s reconstruction evaluation, without evidence of any capability to support resimulation task. OmniLiDAR studies cross-sensor/domain LiDAR but is limited to single-frame and does not support re-simulation. XSIM is a per-scene resimulation method that cannot support any generative task. In this sense, we are the only method capable of those three LiDAR-specific tasks. We will update the paper with this discussion.

Reference:

[1] Chen et al., “LiDAR-GS++: Improving LiDAR Gaussian Reconstruction via Diffusion Priors,” AAAI 2026.

[2] Liu et al., “LiSTAR: Ray-Centric World Models for 4D LiDAR Sequences in Autonomous Driving,” ECCV 2026.

[3] Liu et al., “OmniLiDAR: A Unified Diffusion Framework for Multi-Domain 3D LiDAR Generation,” arXiv preprint, 2026.

[4] Liu et al., “Towards Foundational LiDAR World Models with Efficient Latent Flow Matching,” NeurIPS 2025.

[5] Chen et al., “LiDAR-GS: Real-time LiDAR Re-Simulation using Gaussian Splatting,” CVM 2026.

[6] Jiang et al., “GS-LiDAR: Generating Realistic LiDAR Point Clouds with Panoramic Gaussian Splatting,” ICLR 2025.

[7] Patakin et al., “Unified Sensor Simulation for Autonomous Driving,” IJCAI 2026.

Q2: Privilege Annotations in Resimulation

A2: We would like to clarify that existing baselines, including LiDAR-RT and SplatAD, also use ground-truth future 3D actor boxes and boxes at inference time. LiDAR4D does not, as intended by their formulation, and thus resulting in lack of actor’s controllability and worse performance compared to ours.

We also want to point out that the usage of ground-truth actor boxes are standard practice in autonomous driving resimulation methods [8,9,10,11,12,13] and thus are not considered as privileged information. We will make this clarification clearer in the experiment setting for resimulation.

Reference:

[8] Yang et al., “UniSim: A neural closed-loop sensor simulator,” CVPR 2023

[9] Tonderski et al., “NeuRAD: Neural Rendering for Autonomous Driving,” CVPR 2024

[10] Yan et al., “Street Gaussians: Modeling Dynamic Urban Scenes with Gaussian Splatting,” ECCV 2024

[11] Wu et al., “Dynamic LiDAR Re-simulation using Compositional Neural Fields,” CVPR 2024

[12] Zhou et al., “LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation,” CVPR 2025

[13] Chen et al., “OmniRe: Omni Urban Scene Reconstruction,” ICLR 2025

Q3: Off-trajectory evaluation

A3: To directly test the off-trajectory in conventional sense, we render at different lateral shifts (1m, 2m, and 3m) to the left and right and compute FUD and MMD, following existing methods such as [14,15,16,17]. We use the same evaluation setting as Table 3. We also include comparison with LiDAR-GS as requested, and will include more baselines afterwards. We observe that our methods outperform all resimulation baseline on off-trajectory evaluation, which supports our claim of quality under large off-trajectory viewpoint changes.

Each cell below reports FUD / MMD × 10⁵ (lower is better for all)

Method±1 m±2 m±3 m
LiDAR-GS25.56 / 5.2327.21 / 7.3629.31 / 9.88
LiDAR4D34.50 / 6.3537.12 / 7.6039.76 / 9.18
LiDAR-RT39.25 / 7.3041.72 / 8.7645.32 / 10.23
SplatAD75.94 / 4.4677.82 / 6.1680.06 / 8.36
Ours-Bidir11.73 / 4.1312.96 / 5.8713.79 / 8.15
Ours-SF-DMD25.73 / 4.4326.24 / 5.9227.41 / 7.86

Reference:

[14] Want et al., “FreeVS: Generative View Synthesis on Free Driving Trajectory,” ICLR 2025

[15] Yan et al., “StreetCrafter: Street View Synthesis with Controllable Video Diffusion Models,” CVPR 2025

[16] Che et al., “GenRe: Diffusion-guided Generalizable Enhancer for Urban Scene Reconstruction,” ICRA 2026

[17] Chen et al., “LiDAR-GS++:Improving LiDAR Gaussian Reconstruction via Diffusion Priors,” AAAI 2026

Q4: Which checkpoint can support which task

A4: While cross-sensor transfer and intensity modeling are treated as extension in our paper, a four-step causal student derived from that checkpoint can perform generation, forecasting, resimulation, 32/64/128-beam generation, unseen-pattern transfer, and intensity modeling without further fine-tuning. We deliberately only include 1 dataset in our quantitative experiments to keep the training set fair with the baselines. We plan to release such a checkpoint publicly.

Q5: Efficiency Clarification

A5: If not pre-determined, new ego trajectory can only be injected once after 4-frame generation, resulting in latency of action-conditioning. We provide the full FPS and latency for our 3 modes below.

ModeEnd-to-end FPSBlock (4-frame) latency (ms)
Generation11.18337.78*
Forecasting10.04405.25
Resimulation12.26319.53
Reviewer_RkYpOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper presents LidarVerse, a conditional LiDAR model supporting generation, point-cloud forecasting, and resimulation from sparse observations. It combines a LiDAR range-image tokenizer, a bidirectional diffusion teacher, and a four-step causal student trained through self-forcing distribution matching. A temporal K/V cache provides recent context, while an accumulated spatial cache supplies geometric conditioning. Experiments on Waymo evaluate all three tasks, with generation throughput around 11 FPS on an H100. Separate extensions adapt the tokenizer and bidirectional teacher to multiple sensors and intensity. Generalization to an unseen scanning pattern is demonstrated qualitatively through tokenizer reconstruction.

Critical strengths
  1. The resimulation results make the unified formulation worthwhile. With four observed sweeps per 20-frame chunk, the causal model requires no per-scene optimization and performs particularly well on the top-20 travel-distance subset. Its aggregate CD is 0.294, compared with 0.542 for SplatAD and 0.709 for LiDAR-RT (Table 3). This addresses a practical limitation of reconstruction-based simulation: fitting a separate representation for every captured scene. The student does not dominate on the full validation set, but the quality–setup-cost trade-off is still compelling.

  2. The tokenizer improvement is supported by a useful controlled experiment. At matched compression and latent dimensionality, replacing the convolutional backbone with neighborhood attention reduces reconstruction range error from 0.297 to 0.072 and CD from 0.278 to 0.115, while reducing tokenizer compute from 0.575 to 0.250 TFLOPs/frame (Table 4). Generation FUD also improves from 26.54 to 18.17. These results connect the representation design to both geometric reconstruction and downstream generation quality.

Critical weaknesses
  1. The experiments establish local temporal coherence more convincingly than persistent spatial memory. The TTCE/CTC offsets in Table 2 extend only to four frames, or 0.4 seconds at 10 Hz, within the nine-frame K/V context. Figure 8 likewise evaluates the PSC using CTC@4. Windowed FUD measures distributional degradation, but does not determine whether the model recovers the same geometry after leaving and returning to an area. This matters for identifying the PSC’s contribution. LiDARCrafter, §3.3, already conditions generation on reprojected first-frame and previous-frame geometry. The present comparison against null geometry does not isolate the benefit of maintaining a persistent, updated map over such shorter geometric histories. Appendix A.9 also reports that the least selective cache achieves the best temporal-consistency scores despite worse fidelity. A return-to-view test would therefore be more informative than another local consistency metric. The distinction between local coherence and scene recall is also explicitly evaluated in WorldTrace’s LoopBench. The project’s turning demonstrations are useful, but do not resolve this particular uncertainty.

  2. The claim that all three tasks operate at the native 10 Hz rate needs measurements under their actual configurations. The main generation results use a nine-frame K/V cache, whereas forecasting uses 21 frames (Appendix A.5). Figure 8 places the 21-frame configuration below 10 FPS under the generation protocol. This is not a forecasting speed measurement, but it means the reported 11 FPS cannot simply be carried over to that configuration. Moreover, generating four frames jointly makes steady-state throughput different from response latency to a new ego action. Self Forcing, which this work builds on, reports throughput and first-frame latency separately. The corresponding distinction is important here because interactive sensor simulation is a central motivation.

Questions for authors
  1. Validation of the PSC: Can you provide a return-to-view comparison after the relevant observations have left the rolling K/V context, measuring static-surface consistency with and without the PSC? A matched first-frame/previous-frame reprojection baseline would help establish whether the gain specifically comes from persistent accumulation and updating.

  2. Runtime clarification: What are the end-to-end throughput and block latency for generation, forecasting with the 21-frame cache, and resimulation? Please specify whether timing includes decoding, PSC updating, projection, and condition encoding. When can a newly received ego action first affect an output sweep within the four-frame generation scheme?

2 comments

Thank you for the thoughtful review and valuable feedback. We are pleased that our unified framework is compelling and our tokenizer is well-received.

Q1: Return-to-view Validation of the PSC

A1: We evaluate return-to-view consistency across 10 scenes by editing the recorded ego trajectory to include retracing the route and replaying the original trajectory at identical sensor poses to simulate revisiting. The evaluation follows our Table 2's interactive action-conditioned generation setting. We present the result below.

MethodChamfer ↓Range L1 ↓AbsRel ↓
LaGen5.99111.3790.4781
LiDARCrafter3.0589.9900.5859
Ours (Latest warping)5.3124.9480.2887
Ours (First + latest warping)1.1493.0860.1448
Ours (PSC)0.4961.0590.0402

Using PSC improves return-to-view performance across all metrics, and significantly outperforms the requested baseline (first + latest warping) and other methods (LaGen, LiDARCrafter), suggesting that keeping and updating a persistent memory map is crucial for long horizon generation.

Q2: Runtime Clarification

A2: We provide the end-to-end throughput and block latency for all 3 modes (generation, forecasting with the 21-frame cache, and resimulation), measured with a single H100. We also clarify that our timing is end-to-end, thus including encoding/decoding, DIT denoising, and PSC updating/warping. Note that for forecasting, we do not need to encode the HDMap (Figure 8's experiment requires HDMap encoding), thus bringing our forecasting FPS to be over 10FPS.

ModeEnd-to-end FPSBlock (4-frame) latency (ms)
Generation11.18337.78
Forecasting10.04405.25
Resimulation12.26319.53

Further detailed breakdown of how each component contributes to the final runtime are provided below.

ComponentGeneration (%)Forecasting (%)Resimulation (%)
DiT sampling53.6956.9058.19
VAE decoding14.1415.4013.59
Clean K/V commit13.5610.9511.85
VAE encoding9.697.817.63
PSC warping/projection4.365.876.70
PSC update3.032.010.00
Output XYZ projection0.220.140.31
Noise initialization1.220.781.61
Block setup0.020.070.00
Unattributed0.060.070.12
Total100100100

We look forward to further discussions if you have any further questions or concerns.

We thank the reviewer for taking time in checking our responses. We hope our answers and additional results address your concerns well.

We have updated the paper with the requested experiment along with qualitative results of the return-to-view experiment (Appendix A.12) and runtime breakdown (Appendix A.13).

Please let us know if you have more questions or concerns.

Reviewer_reqBOpenReview ↗
Rating
3(+1)
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes LidarVerse, a real-time unified LiDAR world model. First, a universal tokenizer maps range image sequences from different LiDAR sensors into a shared latent space. Then, a bidirectional teacher model integrates multiple conditioning signals, including historical point clouds, HD maps, spatial caches, and text. Furthermore, a four-step autoregressive student model is developed through knowledge distillation. By incorporating temporal K/V and spatial caches, the model maintains long-horizon consistency while achieving real-time inference at 11 FPS on a single H100 GPU.

Critical strengths

The authors propose a simple and unified LiDAR world model that can adapt to multiple tasks, including point-cloud generation, point-cloud forecasting, and scene resimulation, by incorporating different conditioning information.

Critical weaknesses

The main weaknesses of this work lie in the insufficient clarity of its methodological novelty and the limited experimental validation. For example, the fundamental differences between the proposed method and existing LiDAR world models, and comparisons with recent state-of-the-art methods are lacking.

Questions for authors
  1. The main difference between Generalizable LiDAR Resimulation and generative models seems to lie in the choice of conditioning information. Could the authors further clarify the fundamental methodological differences between them? Additionally, how would existing LiDAR world models perform if applied to this task?
  2. The authors adopt Neighborhood Attention for point-cloud compression, but the ablation studies only compare it with standard Attention and Convolution. Existing methods, such as the Mamba-based scanning strategy proposed in GEM [1], can also effectively reduce computational complexity. It is suggested that the authors include additional comparisons with such methods.
  3. Why do the authors conduct experiments only on the Waymo dataset, rather than evaluating the proposed method on other datasets, such as nuScenes and KITTI Odometry, which have relatively well-established evaluation protocols for generation and forecasting tasks?
  4. Considering the substantial computational capabilities of the H100, is 11 FPS sufficient to support the paper's claim of real-time performance? It is suggested that the authors further evaluate the inference speed on consumer-grade GPUs.
  5. The paper lacks comparisons with several representative state-of-the-art methods for LiDAR generation and forecasting, such as GEM [1], U4D [2], UniScene [3], and Copilot4D [4].

[1] GEM: Generating LiDAR World Model via Deformable Mamba, CVPR 2026. [2] U4D: Uncertainty-Aware 4D World Modeling from LiDAR Sequences, CVPR 2026. [3] UniScene: Unified Occupancy-centric Driving Scene Generation, CVPR 2025. [4] Copilot4D: Learning Unsupervised World Models for Autonomous Driving via Discrete Diffusion, ICLR 2024.

3 comments

We thank the reviewer for the thoughtful review and valuable feedback.

Q1: Fundamental differences between generation and resimulation

A1: The fundamental difference lies in their input and target task:

  • Resimulation methods rely on having captured LiDAR scan and object poses to perform novel view synthesis.
  • Generation methods rely on having conditioning signals (e.g., images, text, or HDMap) to generate LiDAR from noises.
  • Forecasting methods aim to model future conditioning only on past input frames.

Our novelty lies in the fact that, to our understanding, we are the first unified model that can achieve all 3 tasks of long-horizon LiDAR generation, LiDAR forecasting, and generalizable LiDAR resimulation within a single checkpoint and inference interface.

Q2: Other LiDAR world models’ performances on re-simulation

A2: As requested, we present the results of applying other LiDAR world models on re-simulation tasks. Since both LaGen and LiDARCrafter can take warping geometry as input, we use the same geometric conditioning for all methods, effectively mimicking re-simulation in our experiments. We use the same setting as our Table 3 of the main paper. The results are as follows:

MethodRange L1 ↓Chamfer ↓
LaGen3.0381.097
LiDARCrafter2.6360.696
Ours (bidir)1.5360.237
Ours1.7870.294

(best number is bolded, second best is italicized)

We observe that applying other LiDAR world models fails catastrophically at this task, showing that it is nontrivial to apply generation model for resimulation task and highlighting the necessity of a unified conditioning pipeline during training and inference.

Q3: Ablation studies against Mamba-based tokenizer

A3: Based on the reviewer’s suggestion, we conducted an experiment to compare with Mamba-based backbone for both reconstruction and generation. We use matching data and training budget compared with the rest of the ablation experiment, and the experimental setting strictly follows Table 5 in the paper. All tokenizers use the same compression rate and latent dimension size, and capacity-matched DIT. We report the result below:

BackboneRecon. L₁ ↓CD ↓AbsRel ↓FUD ↓CTC@4 ↓TTCE@4 ↓TFLOPs/frame ↓
Conv.0.2970.2780.01526.540.3412.4960.575
Mamba0.0770.1200.00524.580.3182.5030.356
Attn. (Ours)0.0720.1150.00518.170.3392.4630.250

We observe that Mamba backbone’s reconstruction quality matches closely with attention-based, but its generation quality is worse, highlighting the effectiveness of our attention-based tokenizer. We have updated the paper with this experiment.

Q4: Why only Waymo and comparison with more representative works on Nuscenes

A4: We opt to choose Waymo only due to its abundance of annotated data at higher frequencies (10Hz) compared to Nuscenes (2Hz) and KITTI (no HDMap), and wide adoption in both LiDAR generation task [1,2,3] and LiDAR resimulation task [4,5,6].

To demonstrate the effectiveness of our method on other datasets, we conduct experiments to measure the generation quality of our methods on NuScenes, comparing with a few requested baselines. Our setting follows Sec. 4.2 in the paper and generates up to 33 frames autoregressively. We report the results in the following table:

MethodVenueJSD↓MMD² ×10³↓FUD↓FSVD↓FPVD↓TTCE@4↓CTC@4↓ICP Avg.↓
LaGenECCV 20260.05820.117434.6515.3914.393.71660.93380.1701
GEMCVPR 20260.09770.497755.1815.8813.733.71470.82070.1716
UniSceneCVPR 20250.19151.8490106.5644.5340.613.73710.87290.1573
OursN/A0.08220.386621.4712.2010.683.69920.83580.1503

(bold is best, italicized is second-best)

Our generative perceptual quality (FUD, FSVD, FPVD) remains the best across all methods, highlighting the realism of our generated output. Our method also outperforms most other baselines in temporal consistency. We will update the paper with more results against more baselines on nuScenes dataset in the final version.

[1]: Zyrianov et al., “LidarDM: Generative LiDAR Simulation in a Generated World,” ICRA 2025

[2]: Wang et al., “Sensor2Sensor: Cross-Embodiment Sensor Conversion for Autonomous Driving,” CVPR 2026

[3]: Liu et al., “L3DR: 3D-aware LiDAR Diffusion and Rectification,” CVPR 2026

[4]: Huang et al., “NFL: Neural LiDAR Fields for Novel View Synthesis,” ICCV 2023

[5]: Zhou et al., “LiDAR-RT: Gaussian-based Ray Tracing for Dynamic LiDAR Re-simulation,” CVPR 2025

[6]: Hess et al., “SplatAD: Real-Time Lidar and Camera Rendering with 3D Gaussian Splatting for Autonomous Driving,” CVPR 2025

Q5: Does using H100 supports real-time claim

A5: Evaluating real-time inference on H100/200s is an established practice in streaming video generation, spanning across robotics, autonomous driving, and interactive world modelling [1,2,3,4,5,6,7,8]. We summarize a few representative and influential works on real-time video generation and their evaluation hardware below. Our runtime is reported using single H100, thus is similarly hardware-matched.

MethodVenueInference hardwareReported FPS
Self Forcing [1]NeurIPS 20251× H10017.0
OmniDreams [2]arXiv 20261× GB30068
LTX-Video [3]arXiv 20241× H100≈60.5
StreamDiffusionV2 [4]MLSys 20264× H10058.28
MotionStream [5]ICLR 20261× H10029.5
StreamDiT [6]CVPR 20261× H10016
SANA-Video [7]ICLR 20261× H10027
LiveAvatar [8]ECCV 20265× H10045.2

That being said, we do agree that real-time performance in consumer-grade GPU is necessary for downstream usage, but we will leave it as future work. We will adjust the wording accordingly to only mention real-time on a single H100.

Reference:

[1] Huang et al., “Self Forcing: Bridging the Train-Test Gap in Autoregressive Video Diffusion,” NeurIPS 2025.

[2] Basant et al., “NVIDIA OmniDreams: Real-Time Generative World Model for Closed-Loop Autonomous Vehicle Simulation,” arXiv preprint, 2026.

[3] HaCohen et al., “LTX-Video: Realtime Video Latent Diffusion,” arXiv preprint, 2024.

[4] Feng et al., “StreamDiffusionV2: A Streaming System for Dynamic and Interactive Video Generation,” MLSys 2026.

[5] Shin et al., “MotionStream: Real-Time Video Generation with Interactive Motion Controls,” ICLR 2026.

[6] Kodaira et al., “StreamDiT: Real-Time Streaming Text-to-Video Generation,” CVPR 2026.

[7] Chen et al., “SANA-Video: Efficient Video Generation with Block Linear Diffusion Transformer,” ICLR 2026.

[8] Huang et al., “Live Avatar: Streaming Real-time Audio-Driven Avatar Generation with Infinite Length,” ECCV 2026.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.