acceptodds
Withdrawn from ICLR 2027

Generative World Renderer at the Speed of Play

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

G-buffer-conditioned generative rendering synthesizes RGB frames from geometric and material buffers exported by a game engine, with appearance controlled through text prompts. By retaining engine-side geometry, physics, and gameplay logic, this approach provides a foundation for interactive, user-controllable visual worlds. However, the computational cost and fixed-window formulation of existing diffusion-based renderers limit their use in real-time applications. We introduce StreamRenderer, a real-time generative renderer that accelerates a 50-step diffusion-based teacher from 0.56 FPS to 31.54 FPS on a single NVIDIA H200 GPU. StreamRenderer combines autoregressive streaming, progressive four-step distillation, and lightweight distilled codecs for efficient latent encoding and frame reconstruction. It retains the teacher’s G-buffer and text-prompt interfaces while supporting continuous rendering and online prompt updates over input streams of arbitrary length. We evaluate StreamRenderer in terms of content preservation, temporal consistency, cross-window stability, prompt controllability, and runtime efficiency. The results demonstrate substantial computational savings while largely preserving content fidelity and prompt controllability, with a tradeoff in temporal stability relative to the bidirectional teacher. Integrated with a game engine, StreamRenderer enables a fully playable, prompt-controllable generative rendering system running at 30 FPS, bringing generative world rendering to the speed of play.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
forward rendering, diffusion model, model acceleration

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_m7odOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
3
Read review
Paper summary

This paper proposes a generative renderer that takes the G-buffer from modern game engines, including depth, normal, albedo, roughness, and metallicity, as conditional inputs. It aims to replace the traditional shading stage by generating diverse rendering styles from text prompts while preserving the underlying scene structure.
However, the Generative World Renderer system performs rendering on a large-scale dataset of visually complex AAA games, but it relies on a 50-step diffusion process and a heavy VAE codec, making it unsuitable for real-time streaming generation. Therefore, this paper proposes StreamRenderer, which enables infinite-length input streams through autoregressive streaming. It distills the original 50-step diffusion model into a 4-step diffusion model and replaces the G-buffer encoder and Wan VAE decoder with lightweight distilled codecs. These improvements enable real-time rendering at 31.54 FPS.

Critical strengths
  1. The paper successfully incorporates techniques such as model distillation and autoregressive modeling into the generative rendering task. The engineering efforts are impressive, achieving a high rendering FPS and demonstrating the potential of generative rendering for real-time game applications.
  2. The evaluation is comprehensive. The paper conducts experiments on WuKong to validate the effectiveness of the proposed autoregressive design, diffusion distillation, and tiny codec components.
  3. Based on the video results provided in the supplementary material, the proposed method achieves better temporal consistency and visual quality in generated videos compared with the baseline methods, including RGB↔X and FrameDiffuser.
Critical weaknesses
  1. The demo video lacks a real-time demonstration. A real-device demo showing the claimed 30 FPS performance would be important to support the main claim of the paper.
  2. For the model distillation component, the paper lacks details about the training compute and the architecture of the student model, which makes it difficult to fully understand and reproduce the distillation process.
  3. The title and the proposed method name are inconsistent, which is slightly confusing and makes the presentation less clear.
Questions for authors

Could the authors provide a more detailed real-time gameplay demonstration showing user interaction, end-to-end latency, and frame-rate stability under deployment settings?

Reviewer_UDxQOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper targets a clear and well-motivated gap: G-buffer-conditioned generative rendering preserves engine-side geometry and gameplay but remains offline, limited by 50-step denoising, bidirectional fixed-window formulation, and heavy VAE encoding/decoding. The proposed StreamRenderer addresses all three bottlenecks via autoregressive streaming, four-step distillation, and lightweight distilled codecs, achieving a 56× throughput gain (0.56→31.54 FPS) on an H200 and a playable 30 FPS integration in SuperTuxKart. The engineering is solid and the end-to-end deployment is a compelling demonstration. The main weakness is acknowledged and quantified: temporal stability degrades relative to the bidirectional teacher (tLPIPS 0.124→0.155), and the trade-off between streaming throughput and long-horizon consistency is not fully resolved. Comparisons are fair (baselines retrained on the same data), though the paper would benefit from user studies on perceived quality at interactive rates. Overall, a strong systems contribution with honest limitations.

Critical strengths

1.G-buffer conditioned rendering, superior to other types of conditions that rendering game screen. 2.Proposed model aligns auto-regressive inference, light-weight decoder is applied for faster video decoding. Lower denoise steps further accelerate the inference speed at 448 x 832@31fps.

Critical weaknesses

1.The model is fine-tuned on WuKong game, lacking generality on other domains. More examples are needed to evaluate the robustness of the renderer on OOD data. The visualizations on supplement only shows two gameplay cases, how about more cases on different type of games.

2.The interaction latency may be still large for user interaction, worsening the user experience on actual gameplay. Please show evaluate the actual latency for the model and comparison. A full time budget is needed, i.e., breakdown from engine G-buffer rasterization and readback through encoding, denoising, decoding, and display, plus the latency floor implied by the 4-latent-frame chunk size. Also report measured FPS at 720p/1080p and the actual deployment resolution, since 31.54 FPS is only shown at 832×448.

3.What is the model size of the proposed model?

4.How about the model performance on long-term gameplay like 1000 frames or 1e4 frames? More experments are needed on long-term sustainability evaluation.

5.The visualizations are not very clear to see and compare. It is very hard to see the visual improvement on high-frequency details of the model over other methods.

Questions for authors
  1. What is the model size of the proposed model?

  2. How about the model performance on long-term gameplay like 1000 frames or 1e4 frames? More experments are needed on long-term sustainability evaluation.

  3. What is the end-to-end latency of the proposed model? and on different resolutions?

Reviewer_r5UcOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper accelerates a diffusion model that renders RGB video from the G-buffers of a game engine (per-pixel depth, normal, and material maps), with appearance controlled by text prompts, so that it can run during live gameplay. The resulting StreamRenderer generates causally in short chunks conditioned on compressed history, distills the 50 denoising steps of the original model to four, and replaces its video autoencoder with small distilled networks. Most of these components adapt prior techniques from streaming video generation and few-step distillation. On one H200, throughput rises from 0.56 to 31.54 FPS at some cost in temporal stability. Quantitative results cover 5 s clips from Black Myth: Wukong, and a demo in SuperTuxKart runs at 30 FPS without quality metrics.

Critical strengths
  • The paper turns an offline G-buffer renderer that needs 50 denoising steps into a streaming one. On one H200, throughput rises from 0.56 to 31.54 FPS and peak memory falls from 30.1 to 16.2 GB (Table 1). Table 1 also shows that once the step count is reduced, the VAE codecs become the bottleneck, and replacing them alone raises throughput from 6.30 to 31.54 FPS (L305–L308).
  • The external baselines are retrained on the same data (L320–L321). StreamRenderer has lower FVD (384.1 vs. 650.6–1031.3) and tLPIPS_warp (0.155 vs. 0.305–0.440) than RGB↔X and FrameDiffuser, and matches the offline DiffusionRenderer in tLPIPS_warp (0.156) at about 29× its speed (L355–L356). A demo running the full pipeline in SuperTuxKart sustains 30 FPS, including G-buffer readback and display (L428–L431, L463–L467).
Critical weaknesses

Interactivity is claimed but not measured. The title and the claim of a "fully playable" game (L027, L481) concern how quickly the system responds to player input, yet the paper reports only throughput (L463–L467). Each chunk spans four latent frames (L151). Under the 4× temporal compression of the Wan VAE, this is about 16 game frames, or roughly 0.5 s at 30 FPS, and the paper does not report the resulting latency from input to display. All reported timings are also on an H200, and the benchmark timings use 832x448 (L264), below the capture resolution of 1280x720 (L259).

The new components are not ablated. Each row of Table 1 (L271–L278) changes several components at once, so the contributions of the text sink, the tiny G-buffer encoder, and the individual stages of the distillation recipe are not isolated.

Missing references. Several designs follow prior work that the paper does not cite: converting a bidirectional model into one that generates autoregressively in chunks (L148–L151) and the initialization by ODE regression mentioned in L199 [1], the anchor on the first frame and the text sink (L156–L161) [2–4], online prompt switching (L416–L418) [4], and the GAN heads on intermediate features (L226–L229) [5]. The paper also attributes distribution matching distillation only to DMD2 (L115–L116) [6] and defines tLPIPS_warp (L603–L607) without citing earlier temporal metrics [7, 8]. None of these works uses G-buffer conditioning, so no comparison is needed, but citing them would clarify what is new.

[1] CVPR 2025, Yin, From Slow Bidirectional to Fast Autoregressive Video Diffusion Models.

[2] ICLR 2024, Xiao, Efficient Streaming Language Models with Attention Sinks.

[3] ICLR 2026, Liu, Rolling Forcing: Autoregressive Long Video Diffusion in Real Time.

[4] ICLR 2026, Yang, LongLive: Real-time Interactive Long Video Generation.

[5] SIGGRAPH Asia 2024, Sauer, Fast High-Resolution Image Synthesis with Latent Adversarial Diffusion Distillation.

[6] CVPR 2024, Yin, One-step Diffusion with Distribution Matching Distillation.

[7] ECCV 2018, Lai, Learning Blind Video Temporal Consistency.

[8] ACM TOG 2020, Chu, Learning Temporal Coherence via Self-Supervision for GAN-based Video Generation.

Questions for authors
  • What is the latency from G-buffer export to display, and does any mechanism reduce the delay caused by chunks of four latent frames? What are the throughput and latency on a consumer GPU and at 1280x720?
  • How are the five G-buffers exported from SuperTuxKart, and how much of the frame budget do readback and transfer take (L465–L466)?
  • How large is the SuperTuxKart set used for fine-tuning (clips, frames, resolution), and which modules are fine-tuned (L429–L431)?
  • Can you report quality metrics on SuperTuxKart? The 31.54 FPS in L465 equals the value in Table 1. At what input resolution was it measured in the SuperTuxKart deployment?
  • How do prompt control and prompt switching change without the text sink (L161)? What is the M_CLIP for prompt switching defined in L619–L620?
  • What are the separate effects of the tiny encoder and the tiny decoder? Is the transformer updated when the encoder is fine-tuned with "the entire renderer" (L250), and does the gain in Boundary SSIM from 0.371 to 0.430 remain when the transformer is frozen?
  • Can you report results for MFD from an initialization by ODE regression, for training four steps directly, for DMD in place of MFD, and for MFD without GAN heads? L199–L200, L213, L221–L223, and L226–L229 describe these only in text.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.