acceptodds
Withdrawn from ICLR 2027

World Rerenderer: Real-Time Camera-Controlled World Re-rendering via Video Diffusion Transformers

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Real-time camera-controlled world re-rendering aims to synthesize target views from a streaming source video under online user-specified camera trajectories. This task poses two key challenges: maintaining synchronization and content consistency across views, and achieving low-latency rendering with long-horizon stability. Existing approaches are predominantly designed for offline full-sequence processing and rely on bidirectional attention or costly depth-based geometric preprocessing, making them unsuitable for low-latency streaming scenarios. To address these limitations, we develop a teacher-to-student framework that distills a high-fidelity bidirectional re-rendering teacher into a few-step causal autoregressive student. We introduce Cross-Frame Injection to interleave each incoming source frame with its synchronized target latent, thereby establishing direct cross-view correspondence and improving content consistency. To ensure stable long-horizon generation, we further propose Shared Relative RoPE, which bounds temporal indices within the pretrained range, and Cycle Streaming Tuning, which exposes the student to extended cyclic trajectories during streaming training and provides a closed-loop consistency signal. Together, these designs enable low-latency online video re-rendering with precise camera control and sustained spatiotemporal consistency over long streaming rollouts.

Primary area:
generative models
Keywords:
Interactive World Models. World Re-rendering. Streaming Generation.

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_TjLdOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents a streaming framework for camera-controlled video rerendering, world rerenderer. It adapts a pretrained video diffusion model through a bidirectional teacher and a few-step causal student, combining techniques like cross-frame injection, SR-RoPE, and cycle straming tunning. Its main practical contribution lands on processing incrementally arriving reference video without requiring full video geometric preprocessing.

Critical strengths
  1. It works on a meaningful problem, avoiding full reference preprocessing enables ealier output and supports continuously arriving video.
  2. The reported experiment results demonstrate competitive visual quality and source alignment, with favorable camera-control results.
  3. The component ablation partially supports the benefits of SR-RoPE and CST, but they do not fully isolate the underlying mechanisms
Critical weaknesses
  1. The throughput comparison is insufficently specified, the author does not clearly define the FPS denominator (including initial processing or not), timing boundaries, or output frame per chunk. As a result, the claimed approximately 5x throughput improvement over InSpatio-world is not clear enough to be convincing. Author here reports 1.96~2.13 fps for inspatio-world, whereas the latter advertises 24FPS on an H-series GPU and 10FPS on an RTX4090 for its optimized 1.3 B model in its own report. I understand the testing set may be different, and may have different evaluation protocol, but I still need more details on what causes such big difference.

  2. The baseline implementation details are also incomplete. A common two-H20 budget does not establish comparable GPU utilization. Compilation settings and GPU allocation need clarification. If only World Rerenderer benefits from the two-GPU setup, this makes the comparison a bit unfair to other baseline methods. In that case, I would also like to see a single-GPU comparison to ensure fairness. Additionally, in the appendix, the authors mention using Tiny-VAE for acceleration. Was the same Tiny-VAE applied to the other methods? If not, this could also potentially lead to another unfair comparison.

  3. Quantitative evidence for long-horizon stability is limited. Quantitative evaluation extends to 177 frames, while substantially longer rollouts are demonstrated qualitatively. Aggregate clip-level metrics do not establish how camera error, appearance drift, or consistency evolve over extended generation.

  4. The technical contribution relative to existing components needs clearer delineation. The framework combines adapted conditioning, causal distillation, positional indexing, and rollout training. The paper would benefit from a sharper account of what is newly introduced and what follows established approaches.

Questions for authors
  1. What exactly does FPS measure? Please provide the formula, timing boundaries, measurement duration, and averaging procedure. Why Inspatio-world has better time to first chunk gen time, but lower fps than world rerenderer. Please report steady-state and end-to-end throughput separately.

  2. Were Tiny-VAE and compilation applied to all applicable baselines? What model sizes and inference resolutions were used?

  3. How does each method use the two H20 GPUs? Can other method use 2 GPU to speed up initialization, if not streaming? Could you provide a single-GPU comparison?

  4. What explains the discrepancy between your measured InSpatio-World FPS and its self-reported speeds?

  5. How many output frames does each chunk contain, and what is the latency from a new camera command to the corresponding output during interaction with an ongoing stream, rather than at initialization?

  6. Can you distinguish CST’s cycling benefit from longer-rollout training, and clarify what enforces the claimed closed-loop consistency?

  7. What distinguishes the proposed components from their closest predecessors ?

2 comments
AuthorsOpenReview ↗

Inspatio-World’s 24 FPS Excludes Its Depth Reconstruction

===== Part 1/2 =====

The reviewer's central concern, the gap between our measured Inspatio-World speed and its self-reported 24 FPS, has a simple explanation, and it is exactly the core argument of our paper (Sec. 1): Inspatio-World's reported speed excludes its depth reconstruction.


W1 / Q1 / Q4: FPS definition and the Inspatio-World discrepancy

Definition. Every FPS in Table 1 is end-to-end, i.e., measured from receiving the input to emitting the last output frame, and computed identically for all methods on the same two-H20 setup: FPS = N_out / (T_init + T_gen), where T_init is all preprocessing before generation can start and T_gen is the wall-clock generation time including VAE decoding.

Our pipeline. Pose estimation for the incoming stream uses the streaming ViPE frontend on a second GPU, running in parallel at ≈11 FPS, slightly faster than our generator, so it never stalls generation. Our 10 FPS therefore already includes initialization and pose estimation.

Steady-state vs. end-to-end (177-frame set):

MethodBefore first frameSteady-state FPSEnd-to-end FPS
Inspatio-Worlddepth reconstruction of the full reference (79 s)≈15.71.96
Oursnone (1 s init)≈10.610

Steady-state values are backed out of Table 1. Inspatio-World: 177 frames at 1.96 FPS take 90.3 s, of which 79 s is reconstruction, leaving 11.3 s of generation. Ours: 17.7 s in total, minus 1 s of initialization. We will report both columns in the revision.

Why 24 FPS vs. ~2 FPS (Q4). Inspatio-World's report states that "for each video clip, we apply a feedforward reconstruction model to estimate depth information" (their Sec. 3.4), yet its 24 FPS figure does not account for this step; the reconstruction cost is never reported. Its generator-only ≈15.7 FPS on an H20 in our measurement is consistent with 24 FPS on a stronger H-series GPU, since H20 has far less compute than H100/H800. Our measurement reproduces their generator speed; the entire gap is the depth reconstruction they leave out.

Faster first-chunk generation but lower FPS (Q1). Exactly so: Inspatio-World's generator is fast, but its steady state exists only after a pre-observed reference video has been fully captured and reconstructed. On a live stream that point never arrives. End-to-end, ours is ≈5× faster, and unlike Inspatio-World, it needs no pre-observed video at all.


W2 / Q2 / Q3: Tiny-VAE, compilation, and GPU allocation

Tiny-VAE and torch.compile are Inspatio-World's own choices. Its report states that "we replace the original Wan-VAE with a lightweight Tiny-VAE" and uses torch.compile (their Sec. 3.4), and we ran its official pipeline. The streaming comparison is therefore like-for-like. For offline baselines, the VAE is irrelevant: their latency (62–4,140 s, Table 1) is dominated by full-sequence preprocessing and denoising, and no decoder swap changes that.

Two GPUs. Our generation runs on a single GPU; the second runs only the streaming ViPE frontend in parallel. Offline methods cannot exploit extra GPUs to emit frames earlier: they must finish full-sequence preprocessing or generation first (Appendix A.1). Even Inspatio-World cannot remove T_init by adding GPUs, because generation depends on the finished depth. On one GPU, our pose estimation simply runs sequentially; there is still no offline stage.

Settings. All methods, including ours, run at the same 384×672 resolution; baselines otherwise use their official checkpoints and default configurations.


Q5: Chunk size and interactive latency

Each chunk contains 3 latent frames, i.e., 12 output frames after the VAE's 4× temporal decompression. A new camera command takes effect at the next chunk, so command-to-output latency is one to two chunk intervals, ≈1.2–2.4 s at 10 FPS, and it stays constant no matter how long the stream has been running.

AuthorsOpenReview ↗

Long-Horizon Evidence and Contributions

===== Part 2/2 =====

W3: Long-horizon quantitative evidence

177 frames is already more than twice the 81-frame training horizon, and it is where offline baselines visibly break down (ReCamMaster CLIP-V 84.71 → 65.64, Table 1). Table 2 measures long-horizon stability directly at this length: without SR-RoPE and CST, RotErr is 7.105; with both, 1.603. A minute-scale benchmark across all seven methods is impractical, since offline baselines already need up to 4,140 s per 177-frame clip. Beyond 177 frames, Fig. 1 shows a 1,053-frame rollout and Figs. 12–14 show minute-scale rollouts. Because SR-RoPE keeps every temporal index inside the pretrained range regardless of length, there is no positional mechanism by which quality degrades as the stream grows.


W4 / Q7: What is new vs. adopted

Adopted (credited in Sec. 3.2 and Appendix A.1): the Wan-I2V backbone, RoCE camera encoding (ReDirector), Self-Forcing-style DMD, and attention sinks.

New: (1) Cross-Frame Injection. Frame-level interleaving of each source latent with its synchronized target latent, with shared 3D-RoPE coordinates. Prior V2V models use temporal concatenation (ReCamMaster) or channel concatenation, where the source–target distance grows with length. It also defines the natural unit for causal streaming. Table 2 (A/B vs. F): CLIP-V 90.65 / 84.11 → 94.34. (2) SR-RoPE. Existing relative schedules (e.g., Echo-Infinity) are designed for a single stream; applied naively to interleaved tokens, they double the temporal span and give synchronized views different indices. SR-RoPE schedules pairs within the cache and stores RoCE-conditioned but RoPE-free keys, so camera conditioning survives re-indexing without an auxiliary cache. Table 2 (C → D): RotErr 7.105 → 3.636. (3) CST. Long synchronized cyclic streams built from short clips, with teacher supervision only on the final segment, giving train-long–test-long under bounded memory. Table 2 (C → E): RotErr 7.105 → 2.487.

At the system level, this is the first camera-controlled V2V method that streams directly from the incoming video with no 3D reconstruction or depth at any stage.


Q6: CST vs. longer rollouts; closed-loop consistency

CST provides three benefits: (i) longer training streams, closing the short-train/long-test gap; (ii) camera diversity, since forward–backward concatenation yields new trajectories with direction reversals from the same data; and (iii) revisiting earlier viewpoints. The evaluation streams are forward-only, with no reversal, yet CST still cuts RotErr from 7.105 to 2.487, so (i) and (ii) clearly contribute on their own; increasing n_max from 0 to 2 further reduces it from 3.636 to 1.603 (Table 5). The closed-loop signal is implicit: when the stream revisits a viewpoint, the final-segment DMD loss pushes the output toward the same teacher distribution that supervised the earlier visit, while the student is conditioned on its own accumulated, possibly drifted, history. Drift is therefore penalized at revisits.

Reviewer_tDzXOpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper introduces World Renderer, a streaming, camera controllable video-to-video generation model. World Renderer first finetunes a bidirectional Wan-I2V teacher and then distills it into a causal autoregressive student via Self-Forcing DMD. To keep long rollouts stable, World Renderer uses Shared Relative RoPE and Cycle Streaming Tuning. The experiments show that with 10FPS real-time inferences, World Renderer achived competitive or even better video re-rendering quality.

Critical strengths

[S1] World Renderer can inference with low startup time with 10 FPS streaming.

Critical weaknesses

[W1] Small KV cache prevents true long-term scene memory. The model only caches 12 frame pairs and discards all older intermediate frames. This directly weakens the paper’s “long horizon world re-rendering” and “closed-loop consistency” claims. Cycling Streaming Tuning only works because time reversing a clip returns to the first frame, whereas any newly generated novel content through a stream is forgotten as soon as its token leave the window.

[W2] Incremental novelty: The method mostly combines existing ReDirector (Wan-I2V + RoCE + reversal augmentation) and Self-Forcing DMD. Also the 10 FPS speed in Table 1 relies on exchanging the original WAN VAE into Tiny-VAE and running ViPE in parallel.

Questions for authors

[Q1] How does the Stage 1 bidirectional teacher perform in Table 1, compared to the distilled causal student model?

1 comment
AuthorsOpenReview ↗

Response to Reviewer tDzX: Corrections on KV Cache, CST, and Novelty; Teacher Results

We thank the reviewer for their time, However, both weaknesses rest on factual misreadings and are contradicted by results already in the submission.


W1: "Small KV cache prevents true long-term scene memory."

(1) Factual correction. The cache does not "discard all older frames." A sink chunk is retained permanently; only the oldest non-sink chunks are evicted (Sec. 3.3, Fig. 3, Appendix A.1 / Algorithm 1).

(2) The synchronized source stream is itself our history. Long-term memory is the core bottleneck of text/image-initialized world models, whose only history is their own error-prone outputs. Our task is video-conditioned: via Cross-Frame Injection (Eq. 3), every target chunk is paired with its synchronized source chunk, and these clean source latents are cached alongside the generated ones. The scene is thus anchored by a ground-truth record that arrives continuously and never drifts; observed content is re-supplied by the input at every step, not recalled from a degrading memory. This is precisely why a bounded window suffices here.

(3) A bounded KV cache is the de facto standard for streaming video generation: CausVid, Self-Forcing, LongLive, MAGI-1, MotionStream, and Inspatio-World all use it, because it guarantees constant memory and per-chunk latency over unbounded streams. The 12-pair size is a hyperparameter, not an architectural ceiling.

(4) We do not claim persistent memory. Our claim is long-horizon stability: no positional overflow, drift, or collapse beyond the training horizon. This is exactly what Table 2 measures.

(5) "Cycling Streaming Tuning only works because time reversing a clip returns to the first frame." This misreads CST, which provides three benefits; revisiting is only one (Sec. 3.3). (i) Longer training streams: the student is trained on rollouts several times longer than the teacher's horizon, learning from long self-generated contexts under bounded memory and closing the short-train/long-test gap. (ii) Camera diversity: forward–backward concatenation yields new trajectories with direction reversals from the same data, with no extra long videos. (iii) The revisit signal. Our results show (i) and (ii) matter on their own: all evaluation streams are forward-only, with no reversal, yet CST alone cuts RotErr on 177 frames from 7.105 to 2.487 (−65%; Table 2, C→E), and longer streams (n_max 0→2) cut it from 3.636 to 1.603 (Table 5).


W2: "Incremental novelty: mostly combines ReDirector and Self-Forcing DMD."

(1) Our contribution is streaming V2V re-rendering with no reconstruction and no depth. Every prior camera-controlled V2V method is either offline (ReCamMaster, CamCloneMaster, ReDirector, TrajectoryCrafter, NeoVerse), or, in the case of Inspatio-World, streams only after offline depth estimation and warping of a pre-observed reference video (35 s / 79 s). Ours is the first to operate directly on the incoming stream with no 3D reconstruction or depth maps at any stage, emitting its first output in 2.76 s (Table 1). Cross-Frame Injection, SR-RoPE, and CST are what make this possible. The Wan-I2V backbone, RoCE, and reversal augmentation are explicitly credited as adopted teacher components (Sec. 3.2; Appendix A.1) and never claimed.

(2) "ReDirector-style teacher + Self-Forcing DMD" is not hypothetical. It is row (C) of Table 2, and it fails. At 177 frames it yields RotErr 7.105 and TransErr 0.091; our full model reduces these to 1.603 (−77%) and 0.053 (−42%).

(3) Tiny-VAE is standard practice in real-time video generation, including Inspatio-World, the only other streaming baseline, which also relies on a tiny VAE for its speed. It is fully disclosed (Appendix A.1) and orthogonal to our contributions. Our latency advantage is algorithmic: Inspatio-World spends 35 s / 79 s on depth preprocessing alone, and offline methods need 62–4,140 s (Table 1).

(4) ViPE is not an acceleration trick. It is the online pose estimator for the incoming stream, a cost streaming adds; running it on a second GPU only hides this cost. All methods share the same two-H20 budget (Appendix A.1).


Q1: Teacher performance.

For completeness:

MethodCLIP-V↑Sub.↑Bg.↑Aes.↑Img.↑Flick.↑Smooth↑TransErr↓RotErr↓
Teacher (short)92.9893.6494.0355.5160.596.7599.290.0191.463
Ours (short)92.7493.7093.9555.4860.4396.8199.260.01781.530
Teacher (177)94.4793.6694.4559.1062.6897.4799.310.0441.57
Ours (177)94.3493.7594.3159.2162.1997.5499.330.0531.603

The teacher must see the full clip before generating, so it cannot stream; it is a training device, not a competing system.

Reviewer_q16rOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper introduces World Rerenderer, a camera-controlled diffusion video generation framework that enables real-time video reangling for streaming video inputs. It incorporates a teacher-to-student distillation pipeline to convert a bidirectional teacher trained to solve the video reangling task into a few-step autoregressive equivalent. It introduces three core mechanisms. First, Cross-Frame Injection performs frame level interleaving of the source and target latents for synchronized generation. Second, Shared Relative RoPE keeps the temporal RoPE indices within the training range by rolling over the indices to preserve video fidelity over minute-scale video inputs. Third, Cycle Streaming Tuning augments the dataset by concatenating forward and backward video trajectories to train on long autoregressive rollouts and improve generation quality across diverse camera trajectories. Experiments show comparable visual quality and good camera control accuracy relative to baseline multi-step methods on short clips and 177 frame sequences. The framework also achieves real-time generation speed of 10 FPS and a startup latency of 2.76 seconds.

Critical strengths

[S1] The paper addresses a practically meaningful setting, in which the video world model community is interested. Rendering the novel camera viewpoint of incoming source video under online setup. The teacher-to-student pipeline combines synchronized source conditioning with causal generation to support this streaming use case.

[S2] The paper reports competitive visual quality and camera-control accuracy on the evaluated short and 177-frame clips. Especially on the 177-frame setup, the metric degradation is much less compared to the implicit baselines. Also, the reported throughput and startup latency are promising for streaming applications, although the VAE configuration used across quality and efficiency measurements requires clarification.

Critical weaknesses

[W1] Novelty relative to prior methods.

Shared Relative RoPE (L305–323), one of the three core contributions, reads as a straightforward combination of existing techniques rather than a novel contribution.

(a) Bounded, cache-relative temporal indices used identically in training and inference. This essentially reads as the Unified Relative RoPE Recipe of Echo-Infinity (Bian et al., 2026), which also keeps the sink at the initial index and places the newest frame at fmax⁡f_{\max}. While the paper cites this work (L309), it is bit unclear what is the core difference here. WorldPlay (Sun et al., 2025, Fig. 4c) likewise re-assigns the temporal indices of context frames to a bounded relative range ("Temporal Reframing") in a realtime, camera controlled streaming model, but the paper cites it only as an example of camera control (L184).

(b) One shared temporal index per source–target pair. This is the only difference from prior schedules that the paper states (L309–311), and it is the shared 3D RoPE of ReDirector (Park et al., 2026), which the paper's own teacher already adopts. Once the teacher shares indices across views, carrying that convention into the student's relative schedule is the natural default. The ablations show the benefit of enabling SR-RoPE within the proposed pipeline, but do not establish an incremental contribution over a matched combination of prior relative-indexing and shared-index designs. The independent methodological novelty of SR-RoPE therefore remains a bit unclear for me. I hope the authors elaborate more on this distinction.

[W2] Limited quantitative support for long horizon stability.

The paper describes a length agnostic design that "in principle supporting indefinitely long horizons" (Fig. 1 caption, L32–34) and presents long-horizon stability as a key result. However, the quantitative evaluation stops at 177 frames, and the evidence beyond that is qualitative only: the 1053-frame example in Fig. 1 and three minute-scale examples (Fig. 12–14 and the minute-scale supplementary videos). By comparison, Echo-Infinity, whose relative-RoPE recipe SR-RoPE follows, reports quantitative results from 60s or 240s videos.

Even the minute-scale examples show failures. In supplementary video 'h5' (trajectory 2), objects become duplicated around 0:30, and frame-level inconsistencies appear. Also the shown minute-scale examples appear to cover a limited range of source-camera motion and scene dynamics. The moving subject stays near the image center, the source camera barely moves, and object motion over the clip is small (e.g., 'h11'). It is therefore unclear whether the method remains stable on general in-the-wild streams, such as hand-held footage, large source-camera motion, or subjects that leave and re-enter the frame. Supporting the claim would require a quantitative evaluation at minute scale or longer on more diverse sources.

Questions for authors

[Q1] Why does Cycle Streaming Tuning help, and can reversal hurt?

(a) Table 5 only covers nmax⁡∈{0,1,2}n_{\max} \in \{0, 1, 2\}. Do the gains keep increasing with more reversals, or do they saturate or degrade?

(b) What does reversal add beyond longer self-generated context? Algorithm 1 (line 33) has no loss comparing a revisited view with the student’s earlier rendering; earlier outputs affect it only through the generation context, and revisits beyond the 12-pair cache horizon (L409) no longer have the corresponding entries. Moreover, since source and target are reversed jointly, a revisit repeats the same source observations, so recovering source visible content does not by itself show consistency with earlier generated content. A non-cyclic control with matched length, with scene boundary effects controlled, would isolate the claimed closed-loop benefit (L300–302). Does the gain also transfer to forward only streams, where the scene has changed by the time the camera revisits?

(c) Could the benefit of reversal tied to the training data? MultiCamVideo is synthetically rendered with animated motion, so time-reversed clips may still look plausible. Real footage often contains irreversible dynamics (e.g., falling objects, splashes, smoke), whose reversal is physically implausible. Would CST still help, or could it hurt, when training on real-world data, such as the real videos the authors plan to incorporate (Future directions in Appendix A.4)?

[Q2] How does SR-RoPE differ, mathematically or operationally, from applying Echo-Infinity’s bounded relative schedule to synchronized source–target pairs while retaining the shared indices used by ReDirector? Please clarify which part contribute to a new mechanism rather than an adaptation of these existing designs. If there is a substantive difference, a matched comparison on the same student would help establish its contribution. A comparison with inference-only re-anchoring, while keeping source–target indices shared, would additionally clarify the benefit of using the relative schedule during training.

[Q3] Are quality and efficiency evaluated under the same VAE configuration? Appendix A.1 states that the original Wan VAE is replaced with a lightweight Tiny-VAE "when measuring throughput." Please clarify whether this same configuration is used for the quality and camera-control metrics in Table 1, the reported startup latency, and the supplementary videos. It is also unclear which baselines, if any, use this replacement. If quality is evaluated with the original VAE while throughput is measured with Tiny-VAE, the reported results do not establish that the stated quality and speed are achieved simultaneously. Reporting quality, FPS, and startup latency for both VAE configurations would clarify the actual efficiency–quality trade-off and the contribution of the reported speed gains.

[Q4] Why RoCE among recent camera encodings? Several relative camera encodings for camera controlled video generation have appeared recently, e.g., PRoPE (Li et al., 2025), UCPE (Zhang et al., 2026), and CRePE (Jin et al., 2026); PRoPE is also used in the streaming world model WorldPlay. What motivated the choice of RoCE? Is it tied to the streaming design, such as storing RoCE-conditioned but RoPE-free keys in the KV cache (L322–323)? A discussion or a controlled comparison with a representative alternative would clarify whether RoCE offers particular advantages for cache compatibility and efficiency, and whether the benefits of the proposed streaming components generalize beyond this camera encoding.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.