- Rating
- 2
- Confidence
- 2
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
This paper presents a streaming framework for camera-controlled video rerendering, world rerenderer. It adapts a pretrained video diffusion model through a bidirectional teacher and a few-step causal student, combining techniques like cross-frame injection, SR-RoPE, and cycle straming tunning. Its main practical contribution lands on processing incrementally arriving reference video without requiring full video geometric preprocessing.
Critical strengths
- It works on a meaningful problem, avoiding full reference preprocessing enables ealier output and supports continuously arriving video.
- The reported experiment results demonstrate competitive visual quality and source alignment, with favorable camera-control results.
- The component ablation partially supports the benefits of SR-RoPE and CST, but they do not fully isolate the underlying mechanisms
Critical weaknesses
-
The throughput comparison is insufficently specified, the author does not clearly define the FPS denominator (including initial processing or not), timing boundaries, or output frame per chunk. As a result, the claimed approximately 5x throughput improvement over InSpatio-world is not clear enough to be convincing. Author here reports 1.96~2.13 fps for inspatio-world, whereas the latter advertises 24FPS on an H-series GPU and 10FPS on an RTX4090 for its optimized 1.3 B model in its own report. I understand the testing set may be different, and may have different evaluation protocol, but I still need more details on what causes such big difference.
-
The baseline implementation details are also incomplete. A common two-H20 budget does not establish comparable GPU utilization. Compilation settings and GPU allocation need clarification. If only World Rerenderer benefits from the two-GPU setup, this makes the comparison a bit unfair to other baseline methods. In that case, I would also like to see a single-GPU comparison to ensure fairness. Additionally, in the appendix, the authors mention using Tiny-VAE for acceleration. Was the same Tiny-VAE applied to the other methods? If not, this could also potentially lead to another unfair comparison.
-
Quantitative evidence for long-horizon stability is limited. Quantitative evaluation extends to 177 frames, while substantially longer rollouts are demonstrated qualitatively. Aggregate clip-level metrics do not establish how camera error, appearance drift, or consistency evolve over extended generation.
-
The technical contribution relative to existing components needs clearer delineation. The framework combines adapted conditioning, causal distillation, positional indexing, and rollout training. The paper would benefit from a sharper account of what is newly introduced and what follows established approaches.
Questions for authors
-
What exactly does FPS measure? Please provide the formula, timing boundaries, measurement duration, and averaging procedure. Why Inspatio-world has better time to first chunk gen time, but lower fps than world rerenderer. Please report steady-state and end-to-end throughput separately.
-
Were Tiny-VAE and compilation applied to all applicable baselines? What model sizes and inference resolutions were used?
-
How does each method use the two H20 GPUs? Can other method use 2 GPU to speed up initialization, if not streaming? Could you provide a single-GPU comparison?
-
What explains the discrepancy between your measured InSpatio-World FPS and its self-reported speeds?
-
How many output frames does each chunk contain, and what is the latency from a new camera command to the corresponding output during interaction with an ongoing stream, rather than at initialization?
-
Can you distinguish CST’s cycling benefit from longer-rollout training, and clarify what enforces the claimed closed-loop consistency?
-
What distinguishes the proposed components from their closest predecessors ?
2 comments
Long-Horizon Evidence and Contributions
===== Part 2/2 =====
W3: Long-horizon quantitative evidence
177 frames is already more than twice the 81-frame training horizon, and it is where offline baselines visibly break down (ReCamMaster CLIP-V 84.71 → 65.64, Table 1). Table 2 measures long-horizon stability directly at this length: without SR-RoPE and CST, RotErr is 7.105; with both, 1.603. A minute-scale benchmark across all seven methods is impractical, since offline baselines already need up to 4,140 s per 177-frame clip. Beyond 177 frames, Fig. 1 shows a 1,053-frame rollout and Figs. 12–14 show minute-scale rollouts. Because SR-RoPE keeps every temporal index inside the pretrained range regardless of length, there is no positional mechanism by which quality degrades as the stream grows.
W4 / Q7: What is new vs. adopted
Adopted (credited in Sec. 3.2 and Appendix A.1): the Wan-I2V backbone, RoCE camera encoding (ReDirector), Self-Forcing-style DMD, and attention sinks.
New: (1) Cross-Frame Injection. Frame-level interleaving of each source latent with its synchronized target latent, with shared 3D-RoPE coordinates. Prior V2V models use temporal concatenation (ReCamMaster) or channel concatenation, where the source–target distance grows with length. It also defines the natural unit for causal streaming. Table 2 (A/B vs. F): CLIP-V 90.65 / 84.11 → 94.34. (2) SR-RoPE. Existing relative schedules (e.g., Echo-Infinity) are designed for a single stream; applied naively to interleaved tokens, they double the temporal span and give synchronized views different indices. SR-RoPE schedules pairs within the cache and stores RoCE-conditioned but RoPE-free keys, so camera conditioning survives re-indexing without an auxiliary cache. Table 2 (C → D): RotErr 7.105 → 3.636. (3) CST. Long synchronized cyclic streams built from short clips, with teacher supervision only on the final segment, giving train-long–test-long under bounded memory. Table 2 (C → E): RotErr 7.105 → 2.487.
At the system level, this is the first camera-controlled V2V method that streams directly from the incoming video with no 3D reconstruction or depth at any stage.
Q6: CST vs. longer rollouts; closed-loop consistency
CST provides three benefits: (i) longer training streams, closing the short-train/long-test gap; (ii) camera diversity, since forward–backward concatenation yields new trajectories with direction reversals from the same data; and (iii) revisiting earlier viewpoints. The evaluation streams are forward-only, with no reversal, yet CST still cuts RotErr from 7.105 to 2.487, so (i) and (ii) clearly contribute on their own; increasing n_max from 0 to 2 further reduces it from 3.636 to 1.603 (Table 5). The closed-loop signal is implicit: when the stream revisits a viewpoint, the final-segment DMD loss pushes the output toward the same teacher distribution that supervised the earlier visit, while the student is conditioned on its own accumulated, possibly drifted, history. Drift is therefore penalized at revisits.
Inspatio-World’s 24 FPS Excludes Its Depth Reconstruction
===== Part 1/2 =====
The reviewer's central concern, the gap between our measured Inspatio-World speed and its self-reported 24 FPS, has a simple explanation, and it is exactly the core argument of our paper (Sec. 1): Inspatio-World's reported speed excludes its depth reconstruction.
W1 / Q1 / Q4: FPS definition and the Inspatio-World discrepancy
Definition. Every FPS in Table 1 is end-to-end, i.e., measured from receiving the input to emitting the last output frame, and computed identically for all methods on the same two-H20 setup: FPS = N_out / (T_init + T_gen), where T_init is all preprocessing before generation can start and T_gen is the wall-clock generation time including VAE decoding.
Our pipeline. Pose estimation for the incoming stream uses the streaming ViPE frontend on a second GPU, running in parallel at ≈11 FPS, slightly faster than our generator, so it never stalls generation. Our 10 FPS therefore already includes initialization and pose estimation.
Steady-state vs. end-to-end (177-frame set):
Steady-state values are backed out of Table 1. Inspatio-World: 177 frames at 1.96 FPS take 90.3 s, of which 79 s is reconstruction, leaving 11.3 s of generation. Ours: 17.7 s in total, minus 1 s of initialization. We will report both columns in the revision.
Why 24 FPS vs. ~2 FPS (Q4). Inspatio-World's report states that "for each video clip, we apply a feedforward reconstruction model to estimate depth information" (their Sec. 3.4), yet its 24 FPS figure does not account for this step; the reconstruction cost is never reported. Its generator-only ≈15.7 FPS on an H20 in our measurement is consistent with 24 FPS on a stronger H-series GPU, since H20 has far less compute than H100/H800. Our measurement reproduces their generator speed; the entire gap is the depth reconstruction they leave out.
Faster first-chunk generation but lower FPS (Q1). Exactly so: Inspatio-World's generator is fast, but its steady state exists only after a pre-observed reference video has been fully captured and reconstructed. On a live stream that point never arrives. End-to-end, ours is ≈5× faster, and unlike Inspatio-World, it needs no pre-observed video at all.
W2 / Q2 / Q3: Tiny-VAE, compilation, and GPU allocation
Tiny-VAE and torch.compile are Inspatio-World's own choices. Its report states that "we replace the original Wan-VAE with a lightweight Tiny-VAE" and uses torch.compile (their Sec. 3.4), and we ran its official pipeline. The streaming comparison is therefore like-for-like. For offline baselines, the VAE is irrelevant: their latency (62–4,140 s, Table 1) is dominated by full-sequence preprocessing and denoising, and no decoder swap changes that.
Two GPUs. Our generation runs on a single GPU; the second runs only the streaming ViPE frontend in parallel. Offline methods cannot exploit extra GPUs to emit frames earlier: they must finish full-sequence preprocessing or generation first (Appendix A.1). Even Inspatio-World cannot remove T_init by adding GPUs, because generation depends on the finished depth. On one GPU, our pose estimation simply runs sequentially; there is still no offline stage.
Settings. All methods, including ours, run at the same 384×672 resolution; baselines otherwise use their official checkpoints and default configurations.
Q5: Chunk size and interactive latency
Each chunk contains 3 latent frames, i.e., 12 output frames after the VAE's 4× temporal decompression. A new camera command takes effect at the next chunk, so command-to-output latency is one to two chunk intervals, ≈1.2–2.4 s at 10 FPS, and it stays constant no matter how long the stream has been running.