acceptodds
Under review as a conference paper at ICLR 2027

Wonder: Video World Model Done Better

13 views

Abstract

We present Wonder, a general-purpose video world model for real-time, camera-controllable world exploration. Given a single image or a conditioning video, Wonder constructs a playable world in which users navigate by moving the camera, discovering unseen regions and revisiting observed ones over a long horizon. This requires a system-level co-design of control, memory, and training. We introduce a camera conditioning scheme based on a dense pixel-space coordinate field whose renderings supply spatially aligned motion and orientation cues, so the model reads camera motion directly as visual evidence. For fast retrieval over a growing context, we propose a sparse full-fidelity memory mechanism that attends to a small set of relevant tokens at inference, independent of context length. We further rectify the self-forcing distillation pipeline with a timestep-wise mixture-of-students and a camera-aware adversarial regularizer, improving control adherence while retaining the teacher's generation diversity and stability. Together, these components let Wonder synthesize diverse, minute-scale videos at 16 FPS with coherent geometry, appearance, and dynamics across long rollouts. Beyond image-to-video, Wonder natively supports video-conditioned generation, re-shooting existing dynamic scenes in real time.

Primary area:
generative models
Keywords:
Video World Models, Camera-Controllable Video Generation, Long-Term Memory

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_kmwTOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors proposed a camera-controlled video generator for exploration from one image and re-shooting from a source video. It builds on Wan2.1-I2V-14B and combines three contributions. (1) A synthetic lattice and environment map that turns camera motion into image-aligned conditioning. (2) A Sparse retrieval selects historical chunks while retaining their full-resolution attention keys and values. (3) A few-step sampler uses three students and a teacher-feature adversarial loss to preserve camera control during distillation. Wonder improves average video-quality scores and camera-pose errors over the selected baselines on their own benchmarks. Its memory ablation improves revisit consistency over a sliding window, with higher throughput than dense history attention. The central I2V results are encouraging; source-event preservation and interactive runtime need more direct evaluation.

Critical strengths
  1. A coherent design for control, memory, and fast generation. The components address related failure modes: camera cues must survive distillation, and the student must learn to use sparse history. The authors provide enough ablations to back their design choices. e.g. replacing the coordinate field with pose embeddings increases rotational RPE from 0.0784 to 0.1726; removing the control regularizer raises it to 0.1905. These are substantial changes under the reported controlled setup.

2:**Encouraging results and a useful memory trade-off. ** Table 1 reports the best average quality and lowest camera-pose errors among the evaluated methods in both settings. The memory comparison is particularly helpful: sparse retrieval reaches 21.9 dB revisit PSNR at 16 FPS, compared with 22.4 dB at 8.7 FPS for dense attention and 16.3 dB at 16 FPS for a sliding window. This suggests that retaining detailed historical information can improve revisits without paying the full dense-attention cost at the tested horizon.

3: Pixel-Space Coordinate Field provides clear visual cues for camera control. The representation turns camera motion into a signal aligned with the video grid. The lattice supplies translation and parallax cues, while the colored environment map supplies rotation cues. Because the scaffold is synthetic, the signal remains available when the camera explores unseen regions.

Critical weaknesses

(1): Preserving the source video's dynamics is a central V2V claim, but Table 1 reports generic video quality and estimated camera errors. These can improve even when a generated person or object performs a different action from the source. Dynamic degree is 1 for both V2V methods, so it does not distinguish event preservation.

(2): The method uses synthetic geometry, so its scale may not match the actual scene. The evaluation also rescales camera paths, making it hard to tell whether the same command produces consistent movement across scenes. It would help to test scenes with known scale and check how scaffold spacing and focal length affect the results.

(3) Table 2 supports the complete coordinate field, but does not separate the lattice from the environment map. It remains unclear how much of the gain comes from translation cues, rotation cues, or simply providing dense visual conditioning. Separating these two components and comparing against dense camera-ray conditioning through the same input route and under the same training budget would clarify.

Minor: (1)The abstract emphasizes real-time generation and stable latency, but Table 2 measures throughput only at the 60-second mark. Appendix B describes multi-GPU deployment without the inference GPU count, output resolution, memory use, or end-to-end response latency.

Questions for authors

Together with the concerns in the weakness section, I have listed the following questions for the author to answer:

  1. Can you measure whether generated V2V actions and their timing match the source, separately from whether the output simply contains motion?
  2. What are the revisit delays and matching protocol behind Table 2, and how does consistency change as those delays increase?
  3. What GPU configuration and resolution achieve 16 FPS, and what are the measured response latency and total memory use at several rollout lengths?
  4. What retrieval budget and sparsity settings are used, and does Appendix B.4 retain all historical chunks outside the active rolling cache?
2 comments
AuthorsOpenReview ↗

Reply to Reviewer kmwT (Part 1/2)

Thank you for your detailed review and constructive suggestions! Please note our top-level comments and additional experimental results in the updated PDF. Below, we address your questions and concerns.


w.r.t. source video dynamics preservation (weakness 1 & question 1)

Thanks for your suggestions. We extend Table 1 with additional metrics to evaluate dynamic preservation from the source video. We use: (1) A VLM (Qwen3-VL) to evaluate frame-wise dynamics preservation by rating each pair of frames between 0 and 1 (0 means not preserved and 1 means perfectly preserved) and report the average score. (2) Video-level Human Preference Rate from 8 participants by showing them 10 random triplets of source, results from InSpatio-world and Wonder side-by-side. The order of our method and the baseline is randomly determined for each sample. Participants were asked to select a winner based on the dynamic preservation. As shown in the table below, Wonder achieves better dynamic preservation performance. We have included the experimental setup and reported results in Sec. E.5 of the revised manuscript.

ModelVLM frame-level match ↑Human video-level match ↑
InSpatio-World0.67774.3%
Wonder0.79283.1%

w.r.t. scene scale (weakness 2)

We'd like to clarify that our method does not suffer from the scale issue, as it always interprets the input camera path as metric-scale value. This is because (1) we use a metric camera estimator (DA3Metric-Large) to estimate intrinsics and metric-scale camera movement for each training video. (2) The synthetic 3D scaffold is constructed with a fixed grid spacing in metric units (i.e., center-to-center spacing of 1m). (3) To create the conditional video for a training sample, we render the scaffold using the estimated intrinsics (i.e., its focal length) and its metric-scale trajectories. During training, the model therefore learns metric camera motion. During inference, we use the same estimator to estimate intrinsics from the input image or the first frame of the input video, and to render conditioning videos using the estimated intrinsics and user-provided trajectories. Wonder will again treat the input camera as metric-scale values and generate video accordingly. We have revised our manuscript to make this part clearer (Sec. 3.1) and provide samples in Figures 14 & 15.


w.r.t. rescaling camera path in evaluation (weakness 2)

We rescale camera trajectories because other baseline methods did not produce videos in metric scale. Therefore, we follow RELIC (Hong et al., 2025) to adapt an alignment step for a fair comparison. Here, we extend Table 1 and report results in metric scale (without an additional alignment step) below, which shows significant variance between models. We have revised our manuscript to make this part clearer and included this metric (Sec. 4.1, Sec. E.8).

ModelEPE-trans ↓EPE-rot ↓
Wonder(ours)0.449.49
SANA-WM-Streaming7.3415.46
DreamX-World-5B1.3634.09
LingBot-World-Fast16.3347.52

w.r.t. effects of lattice and environment map and comparison with dense-ray baseline (weakness 3)

We conduct additional ablations on each component of the coordinate field and compare it with the pluck ray. As shown in the table below and Figure 18 in the paper, with the environment map but without the lattice, the model fails to respond to translation. With lattice but without an environment map, the accuracy over rotation is reduced. The best performance is obtained by using the complete coordinate field.

Input fieldEPE-trans ↓EPE-rot ↓
Full0.01320.0784
Lattice only0.01290.1248
Environment map only0.05470.2455

We also trained a Plücker-ray variant with the same budget (Table 2), and our representation improves all four metrics:

RepresentationVisual quality ↑Revisit PSNR (dB) ↑RPE trans. ↓RPE rot. ↓
Plücker rays0.841516.40.02150.1479
Coordinate field0.856421.90.01320.0784

We have added the discussion to the revised manuscript (Sec. E.6).


AuthorsOpenReview ↗

Reply to Reviewer kmwT (Part 2/2)

w.r.t. revisit delay and matching protocol for revisit PSNR (questions 2)

The revisit delays in Table 2 are 50s (total length is 60s, 0-5s the camera changes to reach the target position, 5-55s the camera goes to see other areas excluding the target view, 55-60s the camera comes back to the target position). We directly calculate the PSNR between the frames generated by two-time visits since we know which frame is the first visit and which frame is the second one (the matching protocol). In the table below, we additionally study the revisit changes as those delays increase (the first visit happens during 0-5s, and the revisit happens in the last 5s). As shown in the table below, the consistency remains robust against different delays. We have added these discussions to the paper (Sec. E.1).

Total Length123K (20s)246K (40s)369K (60s)
Revisit PSNR22.522.821.9

w.r.t. runtime configuration and measurements (question 3)

The FPS is measured under a single H200 node (8x GPUs). We run our method at a resolution of 480p, with a batch size of 1, together with our runtime optimization discussed in Appendix B.4. The KV cache is distributed over GPUs along the head dimension to reduce per-GPU storage. For I2V, we save the KV cache at BF16 precision. For V2V, we need to additionally store the KV cache for the source video. To accommodate that, we store the two KV caches in the FP8 E4M3 format, following RELIC. This gives the same per-GPU KV-cache memory usage for I2V and V2V (Sec. B.4, Sec. E.1).

The FPS is measured over 10 independent runs. Specifically, for each run, we let Wonder run until it reaches a 1-minute rollout and record the latency for every chunk (i.e a chunk corresponds to 8 RGB frames). We report FPS using the latency of the slowest chunk over 10 runs, which sets a lower bound on throughput. In our experiments, for I2V, the slowest chunk takes 484ms to generate and gives a throughput of 16.5 FPS. For V2V, the slowest chunk takes 497ms to generate and gives a throughput of 16.1 FPS. We round it to 16FPS. We have clarified our setup in the updated manuscript (Sec. E.7).

In the Table below, we additionally report the response latency (max observed latency over 10 rollouts) and total memory usage at several rollout lengths. We have added these discussions to the paper (Table 7).

Number of Tokens (I2V)30K (5s)123K (20s)246K (40s)369K (60s)
Latency (ms)441457462484
Total Mem. per GPU (GB)88.797.4112.2125.3
Number of Tokens (V2V)30K (5s)123K (20s)246K (40s)369K (60s)
Latency (ms)447455472497
Total Mem. per GPU (GB)88.998.7113.0127.5

w.r.t. retrieval budget, sparsity setting, and history cache retention (question 4)

We select top-7 chunks (14 latent frames) for both I2V and V2V setup. For I2V, the model reads the very first sink chunk, the two most recent chunks, and 7 selected chunks, resulting in 10 active chunks read at full resolution. For V2V, the model additionally reads the initial source chunk and the current source chunk, resulting in 12 chunks read. For Appendix B.4, yes, we retain all historical chunks outside the active rolling cache.


Reviewer_PYRhOpenReview ↗
Rating
3
Confidence
4
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents Wonder, a camera-controllable video world model supporting both image-conditioned world exploration and video-conditioned re-shooting. Its central idea is to represent camera motion through a synthetic 3D lattice and a colored environment map. These renderings provide spatially aligned control cues without requiring reconstruction of the input scene. The system also combines sparse retrieval from a full-fidelity historical KV cache with a four-step distillation pipeline using timestep-specialized students and a camera-aware adversarial regularizer. Experiments report improved camera-following accuracy over recent world models, alongside a favorable trade-off between revisit consistency and generation speed.

Critical strengths
  1. The camera-conditioning representation combines explicit geometric cues with reconstruction-free control. This is the contribution I find most interesting. Rendering the lattice and environment map converts camera translation and rotation into spatially aligned visual cues, allowing small pose changes to be reflected in the conditioning signal without relying on an estimated scene reconstruction. The control ablation in Table 2 supports this design, showing better camera-following accuracy than both implicit pose embeddings and point-cloud re-rendering.
  2. The sparse-memory design provides a useful efficiency trade-off. In Table 2, it achieves 21.9 dB revisit PSNR at 16 FPS, compared with 22.4 dB at 8.7 FPS for dense full-history attention. This suggests that selective retrieval can preserve much of the revisit fidelity while reducing computation.
Critical weaknesses
  1. Potential periodic ambiguity in the lattice representation needs clarification. If the lattice geometry and appearance repeat spatially, camera positions separated by lattice periods may produce similar renderings at the same orientation. The environment map would not resolve this translational ambiguity. Section 3.1 does not explain whether the lattice breaks this symmetry, and the experiments do not report translation ranges relative to lattice spacing. Evaluating trajectories across multiple cells would help establish the effective control range and whether temporal context resolves the ambiguity.
  2. Public benchmark evaluation would strengthen the evidence. The experiments mainly use an author-constructed collection and selected camera trajectories. Complementary evaluation on public benchmarks or datasets, such as https://haoyi-duan.github.io/WorldScore/ for I2V and https://kair-bair.github.io/dycheck/ for V2V, would make the comparisons more convincing.
Questions for authors
  1. Lattice ambiguity: Is the rendered lattice spatially periodic? If so, how are positions in different cells disambiguated, and does control remain accurate over trajectories spanning multiple cells?
  2. Runtime settings: What hardware, resolution, and batch size were used for the reported FPS? Please clarify whether these are end-to-end measurements and report I2V and V2V throughput and latency separately.
1 comment

Thank you for your detailed review and constructive suggestions! Please note our top-level comments and additional experimental results in the updated PDF. Below, we address your questions and concerns.


w.r.t. lattice ambiguity (weakness 1 & question 1)

The lattice representation is indeed periodic by design and will have the same renderings if the two cameras are in different lattice cells by one lattice period. Yet, the ambiguity is removed by the temporal context because: 1. the rendered frames reflect a visually continuous movement and 2. the model learns to interpret two adjacent conditioning frames as relative motion using the strong spatio-temporal cues. And thus, the control can remain accurate over trajectories spanning multiple cells. We have updated the paper (Sec. E.2, Fig. 14 & 15) with paired control video and results to demonstrate this, which contains trajectories across multiple cells. There is no strict constraint on the translation range, and the control remains accurate as the user keeps moving.


w.r.t. public benchmarks (weakness 2)

Thanks for your suggestion. We conducted the requested evaluations on WorldScore and DyCheck and compared our method with the state-of-the-art world models. We report quantitative results for WorldScore in the Tables below. More details and qualitative results can be found in Sec. E.3, E.4 & Fig. 17 of the updated manuscript. Our method achieves the best overall results.

ModelOverall ↑Camera control ↑3D consistency ↑
Wonder71.169.281.7
DreamX-World-5B61.045.872.9
SANA-WM-Streaming58.550.175.1
LingBot-World-Fast55.442.472.7

w.r.t. runtime settings (question 2)

The FPS is measured under a single H200 node (8x GPUs), at a resolution of 480p, with a batch size of 1, together with our runtime optimization discussed in Appendix B.4. The FPS is measured over 10 independent runs. Specifically, for each run, we let Wonder run until it reaches 1 minute and record the latency for every chunk (i.e a chunk corresponds to 8 RGB frames). We compute FPS using the latency of the slowest chunk over 10 runs, which sets a lower bound on throughput. In our experiments, for I2V, the slowest chunk takes 484ms to generate and gives a throughput of 16.5 FPS. For V2V, the slowest chunk takes 497ms to generate and gives a throughput of 16.1 FPS. We round it to 16FPS. We have clarified our setup in the updated manuscript (Sec. B.4, Sec. E.7).


Reviewer_11fROpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces Wonder, a camera-controllable video world model that supports real-time inference.

For camera control, Wonder uses pixel-space conditioning signals obtained by rendering a synthetic 3D scaffold and an environment map.

For efficient streaming generation, Wonder combines few-step autoregressive distillation with top-k retrieval of historical KV chunks. Sparse context forcing, a mixture of students, and camera-aware adversarial regularization help preserve generation quality and controllability. The model demonstrates good visual quality in both I2V and V2V tasks.

Critical strengths
  1. Wonder achieves lower camera-pose errors and higher average visual-quality scores than the evaluated baselines.
  2. Each proposed component of Wonder is effective and well motivated.
  3. The paper is clearly written and easy to follow.
Critical weaknesses

The paper does not sufficiently discuss related work or clarify its relationship to prior approaches. Specifically, Mixture of Contexts for Long Video Generation [1] has already explored top-k retrieval and sparse-memory mechanisms. A clear discussion or relevant experiments, if possible, could help better demonstrate the effectiveness of the proposed method.

[1] Mixture of Contexts for Long Video Generation. ICLR 2026

Questions for authors
  1. Please see the weakness section. I will raise scores if it is addressed.
  2. Since the proposed method uses top-kk chunk selection, how do end-to-end latency, KV-cache memory usage, and revisit consistency change as the rollout length increases? Evaluating the method across different sequence lengths with the same image conditions would help clarify these trends.
  3. The pixel-space conditioning signals seem to be dependent on the scene scales and 3D scaffold's resolutions. Does the model perform robustly under different hyper-parameters?
3 comments

Thank you for your detailed review and constructive suggestions! Please note our top-level comment, along with additional experimental results, in the updated PDF. Below, we address your questions and concerns.


w.r.t. relationship to Mixture of Contexts (weakness & question 1)

Thanks for pointing this out. Our method shares a similar design to MoC [1], i.e. both use top-k based retrieval and sparse memory mechanisms, but differs in the way the top-k tokens are selected. Specifically, MoC is purely semantic-biased: it computes top-k based on dot-product similarities over pooled queries and keys. Wonder is both semantic and camera-biased. Following RELIC (Hong et al, 2025), we additionally injects absolute camera pose to queries and keys before similarities are measured. This allows Wonder to additionally use spatial information to retrieve relevant tokens when revisit happens, which is crucial for world models. In the table below, without camera pose injection, revisit accuracy is reduced (Fig. 13 gives qualitative comparisons). Further sparse context forcing design enables this operation during distillation (Table 2, Fig. 7). We have added more discussion and implementation details in the updated manuscript (Sec. 3.2, Sec. E.1, Table 3, Fig. 13).

Method (60s Rollout)Revisit PSNR
Wonder (full)21.9
w/o camera Injection19.3
w/o sparse context forcing19.1

w.r.t. latency, KV-cache memory usage and revisit consistency (question 2)

We report the requested evaluations in the Table below. In the evaluation, we gradually increase the rollout length from 30K tokens (5s video) to 369K tokens (60s video). The results are measured under our inference setup described in Appendix B.4 & E.7, and top-level comments.

  1. Latency stays roughly constant and only increases very slightly with respect to the rollout length. This is because the additional computation induced by rollout length comes from selection overhead over lightweight pooled tokens. This overhead is negligible. Specifically, we spatially pool each latent by 8x, resulting in a 64x token reduction. Therefore, rolling to 369K from 30K is equivalent to an increase of 5.3K uncompressed tokens ((369K−30K)/64)((369K-30K)/64) in terms of compute.
  2. KV-cache memory usage grows linearly with respect to rollout length as we store the generated history at full resolution for retrieval. As discussed in Appendix B.4, the KV-cache memory is distributed along the head dimension over the available GPUs. In our experiment, we use 8x H200 for I2V and save the KV cache at BF16 precision. For V2V, we need to additionally store the KV cache for the source video. To accommodate for that, we store the two KV cache as FP8 E4M3 format following RELIC (Hong et al, 2025). This gives the same per-GPU memory usage for I2V and V2V.
  3. Revisits consistency remains robust with respect to rollout length, and we only observe a slight decrease. This demonstrates the effectiveness of sparse attention.

We have updated our manuscript with the detailed discussions and experiments (Table 3, Sec. E.1).

30K (5s)123K (20s)246K (40s)369K (60s)
Latency (ms)441457462484
KV Mem. per GPU (GB)3.212.125.237.3
Revisit PSNR22.722.522.821.9

w.r.t. scene scale and scaffold resolution (question 3)

We'd like to clarify that our method always generates videos in metric scale. Metric-scale prediction is enforced by design during training. As mentioned in Appendix A, we use Depth-Anything-3 (DA3Metric-Large specifically) to estimate metric-scale camera poses for our training set. We construct the synthetic scene of the 3D scaffold with a fixed configuration (i.e., center-to-center spacing of 1m) and render the pixel-space conditioning signals using the metric camera intrinsics and extrinsics from a corresponding training video. During training, the model learns metric camera motion leveraging local spatio-temporal cues in consecutive frames from the visual camera signal.

During inference, we use the same estimator to obtain the camera intrinsics of the input image, which are used to render the pixel-space conditioning signals. Input cameras are interpreted in the same metric-scale convention used during training, regardless of their original scale. This ensures a consistent interpretation of camera motion and eliminates scene-scale ambiguity. We have updated our manuscript to make this part clearer (Sec. 3.1) and to present outdoor and indoor examples (Fig. 14 & 15).


Reviewer_11fROpenReview ↗

Thanks for the rebuttal

Thank you for the rebuttal. Most of my concerns have been addressed.

I have one minor remaining question regarding scene scale and scaffold resolution. My concern extends beyond the use of metric-scale backbones to the expressiveness of the proposed modules across different spatial scales. For example, with a fixed center-to-center spacing of 1 m, how well can the scaffold represent small-scale scenes (such as tabletop setups or kitchen activities)? The same concern applies to video generation involving transitions between indoor and outdoor environments, where the relevant spatial scales can vary substantially.

AuthorsOpenReview ↗

Thanks for the Reply

Thanks for your reply, and we are glad to hear that our reply addressed most of your concerns.

Yes, our representation is generally robust to different scene scales. This is because: (1) the model can infer the metric scale of the scene from the input image, leveraging the extensive metric-scale supervision provided during training; (2) our synthetic 3D scaffold is “continuous” in space (ie vertices are connected by edges), and both small-scale and large-scale movement can be reflected as visual spatiotemporal clues. We have provided an example for the tabletop case in Figure 19 and the indoor-outdoor transition in Figure 20. In both cases, our method works. We will include these video results on the public webpage as well for the final versions. But we do admit that, as our model was trained on data with realistic and common scales, there might be challenging cases with unusual scales, like macro photography, that our models might not generalize well. We have added it as a potential limitation in the revised script.

Comments on the paper

1 comment

We sincerely thank all reviewers (R1: 11fR, R2: PYRh, and R3: kmwT) for providing constructive feedback that helped us improve the paper. We are encouraged that the reviewers recognize:

  • Our camera-conditioning representation provides reconstruction-free, spatially aligned geometric cues (R2, R3);
  • Wonder achieves better visual quality and camera-following accuracy than the evaluated baselines (R1, R3);
  • The sparse-memory design offers a useful trade-off between revisit consistency and generation speed (R2, R3);
  • The components are well motivated and effective, with supporting ablations (R1, R3);
  • The paper is clearly written and easy to follow (R1).

We have been working diligently on improving the paper on several fronts, addressing the critique.

Please note that the additional figures, tables, and discussions are included in the revised manuscript (marked in blue).

We address the common questions from the reviewers as follows:

w.r.t. scale ambiguity in lattice representation

We'd like to clarify that our method always generates videos in metric scale. Metric scale prediction is enforced by design during training. As mentioned in Appendix A, we use DepthAnything3 (DA3Metric-Large specifically) to estimate metric-scale camera poses for our training set. We construct the synthetic scene of the 3D scaffold with a fixed configuration (i.e., center-to-center spacing of 1m) and render the pixel-space conditioning signals using the metric camera intrinsics and extrinsics from a corresponding training video. During training, the model learns metric camera motion leveraging local spatio-temporal cues in consecutive frames from the visual camera signal.

During inference, we use the same estimator to obtain the camera intrinsics of the input image, which are used to render the pixel-space conditioning signals. Input cameras are interpreted in the same metric-scale convention used during training, regardless of their original scale. This ensures a consistent interpretation of camera motion and eliminates scene-scale ambiguity. We have updated our manuscript to make this part clearer (Sec. 3.1) and present outdoor and indoor examples (Fig. 14 & 15). We also extended Table 1 to report results at the metric scale (Sec. E.8).

w.r.t. runtime configuration, throughput, latency, and memory usage

The FPS is measured on a single H200 node (8x GPUs), at a resolution of 480p, with a batch size of 1, together with our runtime optimization discussed in Appendix B.4. The FPS is measured over 10 independent runs. Specifically, for each run, we let Wonder run until it reaches a 1-minute rollout and record the latency for every chunk (i.e a chunk corresponds to 8 RGB frames). We compute FPS using the latency of the slowest chunk over 10 runs, which sets a lower bound on throughput. In our experiments, for I2V, the slowest chunk takes 484ms to generate and gives a throughput of 16.5 FPS. For V2V, the slowest chunk takes 497ms to generate and gives a throughput of 16.1 FPS. We round it to 16FPS.

For memory storage, as discussed in Appendix B.4, the KV-cache memory is distributed along the head dimension over the available GPUs. In our experiment, we save the KV cache at BF16 precision for I2V. For V2V, we need to additionally store the KV cache for the source video. To accommodate for that, we store the two KV caches as FP8 E4M3 format following RELIC (Hong et al, 2025). This gives the same per-GPU KV cache memory usage for I2V and V2V. We replicate the model (i.e. mixture-of-students) for each GPU to avoid extra parameter broadcasting. We have clarified our setup in the updated manuscript (Sec. B.4, E.1 & E.7). In the Table below, we report the latency, GPU memory usage, and KV cache memory usage with respect to the rollout length.

Number of Tokens (I2V)30K (5s)123K (20s)246K (40s)369K (60s)
Latency (ms)441457462484
Total Mem. per GPU (GB)88.797.4112.2125.3
KV cache Mem. per GPU (GB)3.212.125.237.3
Number of Tokens (V2V)30K (5s)123K (20s)246K (40s)369K (60s)
Latency (ms)447455472497
Total Mem. per GPU (GB)88.998.7113.0127.5
KV cache Mem. per GPU (GB)3.212.125.237.3

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.