- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
VivaWorld addresses geometric inconsistency and camera drift in long video generation from a single image. It combines reconstructed point clouds with generated object meshes, introduces mesh-seeded point-cloud filtering, and augments global camera embeddings with object-relative information to guide a pretrained video model. Experiments demonstrate improved long-video fidelity and camera accuracy, together with better downstream 3D reconstruction, compared with the evaluated baselines.
Critical strengths
-
The combination of reconstructed point clouds and generated object meshes is well motivated, providing complementary guidance for observed and unseen regions. Object-relative camera embeddings further incorporate camera–object relationships.
-
The reported experiments show improvements in long-video fidelity and camera accuracy over several baselines, supported by downstream 3D reconstruction results and supplementary visual comparisons.
Critical weaknesses
-
Unexplained differences between ablation results. Tables 3 and 4 report different results for their reference configurations despite both evaluating on DL3DV. For example, long-video LPIPS differs substantially (0.488 vs. 0.545), exceeding the Global-to-Hybrid improvement in Table 3. The paper should clarify whether these configurations share the same components, evaluation scenes, checkpoints, and sampling settings.
-
Incomplete justification of object-anchored camera conditioning. Table 3 shows positive gains, but the explanation that global embeddings accumulate errors is unclear: the embeddings are computed analytically, while drift occurs during generation. Comparisons with simpler object-center conditioning and sensitivity analyses for anchor count and localization errors would better establish the specific benefit of the proposed representation.
-
Limited evaluation of robustness and computational cost. The mesh-seeded filter may discard valid geometry disconnected from registered meshes, yet its effect is evaluated only qualitatively. Robustness to inaccurate meshes and registration failures is also unexplored. End-to-end runtime and memory costs should be reported, and releasing generated training assets would help address reproducibility concerns associated with proprietary mesh generation.
Questions for authors
-
What explains the differences between the reference results in Tables 3 and 4? Do the model configurations, evaluation scenes, or sampling settings differ?
-
How does object-anchored conditioning help reduce generation drift, and how sensitive is its benefit to the number and accuracy of the anchors?
-
How does the method handle inaccurate meshes, registration failures, or valid geometry disconnected from mesh seeds? What are its end-to-end inference time and memory requirements?
4 comments
Response to review (2/n)
For mesh-seeded filtering, we evaluate conditioning quality across 5 cases by comparing renderings of progressively reconstructed point clouds with ground-truth images. The table reports mean PSNR gains over unfiltered point clouds. Confidence filtering at 0.01 improves conditioning quality by suppressing reconstruction noise, while mesh-seeded filtering achieves the largest gain with a 128 × 128 × 128 voxel grid and r = 1. We nevertheless choose r = 2, which achieves a nearly identical gain while allowing connections across a one-voxel gap, providing greater tolerance to fragmented geometry.
| Conf ≥0.01 | Grid32/r1 | Grid32/r2 | Grid64/r1 | Grid64/r2 | Grid128/r1 | Grid128/r2 |
|---|---|---|---|---|---|---|
| +0.0512 | +0.0028 | +0.0080 | +0.0532 | +0.0531 | +0.1402 | +0.1387 |
In addition to using 2 grids, we use multiple object meshes derived from SAM3’s “everything” mode as seeds to preserve valid regions disconnected by occlusion or sparse sampling, increasing coverage across the scene. We also acknowledge that mesh-seeded filtering parameters (voxel sizes and connectivity grid) may require adjustment for complex scenes or inaccurate results/failure cases and users could use it as an optional choice. Each filtering run takes an average of 1 seconds, making it practical to evaluate several parameter settings and select the most suitable result (+0.2421 in those 5 test cases).
Q. Runtime and Memory.
On a single NVIDIA A100 80GB GPU, our method generates 1,000 frames in 166.4 minutes with a peak memory usage of 44.76 GiB, running faster than VerseCrafter, Lyra2, NeoVerse, and Voyager but slower than Gen3C, AnyRecon in the same 832 * 480 resolution. Diffusion denoising accounts for 150.27 minutes, other chunk processing takes 7.31 minutes, and preprocessing and remaining overhead take 8.82 minutes.
| Method | Model Time (min)↓ | Total Time (min)↓ | Peak Memory (GiB) |
|---|---|---|---|
| Ours | 157.6 | 166.4 | 44.76 |
| Gen3C | 38.9 | 41.0 | 21.98 |
| AnyRecon | 50.6 | 54.1 | 40.38 |
| VerseCrafter | 148.1 | 172.6 | 54.87 |
| Lyra2 | 208.7 | 225.9 | 66.73 |
| NeoVerse | 239.1 | 254.8 | 46.54 |
| Voyager | 457.1 | 514.3 | 42.28 |
| Component / Step | Time (min) | Description / Details |
|---|---|---|
| Video diffusion denoising | ~150.27 | 13 chunks, 30 steps each |
| Other processing across chunks | ~7.31 | Input preparation, text/VAE processing, video encoding and saving |
| Processing outside chunks | 4.06 | Model initialization, initial and cumulative Pi3 control updates, stitching and validation |
| First-frame Pi3 | 1.79 | Including model loading |
| First-frame SAM3 segmentation | 0.74 | Including model loading |
| First-frame RecGen | 1.93 | Model loading, mesh generation |
| Voxel-based filtering | 0.30 | Point-cloud filtering |
| Total | 166.40 | Approximately 2 hours 46 minutes |
| Astra: 1 iteration (optional) | 2.89 | Excluded from the total |
| Astra: 2 iterations (optional) | 6.23 | Excluded from the total |
Q: Data Release.
We are happy to release the data curation method and plan to release some data and code.
Reminder to the reviewer
Thanks again for your insightful feedback on our work! We've carefully worked to address your comments/questions and update the draft and images in the revision pdf. We are happy to offer more details if helpful. Are there any further questions or concerns we should discuss?
Further reponse to the reviewer
Hi, we thank the reviewer for the great suggestions! Feel free to ask more questions if it helps in increasing your confidence in our work. We also added the new Table 9-13 and new Fig.14, 15, 16 and 17 in page 20-28, including more diverse cases, including driving scenes, landscapes with full/half/inaccurate mesh or point cloud conditioning, and imperfect hybrid embeddings. Please see more details in page 20-28 in the revised pdf. Thanks again for your valuable suggestions!
Response to reviewer (1/n)
We thank the reviewer for the insightful comments and are encouraged by their recognition of (1) our novel combination of point-cloud and mesh-based visual guidance with hybrid camera embeddings and (2) the effectiveness of our method in long-video generation and 3D reconstruction.
Q:Inconsistent Tables.
Thank you for pointing out this inconsistency. Two factors explain the differences. First, Table 1 reports frame-count-weighted averages over both DL3DV and Tanks and Temples, whereas Tables 3 and 4 evaluate DL3DV scenes only. Second, at submission, Table 3 used checkpoints trained for 80% iterations, while Table 4 used checkpoints trained for full iterations, because the ablation runs had reached different training stages by the deadline. We acknowledge that this distinction was insufficiently documented and causes misunderstanding. We have now completed the remaining ablation training and the updated results are presented in the table below. The current setting still outperforms the other visual guidance configurations and the global embedding baseline.
Q: Object Anchor Ablation and Failure Case Analysis
Thank you for suggesting an ablation on the number of object anchors. We acknowledge that global camera embeddings are computed analytically and do not themselves accumulate drift and drift arises during video generation using global embeddings. Object-anchored conditioning explicitly encodes camera–object relationships relative to fixed 3D anchors, providing persistent spatial references that encourage consistent object placement across autoregressive chunks. In addition, our curated training data include scenes with only one or two object anchors. We pad these inputs to the required size during training, allowing the model to handle varying numbers of available anchors.
We reduce the number of anchors from 3, 2 and 1 to evaluate how anchor count affects generation quality in 5 cases. We additionally evaluate 5 test cases (mesh numbers range from 3 to 17) with inaccurate conditions by reducing geometric coverage by retaining approximately half the meshes, reducing the average from 9 to 4 objects per scene, or applying a point-cloud (PLY) confidence threshold of 0.3, which removes approximately 44.7% of the points. It shows modest performance degradation with fewer anchors while retaining the ability to generate long videos. In the table, it could be noticed that 47% removing point-cloud points substantially reduces conditioning PSNR (-2.0) but only lowers a little generation PSNR(-0.2). Shifting mesh positions only degrades performance a little as well. Using one or two anchors causes relatively small changes (since in our training dataset, some scenes only have 1-2 anchors), and retaining half the meshes has a modest impact. In terms of inaccurate meshes, registration errors, and incomplete geometry, our model is trained to handle imperfect inputs through data augmentation and diverse conditioning inputs, including sparser RecGen meshes (in the training data curation, the feed forward mesh generation might have some inaccurate results as well), denser Astra meshes (might include some coarse mesh shape from parametric code-based modelling), and both point clouds progressively reconstructed from previously generated frames and full point clouds.