- Rating
- 2
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 1
Read review
Paper summary
This paper proposes PixWorld, a pixel-space diffusion framework that unifies feed-forward 3D scene reconstruction and generation in a single model. PixWorld replaces the VAE latent interface with a 32×32 pixel patch embedding, which the authors initialize through latent-to-pixel (L2P) input alignment. The input views are partitioned into clean and noisy subsets. Clean views drive reconstruction and noisy views are generated, and a shared DiT predicts both RGB images and pixel-aligned 3D Gaussians. Following FlashWorld, the RGB branch then serves as the teacher for DMD distillation into a 4-step 3DGS generator. The authors further introduce PixWorld-Streaming, which combines Causal Forcing++ initialization, Rolling Forcing rollout distillation, and ReCoSplat-style incremental Gaussian assembly for online reconstruction and generation.
Critical strengths
- Unified framework for generation and reconstruction with streaming support.
A single model performs feed-forward reconstruction and generation through the clean/noisy view partition. It outputs an explicit 3DGS scene in both cases. PixWorld-Streaming extends this to an autoregressive setting, so the model can take streaming inputs and incrementally reconstruct or generate minute-long sequences.
- Efficiency via few-step distillation.
The RGB-to-3DGS DMD distillation produces a few step generator. It outperforms baselines over generation speed.
Critical weaknesses
- Limited novelty: most components are combinations of existing methods.
The core pipeline closely follows FlashWorld. FlashWorld already trains a 3DGS decoder by cross-mode DMD from a multi-view RGB teacher and distills it into a few-step generator. Relative to FlashWorld, the contributions reduce to three extensions, none of which is sufficiently justified:
(1) Pixel-space diffusion loss: Replacing the latent space with pixel space is the central claim, but it is not supported by a comparison with the original Wan2.2 3D VAE. See Weakness 2 and Q2.
(2) Extension to an AR video diffusion model: PixWorld-Streaming is essentially a straightforward combination of the proposed 3D generator with existing autoregressive video diffusion recipes, rather than a new streaming formulation. Each stage is taken directly from prior works. Autoregressive generation of 3D representations has also been explored before, e.g., VGGT-World, which the paper itself cites in App. A.1. The paper should therefore clarify what, beyond the combination itself, is technically new. It should also position this extension with respect to prior AR video world models and streaming 3D generation and reconstruction methods.
(3) Joint generation and reconstruction: Combining generation and reconstruction in a single model has been explored before (e.g., Gen3r, Aether). Although these methods differ in some implementation details, the paper does not position its unification against them. Moreover, the reconstruction evaluation (Table 1) compares only against reconstruction-only methods. So the benefit of the proposed unification remains unclear.
- The central claim of the paper is not supported by any experiment.
The paper’s main contribution is the claim that operating in pixel space, rather than in a VAE latent space, benefits both tasks. According to the paper, removing the VAE (i) avoids the information loss of autoencoder compression and (ii) allows the diffusion objective to directly optimize the 3D scene. However, no experiment isolates this effect against the most relevant alternative, the latent space of the pretrained video VAE that the model is built upon. The core contribution therefore remains unverified.
(1) The only direct evidence (Tables 6–7, 11–12) uses a 1.3B DiT trained from scratch at 256×256 for 100K steps. It compares only against image autoencoders (SD-VAE, RAE). The final model, however, is initialized from Wan2.2-TI2V-5B. Its own pretrained video VAE is therefore the natural and necessary baseline: the same training setting with a Gaussian head on Wan-VAE latents (or on VAE-decoded RGB). Comparisons with external latent-space methods such as FlashWorld do not isolate this effect either, since they differ in training setup.
(2) Prior work suggests the opposite. VIST3A reports that stitching a reconstructor onto a video VAE’s latent space improves novel-view synthesis over the pixel input reconstructor (see VIST3A Table 3), attributing the gain to the richer appearance representation of video VAE latents. It further shows that reconstructing directly from latents is more robust to latent perturbations than sequential RGB decoding followed by reconstruction (see VIST3A Appendix D.2, Fig. 8). Under noise injected into the latents, the latent space reconstructor consistently outperforms the decode-then-reconstruct pipeline, even at perturbation levels where visual artifacts are hardly perceptible. The claim that performance improves because of pixel space, made without any comparison against the pretrained Wan2.2 VAE latent space, is therefore not convincing.
Questions for authors
- Fairness of the Table 1 pose-free methods evaluation.
Table 1 places pose-free methods (AnySplat, YoNoSplat) alongside PixWorld, which uses ground-truth poses as input. Please clarify whether the pose-free baselines were given ground-truth poses, or whether their predicted camera frames were aligned to the ground-truth target poses before rendering (e.g., by test-time target-pose optimization). If neither was done, the baselines should be re-evaluated for a fair comparison (e.g., refer to NoPoSplat, AnySplat, InstantSplat evaluation setting). AnySplat’s notably low score suggests that the evaluation may be affected by such misalignment.
- How is L2P alignment (Eq. 5) performed with a 3D VAE?
The Wan2.2 VAE is a causal 3D VAE that compresses four frames into one latent frame ( latent frames). PixWorld instead maps each view to its own token frame with a (1, 32, 32) patch. Is the alignment target computed by encoding each view independently as a single-frame latent (i.e., aligned one latent at a time)? How are the ten non-RGB input channels handled during alignment?
9 comments
Response to Reviewer jYUh (Part 2/4)
W2. Pixel-space modeling and the Wan-VAE comparison
“The central claim of the paper is not supported by any experiment.”“The paper’s main contribution is the claim that operating in pixel space, rather than in a VAE latent space, benefits both tasks. According to the paper, removing the VAE (i) avoids the information loss of autoencoder compression and (ii) allows the diffusion objective to directly optimize the 3D scene. However, no experiment isolates this effect against the most relevant alternative, the latent space of the pretrained video VAE that the model is built upon. The core contribution therefore remains unverified.”
“(1) The only direct evidence (Tables 6–7, 11–12) uses a 1.3B DiT trained from scratch at 256×256 for 100K steps. It compares only against image autoencoders (SD-VAE, RAE). The final model, however, is initialized from Wan2.2-TI2V-5B. Its own pretrained video VAE is therefore the natural and necessary baseline: the same training setting with a Gaussian head on Wan-VAE latents (or on VAE-decoded RGB). Comparisons with external latent-space methods such as FlashWorld do not isolate this effect either, since they differ in training setup.”
“(2) Prior work suggests the opposite. VIST3A reports that stitching a reconstructor onto a video VAE’s latent space improves novel-view synthesis over the pixel input reconstructor (see VIST3A Table 3), attributing the gain to the richer appearance representation of video VAE latents. It further shows that reconstructing directly from latents is more robust to latent perturbations than sequential RGB decoding followed by reconstruction (see VIST3A Appendix D.2, Fig. 8). Under noise injected into the latents, the latent space reconstructor consistently outperforms the decode-then-reconstruct pipeline, even at perturbation levels where visual artifacts are hardly perceptible. The claim that performance improves because of pixel space, made without any comparison against the pretrained Wan2.2 VAE latent space, is therefore not convincing.”
We agree that Wan2.2-VAE is the most relevant comparison and have added it to both tasks.
1. Direct Wan2.2-VAE comparison.
We add Wan2.2-VAE, the latent space of our backbone, to the controlled 1.3B ablations under the same training setting. Pixel space outperforms SD-VAE, RAE and Wan2.2-VAE on all generation and reconstruction metrics (Tables C–D).
Table C. Single-view conditional generation (extended Table 6).
| Representation | PSNR ↑ | LPIPS ↓ | IQ ↑ | AUC@5 ↑ |
|---|---|---|---|---|
| SD-VAE | 16.04 | 0.447 | 0.493 | 0.335 |
| RAE | 15.46 | 0.486 | 0.501 | 0.372 |
| Wan2.2-VAE | 16.37 | 0.615 | 0.538 | 0.364 |
| Pixel space | 17.95 | 0.353 | 0.603 | 0.538 |
Table D. RE10K reconstruction with eight input views (extended Table 7).
| Representation | PSNR ↑ | SSIM ↑ | LPIPS ↓ |
|---|---|---|---|
| SD-VAE | 27.47 | 0.865 | 0.172 |
| RAE | 26.89 | 0.853 | 0.188 |
| Wan2.2-VAE | 27.68 | 0.871 | 0.145 |
| Pixel space | 28.16 | 0.885 | 0.106 |
Response to Reviewer jYUh (Part 3/4)
W2 (continued). Relation to VIST3A and robustness
2. Relation to VIST3A.
- Training difference. In VIST3A Table 3, the latent-stitched reconstructor is further fine-tuned for 50 epochs, while the AnySplat baseline is used with its original DINOv2 encoder, so part of the gain may also come from the additional training.
- Matched comparison. We run the same AnySplat comparison under identical training: with DINO removed, the full AnySplat backbone takes either pixel-unshuffled RGB or frozen Wan2.2-VAE latents through a linear adapter, with the same cameras, losses, optimizer and 10K-step budget, fitted on each of 10 RealEstate10K scenes.
- Result. Pixel input reaches higher rendering quality (30.36 vs. 30.17 dB; first two rows of Table E).
3. Robustness to input perturbations.
- Comparability. In VIST3A's perturbation study, noise is added to the latent and the decoded noisy image serves as the RGB input, so the latent and the image do not lose the same amount of information.
- Matched perturbation. We instead freeze both models and add Gaussian noise to each branch's own input, calibrated so that the perturbed inputs have the same image PSNR: the perturbed latent is decoded by the Wan2.2 VAE and compared with the decoded clean latent, and the RGB noise is scaled to match this PSNR.
- Result. At matched input PSNRs of 35 and 30 dB, averaged over three seeds, pixel input degrades less (0.78/1.50 vs. 0.87/2.82 dB PSNR drop; Table E), showing that it is also more robust.
Table E. Rendering quality under matched input perturbations (mean over 10 scenes).
| Input PSNR | Representation | PSNR ↑ | SSIM ↑ | LPIPS ↓ | ΔPSNR ↓ |
|---|---|---|---|---|---|
| Clean | Wan latent | 30.17 | 0.9362 | 0.0320 | – |
| Clean | Pixel | 30.36 | 0.9387 | 0.0306 | – |
| 35 dB | Wan latent | 29.30 | 0.9156 | 0.0592 | 0.87 |
| 35 dB | Pixel | 29.58 | 0.9223 | 0.0470 | 0.78 |
| 30 dB | Wan latent | 27.35 | 0.8761 | 0.1121 | 2.82 |
| 30 dB | Pixel | 28.86 | 0.9093 | 0.0617 | 1.50 |
Response to Reviewer jYUh (Part 1/4)
We thank the reviewer for the detailed feedback and for recognizing our unified framework, streaming support, and inference efficiency. We address the concerns about novelty, pixel-space modeling, evaluation fairness, and L2P alignment below.
We thank the reviewer for the detailed comparison with FlashWorld. Beyond FlashWorld, PixWorld changes where 3D is modeled: a pretrained video diffusion model operates in pixel space and outputs a renderable 3DGS scene that serves both reconstruction and generation, offline and in streaming. Table A summarizes this positioning, and we have added it to the related work of the revised paper (Table 9 in Appendix A).
Table A. Capability comparison.
1. Pixel-space modeling vs. FlashWorld.
FlashWorld denoises VAE latents, and its distillation gradients must pass through the lossy VAE encoder. PixWorld applies diffusion and distribution matching directly to Gaussian renderings, with gradients reaching the Gaussians only through the renderer. The requested Wan2.2-VAE control (W2) shows that pixel space outperforms SD-VAE, RAE, and Wan2.2-VAE in both generation and reconstruction.
2. Streaming as a new formulation.
AR video world models generate frames without a persistent 3D state, VGGT-World reconstructs and forecasts geometry without a renderable appearance model, and streaming 3DGS methods such as ReCoSplat only reconstruct. PixWorld-Streaming maintains one renderable Gaussian scene that is both reconstructed and generated, and its chunk-rollout distillation matches renderings of this accumulated scene to the RGB teacher. Revisit PSNR rises to 20.33 dB vs. 14.18/14.04 for LingBot-World/SANA-WM, and streaming reconstruction surpasses ReCoSplat (22.17 vs. 22.00 dB).
3. Unification vs. Gen3R and Aether.
Gen3R and Aether denoise appearance and geometry as latents and decode 3D afterwards. PixWorld uses one pixel-space forward pass: all-clean views give feed-forward reconstruction, and noisy views add generation. PixWorld outperforms both in generation (AUC@5 0.622 vs. 0.148 for Gen3R on RealEstate10K, Table 2; WorldScore 73.11 vs. 55.64 for Aether, Table 3), and rendering Gaussians during denoising beats generate-then-reconstruct (AUC@5 0.568 vs. 0.493). The point-cloud reconstruction comparison with Gen3R and Aether on TartanAir is reported in Table B.
Table B. TartanAir point-cloud reconstruction (50 test samples).