acceptodds
Withdrawn from ICLR 2027

PixWorld: Unifying 3D Scene Generation and Reconstruction in Pixel Space

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

3D reconstruction and generation typically follow separate paradigms: pixel-based regression for reconstruction and latent diffusion for generation. Recent methods seek to unify them in a shared latent space. However, autoencoder compression introduces information loss in both tasks, while latent-space diffusion objectives supervise encoded latents rather than the resulting 3D scenes. To address these limitations, we introduce PixWorld, a single model that unifies 3D reconstruction and generation under a pixel-space diffusion paradigm. By supervising diffusion on rendered images rather than encoded latents, PixWorld eliminates the autoencoder bottleneck and directly optimizes the 3D scene. Furthermore, we introduce PixWorld-Streaming through autoregressive diffusion distillation, extending this unified framework beyond a fixed view budget to support online reconstruction and generation over minute-long sequences. Experiments show that PixWorld outperforms latent-space generation methods and matches state-of-the-art reconstruction, while PixWorld-Streaming retains comparable quality under causal streaming inputs, demonstrating the effectiveness of a unified pixel-space approach.

Primary area:
generative models
Keywords:
3D Generation, 3D Reconstruction, Pixel Space Diffusion

Reviews

Among:
Shape:
1234this paper, Top 49%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.33 over 3 reviews, above 51% of the others.
Reviewer_jYUhOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

This paper proposes PixWorld, a pixel-space diffusion framework that unifies feed-forward 3D scene reconstruction and generation in a single model. PixWorld replaces the VAE latent interface with a 32×32 pixel patch embedding, which the authors initialize through latent-to-pixel (L2P) input alignment. The input views are partitioned into clean and noisy subsets. Clean views drive reconstruction and noisy views are generated, and a shared DiT predicts both RGB images and pixel-aligned 3D Gaussians. Following FlashWorld, the RGB branch then serves as the teacher for DMD distillation into a 4-step 3DGS generator. The authors further introduce PixWorld-Streaming, which combines Causal Forcing++ initialization, Rolling Forcing rollout distillation, and ReCoSplat-style incremental Gaussian assembly for online reconstruction and generation.

Critical strengths
  1. Unified framework for generation and reconstruction with streaming support.

A single model performs feed-forward reconstruction and generation through the clean/noisy view partition. It outputs an explicit 3DGS scene in both cases. PixWorld-Streaming extends this to an autoregressive setting, so the model can take streaming inputs and incrementally reconstruct or generate minute-long sequences.

  1. Efficiency via few-step distillation.

The RGB-to-3DGS DMD distillation produces a few step generator. It outperforms baselines over generation speed.

Critical weaknesses
  1. Limited novelty: most components are combinations of existing methods.

The core pipeline closely follows FlashWorld. FlashWorld already trains a 3DGS decoder by cross-mode DMD from a multi-view RGB teacher and distills it into a few-step generator. Relative to FlashWorld, the contributions reduce to three extensions, none of which is sufficiently justified:

(1) Pixel-space diffusion loss: Replacing the latent space with pixel space is the central claim, but it is not supported by a comparison with the original Wan2.2 3D VAE. See Weakness 2 and Q2.

(2) Extension to an AR video diffusion model: PixWorld-Streaming is essentially a straightforward combination of the proposed 3D generator with existing autoregressive video diffusion recipes, rather than a new streaming formulation. Each stage is taken directly from prior works. Autoregressive generation of 3D representations has also been explored before, e.g., VGGT-World, which the paper itself cites in App. A.1. The paper should therefore clarify what, beyond the combination itself, is technically new. It should also position this extension with respect to prior AR video world models and streaming 3D generation and reconstruction methods.

(3) Joint generation and reconstruction: Combining generation and reconstruction in a single model has been explored before (e.g., Gen3r, Aether). Although these methods differ in some implementation details, the paper does not position its unification against them. Moreover, the reconstruction evaluation (Table 1) compares only against reconstruction-only methods. So the benefit of the proposed unification remains unclear.

  1. The central claim of the paper is not supported by any experiment.

The paper’s main contribution is the claim that operating in pixel space, rather than in a VAE latent space, benefits both tasks. According to the paper, removing the VAE (i) avoids the information loss of autoencoder compression and (ii) allows the diffusion objective to directly optimize the 3D scene. However, no experiment isolates this effect against the most relevant alternative, the latent space of the pretrained video VAE that the model is built upon. The core contribution therefore remains unverified.

(1) The only direct evidence (Tables 6–7, 11–12) uses a 1.3B DiT trained from scratch at 256×256 for 100K steps. It compares only against image autoencoders (SD-VAE, RAE). The final model, however, is initialized from Wan2.2-TI2V-5B. Its own pretrained video VAE is therefore the natural and necessary baseline: the same training setting with a Gaussian head on Wan-VAE latents (or on VAE-decoded RGB). Comparisons with external latent-space methods such as FlashWorld do not isolate this effect either, since they differ in training setup.

(2) Prior work suggests the opposite. VIST3A reports that stitching a reconstructor onto a video VAE’s latent space improves novel-view synthesis over the pixel input reconstructor (see VIST3A Table 3), attributing the gain to the richer appearance representation of video VAE latents. It further shows that reconstructing directly from latents is more robust to latent perturbations than sequential RGB decoding followed by reconstruction (see VIST3A Appendix D.2, Fig. 8). Under noise injected into the latents, the latent space reconstructor consistently outperforms the decode-then-reconstruct pipeline, even at perturbation levels where visual artifacts are hardly perceptible. The claim that performance improves because of pixel space, made without any comparison against the pretrained Wan2.2 VAE latent space, is therefore not convincing.

Questions for authors
  1. Fairness of the Table 1 pose-free methods evaluation.

Table 1 places pose-free methods (AnySplat, YoNoSplat) alongside PixWorld, which uses ground-truth poses as input. Please clarify whether the pose-free baselines were given ground-truth poses, or whether their predicted camera frames were aligned to the ground-truth target poses before rendering (e.g., by test-time target-pose optimization). If neither was done, the baselines should be re-evaluated for a fair comparison (e.g., refer to NoPoSplat, AnySplat, InstantSplat evaluation setting). AnySplat’s notably low score suggests that the evaluation may be affected by such misalignment.

  1. How is L2P alignment (Eq. 5) performed with a 3D VAE?

The Wan2.2 VAE is a causal 3D VAE that compresses four frames into one latent frame (1+(T−1)/41+(T-1)/4 latent frames). PixWorld instead maps each view to its own token frame with a (1, 32, 32) patch. Is the alignment target computed by encoding each view independently as a single-frame latent (i.e., aligned one latent at a time)? How are the ten non-RGB input channels handled during alignment?

9 comments
AuthorsOpenReview ↗

Response to Reviewer jYUh (Part 1/4)

We thank the reviewer for the detailed feedback and for recognizing our unified framework, streaming support, and inference efficiency. We address the concerns about novelty, pixel-space modeling, evaluation fairness, and L2P alignment below.


W1. Contributions and positioning
“Limited novelty: most components are combinations of existing methods.”

“The core pipeline closely follows FlashWorld. FlashWorld already trains a 3DGS decoder by cross-mode DMD from a multi-view RGB teacher and distills it into a few-step generator. Relative to FlashWorld, the contributions reduce to three extensions, none of which is sufficiently justified:”

“(1) Pixel-space diffusion loss: Replacing the latent space with pixel space is the central claim, but it is not supported by a comparison with the original Wan2.2 3D VAE. See Weakness 2 and Q2.”

“(2) Extension to an AR video diffusion model: PixWorld-Streaming is essentially a straightforward combination of the proposed 3D generator with existing autoregressive video diffusion recipes, rather than a new streaming formulation. Each stage is taken directly from prior works. Autoregressive generation of 3D representations has also been explored before, e.g., VGGT-World, which the paper itself cites in App. A.1. The paper should therefore clarify what, beyond the combination itself, is technically new. It should also position this extension with respect to prior AR video world models and streaming 3D generation and reconstruction methods.”

“(3) Joint generation and reconstruction: Combining generation and reconstruction in a single model has been explored before (e.g., Gen3r, Aether). Although these methods differ in some implementation details, the paper does not position its unification against them. Moreover, the reconstruction evaluation (Table 1) compares only against reconstruction-only methods. So the benefit of the proposed unification remains unclear.”

We thank the reviewer for the detailed comparison with FlashWorld. Beyond FlashWorld, PixWorld changes where 3D is modeled: a pretrained video diffusion model operates in pixel space and outputs a renderable 3DGS scene that serves both reconstruction and generation, offline and in streaming. Table A summarizes this positioning, and we have added it to the related work of the revised paper (Table 9 in Appendix A).

Table A. Capability comparison.

MethodVideo priorPixel spaceRenderable 3DGSRecon. + Gen.Streaming
DiffusionGS✗✓✓✓✗
FlashWorld✓✗✓✗✗
Gen3R✓✗✗✓✗
Aether✓✗✗✓✗
VGGT-World✗✗✗✓✓
Rolling Forcing / SANA-WM✓✗✗✗✓
ReCoSplat✗–✓✗✓
PixWorld (ours)✓✓✓✓✓

1. Pixel-space modeling vs. FlashWorld.

FlashWorld denoises VAE latents, and its distillation gradients must pass through the lossy VAE encoder. PixWorld applies diffusion and distribution matching directly to Gaussian renderings, with gradients reaching the Gaussians only through the renderer. The requested Wan2.2-VAE control (W2) shows that pixel space outperforms SD-VAE, RAE, and Wan2.2-VAE in both generation and reconstruction.

2. Streaming as a new formulation.

AR video world models generate frames without a persistent 3D state, VGGT-World reconstructs and forecasts geometry without a renderable appearance model, and streaming 3DGS methods such as ReCoSplat only reconstruct. PixWorld-Streaming maintains one renderable Gaussian scene that is both reconstructed and generated, and its chunk-rollout distillation matches renderings of this accumulated scene to the RGB teacher. Revisit PSNR rises to 20.33 dB vs. 14.18/14.04 for LingBot-World/SANA-WM, and streaming reconstruction surpasses ReCoSplat (22.17 vs. 22.00 dB).

3. Unification vs. Gen3R and Aether.

Gen3R and Aether denoise appearance and geometry as latents and decode 3D afterwards. PixWorld uses one pixel-space forward pass: all-clean views give feed-forward reconstruction, and noisy views add generation. PixWorld outperforms both in generation (AUC@5 0.622 vs. 0.148 for Gen3R on RealEstate10K, Table 2; WorldScore 73.11 vs. 55.64 for Aether, Table 3), and rendering Gaussians during denoising beats generate-then-reconstruct (AUC@5 0.568 vs. 0.493). The point-cloud reconstruction comparison with Gen3R and Aether on TartanAir is reported in Table B.

Table B. TartanAir point-cloud reconstruction (50 test samples).

MethodAccuracy ↓Completeness ↓CD ↓
Aether3.462.893.18
Gen3R2.341.561.95
PixWorld (Ours)1.871.911.89
AuthorsOpenReview ↗

Response to Reviewer jYUh (Part 2/4)

W2. Pixel-space modeling and the Wan-VAE comparison
“The central claim of the paper is not supported by any experiment.”

“The paper’s main contribution is the claim that operating in pixel space, rather than in a VAE latent space, benefits both tasks. According to the paper, removing the VAE (i) avoids the information loss of autoencoder compression and (ii) allows the diffusion objective to directly optimize the 3D scene. However, no experiment isolates this effect against the most relevant alternative, the latent space of the pretrained video VAE that the model is built upon. The core contribution therefore remains unverified.”

“(1) The only direct evidence (Tables 6–7, 11–12) uses a 1.3B DiT trained from scratch at 256×256 for 100K steps. It compares only against image autoencoders (SD-VAE, RAE). The final model, however, is initialized from Wan2.2-TI2V-5B. Its own pretrained video VAE is therefore the natural and necessary baseline: the same training setting with a Gaussian head on Wan-VAE latents (or on VAE-decoded RGB). Comparisons with external latent-space methods such as FlashWorld do not isolate this effect either, since they differ in training setup.”

“(2) Prior work suggests the opposite. VIST3A reports that stitching a reconstructor onto a video VAE’s latent space improves novel-view synthesis over the pixel input reconstructor (see VIST3A Table 3), attributing the gain to the richer appearance representation of video VAE latents. It further shows that reconstructing directly from latents is more robust to latent perturbations than sequential RGB decoding followed by reconstruction (see VIST3A Appendix D.2, Fig. 8). Under noise injected into the latents, the latent space reconstructor consistently outperforms the decode-then-reconstruct pipeline, even at perturbation levels where visual artifacts are hardly perceptible. The claim that performance improves because of pixel space, made without any comparison against the pretrained Wan2.2 VAE latent space, is therefore not convincing.”

We agree that Wan2.2-VAE is the most relevant comparison and have added it to both tasks.

1. Direct Wan2.2-VAE comparison.

We add Wan2.2-VAE, the latent space of our backbone, to the controlled 1.3B ablations under the same training setting. Pixel space outperforms SD-VAE, RAE and Wan2.2-VAE on all generation and reconstruction metrics (Tables C–D).

Table C. Single-view conditional generation (extended Table 6).

RepresentationPSNR ↑LPIPS ↓IQ ↑AUC@5 ↑
SD-VAE16.040.4470.4930.335
RAE15.460.4860.5010.372
Wan2.2-VAE16.370.6150.5380.364
Pixel space17.950.3530.6030.538

Table D. RE10K reconstruction with eight input views (extended Table 7).

RepresentationPSNR ↑SSIM ↑LPIPS ↓
SD-VAE27.470.8650.172
RAE26.890.8530.188
Wan2.2-VAE27.680.8710.145
Pixel space28.160.8850.106
AuthorsOpenReview ↗

Response to Reviewer jYUh (Part 3/4)

W2 (continued). Relation to VIST3A and robustness

2. Relation to VIST3A.

  • Training difference. In VIST3A Table 3, the latent-stitched reconstructor is further fine-tuned for 50 epochs, while the AnySplat baseline is used with its original DINOv2 encoder, so part of the gain may also come from the additional training.
  • Matched comparison. We run the same AnySplat comparison under identical training: with DINO removed, the full AnySplat backbone takes either pixel-unshuffled RGB or frozen Wan2.2-VAE latents through a linear adapter, with the same cameras, losses, optimizer and 10K-step budget, fitted on each of 10 RealEstate10K scenes.
  • Result. Pixel input reaches higher rendering quality (30.36 vs. 30.17 dB; first two rows of Table E).

3. Robustness to input perturbations.

  • Comparability. In VIST3A's perturbation study, noise is added to the latent and the decoded noisy image serves as the RGB input, so the latent and the image do not lose the same amount of information.
  • Matched perturbation. We instead freeze both models and add Gaussian noise to each branch's own input, calibrated so that the perturbed inputs have the same image PSNR: the perturbed latent is decoded by the Wan2.2 VAE and compared with the decoded clean latent, and the RGB noise is scaled to match this PSNR.
  • Result. At matched input PSNRs of 35 and 30 dB, averaged over three seeds, pixel input degrades less (0.78/1.50 vs. 0.87/2.82 dB PSNR drop; Table E), showing that it is also more robust.

Table E. Rendering quality under matched input perturbations (mean over 10 scenes).

Input PSNRRepresentationPSNR ↑SSIM ↑LPIPS ↓ΔPSNR ↓
CleanWan latent30.170.93620.0320–
CleanPixel30.360.93870.0306–
35 dBWan latent29.300.91560.05920.87
35 dBPixel29.580.92230.04700.78
30 dBWan latent27.350.87610.11212.82
30 dBPixel28.860.90930.06171.50
Reviewer_4eLmOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper presents PixWorld, a pixel-space diffusion framework that unifies multi-view 3D reconstruction and generation. The method split input views into clean and noisy subsets: clean views provide reconstruction evidence, while noisy views are generated conditioned on the available observations. PixWorld adapts a Wan2.2-5B backbone from latent to pixel inputs and uses a shared DiT to predict both RGB images and pixel-aligned 3D Gaussians. Experiments cover multi-view reconstruction, one- and two-view conditional generation, WorldScore, streaming reconstruction, long-sequence generation, inference efficiency, and ablation studies.

Critical strengths
  1. The clean/noisy view formulation provides a simple and practically useful way to unify reconstruction and generation within a single model and representation.
  2. The method covers a broad range of settings, including offline reconstruction, conditional generation, text-to-3D, incremental reconstruction, and long-horizon generation.
Critical weaknesses
  1. In the supplementary material, the bed in the first example, the building in the second, and the door handle in the third exhibit noticeable artifacts in fine details, suggesting poor geometric quality. For the last tw landscape examples, it is unclear why the baselines are considered worse, since they appear to involve substantially larger camera movements.
  2. The visual artifacts mentioned in my first point challenge whether the pixel-space approach actually works, even though this is the authors’ primary claim. Given the current visual results, I am not convinced that pixel-space modeling is effective.
  3. The methods are largely inherited from prior work, making the contribution a system integration. The offline model combines L2P initialization, existing pixel-space heads and perceptual objectives, and FlashWorld-style cross-mode DMD. The streaming extension further combines Causal Forcing++, Rolling Forcing, and Reco Splat-style scene assembly. The current ablations are inefficient.
  4. The long-sequence experiment verifies that the model can complete a 60s rollout, but not that it maintains a globally consistent 3D scene. There is no evaluation of cross-chunk geometric alignment, accumulated 3D drift, duplicated Gaussians, geometric loop closure, or quality degradation over time. Revisit PSNR only measures appearance consistency, while all Gaussians are retained and scene memory grows substantially with sequence length. Robustness to errors in the externally estimated camera poses is also not evaluated.
Questions for authors

The offline PSNR, SSIM, and LPIPS comparisons are computed after center-cropping both the predictions and ground-truth images to a square and resizing them to 256*256. Given that PixWorld is motivated by preserving information and fine-grained details in pixel space, can evaluation at this relatively low resolution faithfully reflect its claimed advantages and the models’ actual performance at their native output resolutions? Could the authors additionally report native-resolution, full-frame results without center cropping? Other metrics, including VBench, pi3based pose AUC, and WorldScore, follow their respective evaluation protocols

4 comments
AuthorsOpenReview ↗

Response to Reviewer 4eLm (Part 1/3)

We thank the reviewer for the constructive feedback and for recognizing the simple and practical clean/noisy formulation and the broad range of settings it supports. We address the concerns about visual quality, pixel-space evidence, contributions, evaluation resolution and long-sequence consistency below.


W1. Fine-detail artifacts and camera motion
“In the supplementary material, the bed in the first example, the building in the second, and the door handle in the third exhibit noticeable artifacts in fine details, suggesting poor geometric quality. For the last two landscape examples, it is unclear why the baselines are considered worse, since they appear to involve substantially larger camera movements.”

We thank the reviewer for the careful inspection of the supplementary examples.

1. Challenge of 3DGS generation.

Inferring entirely unobserved geometry while maintaining 3D consistency is inherently difficult in 3DGS generation. Figs. 9–11 in Appendix D show similar, sometimes more severe, artifacts in FlashWorld and DiffusionGS, highlighting this shared challenge. Despite this difficulty, PixWorld achieves higher WorldScore-Static 3D consistency than latent-space FlashWorld (88.94 vs. 84.93; Table 3), supporting the effectiveness of our pixel-space formulation.

2. Landscape comparisons.

For these landscapes, all methods use the same small-range ACID camera trajectories. Uni3C overshoots the intended return trajectory and loses detail to haze; Voyager shows inaccurate camera motion and grainy, speckled surfaces.


W2. Evidence for pixel-space modeling
“The visual artifacts mentioned in my first point challenge whether the pixel-space approach actually works, even though this is the authors’ primary claim. Given the current visual results, I am not convinced that pixel-space modeling is effective.”

We thank the reviewer for raising this point.

1. Shared challenge of 3DGS generation.

Predicting unseen geometry that remains consistent across views is difficult, particularly with large camera changes and return trajectories, making 3DGS generation prone to artifacts. FlashWorld and DiffusionGS exhibit similar, sometimes more severe, artifacts (Figs. 9–11 in Appendix D), underscoring this shared challenge. Despite this, PixWorld achieves higher WorldScore-Static 3D consistency than latent-space FlashWorld (88.94 vs. 84.93; Table 3).

2. Effectiveness of pixel-space modeling.

Tables A–B, now including Wan2.2-VAE, show that pixel-space modeling outperforms SD-VAE, RAE and Wan2.2-VAE across all reported generation and reconstruction metrics. These controlled ablations directly support the effectiveness of pixel-space modeling.

Table A. Single-view conditional generation.

RepresentationPSNR ↑LPIPS ↓IQ ↑AUC@5 ↑
SD-VAE16.040.4470.4930.335
RAE15.460.4860.5010.372
Wan2.2-VAE16.370.6150.5380.364
Pixel space17.950.3530.6030.538

Table B. RE10K reconstruction with eight input views.

RepresentationPSNR ↑SSIM ↑LPIPS ↓
SD-VAE27.470.8650.172
RAE26.890.8530.188
Wan2.2-VAE27.680.8710.145
Pixel space28.160.8850.106
AuthorsOpenReview ↗

Response to Reviewer 4eLm (Part 2/3)

W3. Contribution and the role of existing components
“The methods are largely inherited from prior work, making the contribution a system integration. The offline model combines L2P initialization, existing pixel-space heads and perceptual objectives, and FlashWorld-style cross-mode DMD. The streaming extension further combines Causal Forcing++, Rolling Forcing, and Reco Splat-style scene assembly. The current ablations are inefficient.”

We thank the reviewer for this comment. PixWorld's novelty lies in its formulation: a pretrained video diffusion model becomes a pixel-space 3D model whose denoising directly outputs a renderable 3DGS scene, unifying reconstruction and generation in offline and streaming settings.

1. Contributions of PixWorld

Pixel-space 3D modeling with a video prior. Prior video-prior 3D methods denoise VAE latents and decode 3D afterwards. To our knowledge, PixWorld is the first to bring a video diffusion prior into pixel space for 3D, with RGB, depth, and Gaussians sharing one backbone and the diffusion loss directly supervising Gaussian renderings.

Pixel-space cross-mode distillation. FlashWorld distills on VAE-encoded renderings, so student gradients must pass through the lossy VAE encoder. PixWorld matches the RGB teacher directly on rendered pixels, letting gradients reach the Gaussians only through the renderer; the 4-step student even surpasses the 50-step model (19.21 vs. 18.82 dB).

2. Contributions of PixWorld-Streaming

Streaming 3DGS reconstruction and generation. AR video models keep no persistent 3D state, and streaming 3DGS methods only reconstruct. PixWorld-Streaming maintains one renderable Gaussian scene that is reconstructed from incoming views and expanded with generated content.

Distribution matching on the accumulated scene. Chunk-rollout distillation matches renderings of the merged scene, including its history, to the RGB teacher, which therefore supervises the persistent scene rather than isolated frames. Revisit PSNR reaches 20.33 dB vs. 14.18/14.04 for LingBot-World/SANA-WM (Table 5).

3. Comprehensive ablations

Tables 6–7, 8(a–e) and 14 systematically evaluate representation space, losses, input alignment, prediction heads, initialization and distillation, fine-tuning scope, and RGB/3DGS switching. We further add Wan2.2-VAE and generate-then-reconstruct controls: pixel space outperforms all three autoencoders, and rendering Gaussians during denoising improves AUC@5 from 0.493 to 0.568.

We have clarified these contributions in the introduction and related work of the revised paper (Table 9 in Appendix A).


Q1. Native-resolution, full-frame evaluation
The reviewer requests PSNR, SSIM, and LPIPS at native resolution on the full frame, without center cropping, in addition to the reported square-crop evaluation.

We thank the reviewer for the question.

1. Evaluation resolution.

The compared methods produce outputs at different native resolutions and aspect ratios, so we score all methods at a common 256×256 resolution, following pixelSplat and MVSplat. A common resolution is necessary because PSNR, SSIM and LPIPS depend on the scoring resolution itself: scoring the same reconstructions at 448×448 instead of 224×224 changes LPIPS by 0.07 (Table C), more than the gap between the leading methods in Table 1. Native-resolution scores would therefore mix model quality with each method's output resolution.

2. High-resolution generation-quality evaluation.

IQ and other VBench metrics, as well as WorldScore, evaluate high-resolution generated outputs using their own official preprocessing, rather than the 256×256 fidelity-scoring images.

Table C. The same 400 target views scored at two resolutions.

Scoring resolutionPSNR ↑SSIM ↑LPIPS ↓
448×44824.28200.82210.2253
224×22424.54620.82240.1552
Δ (224 − 448)+0.2642+0.0003−0.0701
AuthorsOpenReview ↗

Response to Reviewer 4eLm (Part 3/3)

W4. Long-sequence geometry, memory, and pose robustness
“The long-sequence experiment verifies that the model can complete a 60s rollout, but not that it maintains a globally consistent 3D scene. There is no evaluation of cross-chunk geometric alignment, accumulated 3D drift, duplicated Gaussians, geometric loop closure, or quality degradation over time. Revisit PSNR only measures appearance consistency, while all Gaussians are retained and scene memory grows substantially with sequence length. Robustness to errors in the externally estimated camera poses is also not evaluated.”

We thank the reviewer for these suggestions. We add geometric, temporal, memory and pose-robustness evaluations of PixWorld-Streaming on 60-second rollouts.

1. Cross-chunk alignment and accumulated drift.

We unproject 3DGS-rendered depth (PixWorld) or DA3 depth (SANA-WM) into point clouds and compute the Chamfer distance (CD) over the common visible region, without per-pair alignment. PixWorld achieves lower CD than SANA-WM in every interval and stays below 8.5 cm throughout the 60-second rollout, indicating limited drift accumulation.

Table D. CD ↓ (estimated cm) by return-time interval.

Method / time (s)0–1010–2020–3030–4040–5050–60
SANA-WM8.939.179.818.659.2711.01
PixWorld5.896.756.167.376.828.45

2. Geometric loop closure.

For first-visit/return pairs selected from camera poses, PixWorld reduces CD from 11.30 to 8.19 cm for returns during 30–35 s and from 18.84 to 11.32 cm for returns during 55–60 s. Together with revisit PSNR (Table 5), this shows that the persistent Gaussian scene keeps revisited regions consistent in both geometry and appearance.

Table E. Geometric revisit consistency: CD ↓ (estimated cm).

MethodReturns during 30–35 sReturns during 55–60 s
SANA-WM11.3018.84
PixWorld8.1911.32

3. Quality over time.

IQ remains stable over the first 30 s and decreases moderately to 60.45 in the last window, without collapse.

Table F. VBench IQ over six 10-second windows of 60-second rollouts.

Time (s)0–1010–2020–3030–4040–5050–60
IQ ↑65.2965.1364.9763.3861.1460.45

4. Memory and duplicated Gaussians.

Table G compares PixWorld-Streaming with the bidirectional offline model, whose full attention runs out of memory at 128 views. For a fair comparison, both settings retain all Gaussians in this measurement; even so, streaming peak memory grows only from 16.21 to 44.96 GiB as the number of views increases 16×. In actual streaming, voxel-based selection removes duplicated Gaussians from overlapping chunks, so the last 50 s add less 3DGS storage than the first 10 s (1122 vs. 1315 MiB; Table H).

Table G. Peak GPU memory (GiB) on one H100 at 480×832, retaining all Gaussians (Table 20).

Input viewsBidirectional (offline)Streaming (online)
1627.0616.21
3244.4718.56
6470.9222.44
128OOM30.21
256OOM44.96

Table H. Cumulative 3DGS storage (MiB) at the end of each 10-second interval.

Time (s)0–1010–2020–3030–4040–5050–60
Storage1315.401702.961732.672136.132258.542437.39

5. Robustness to estimated poses.

Replacing the training-consistent ViPE poses with LingBot-Map poses changes CMC, IQ and revisit PSNR only marginally.

Table I. Streaming generation with different pose sources.

Pose sourceCMC ↓IQ ↑Revisit PSNR ↑
ViPE1.8563.3920.33
LingBot-Map1.9363.6719.86
AuthorsOpenReview ↗

Follow-up on Our Rebuttal

Dear Reviewer 4eLm,

Thank you for your time and effort in reviewing our paper. Your comments have been valuable in helping us improve its quality. As the author–reviewer discussion period will end in about three days, we would appreciate it if you could read our rebuttal and let us know whether your concerns have been addressed. Thank you again for your time and constructive feedback.

Best regards, The Authors

Reviewer_X5cUOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes PixWorld, a single diffusion model that does both 3D scene reconstruction and 3D scene generation in pixel space. Starting from a pretrained video model (Wan2.2), it takes posed views where some are clean and some are noisy, and predicts both an RGB image and per-pixel 3D Gaussians for every view. The rendered Gaussians are supervised with the same diffusion loss, so the 3D scene is optimized directly. The model is then distilled to four steps, and a streaming version extends it to minute-long sequences. Experiments cover reconstruction, one/two-view generation, WorldScore and streaming.

Critical strengths
  • One model handles reconstruction, generation and streaming, with strong results and fast inference. The demo video shows clearly sharper results than FlashWorld. I find the streaming part the most interesting. PixWorld-Streaming reconstructs incoming views and generates unseen regions into one growing Gaussian scene, and the rooms stay consistent when the camera revisits them. To my knowledge, doing both online in one model has not been shown before, and I would suggest the authors make this a main contribution of the paper.
  • The design choices are well studied (pixel vs. latent space, losses, fine-tuning scope), and converting a latent video model into a pixel-space 3D model is a useful recipe for the community.
Critical weaknesses
  • The central idea has clear prior work that is not cited, and it is not isolated. Diffusion models that predict a 3D representation and supervise its renders have been explored before, e.g., RenderDiffusion, Viewset Diffusion, DMV3D, and especially DiffusionGS, which also uses clean and noisy views with per-pixel Gaussians on the same datasets. None of these is discussed. Unifying reconstruction and generation by treating some views as clean and the rest as noisy is also not new: Matrix3D already does this in one model for images, poses and depth, and it is not cited either. FlashWorld, the main baseline, also already supervises its Gaussian renders in pixel space. In addition, the final four-step model mostly runs in RGB mode and predicts Gaussians only at the last step, so a "generate, then reconstruct" baseline is the key missing comparison. I think the paper should position itself as scaling this idea with a video prior, rather than as a new principle.
  • The evaluation does not yet support the claim of matching state-of-the-art reconstruction. Reconstruction uses a custom protocol, and the baseline numbers are much lower than in their own papers, so they cannot be compared with published results. Strong feed-forward methods such as LVSM and DA3 are missing, and LVSM is even better than PixWorld on one of the paper's own settings. For generation there is no distribution metric (e.g., FID or FVD), and for streaming there is no analysis of quality over time.
Questions for authors
  1. In the pixel vs. latent ablation, how do the latent variants produce Gaussians, and do they use the same losses?
  2. Does supervising diffusion through the rendered Gaussians actually help? Table 8(e) only changes the render loss weight (which also scales the held-out view loss), and Table 13 only changes when sampling switches from RGB to Gaussians, so neither shows what the render supervision adds. Could the authors add a "generate, then reconstruct" control with the existing weights: generate all views in RGB mode only, then reconstruct them with PixWorld's all-clean mode, and compare with the default sampling?
  3. Could the authors report reconstruction on the standard protocols, and add LVSM and DA3?
  4. The paper claims benefits from unifying the two tasks, but no experiment compares the joint model with a model trained on only one task. Does joint training actually improve generation or reconstruction? Even a small comparison in the 1.3B setting used for the pixel vs. latent ablation would help.
  5. In the demo video, PixWorld's renders show ghost effects, for example duplicated, stair-stepped mountain edges and floating fragments in the sky in the generation results (around 1:01 to 1:03), and smeared surfaces near the camera in the streaming results. What causes this? Is it because each view predicts its own pixel-aligned Gaussians, so the same surface ends up at slightly different depths across views? Do the authors fuse or prune Gaussians across views, and could they show depth maps and renders from viewpoints far from the input or generated views?
  6. Could the authors discuss streaming reconstruction methods such as LingBot-Map, and if possible compare geometry accuracy on the same streams, or show results on unposed streams using estimated poses?

Overall: PixWorld is a fast and capable system, and the visual results are convincing. I find the streaming part, which reconstructs and generates one growing 3D scene online, the most valuable contribution, and to my knowledge it has not been shown before. However, the paper frames pixel space as its main idea, which has clear prior work that is not discussed and is not isolated, and the reconstruction claim relies on a custom protocol. I currently lean toward a weak acceptance, mainly because of the streaming results. I hope the authors reposition the paper around streaming, add the missing prior work, and address the questions above, especially the "generate, then reconstruct" control and the ghost effects. If these points are not addressed, I may lower my score.

5 comments
AuthorsOpenReview ↗

Response to Reviewer X5cU (Part 1/4)

We thank the reviewer for the thoughtful feedback and for highlighting the value of our streaming capability. We address each point below, including our relation to prior work, experimental comparisons, and streaming evaluation.


W1. Prior work and positioning
“The central idea has clear prior work that is not cited, and it is not isolated. Diffusion models that predict a 3D representation and supervise its renders have been explored before, e.g., RenderDiffusion, Viewset Diffusion, DMV3D, and especially DiffusionGS, which also uses clean and noisy views with per-pixel Gaussians on the same datasets. None of these is discussed. Unifying reconstruction and generation by treating some views as clean and the rest as noisy is also not new: Matrix3D already does this in one model for images, poses and depth, and it is not cited either. FlashWorld, the main baseline, also already supervises its Gaussian renders in pixel space. In addition, the final four-step model mostly runs in RGB mode and predicts Gaussians only at the last step, so a "generate, then reconstruct" baseline is the key missing comparison. I think the paper should position itself as scaling this idea with a video prior, rather than as a new principle.”

We thank the reviewer for these references. We have added them to the related work of the revised paper, and clarify their relationship to PixWorld below.

1. RenderDiffusion, Viewset Diffusion and DMV3D.

Early diffusion models, for both images and 3D, operated in pixel space, and limitations of the pixel-space models available at the time restricted them to very low resolution. RenderDiffusion, Viewset Diffusion and DMV3D belong to this stage. Both image and 3D diffusion were subsequently largely replaced by latent-space models.

2. DiffusionGS.

DiffusionGS also diffuses in pixel space, predicting pixel-aligned Gaussians from clean and noisy views. It is trained from scratch, and its patch size of 8 keeps the token count high, which makes high resolution difficult. On scenes, both its training and evaluation involve small viewpoint changes, so the task is closer to reconstructing nearby views than to generating unseen regions, and its generation quality remains limited. Please see Table A for the RE10K one-view generation results and Fig. 11 in Appendix D for qualitative comparisons.

3. Gen3R, Matrix3D and FlashWorld.

These pixel-space approaches were subsequently replaced by latent-space methods, many of which build on pretrained image and video priors. Gen3R (cited in the submission) and Matrix3D (added in the revision) unify reconstruction and generation in a latent space. FlashWorld adds pixel-space losses on its Gaussian renders, but its main flow-matching objective is still defined on latents, whereas every PixWorld loss, including the diffusion loss, is computed in pixel space.

We have revised the introduction to follow this progression. Early diffusion models that predict a 3D representation and supervise its renderings operated in pixel space, but they were replaced for two reasons: early pixel-space models were limited to low resolution, and training from scratch gave limited quality.

Later methods moved to the latent space of pretrained image and video generators. With strong video generation priors now available, PixWorld studies whether such a prior can also support unified generation and reconstruction directly in pixel space, and this is our contribution.

Table A. PixWorld vs. DiffusionGS on RE10K (one-view generation).

MethodPSNR ↑LPIPS ↓IQ ↑AUC@5 ↑
DiffusionGS16.350.6190.5380.541
PixWorld19.210.3120.6220.622
AuthorsOpenReview ↗

Response to Reviewer X5cU (Part 2/4)

W1 / Q2. Generate-then-reconstruct control
“Does supervising diffusion through the rendered Gaussians actually help? Table 8(e) only changes the render loss weight (which also scales the held-out view loss), and Table 13 only changes when sampling switches from RGB to Gaussians, so neither shows what the render supervision adds. Could the authors add a "generate, then reconstruct" control with the existing weights: generate all views in RGB mode only, then reconstruct them with PixWorld's all-clean mode, and compare with the default sampling?”

We thank the reviewer for suggesting this control.

1. Sampling procedure.

Our default sampler switches to Gaussian renderings before RGB denoising is complete. The Gaussian branch therefore receives noisy inputs and participates in denoising.

2. Controlled comparison.

We add two settings to the original experiment in Table 14: σsw\sigma_{\mathrm{sw}} = 0.2 and σsw\sigma_{\mathrm{sw}} = 0. In the latter, all views are generated in RGB mode and then reconstructed using the all-clean mode. Table B shows that the default setting outperforms both added controls across all reported metrics, supporting the use of Gaussian renderings during denoising. RGB-only diffusion may produce cross-view inconsistencies that weaken the 3D consistency of subsequent reconstruction, consistent with the lower pose AUC of the generate-then-reconstruct control.

Table B. Switching timestep and generate-then-reconstruct control.

σsw\sigma_{\mathrm{sw}}PSNR ↑SSIM ↑LPIPS ↓I.Q. ↑AUC@5 ↑
0.717.530.6410.3750.5720.476
0.618.160.6690.3580.5950.549
0.5 (default)18.530.6790.3520.6090.568
0.418.270.6610.3830.6020.537
0.218.230.6630.3890.6050.516
0.0 (generate, then reconstruct)18.180.6580.3920.5980.493

W2 / Q3. Reconstruction under standard protocols
“Reconstruction uses a custom protocol, and the baseline numbers are much lower than in their own papers … Strong feed-forward methods such as LVSM and DA3 are missing, and LVSM is even better than PixWorld on one of the paper’s own settings.”

“Could the authors report reconstruction on the standard protocols, and add LVSM and DA3?”

We thank the reviewer for this suggestion. We now evaluate reconstruction under the RE10K protocol of YoNoSplat, which uses its official test index (1,600 scenes), six context views and 224×224 resolution.

1. Baselines.

Results for DepthSplat and YoNoSplat are taken from Table 2 of the YoNoSplat paper. Since PixWorld takes camera poses and intrinsics as input, we report YoNoSplat in its setting with ground-truth poses and intrinsics for a fair comparison. LVSM and DA3-Splat are evaluated by us with their official checkpoints under the same protocol; both take the ground-truth context poses and intrinsics as input.

2. Results.

As shown in Table C, PixWorld achieves reconstruction quality comparable to LVSM and YoNoSplat under this standard protocol, while unifying reconstruction and generation in a single model.

Table C. Reconstruction under the standard YoNoSplat RE10K protocol.

MethodPoseIntrinsicsPSNR ↑SSIM ↑LPIPS ↓
DepthSplat✓✓24.1560.8460.145
YoNoSplat✓✓25.0370.8480.134
LVSM✓✓24.7510.8320.131
DA3-Splat✓✓21.7900.7630.188
PixWorld✓✓24.7560.8410.133

W2. Distributional generation metrics
“For generation there is no distribution metric (e.g., FID or FVD)”

We thank the reviewer for the suggestion. Many 3D scene generation methods take a long time to generate each scene, so evaluations typically use a limited number of samples (200 test scenes per dataset in our case), and FID and FVD can be strongly biased at this sample size.

We therefore chose WorldScore, a comprehensive benchmark for world generation, and evaluated generation along several complementary axes: visual quality with VBench, camera-pose control with pose AUC, and fidelity to target views with PSNR, SSIM and LPIPS. This is in line with recent works such as FlashWorld, FantasyWorld and Gen3R.

AuthorsOpenReview ↗

Response to Reviewer X5cU (Part 3/4)

W2. Streaming quality over time
“for streaming there is no analysis of quality over time.”

We thank the reviewer for the suggestion. We analyze streaming reconstruction and generation separately.

1. Streaming reconstruction.

On DL3DV, we reconstruct each scene using a fixed set of 256 input views and evaluate novel-view synthesis over four consecutive view intervals: 0–64, 64–128, 128–192 and 192–256. We report PSNR, SSIM and LPIPS on held-out views in each interval. As shown in Table D, all three metrics vary only slightly across intervals, with no consistent degradation toward later views.

2. Streaming generation.

Following SANA-WM, we divide each 60-second rollout from the test set used in Table 5 into six non-overlapping 10-second windows and report VBench imaging quality (IQ) for each window. As shown in Table E, IQ remains relatively stable in the first half of the rollout and declines more noticeably in the second half.

Table D. DL3DV novel-view synthesis across sequence intervals with 256 input views.

View intervalPSNR ↑SSIM ↑LPIPS ↓
0–6422.090.7570.189
64–12821.790.7530.192
128–19222.430.7660.182
192–25622.360.7640.181

Table E. Streaming generation: IQ over six 10-second windows.

Time (s)0–1010–2020–3030–4040–5050–60
IQ ↑65.2965.1364.9763.3861.1460.45

Q1. Gaussian prediction in latent variants
“In the pixel vs. latent ablation, how do the latent variants produce Gaussians, and do they use the same losses?”

We thank the reviewer for the question. In the pixel vs. latent ablation (Tables 6–7, 1.3B DiT trained from scratch), our latent variants use a latent-to-Gaussian design inspired by OneWorld (Gao et al., 2026) and GLD (Jang et al., 2026).

The SD-VAE variant uses its pretrained encoder, while the RAE variant uses Pi3 as its encoder. We freeze the encoders and adapt the corresponding decoders to predict 3D Gaussians. The DiT is then trained in the corresponding latent space, conditioned on the clean latents of the context views to generate the whole scene, and supervised with a flow-matching loss. This loss is defined on latents, whereas PixWorld defines its diffusion loss on pixels and rendered Gaussians; this difference is the factor the ablation compares.


Q4. Joint versus single-task training
“The paper claims benefits from unifying the two tasks, but no experiment compares the joint model with a model trained on only one task. Does joint training actually improve generation or reconstruction? Even a small comparison in the 1.3B setting used for the pixel vs. latent ablation would help.”

We apologize for the ambiguity in our wording. Our focus is on adapting a pretrained video diffusion prior for unified reconstruction and generation. Our intended claim is that modeling these tasks directly in pixel space provides better results than latent-space approaches, not that jointly training the two tasks improves either one over single-task training.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.