acceptodds
Under review as a conference paper at ICLR 2027

SceneAE: Scene Generation and Reconstruction in Queryable 3D-Anchored Latent Space

Abstract

Recent video world models have increasingly adopted 3D memory mechanisms to integrate historical observations into a lightweight and persistent scene representation. This naturally raises a complementary question: if 3D representations can serve as an effective mechanism for memorizing observed scenes, can they also be directly used as the latent space for reconstructing and generating scenes? Motivated by this idea, we represent a scene as a compact and unstructured 3D-anchored latent space (SceneAE). Each latent combines an explicit 3D position with an implicit content feature and is spatially grounded to describe its local scene region, forming a sparse latent point cloud. Unlike conventional video-VAE latents that describe how a scene appears from particular viewpoints, our representation directly describes the scene itself, making it naturally applicable to both reconstruction and generation. For reconstruction, once the scene has been generated, our persistent 3D representation can be projected and decoded into RGB-D observations by different target cameras, without the need to regenerate a video for each new camera trajectory. For generation, we leverage the geometry-content separation to first generate camera-controlled 3D geometry and then synthesize content conditioned on it, reducing camera-appearance entanglement and improving camera controllability. Experiments show that this 3D latent achieves superior generation quality, computational cost, and camera controllability.

Primary area:
generative models
Keywords:
3D SCENE GENERATION, 3D SCENE RECONSTRUCTION, WORLD MODEL

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_JcsQOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This work introduces SceneAE, which represents a scene using sparse 3D anchors with learned content features. An encoder aggregates geometry-foundation-model features at these anchors, and a projection-based decoder renders queried RGB-D views. For generation, a point model first predicts the anchor layout conditioned on a camera trajectory, followed by content diffusion with geometry refinement. Experiments evaluate scene generation, reconstruction, camera following, and consistency, with additional out-of-distribution tests and component ablations.

Critical strengths
  1. The representation provides a useful connection between compact generative latents and queryable scene reconstruction. Explicit anchors support spatially localized feature aggregation and decoding, while allowing the same scene representation to serve multiple viewpoints. Table 5 shows improvements over SceneTok and LVSM on novel-view synthesis.

  2. The geometry-first generation design has encouraging empirical support. Table 9 reports improvements over joint and content-only alternatives, while Table 3 shows better trajectory following. Together, these results support explicit geometry as a useful intermediate variable for camera-controlled generation.

Critical weaknesses
  1. The geometry evaluation does not establish a consistent comparison across methods. Table 2 evaluates Gen3R against VGGT-derived geometry, but SceneAE and GLD against DA3-derived geometry. These scores measure agreement with different references and therefore cannot directly establish the reported ranking in geometric accuracy. The shared-reference comparison with GLD remains informative, but an independent common reference is needed to substantiate the broader claim.

  2. Important details needed to assess the generation and efficiency claims remain unclear. Equation (7) interpolates between separately sampled reference and target point sets, but does not explain their correspondence or ordering. Equation (10) specifies a residual target without explaining its timestep scaling or sampling update. Meanwhile, Table 1 reports per-step cost despite the method using two generation stages and subsequent geometry correction. Without the correspondence rule, sampling procedure, and total inference cost, reproducibility and the claimed efficiency advantage remain insufficiently established.

Questions for authors
  1. Can you evaluate all methods against the same geometry reference, preferably independent measured geometry, and report the alignment and scale-normalization protocol?

  2. How are reference and target anchors paired for Equation (7)? Please specify the inference updates and sampling steps for both stages, clarify the timestep treatment in Equation (10), and report total latency including reference encoding, geometry generation, content generation, correction, and decoding.

1 comment

Dear Reviewer JcsQ,

We sincerely thank you for your valuable feedback and thoughtful comments on our work. We have carefully addressed your concerns and provided detailed responses in our rebuttal.

We would greatly appreciate it if you could take a moment to review our responses and let us know whether they adequately address your concerns. We would be happy to provide further clarification if needed.

Thank you again for your time and consideration.

Best regards,

The Authors

Reviewer_9d5bOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes Scene-AE, which represents a static scene as a sparse set of 3D anchor points, each carrying a learned content feature. A frozen Depth Anything 3 backbone provides depth and image features. Anchors are obtained by back-projecting depth and applying farthest-point sampling, and the encoder gathers image features at each anchor’s projected location. To render a view, the decoder splats the anchors into a low-resolution feature map, then refines and upsamples it into RGB-D with attention layers.

For generation, a Point DiT first predicts anchor positions from the reference view(s) and target cameras. A Content DiT then generates features on that geometry, with a position-correction step described in Appendix A.4. Experiments cover 1- and 2-view generation, novel-view reconstruction, geometry, camera following, round-trip consistency, efficiency, and ablations.

Overall I think the method is sound and the 2 stage pipeline separating reconstruction and generation also make sense to me. I think the novelty is moderate as anchor-based reconstruction is not a new thing. I have some questions about the method and the experiments I would like the authors to clarify. Specifically I think the authors should show more true novel view rendering results outside of the input camera trajectory and clarify their claims about having lower computational cost, which I think is the main benefit of this method. I will currently give weak reject but I might raise my rating if the author's responses are satisfactory.

Critical strengths
  • The authors propose the feed-forward, 3D-anchored latent that can be used both as an autoencoder and generative latent space, in my view this is a fairly new approach that pushes anchored-based 3D reconstruction that have already worked in optimization-based 3DGS to feed-forward paradigm.
  • The camera following improvements seems promising and show real improvement in trajectory accuracy.
  • The method shows good generalization on out-of-distribution TartanAir and ScanNet++ data.
Critical weaknesses
  • The abstract claims that once a scene is generated, it can be rendered from new cameras without regeneration. However, the only novel-view experiment (Table 5) uses latents encoded from nine real images, not generated latents. And the generated experiments only evaluate on the input trajectory cameras. What happens when we are extrapolating the scene with a camera outside the trajectory?
  • The authors report per-sampling step costs, but the total number of steps needed is not specified. The authors should also clarify runtime for the stage 2 diffusion stage. Table 2 also evaluates Gen3R against VGGT-derived geometry and SceneAE and GLD against DA3-derived geometry. SceneAE builds its anchors from DA3 depth, which may favor it when DA3 is the reference.
  • Several core components have close precedents that are missing from the paper. Anchors with learned local features decoded for rendering are central to Scaffold-GS (per-scene and non-generative). The FPS-anchored encoder resembles 3DShape2VecSet, and the splat-then-refine decoder resembles neural point-based rendering (e.g., NPBG). The authors should disscuss these related works and frame its contribution in the context of these previous works.
Questions for authors
  • How are far away regions like sky are handled?
  • Have the authors tested what happens when the scene is challenging for the geometry foundation model fails such as scene with transparency.
  • Please clarify the total runtime and resource costs for both reconstruction and generation stages?
Reviewer_tS11OpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper proposes SceneAE, a sparse 3D latent representation for scene reconstruction and camera-controlled generation. Each latent combines an explicit 3D position with a learned content feature. The encoder aggregates DA3 features at sampled 3D anchors, while a splat-based decoder renders RGB-D observations from target viewpoints. Generation proceeds in two stages: Point DiT generates camera-conditioned geometry, and Content DiT synthesizes the corresponding features. Experiments on DL3DV and RealEstate10K evaluate generation, reconstruction, camera control, and revisit consistency.

Critical strengths
  1. A well-motivated scene-centric representation. Sharing a 3D representation across observations reduces redundant encoding and enables novel-view rendering. Combining explicit positions with implicit content is a sensible design for both reconstruction and generation.
  2. Consistent use of spatial structure. Anchor projection, depth-aware feature selection, splat-based initialization, and local cross-attention use 3D positions to guide information access throughout the encoder and decoder.
Critical weaknesses
  1. Geometry metrics are not directly comparable across all methods. Table 2 evaluates Gen3R against VGGT-derived depth, while GLD and SceneAE use DA3-derived depth. Different reference geometries make the reported errors difficult to compare fairly.

  2. Per-step costs do not establish overall efficiency. Table 1 reports GFLOPs and latency per step, but SceneAE sequentially executes Point DiT and Content DiT, with additional geometry correction during the final ten content sampling steps. The total generation cost remains unclear.

Questions for authors
  1. Table 2 evaluates Gen3R against VGGT-derived depth, while GLD and SceneAE are evaluated against DA3-derived depth, making the errors difficult to compare directly. Could the authors double-check the geometry evaluation or use the same geometry reference for all methods?
  2. In Figure 4, the baseline results appear to show substantial discrepancies in camera viewpoint and scene scale relative to the ground truth. Please explain how camera poses, translation scales, and scene coordinates were normalized for each baseline, and verify that these discrepancies are not caused by inconsistent preprocessing.
  3. Please report the sampling steps used by Point DiT and Content DiT and clarify whether Table 1 includes both stages and geometry correction. Please also provide total generation latency and peak GPU memory, including reference encoding and RGB-D decoding, under the same hardware, precision, output resolution, and number of views as the baselines. This would establish whether the reported per-step advantage translates into an overall efficiency gain.
  4. Table 5 shows substantially lower reconstruction fidelity than Wan-VAE and GLD, while Figure 5 suggests smoothing of textures and boundaries. Please vary the anchor count and content feature dimension around the current settings of 3,000 and 64, respectively, and report reconstruction quality, generation quality, latency, and memory usage. This would clarify whether increasing capacity can close the fidelity gap while preserving the efficiency advantage.

If the authors satisfactorily address these concerns, I would be willing to raise my score.

5 comments

Thank you for the detailed explanation. However, I still have an unresolved question.

Regarding Q2. Alignment between predicted geometry and GT cameras

I understand that each method follows its official input preprocessing protocol. However, my concern is whether each method’s output geometry is aligned with the coordinate system and scale of the GT cameras. RE10K and DL3DV have scene-specific coordinate scales. Before rendering with GT cameras, each baseline’s predicted geometry must therefore be brought into the corresponding GT coordinate system and scale. The current explanation does not clarify how this potential mismatch is resolved.

Specifically, a per-scene scale factor mapping the model’s output scale to the GT dataset scale should be estimated using the Umeyama algorithm on the predicted and GT camera translations, both expressed relative camera pose to the same reference camera (e.g., the first frame). This factor would then be applied to the predicted depths and predicted camera translations, bringing the reconstructed geometry into the GT dataset scale before rendering with the GT cameras. Was this procedure applied consistently to all methods during inference and evaluation, including the results in Fig. 4 and all experiments?

Could you also clarify more precisely how the previously mentioned Umeyama alignment is performed—specifically, which correspondences are used and which camera scale is aligned to which reference scale?

We thank the reviewer for this precise question. We clarify three main points:

  1. Camera-conditioning consistency: Gen3R, SceneAE and GLD were trained and evaluated on RE10K and DL3DV. Therefore, for each method, we processed the GT camera data according to the corresponding preprocessing flow, so that during the evaluation, the camera scale convention expected during training could be maintained and aligned with the dataset GT. For Aether, we acknowledge that there is still uncertainty regarding whether its original camera scale convention is applicable to these datasets.

  2. Fig. 4 does not involve rendering the final predicted depth with GT cameras: RGB outputs are directly decoded conditioned on the preprocessed GT cameras. Consequently, post-hoc alignment of the predicted output geometry does not change these RGB results.

  3. Geometry alignment is consistent across methods: We apply Gen3R’s released similarity-alignment routine independently to each scene, mapping predicted geometry to reference geometry before computing geometric metrics.

1. Camera preprocessing and training conventions.

For Gen3R, SceneAE, and GLD, we preserve their respective camera-conditioning conventions:

  • SceneAE: Poses are expressed relative to the first camera. Depth/3D coordinates and camera translations are divided by the same per-sample factor derived from the first-frame DA3 depth. This preserves their relative geometric scale; reversing the normalization restores the original units.
  • Gen3R: We follow its released preprocess_poses and camera-ray construction. The pose preprocessing makes the first camera the reference without introducing an additional translation-scale normalization in that function.
  • GLD: We follow its released reference-camera conversion and camera encoding, including translation normalization by the maximum absolute translation component over the input batch.

Different numerical camera scales therefore reflect the conventions expected by the respective models. Using each method’s corresponding preprocessing preserves its training convention.

Aether was trained on synthetic data. Although we follow its released preprocessing, we cannot conclusively exclude a camera-scale adaptation issue on RE10K and DL3DV. The first-frame PSNR/SSIM results in Table R2 provide an additional sanity check for conditioning-image preservation, but do not independently establish correct translation scaling throughout a trajectory.

2. RGB generation versus output-geometry alignment.

The proposed procedure—fitting a scale from camera trajectories, rescaling predicted geometry, and rendering it with GT cameras—addresses a different output pipeline.

For Fig. 4 and the RGB evaluation, the prescribed GT cameras are supplied as model conditioning/query cameras after the corresponding preprocessing. The models directly decode RGB images; no additional rendering stage uses their final predicted depth maps to produce these images. Therefore, output-depth alignment cannot alter the viewpoints or apparent scene scales in Fig. 4.

SceneAE’s decoder internally projects 3D latents using correspondingly normalized query cameras. This internal operation is distinct from the post-hoc geometry alignment used for metric computation. Camera following is evaluated separately in Table 3 using poses estimated from generated RGB videos.

3. Geometry references, correspondences, and alignment direction.

RE10K and DL3DV do not provide measured GT depth. Our references are pseudo-GT geometry reconstructed from the original videos using the geometry foundation model and the corresponding dataset cameras. Under the unified evaluation protocol, all methods use the same reference geometry for each sample.

We apply Gen3R’s released alignment implementation independently to each scene. All frames are first expressed in a common reference coordinate system. Correspondences are formed between predicted and reference 3D points at matching valid pixel locations across the sequence. A single scale, rotation, and translation are then estimated jointly and applied to all predicted frames before computing the geometric metrics.

This procedure removes the sequence’s global coordinate and scale ambiguity while preserving inter-frame scale drift and relative geometric errors. The same protocol is applied to every method. This is geometry-based alignment, rather than alignment fitted from camera-center trajectories, and it does not modify the generated RGB images.

Furthermore, in order to prove the correctness of the above statement. We therefore perform a relative-pose audit after unifying the axis convention, and check each method against its own expected normalization factor.

1.Protocol

All cameras are first converted to the same camera-to-world (c2w) axis convention. For each method and each chunk, we compute

\qquad \Delta \tilde T_v = \tilde T_0^{-1} \tilde T_v,$$ where $T_v$ is the original GT c2w and $\tilde T_v$ is the c2w after the method's official preprocessing. Let $R_v$ and $b_v$ denote the rotation and translation of $\Delta T_v$, and $\tilde R_v$, $\tilde b_v$ those of $\Delta \tilde T_v$. We compare: - the geodesic rotation error between $R_v$ and $\tilde R_v$; - the angle between the translation directions $b_v$ and $\tilde b_v$; - the translation scale ratio $$s_v = \frac{\Vert \tilde b_v \Vert}{\Vert b_v \Vert}.$$ We exclude the first frame $v=0$ and frames with near-zero displacement, because direction and scale ratio are unstable there. We do not compare raw translation components, and we do not use the translation column of a w2c matrix as a camera center. For each method, let $\alpha_m$ be its expected normalization factor in the current chunk. We verify $$\tilde b_v \approx \alpha_m b_v, \qquad \frac{s_v}{\alpha_m} \approx 1,$$ and report $$E_{\mathrm{trans}} = \sqrt{ \frac{\sum_v \Vert \tilde b_v - \alpha_m b_v \Vert^2} {\sum_v \Vert \alpha_m b_v \Vert^2} }.$$ This checks both the direction and the magnitude of the translation. Since SceneAE and GLD normalize per sample/chunk, we compute $s_v/\alpha_m$ within each chunk and then aggregate $E_{\mathrm{trans}}$; we do not mix scenes before computing the scale statistics. ### 2. Results | Method | Rotation error (deg)↓ | Translation direction error (deg)↓ | $s_v/\alpha_m$ (per-chunk mean ± std) | $E_{\mathrm{trans}}$ (%)↓ | Expected factor $\alpha_m$ | |---|---:|---:|---:|---:|---| | SceneAE | 0.00 | 0.02 | 1.000 ± 0.001 | 0.2 | $1/d_{\max}^{(0)}$: first-frame max DA3 depth | | GLD | 0.00 | 0.03 | 1.000 ± 0.002 | 0.4 | $1/\max \Vert t \Vert$: max abs translation over input batch | | Gen3R | 0.00 | 0.02 | 1.000 ± 0.001 | 0.3 | $1$: reference camera change only, no extra scale | | Aether | 0.00 | 0.05 | 1.020 ± 0.030 | 1.5 | official preprocessing (scale convention uncertain) | ### 3.Analysis - **Rotation and direction are preserved.** After unifying the axis convention, all methods have rotation errors of about $0^\circ$ and translation-direction errors below $0.05^\circ$. This confirms that the preprocessing changes only the reference camera and the scale, not the relative camera motion. - **SceneAE, GLD, and Gen3R match their expected factors.** Their per-chunk $s_v/\alpha_m$ is $1.000$ with a within-chunk std of $0.001$–$0.002$, and $E_{\mathrm{trans}}$ is below $0.5\\%$. The large variation of $\alpha_m$ across scenes is expected: SceneAE uses the first-frame max depth and GLD uses the batch max translation; the relevant check is that $s_v/\alpha_m$ is stable within each chunk, which it is. - **Aether remains uncertain in scale, not in axis or rotation.** Its direction error is still small ($0.05^\circ$), but $s_v/\alpha_m \approx 1.02$, the within-chunk std is $0.03$, and $E_{\mathrm{trans}} \approx 1.5\\%$, clearly larger than the other methods. Aether exhibits a larger residual under the current audit formulation. Its expected transformation and encoding inversion require further verification; this residual alone does not establish a dataset-scale adaptation issue. ### 4.Conclusion The audit supports that the implemented extrinsic-camera preprocessing for SceneAE, GLD, and Gen3R preserves relative rotations and translation directions, while matching the expected method-specific translation scaling with a relative residual below 0.5%. These results provide direct evidence against a camera-conversion or scale-normalization error in the audited evaluation paths. They do not, by themselves, establish that the generated outputs accurately follow the conditioning trajectory; that behavior is evaluated separately in Table 3.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.