- Rating
- 4
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 3
Read review
Paper summary
This paper propose the direct generation of renderable, dynamic driving scenes from inputs. The proposed method generates scenes within the state space of a pre-trained DGGT reconstruction model, while simultaneously predicting the metric scale and the decoder’s field of view for each segment. It first localizes the content using 2D projections of maps and trajectories, then translates this layout into ray and depth conditions within the reconstruction space. A HDS mechanism is employed to impose constraints on features, geometry, and rendered images via the frozen decoder. Experimental results demonstrate high-quality scene generation with effective metric-scale control.
Critical strengths
- Sufficient preliminary study, and the connection between the problem and the methodology is logical, and the motivation is clear.
- The ablation design is targeted.
- The paper is easy to read and follow
Critical weaknesses
- This paper emphasizes the generation of 3D scenes based on metric-scale maps and trajectories. While scale and field of view (FOV) primarily govern the transformation of coordinate systems, the scene layout is influenced by an adapter during the generation process. Furthermore, the geometric targets for HDS are derived from DGGT predictions rather than ground-truth metric geometry. Although Table 3 quantifies the associated errors, the precise relationship between the errors inherent in this method and DGGT remains to be clarified, as well as error propagation mechanism.
- The C=1024 setting in Section 3 is somewhat peculiar. It appears to be an experimental setup choice rather than an inherent aspect of the method itself. The impact of this specific value warrants discussion.
Questions for authors
-
How do errors in DGGT’s geometric predictions affect the accuracy of the scenes generated by DriveReGen?
-
How sensitive is the method to the chosen 1,024-dimensional tokenizer representation?
4 comments
Rebuttals (2/2) - W2/Q2 & Summary
W2 & Q2. The C = 1024 setting and sensitivity to the 1,024-dimensional latent
Thanks for this comment. C = 1024 is the native token width of the frozen DGGT, and our latent adopts this width. This width is not a tight bottleneck: in the trained model, retaining 512 principal directions of the latent preserves rendering quality and decoded depth.
- Origin of 1,024. In Sec. 3.1, C is the width of one DGGT token stream, fixed by its pretrained backbone. Each patch holds twelve such streams (three at each of four levels), 12,288 values in total. Our tokenizer compresses them 12:1 into one latent that keeps this native width. The round trip preserves DGGT's decoding: RGB–3D AbsRel is 0.1512 vs. 0.1496 for DGGT's direct reconstruction (Table 1).
- Robustness to latent truncation (Table R2). With the trained model fixed, we project generated latents from all 404 benchmark clips onto the top-k principal directions fitted on 200 training scenes, map them back to 1,024 dimensions, and decode them with the same frozen tokenizer decoder and DGGT heads. With k = 512, the renders match those of the full latent at 62.8 dB PSNR with unchanged FID. They stay close at k = 256 (39.8 dB) and show larger differences at k = 128 (28.7 dB). The decoded scenes thus rely on about half of the latent's directions, which suggests that 1,024 leaves a wide margin and that its exact value is not critical.
- Cost. Lowering the latent width alone would shrink the generator's input/output projections while leaving its main denoising blocks unchanged. We therefore retain the native width without imposing a tighter bottleneck on the representation.
We have added the origin of the latent width and the truncation results to App. B.5 (Table 14).
Table R2. Decoding generated latents using the top-k principal directions fitted on 200 training scenes.
| Kept directions k | Render PSNR vs. full latent (dB) ↑ | Render LPIPS vs. full latent ↓ | Depth AbsRel vs. full latent ↓ | FID change from full latent |
|---|---|---|---|---|
| 512 | 62.75 | 0.0001 | 0.0001 | +0.00 |
| 256 | 39.76 | 0.0100 | 0.0119 | −0.16 |
| 128 | 28.68 | 0.0816 | 0.0377 | +6.56 |
Summary
We thank the reviewer again for the supportive and constructive feedback. Our responses clarify how DGGT's errors affect the generated scenes, show that our scenes remain within 0.013 LiDAR AbsRel of DGGT's reconstructions, and explain our choice of the native latent width alongside evidence of decoding robustness to latent truncation. We hope these responses address the reviewer's questions and further strengthen their positive assessment of our work. We welcome further discussion and would be glad to provide additional analyses if helpful.
I have already explain my support for this work in the initial review, and the rebuttal infomation do help for this paper. I believe this paper is worth to be accepted
Thanks for your continued support!
Dear Reviewer diRC,
Thank you for the encouraging feedback! We’re really glad that you found our rebuttal helpful, and your continued support means a lot to us. We’ll keep working on the suggestions we’ve received to make the paper even better.
Best regards,
The Authors
Rebuttals (1/2) - Overview & W1/Q1
We thank the reviewer for the positive assessment and constructive feedback! We are glad that the reviewer found that our experiments "demonstrate high-quality scene generation with effective metric-scale control", found the paper clear and easy to follow, and had no substantive concerns about its soundness. We also appreciate the questions on DGGT's errors and the latent width, which helped us explain our design more clearly.
We have made the following changes to the manuscript:
Table numbers below follow the revised paper: Tables 5 and 6 of the submission are now Tables 6 and 7.
Below, we respond to each comment in turn.
W1 & Q1. Effect of DGGT's errors on the generated scenes
Thanks for this question. Our generator is trained on DGGT's reconstructions of recorded videos, so DGGT's geometric errors can carry over to our scenes. Measured against LiDAR, our generated scenes remain within 0.013 AbsRel of DGGT's own reconstructions (0.0621 vs. 0.0494). Metric scale does not come from DGGT: we generate it jointly with the scene and supervise it with LiDAR.
We have added the tokenizer round trip to Table 3 and discuss it in Sec. 4.2.
Table R1. LiDAR error after each stage from DGGT to DriveReGen (road and actor depth vs. real Waymo LiDAR; clips with ego motion).
[Continued below...]