acceptodds
Under review as a conference paper at ICLR 2027

DriveReGen: Generating Driving Scenes as Gaussians in Reconstruction Space

Abstract

Collecting real driving data is costly. Video models offer an alternative by generating driving scenes from text and layouts, yet they control these layouts only in 2D space and cannot ensure consistent geometry across frames and views. Recent methods generate Gaussian scenes from generated videos or video latents, but the reconstructed geometry may not fully match the generated appearance. General-scene methods instead generate in feed-forward reconstruction state space to jointly model geometry and appearance. However, conditioning on metric driving layouts requires the frozen decoder's scale and FOVs. We introduce DriveReGen, which generates driving scenes in reconstruction space with scene-specific scale and decoder fields of view for metric layout conditioning. Our two-stage layout control first aligns content with projected HD maps and actor tracks, then uses decoded calibration to encode metric layouts as reconstruction rays and depths. To supervise decoded outputs, Hierarchical Decoding Supervision (HDS) backpropagates feature, geometry, and image errors through frozen decoding. Given text, cameras, and layouts, DriveReGen generates 3D Gaussians without target images or LiDAR. On Waymo, DriveReGen reduces FVD by 58.0% and improves lane F1 by 3.77 percentage points over InfiniCube. Under physical cameras, its road/actor LiDAR AbsRel is 17.0% lower. Code will be made publicly available.

Primary area:
applications to computer vision, audio, language, and other modalities
Keywords:
Diffusion Models, 3D Gaussian Scene Generation, Autonomous Driving

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_diRCOpenReview ↗
Rating
4
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

This paper propose the direct generation of renderable, dynamic driving scenes from inputs. The proposed method generates scenes within the state space of a pre-trained DGGT reconstruction model, while simultaneously predicting the metric scale and the decoder’s field of view for each segment. It first localizes the content using 2D projections of maps and trajectories, then translates this layout into ray and depth conditions within the reconstruction space. A HDS mechanism is employed to impose constraints on features, geometry, and rendered images via the frozen decoder. Experimental results demonstrate high-quality scene generation with effective metric-scale control.

Critical strengths
  1. Sufficient preliminary study, and the connection between the problem and the methodology is logical, and the motivation is clear.
  2. The ablation design is targeted.
  3. The paper is easy to read and follow
Critical weaknesses
  1. This paper emphasizes the generation of 3D scenes based on metric-scale maps and trajectories. While scale and field of view (FOV) primarily govern the transformation of coordinate systems, the scene layout is influenced by an adapter during the generation process. Furthermore, the geometric targets for HDS are derived from DGGT predictions rather than ground-truth metric geometry. Although Table 3 quantifies the associated errors, the precise relationship between the errors inherent in this method and DGGT remains to be clarified, as well as error propagation mechanism.
  2. The C=1024 setting in Section 3 is somewhat peculiar. It appears to be an experimental setup choice rather than an inherent aspect of the method itself. The impact of this specific value warrants discussion.
Questions for authors
  1. How do errors in DGGT’s geometric predictions affect the accuracy of the scenes generated by DriveReGen?

  2. How sensitive is the method to the chosen 1,024-dimensional tokenizer representation?

4 comments
AuthorsOpenReview ↗

Rebuttals (1/2) - Overview & W1/Q1

We thank the reviewer for the positive assessment and constructive feedback! We are glad that the reviewer found that our experiments "demonstrate high-quality scene generation with effective metric-scale control", found the paper clear and easy to follow, and had no substantive concerns about its soundness. We also appreciate the questions on DGGT's errors and the latent width, which helped us explain our design more clearly.

We have made the following changes to the manuscript:

  • Tokenizer depth fidelity (Table 3). We add the tokenizer round trip to the LiDAR evaluation and discuss how closely it preserves DGGT's depth accuracy.
  • Latent width (App. B.5). We state where the width 1,024 comes from and test decoding robustness to latent truncation in the trained model (Table 14).

Table numbers below follow the revised paper: Tables 5 and 6 of the submission are now Tables 6 and 7.

Below, we respond to each comment in turn.


W1 & Q1. Effect of DGGT's errors on the generated scenes

Thanks for this question. Our generator is trained on DGGT's reconstructions of recorded videos, so DGGT's geometric errors can carry over to our scenes. Measured against LiDAR, our generated scenes remain within 0.013 AbsRel of DGGT's own reconstructions (0.0621 vs. 0.0494). Metric scale does not come from DGGT: we generate it jointly with the scene and supervise it with LiDAR.

  1. Propagation mechanism. We train the generator on the features that DGGT extracts from recorded videos, compressed by our tokenizer. The generator therefore learns DGGT's geometry together with its errors, and this is how DGGT's errors reach our scenes. In HDS, the feature and geometry terms take their targets from the same features, while the image term compares the rendered scene with the real video (Sec. 3.4).
  2. Error after each stage (Table R1). On the same clips, we compare three results with real LiDAR: (a) DGGT's reconstruction of a recorded video (AbsRel 0.0494), (b) the same reconstruction after our tokenizer (AbsRel 0.0548), and (c) our scene generated without target images as model inputs (AbsRel 0.0621). (a) and (b) share the same video, so their gap of 0.005 is due to the tokenizer, and (c) remains within 0.013 of (a). For comparison, when the same DGGT reconstructs generated videos instead of real ones, AbsRel is 0.2365 (OpenDWM) and 0.4024 (Cosmos3) (Table 3).
  3. Metric scale is anchored by LiDAR. DGGT's reconstruction space has no metres (Eq. 2). We therefore generate the metric scale jointly with the scene and supervise it with the LiDAR-to-DGGT depth ratio of each clip (Eq. 9). Replacing it with a training-set constant doubles LiDAR AbsRel from 0.0810 to 0.1705 (Table 3).

We have added the tokenizer round trip to Table 3 and discuss it in Sec. 4.2.

Table R1. LiDAR error after each stage from DGGT to DriveReGen (road and actor depth vs. real Waymo LiDAR; clips with ego motion).

StageInputAbsRel ↓AbsRel gap to DGGT directRMSE (m) ↓
(a) DGGT directRecorded video0.0494n/a1.1258
(b) + our tokenizer (round trip)Recorded video0.0548+0.00541.2098
(c) + our generator (DriveReGen)Text, cameras, and layouts0.0621+0.01271.4376

[Continued below...]

AuthorsOpenReview ↗

Rebuttals (2/2) - W2/Q2 & Summary

W2 & Q2. The C = 1024 setting and sensitivity to the 1,024-dimensional latent

Thanks for this comment. C = 1024 is the native token width of the frozen DGGT, and our latent adopts this width. This width is not a tight bottleneck: in the trained model, retaining 512 principal directions of the latent preserves rendering quality and decoded depth.

  1. Origin of 1,024. In Sec. 3.1, C is the width of one DGGT token stream, fixed by its pretrained backbone. Each patch holds twelve such streams (three at each of four levels), 12,288 values in total. Our tokenizer compresses them 12:1 into one latent that keeps this native width. The round trip preserves DGGT's decoding: RGB–3D AbsRel is 0.1512 vs. 0.1496 for DGGT's direct reconstruction (Table 1).
  2. Robustness to latent truncation (Table R2). With the trained model fixed, we project generated latents from all 404 benchmark clips onto the top-k principal directions fitted on 200 training scenes, map them back to 1,024 dimensions, and decode them with the same frozen tokenizer decoder and DGGT heads. With k = 512, the renders match those of the full latent at 62.8 dB PSNR with unchanged FID. They stay close at k = 256 (39.8 dB) and show larger differences at k = 128 (28.7 dB). The decoded scenes thus rely on about half of the latent's directions, which suggests that 1,024 leaves a wide margin and that its exact value is not critical.
  3. Cost. Lowering the latent width alone would shrink the generator's input/output projections while leaving its main denoising blocks unchanged. We therefore retain the native width without imposing a tighter bottleneck on the representation.

We have added the origin of the latent width and the truncation results to App. B.5 (Table 14).

Table R2. Decoding generated latents using the top-k principal directions fitted on 200 training scenes.

Kept directions kRender PSNR vs. full latent (dB) ↑Render LPIPS vs. full latent ↓Depth AbsRel vs. full latent ↓FID change from full latent
51262.750.00010.0001+0.00
25639.760.01000.0119−0.16
12828.680.08160.0377+6.56

Summary

We thank the reviewer again for the supportive and constructive feedback. Our responses clarify how DGGT's errors affect the generated scenes, show that our scenes remain within 0.013 LiDAR AbsRel of DGGT's reconstructions, and explain our choice of the native latent width alongside evidence of decoding robustness to latent truncation. We hope these responses address the reviewer's questions and further strengthen their positive assessment of our work. We welcome further discussion and would be glad to provide additional analyses if helpful.

I have already explain my support for this work in the initial review, and the rebuttal infomation do help for this paper. I believe this paper is worth to be accepted

AuthorsOpenReview ↗

Thanks for your continued support!

Dear Reviewer diRC,

Thank you for the encouraging feedback! We’re really glad that you found our rebuttal helpful, and your continued support means a lot to us. We’ll keep working on the suggestions we’ve received to make the paper even better.

Best regards,

The Authors

Reviewer_k7RvOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper studies generation of dynamic 3D Gaussian driving scenes from text, camera trajectories, HD maps, and actor tracks without target images at inference. It proposes DriveReGen, which generates DGGT reconstruction states jointly with scene-specific metric scale and decoder FOV calibration, uses calibration-aware two-stage layout conditioning, and trains with decoded feature, geometry, and rendering supervision. Experiments on Waymo compare visual quality, layout adherence, depth accuracy, and shifted-view rendering.

Critical strengths
  • Sections 3.1 and 3.3 identify that a frozen reconstruction decoder has clip-dependent scale and decoder intrinsics, then use generated calibration to transform metric layout points into decoder rays and depths. This is a technically meaningful interface between metric driving controls and reconstruction latents, rather than merely attaching 2D layout conditioning to a video generator.
  • The component evidence is unusually well targeted, provides evidence that calibration-aware reconstruction-space conditioning.
Critical weaknesses
  • Although the paper evaluates camera shifts of 2 m and 4 m and provides longer sequence examples, the quantitative evaluation remains focused on relatively short, front-view sequences. More extensive evaluation under larger viewpoint changes, diverse camera trajectories, and longer temporal horizons would further strengthen the claims regarding 3D consistency and scene generation.
  • The method relies on a pretrained DGGT reconstruction backbone, and the experiments are primarily conducted on Waymo. Could the authors proivde how well the framework generalizes to different datasets, camera configurations, and scene distributions without adapting the reconstruction backbone?
Questions for authors
  • How many independent generation samples are evaluated per clip and method? Could the authors provide scene-level paired confidence intervals, or an equivalent statistical analysis?
  • Appendix B.1 states that captions are generated from all ten reference frames of each evaluated clip. Can the authors clarify whether the captions contain scene-specific details that are unavailable from the supplied text/layout controls alone, and whether evaluation with less reference-derived prompts changes the relative ranking?
5 comments
AuthorsOpenReview ↗

Rebuttals (1/2) - Overview, W1 & W2

We thank the reviewer for the careful review and constructive feedback! We are glad the reviewer found our generated calibration "technically meaningful" and the component evidence "unusually well targeted".

We have made the following changes to the manuscript:

  • Larger camera shifts (App. B.8). We add ±6 m lateral and +4 m forward shifts.
  • Longer horizons (App. B.7). We add lane and LiDAR metrics over 29 frames.
  • Transfer (App. B.5, App. C). We test the frozen DGGT and our tokenizer on nuScenes.
  • Statistics (App. B.1, B.9). We add the sampling protocol, three seeds, and paired tests.
  • Captions (Sec. 4.3, Table 5). We compare all methods with reduced prompts.

W1. Larger viewpoint changes, diverse trajectories, and longer horizons

Thanks for this suggestion. DriveReGen keeps the best lane alignment and video quality under ±6 m lateral and +4 m forward shifts, and stays ahead of InfiniCube throughout 29-frame clips, with LiDAR error ending lower than it starts.

  1. Larger shifts (Table R1). As in Table 4, we render each method's fixed Gaussian scene from the shifted cameras. All methods degrade at ±6 m. Ours remains the best: lane F1 is 3.14% vs. at most 1.13% for the baselines, still above every baseline's F1 at 4 m (at most 2.67%, Table 4), with the lowest FVD (1038 vs. 1338–2235). At +4 m forward, it is 17.16% vs. at most 12.18%, with the lowest FVD (281 vs. 412–789).
  2. Trajectory types. The benchmark takes two clips per validation scene at fixed positions (App. B.1), so it follows the split's trajectory mix: 268 straight, 20 turning, and 116 slow or stopped clips. On turning clips, DriveReGen has the lowest full-frame LiDAR AbsRel (0.189 vs. 0.252–0.372). On slow or stopped clips, it is close to InfiniCube (0.203 vs. 0.214).
  3. Longer horizons (Table R2). On the 29-frame clips of App. B.7, about three times the training length, we score lane F1 and LiDAR AbsRel over all frames and three time windows. DriveReGen leads in every window, and its last-window AbsRel is below its first (0.181 vs. 0.193).

Other camera orientations would need multi-camera training data, as our generator is trained on front-camera videos.

We have added Table R1 and the trajectory breakdown to App. B.8, and Table R2 to App. B.7.

Table R1. Lane alignment and video quality under larger camera shifts.

Method±6 m lateral: Lane F1 (%) ↑±6 m lateral: Lane X-Err (m) ↓±6 m lateral: FVD ↓+4 m forward: Lane F1 (%) ↑+4 m forward: Lane X-Err (m) ↓+4 m forward: FVD ↓
InfiniCube1.131.0901337.929.940.824677.74
OpenDWM + DGGT0.781.1381536.7312.080.737411.65
Cosmos3 + DGGT0.451.1782235.3012.180.769788.99
DriveReGen3.140.9241038.2617.160.704281.40

Table R2. Time windows within jointly generated 29-frame clips.

FramesLane F1 (%) ↑: InfiniCubeLane F1 (%) ↑: DriveReGenLiDAR AbsRel ↓: InfiniCubeLiDAR AbsRel ↓: DriveReGen
0–28 (all)12.2616.240.22330.1824
0–913.4317.790.22240.1928
10–1912.7816.410.22080.1736
20–2810.2914.330.22700.1805

W2. Generalization without adapting the reconstruction backbone

Thanks for this question. The frozen DGGT and our tokenizer carry over to nuScenes without adaptation. This evidence is at the reconstruction level: generating nuScenes scenes still requires training the generator on its data, but not adapting the backbone.

  1. Frozen backbone. Trained only on Waymo and without fine-tuning, DGGT reaches 25.31 / 26.34 dB PSNR on nuScenes / Argoverse2, close to the 26.63 / 26.96 dB when trained on each dataset (DGGT paper, Table 2).
  2. Our latent space on nuScenes. We pass 100 nuScenes validation clips through the frozen DGGT and our Waymo-trained tokenizer. The round trip reaches 26.06 dB PSNR at the input views vs. 26.16 dB for DGGT's direct decoding, with LPIPS nearly unchanged (−0.004).
  3. Lighting and weather. With clips grouped by caption keywords, DriveReGen has the lowest full-frame LiDAR AbsRel at night, dusk, or dawn (92 clips; 0.177 vs. 0.222–0.406) and in rain (10 clips; 0.277 vs. 0.371–0.530).

For generation, our generator follows its training data, and nuScenes differs from Waymo in cameras, road layouts (e.g., left-hand traffic in Singapore), and appearance. Our Waymo-trained generator is therefore not well suited to direct use on nuScenes, as noted in the limitations of our submission (App. C). These results indicate that the frozen DGGT and tokenizer can supply its training targets there. Cameras far from Waymo's, such as wide-angle dashcam views, may still need DGGT fine-tuning (App. C).

We have added the nuScenes results to App. B.5 (Fig. 8) and expanded the discussion in App. C.


[Continued below...]

AuthorsOpenReview ↗

Rebuttals (2/2) - Q1, Q2 & Summary

Q1. Samples per clip and statistical analysis

Thanks for this question. Each method is evaluated with one sample per clip (404 clips from 202 scenes). Our results are stable across three seeds, and scene-level paired tests support our advantages on the main metrics.

  1. Seeds (Table R3). We regenerated all 404 clips with two additional seeds. The standard deviations (0.13 FID, 5.9 FVD, 0.14 lane F1 points, 0.004 actor mAP) are smaller than our margins over the strongest baseline, OpenDWM + DGGT (15.6, 150, 1.35 points, 0.014). Every seed outperforms it on all four metrics.
  2. Scene-level paired tests (Table R4). We resample or permute whole scenes, keeping the two clips of a scene together. Our FID and FVD are lower than those of every baseline (p ≤ 0.001). The 95% CIs exclude zero against every baseline for lane F1 at 0, 2, and 4 m, for road and actor LiDAR AbsRel, and for actor recall. These metrics are built from per-scene counts or means, so we resample them directly. Actor mAP instead comes from a precision–recall curve ranked over all clips and has no per-scene decomposition, so we report its lead over OpenDWM + DGGT across seeds: 0.008–0.015 (Table R3). When clips with a static ego vehicle are also included (Table 3 rows marked (all)), our AbsRel stays lower than InfiniCube's in 97 of 159 scenes (Wilcoxon p = 0.028), although the CI of the mean gap narrowly includes zero (−0.016 [−0.033, +0.0003]).

We have stated the sampling protocol in App. B.1 and added Tables R3 and R4 to App. B.9.

Table R3. Sampling seeds of DriveReGen (404 clips; Seed 1 is the sample set of Tables 1 and 2).

RunFID ↓FVD ↓Lane F1 (%) ↑Actor mAP ↑
Seed 137.15260.7712.430.125
Seed 236.96259.7012.160.119
Seed 336.89249.9812.260.126
Mean ± std37.00 ± 0.13256.8 ± 5.912.28 ± 0.140.123 ± 0.004
OpenDWM + DGGT52.72410.6611.080.111

Table R4. DriveReGen minus each baseline (FID/FVD: scene permutation p-values; others: 95% CIs from paired scene bootstraps; †vehicles only, as InfiniCube controls only vehicles).

Metricvs. InfiniCubevs. OpenDWM + DGGTvs. Cosmos3 + DGGT
FID ↓−22.6 (p = 0.001)−15.6 (p = 0.001)−27.7 (p = 0.001)
FVD ↓−360 (p < 0.001)−150 (p < 0.001)−522 (p < 0.001)
Lane F1, 0 m (points) ↑+3.8 [+2.2, +5.4]+1.4 [+0.2, +2.6]+2.3 [+0.8, +3.8]
Lane F1, 2 m (points) ↑+5.8 [+4.4, +7.3]+5.9 [+4.5, +7.4]+7.1 [+5.7, +8.6]
Lane F1, 4 m (points) ↑+4.6 [+3.6, +5.7]+5.1 [+3.9, +6.3]+5.8 [+4.5, +7.1]
Road and actor LiDAR AbsRel ↓−0.035 [−0.052, −0.021]−0.174 [−0.231, −0.127]−0.340 [−0.624, −0.175]
Actor recall, 0 m ↑+0.302 [+0.257, +0.348]†+0.030 [+0.006, +0.054]+0.216 [+0.174, +0.256]

Q2. Reference-derived captions

Thanks for this question. Yes, the captions contain details that the controls do not specify. After removing them, DriveReGen still ranks first in FID, FVD, lane F1, and vehicle mAP.

  1. Caption content. Each caption describes its reference clip, including appearance details absent from the layouts; e.g., 158 of the 404 captions name a vehicle's color. All methods receive the same caption.
  2. Reduced prompts (Table R5). For one clip from each of 101 validation scenes, we compare the full caption with a reduced prompt that keeps only scene type, weather, time of day, and lighting (15 vs. 105 words on average). Without appearance details, our FID and FVD rise by 6.3 and 68, while our lane F1 and vehicle mAP remain the highest, so our layout control depends little on caption details.

We have added Table R5 as Table 5 (Sec. 4.3) and the caption statistics to App. B.1.

Table R5. Full vs. reduced prompts (101 clips; vehicle-only mAP as in App. B.4).

PromptMethodFID ↓FVD ↓CLIP-T ↑Lane F1 (%) ↑Vehicle mAP ↑
FullInfiniCube99.77970.6630.457.830.140
FullOpenDWM + DGGT95.07814.3631.1010.710.292
FullCosmos3 + DGGT111.421184.8930.399.390.148
FullDriveReGen66.26583.3830.6913.440.317
ReducedInfiniCube101.01877.7929.9210.160.100
ReducedOpenDWM + DGGT106.75867.3330.2410.680.315
ReducedCosmos3 + DGGT105.511259.2329.8010.380.156
ReducedDriveReGen72.58651.6430.3612.540.331

Summary

We thank the reviewer again for the constructive feedback. The added results extend camera shifts to 6 m lateral and 4 m forward, check the frozen representation on nuScenes, quantify seed and scene-level uncertainty, and repeat the comparison with reduced prompts. We hope these responses address the reviewer's concerns and further strengthen their positive assessment of our work.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses

Dear Reviewer k7Rv,

Could you please let us know whether our rebuttal addresses your main concerns? We hope our responses will be helpful in your final assessment.

Thank you for your time.

Best wishes,
The authors

Reviewer_BjbrOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper presents DriveReGen, a method for generating text- and layout-conditioned driving scenes directly in the reconstruction state space of DGGT, then decoding them into renderable 3D Gaussian scenes. The authors introduce a two-stage layout control and Hierarchical Decoding Supervision (HDS) for generation. The paper reports improvements on multiple benchmarks.

Critical strengths
  • Generate driving scenes in the latent reconstruction space is useful and interesting.
  • The evaluation covers multiple objectives. In addition to FID/FVD and text alignment, the paper evaluates lane and vehicle layout, LiDAR depth consistency,
  • The experiment demonstrates the performance of DriveReGen.
Critical weaknesses
  • The contributions might be engineering and have limit novelty. The performance of DriveReGen might mainly depend on DGGT.
  • The reconstruction-feature loss in HDS seems not effective. In Table 6, when adding reconsturction-feature loss, the improvement in each metric is not obvious.
  • The depth evaluation in Table 3 might be not reasonable. DA2 is an old model and the output depth might be inaccurate. Therefore, the depth evaluation for DriveReGen is unconvincing.
Questions for authors

Please see the weakness.

4 comments
AuthorsOpenReview ↗

Rebuttals (1/2) - Overview & W1

We thank the reviewer for the thoughtful review and constructive feedback! We are glad that the reviewer found generating driving scenes in the reconstruction space "useful and interesting", noted that our evaluation "covers multiple objectives", and found that our experiments demonstrate the performance of DriveReGen. We also appreciate the questions on novelty, HDS, and depth evaluation, which helped us clarify our contributions and strengthen our evidence.

We have made the following changes to the manuscript:

  • Contributions (Sec. 1). We state the problem behind our method and rewrite the contribution list around it.
  • LiDAR evaluation of all baselines (Table 3). OpenDWM + DGGT and Cosmos3 + DGGT are now compared with real Waymo LiDAR, together with InfiniCube.
  • Robustness to the depth estimator (App. B.1, Table 9). We repeat the RGB–3D consistency check with Depth Anything 3.
  • Feature supervision in HDS (Sec. 4.4). We report its AbsRel and lane F1 gains, their rationale, and its cost. We also revised Table 7 to make the cumulative addition of HDS losses explicit.

Table numbers below follow the revised paper: Tables 5 and 6 of the submission are now Tables 6 and 7.

Below, we respond to each comment in turn.


W1. Novelty and dependence on DGGT

We would like to clarify our novelty. DriveReGen builds on established tools such as flow matching and adapters, but its contribution is not their combination. Our novelty is a problem that prior work did not face, and our solution to it. DGGT's reconstruction space has no metres. Placing a metric layout in it requires the scene's calibrations (scale and FOVs), but these calibrations depend on the scene that is still being generated.

The problem. DGGT inherits VGGT's per-scene normalization, so metres per DGGT unit vary across scenes, and its decoder FOVs differ from those of the physical cameras (Fig. 2a, b). General-scene methods that generate in reconstruction space (e.g., OneWorld, Gen3R) take no metric layouts.

Our solution.

  1. Scale and FOVs are generated with the scene, as they cannot be given as inputs. The generated scale is three times closer to its LiDAR-calibrated reference than a training-set constant (Table 6). Using constants instead raises AbsRel from 0.1581 to 0.2413 (Fixed units, Table 7).
  2. The layout enters in two stages. Its 2D projection needs neither scale nor decoder FOVs, so Stage 1 uses it before they are predicted. Its depth and size need both, so Stage 2 converts them with the scale and FOVs predicted in the same flow step. With the same modules and inputs, using constants only in this conversion lowers actor mAP from 0.1251 to 0.0873 (Fixed S2 γ\gamma, Table 7).
  3. The generator is supervised on the decoded scene. HDS backpropagates feature, geometry, and image losses through the frozen decoder, whereas prior driving Gaussian pipelines (WorldSplat, CVD-STORM, ScenDi) train their generators with latent losses only (Sec. 2.2). Without HDS, actor mAP drops to 0.0919 and FVD rises from 260.77 to 288.33 (Table 7).

Dependence on DGGT. We agree that DGGT matters: it defines the space we generate in and decodes our states, so its reconstruction quality carries over to our scenes (LiDAR AbsRel 0.0494 on recorded videos vs. our 0.0621, Table 3). However, DGGT alone does not explain our gains, since two baselines use the same frozen DGGT.

  1. Visual quality. With the same DGGT, OpenDWM and Cosmos3 reach FID 52.72 and 64.83 vs. our 37.15 (Table 1). Our renderings are also better than their original videos before DGGT reconstruction (FID 44.74 and 60.92, Table 11).
  2. Metric scale. DGGT's outputs have no metric scale. On the same generated scenes, replacing our generated scale with a training-set constant doubles LiDAR AbsRel from 0.0810 to 0.1705 (Table 3).
  3. Layout and geometry. With the same DGGT, OpenDWM + DGGT reaches 5.63% lane F1 under a 2 m camera shift vs. our 11.52% (Table 4), and 0.2365 LiDAR AbsRel vs. our 0.0621 (Table 3).

Our submission listed the three contributions side by side, which may have hidden how they are connected. We have revised Sec. 1 to make this clear.


[Continued below...]

AuthorsOpenReview ↗

Rebuttals (2/2) - W2, W3 & Summary

W2. Feature loss in HDS

Thanks for this comment. We would like to clarify that the feature term's gains are larger than they may appear. It is added last, after the image and geometry terms have already improved the model and left less room for gains. It still improves all four metrics, and its AbsRel and lane F1 gains are no smaller than those of the other two terms.

  1. Ablation order. Its gains come on top of the image and geometry terms. Even so, its lane F1 gain (+0.35 points) exceeds that of the first two terms together (+0.27).
  2. Gains (Table R1). The feature term lowers AbsRel by 2.2% and raises lane F1 by 2.9% and actor mAP by 6.1%, each under an independent frozen evaluator with the same clips and seeds. It gives 38% of the total HDS gain in AbsRel and 56% in lane F1.
  3. Mechanism. It supervises the generated DGGT features at the four levels that the frozen heads read (Sec. 3.4). The depth and Gaussian heads predict geometry from these features, so feature errors propagate to the decoded depth and Gaussians. The gains are consistent with this mechanism: they appear on AbsRel, lane F1, and mAP, which depend on decoded geometry, while FVD reflects appearance, already supervised by the image term.
  4. Cost. It reuses features already decoded for the geometry term, adding only a distance computation during training. Inference is unchanged, so we keep the term for its gains across metrics.

We have revised Sec. 4.4, which discussed only the FVD change of this term, to report these gains, their rationale, and the cost.

Table R1. Gain of each HDS term over the previous row of Table 7 (Table 6 in the submission; relative change in parentheses).

Added term (cumulative)FVD ↓AbsRel ↓Lane F1 ↑mAP ↑
+ RGB−13.48 (−4.7%)−0.0032 (−1.9%)−0.04 (−0.3%)+0.0231 (+25.1%)
+ Geometry−11.73 (−4.3%)−0.0024 (−1.5%)+0.31 (+2.6%)+0.0029 (+2.5%)
+ Feature−2.35 (−0.9%)−0.0035 (−2.2%)+0.35 (+2.9%)+0.0072 (+6.1%)

W3. Depth evaluation and DA2

We appreciate this concern. We would like to clarify that Table 3 uses real Waymo LiDAR as the depth reference, with no depth estimator. We believe the reviewer may be referring to Table 1, where DA2 measures RGB–3D consistency. To evaluate depth without an estimator, we extend our LiDAR evaluation to all baselines. To test a newer estimator, we repeat the consistency check with DA3. DriveReGen remains the best generator in both.

  1. LiDAR for all baselines (Table R2). Table 3 of our submission already compared DriveReGen and InfiniCube against real LiDAR. We now add OpenDWM + DGGT and Cosmos3 + DGGT. DriveReGen has the lowest AbsRel and RMSE and the highest δ1\delta_1 among generators (AbsRel 0.0621 vs. 0.0975 for the strongest baseline, InfiniCube). On all clips, our AbsRel also stays below InfiniCube's (0.0802 vs. 0.0966, Table 3). The DGGT-paired baselines use a trajectory-based scale (Sec. 4.1); this rule gives DGGT 0.0494 on recorded videos, so their errors come from the generated videos.
  2. A newer estimator, DA3 (Table R3). We used DA2 following CVD-STORM and WorldSplat. First, we test both estimators on the real images of benchmark clips: DA3 is closer to LiDAR than DA2 (AbsRel 0.107 vs. 0.141). Then, we repeat the consistency check with DA3: DriveReGen retains the best AbsRel and δ1\delta_1, ahead of the strongest baseline, InfiniCube (0.1241 vs. 0.1903 and 0.8601 vs. 0.6840). The ranking matches Table 1, so our conclusion is not specific to DA2.

We have added Table R2 to Table 3 and Table R3 to App. B.1 (Table 9).

Table R2. Road and actor depth vs. real Waymo LiDAR (clips with ego motion).

MethodAbsRel ↓RMSE (m) ↓δ1\delta_1 ↑
InfiniCube0.09752.15490.8967
OpenDWM + DGGT0.23653.79310.6925
Cosmos3 + DGGT0.40246.12070.6295
DriveReGen0.06211.43760.9608
DGGT direct‡0.04941.12580.9731

‡DGGT reconstruction of the recorded video, for reference.

Table R3. RGB–3D consistency with DA3 in place of DA2.

MethodAbsRel ↓δ1\delta_1 ↑
InfiniCube0.19030.6840
OpenDWM + DGGT0.26140.6034
Cosmos3 + DGGT0.41810.4984
DriveReGen0.12410.8601

Summary

We thank the reviewer again for the constructive feedback, which has helped us articulate our novelty, extend the LiDAR evaluation to all baselines, confirm our depth results with DA3, and clarify the gains of feature supervision. We hope these responses address the reviewer's concerns, and we would be grateful if the reviewer could take the updated results into account in the final assessment. We are happy to discuss any remaining questions.

AuthorsOpenReview ↗

Follow-up on Our Rebuttal Responses

Dear Reviewer Bjbr,

Could you please let us know whether our rebuttal addresses your main concerns? We hope our responses will be helpful in your final assessment.

Thank you for your time.

Best wishes,
The authors

AuthorsOpenReview ↗

A follow-up on our rebuttal

Dear Reviewer Bjbr,

Thank you again for your constructive feedback! With the discussion period ending soon, we wanted to briefly highlight our responses to your three concerns:

  • Novelty and DGGT: We clarify how jointly generating scale and FOVs enables metric layout control, and show gains over baselines using the same frozen DGGT.
  • HDS: Added after image and geometry supervision, the feature term still lowers AbsRel by 2.2% and raises lane F1 by 2.9%, without extra inference cost.
  • Depth evaluation: We now evaluate all baselines against real Waymo LiDAR and repeat the RGB–3D consistency check with DA3. DriveReGen leads the compared generators in both.

We’d really appreciate hearing whether these responses address your concerns. We hope you’ll take these clarifications and additional results into account in your final assessment.

Best regards,
The Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses and Paper Revisions

Dear Reviewers and Area Chair,

We thank all reviewers for their constructive feedback. We have replied to each review and uploaded a revised paper, with changes highlighted in blue.

Paper summary. DriveReGen generates driving scenes as 3D Gaussians in the reconstruction space of a frozen feed-forward model, DGGT. This space has no metres, so placing metric HD maps and actor tracks in it requires the scene's scale and decoder FOVs, which depend on the scene being generated. DriveReGen generates them jointly with the scene, uses them in a two-stage layout control, and trains the generator on the decoded scene with Hierarchical Decoding Supervision (HDS).

Main responses and revisions.

  • Novelty and the role of DGGT (Bjbr, diRC): Our novelty lies in placing metric layouts in a reconstruction space without metres, a problem that prior work did not face, and in our solution to it (revised Sec. 1). DGGT alone does not explain our gains: baselines using the same frozen DGGT have worse FID and FVD (Table 1). In LiDAR depth, our generated scenes stay close to DGGT's own reconstructions of the recorded videos (Table 3).
  • Depth evaluation (Bjbr): Table 3 uses real Waymo LiDAR and now includes all baselines, with DriveReGen the most accurate generator; a DA3 check gives the same ranking as DA2 (App. B.1).
  • Feature loss in HDS (Bjbr): Although added last in the ablation, the feature loss still improves depth and layout metrics (Sec. 4.4, Table 7; Table 6 in the submission).
  • Larger shifts, longer clips, and nuScenes (k7Rv): DriveReGen stays ahead under ±6 m lateral and +4 m forward shifts and on 29-frame clips, and the frozen DGGT and our tokenizer transfer to nuScenes without adaptation (App. B.5, B.7, B.8).
  • Seeds and captions (k7Rv): Our main advantages hold across three sampling seeds, under scene-level paired tests (App. B.9), and with reduced prompts (Sec. 4.3).
  • Latent width (diRC): C = 1024 is DGGT's native token width, and decoding is robust to truncating the latent (App. B.5).

We would appreciate it if you could review our responses and the revised manuscript and share whether they address your concerns. We would be glad to discuss any remaining questions.

Best regards, The Authors

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.