acceptodds
Under review as a conference paper at ICLR 2027

TerrainForge: Physics-Grounded road geometry Editing for Counterfactual Autonomous Driving

Abstract

Road geometry (e.g., crests, sags, and speed humps) and surface conditions (e.g., wet or icy pavement) affect how vehicles move, what drivers and onboard cameras observe, and how much clearance remains between vehicles. Editing these properties in a driving scene therefore requires corresponding changes in vehicle motion. Capturing these differences in a driving video requires a road edit to propagate to vehicle motion, camera viewpoint, and the clearance between vehicles. We present TerrainForge, a framework for generating road geometry-focused counterfactuals from reconstructed multi-vehicle driving episodes. A unified road model connects scene deformation with four-wheel vehicle dynamics, allowing crests, sags, speed humps, and friction changes to propagate through vehicle motion, camera viewpoint, and inter-vehicle clearance. Vehicle dynamics are evaluated against CarSim, and prescribed road geometry is verified in reconstructed Waymo scenes. Across 18 episodes, leaving surrounding vehicles on their recorded trajectories instead of recomputing their responses produces median peak differences in predicted ego–lead distance of 1.52 m for crests and 1.41 m for sags. We further simulate the ego response to 15,758 road edits across 983 braking episodes, pairing each edit with its safety outcomes relative to an unedited replay. These pairs train a first-stage screening surrogate that takes the original driving context and candidate road-edit parameters as input and predicts the resulting change in the ego's terminal gap. On held-out scenes, this prediction achieves 22–40% lower mean absolute error than predicting no change, so candidates can be screened cheaply before the full multi-vehicle rollout.

Primary area:
applications to robotics, autonomy, planning
Keywords:
Counterfactual driving; Physics-aware generation; Road geometry editing; Vehicle dynamics ;3D Gaussian Splatting

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_E38JOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

TerrainForge studies road-geometry and friction counterfactuals in reconstructed driving episodes. A shared road field drives Gaussian-scene deformation and vehicle dynamics; inferred controls are held fixed across paired baseline and edited runs, and their motion difference updates the recorded vehicle poses and camera trajectory. The main multi-vehicle experiments replay plan-view paths while recomputing longitudinal and vertical response, rather than modeling reactive drivers. The evaluation combines CarSim comparisons, ZOD/LiDAR road-profile checks, geometry verification in Waymo reconstructions, and multi-vehicle gap analysis. A scene-disjoint bank of 15,758 ego-only road edits trains surrogates for terminal-gap change and consequential-edit detection. The reported surrogate results support prediction under this intermediate protocol; their transfer to the proposed full multi-vehicle screening pipeline remains to be established.

Critical strengths
  1. A useful, coherent formulation of fixed-control road counterfactuals. The shared road representation connects geometric editing, vehicle response, and rendering instead of treating terrain as appearance alone. Under the primary edits, recomputing surrounding vehicles changes the estimated ego–lead gap response: the median peak discrepancy from ego-only propagation is 1.52 m for crests and 1.41 m for sags across 18 episodes (Section 4.3). These are peak gap discrepancies, not demonstrated safety improvements, but they substantiate the importance of propagating a shared road edit beyond the ego vehicle.

  2. Substantial component validation and transparent reporting. The CarSim study distinguishes development from held-out cases, includes steering and vertical-alignment tests, and reports raw-time errors and numerical checks. Independent scene-level splits and bootstrap intervals strengthen the surrogate evaluation, while analytical and statistical baselines help identify what is learned. The paper also reports model-domain failures, weak minimum-clearance prediction, and differences among experimental protocols rather than presenting all results as a single end-to-end validation. This supports confidence in the demonstrated component-level findings.

Critical weaknesses
  1. Screening effectiveness is not established for the final multi-vehicle outcomes. Section 4.5 and Appendix H train and evaluate the surrogate with recorded target motion, whereas Section 4.3 and Appendix E.2 show that allowing the lead to respond can substantially change, and in the illustrated episode reverse, the gap-change estimate. Recall@20% = 0.80 in Table 15 therefore measures retention of ego-only labels, not retention of consequential edits under full multi-vehicle simulation. Re-simulating only shortlisted candidates cannot recover important edits discarded by the screen. Moreover, the label is direction-agnostic, minimum-clearance prediction does not improve reliably over no change (Table 14), and only 7.0% of bank rows have both rollouts stop (Appendix H), so terminal-gap accuracy should not be equated with stopping-margin or collision-risk accuracy. A bounded end-to-end shortlist evaluation, including total computational cost, is needed to support the proposed two-stage use.

  2. Accuracy of the coupled model is better supported than its incremental benefit for the reported counterfactual tasks. The CarSim holdouts validate the 32-state model, and Appendix G demonstrates lateral/load responses; my concern is not an absence of dynamics validation. However, the real-road test uses the inherited half-car, and kinematic road following has numerically lower test world-frame pitch RMSE than that half-car (0.236 versus 0.258 degrees; Table 8). As the authors note, this supports road-attitude recovery, not an advantage of richer dynamics. The main multi-vehicle bank fixes plan-view paths, and Appendix E.2 already explains much of the load response through a curvature-based formula. What remains insufficiently isolated is when richer coupling materially improves gap, wheel-load, or screening decisions relative to a matched reduced model. A small comparison on existing cases would clarify the added value and appropriate operating regime of this central component.

Questions for authors
  1. On a representative held-out subset, how much of the full multi-vehicle consequential-edit set is retained by the surrogate shortlist? Please report recall at a fixed simulation budget against random selection and a simple physics/context baseline, distinguish harmful outcomes where sample support permits, and include feature-extraction and rollout costs. This targeted test would determine whether the current intermediate-label results support the proposed screening use; otherwise, would the authors restrict that claim to ego-response screening?

  2. Beyond the existing CarSim accuracy tests and curvature-formula analysis, which outcomes benefit materially from the 32-state model over reduced longitudinal/half-car alternatives? If no direct comparison exists, could a bounded ablation on existing crest/sag and hump cases, with the same interventions, controls, and horizons, quantify changes in gap and wheel-load extrema relative to CarSim where available? This would clarify where the additional coupling is necessary without requiring a new benchmark.

4 comments

We thank the reviewer for recognizing our unified road representation and component validation. We address both concerns with new experiments.

1. Full multi-vehicle screening. We appreciate the distinction between ego-response and all-vehicle screening. We generated 28,736 all-vehicle labels across 1,922 scenes: 21,364 edits/1,422 scenes for development and 7,372 edits/500 scene-disjoint B2 scenes for evaluation. R@20 is recall in the top 20%; B@80 is the budget for 80% recall.

MethodConsequential R@20 / AP / B@80Harmful R@20 / AP / B@80
Random (measured).192 / .346 / 80.5%.182 / .093 / 81.7%
Context logistic regression.374 / .622 / 56.8%.533 / .313 / 50.0%
Two-car physics prior.448 / .686 / 61.8%.563 / .370 / 57.0%
Original ego-only surrogate.403 / .654 / 55.6%.498 / .306 / 50.9%
All-vehicle surrogate.503 / .813 / 43.8%.544 / .352 / 43.2%
Interaction-aware (3c / H4).517 / .860 / 35.9%.775 / .637 / 22.9%

For consequential edits, 3c reduces B@80 from 55.6% (original) to 35.9% (random: 80.5%). With 34.9% positives, the R@20 ceiling is .573. With identical training scenes and inputs, changing only ego-only to all-vehicle labels improves R@20 .255→.482 and AP .435→.781. Adding the two-car prior raises R@20 to .503; pre-intervention interaction features then improve AP .813→.860 and B@80 43.8%→35.9%. Features use logs and edit-free replay only. Pre-registered paired 95% CIs: [+.031,+.064] for consequential AP (3c vs all-vehicle) and [+.138,+.291] for harmful R@20 (H4 vs two-car prior).

Harmful edits. We define harmful edits as a new footprint collision or a ≥0.1 m decrease in minimum signed ego-to-other-vehicle footprint clearance (edited minus baseline; common horizon; all movers re-simulated with frozen controls). Terminal gap is excluded. Among 719 positives, 196 are new collisions; the remaining 523 have a median clearance decrease of 0.45 m. Raising the threshold to 0.5 m preserves H4's advantage over the two-car prior (R@20 .845 vs .658; 95% CI of difference [+.107,+.279]). For new collisions or clearance deterioration ending within 1 m, the R@20 difference is not statistically significant (.692 vs .638; CI [-.020,+.153]), while the prior has higher AP (.413 vs .390). Thus, we claim improved clearance-deterioration screening, not superior near-collision or collision-risk prediction.

Computational cost. Costs were measured on 335 B-group scenes/5,088 edits (x86); B2 used different hardware. Screening costs 10.3 CPU-s/edit: 9.85 s for shared edit-free ego baseline replay, 0.45 s for interaction features, and ~0.02 s for other features/prior/inference. A full all-vehicle rollout costs 662 CPU-s, baseline included. We charge scene-fixed work once per selected scene without double-counting. At matched 80% recall, 3c uses 41% of exhaustive compute (2.4×) for consequential edits, versus 66% (1.5×) for the original model and 83% (1.2×) for random. H4 uses 29% (3.5×) for harmful edits, versus 69% (1.5×) for the two-car prior. These are modeled CPU-time speedups, not wall-clock measurements; claims concern lead-following scenes.

2. Incremental value of 32-state dynamics. We compared kinematic (Kin), point-mass (PM), half-car (HC), and full 32-state models with matched edits, controls, initial states, and horizons in 34 CarSim cases (12 OOD holdouts). Errors are against CarSim.

RegimeSpeed RMSE PM/32 (m/s)Wheel-load RMSE PM/HC/32 (N)Pitch RMSE Kin/HC/32 (°)
Long curves (7).030/.00720/58/14.024/.015/.014
Braking (11).010/.001219/90/24.243/.108/.020
Hump/depression (4).107/.0201264/276/132.390/.084/.037
OOD bumps (4).051/.0101000/171/112.803/.055/.051
Dry stopping (3).015/.014835/167/521.446/.212/.069
Low-friction ABS (5).281/.092692/199/521.222/.264/.089

Across all reported regimes and metrics, the 32-state model achieves the lowest errors. Although reduced models already approximate smooth-curve longitudinal motion well, the benefits of 32-state dynamics are more pronounced under braking, bumps, and low friction, where suspension, tire forces, and load transfer are strongly coupled. Across 3,924 speed-bump edits, the median PM/32-state terminal-gap response ratio is 5% (IQR 2–14%). Top-ranked gap-increasing edits agree in 41.9% of 1,572 candidate groups (788 scenes) (95% CI 38.6–45.4%; random: 10.8%). Fidelity changes edit priorities, though disagreement alone does not prove superiority. As the reviewer notes, Table 8 evaluates real-road attitude recovery with the inherited half-car; the new matched CarSim comparison instead identifies where higher-order coupling provides a measurable benefit.

Thank you for the additional experiments. The all-vehicle screening evaluation addresses the original label mismatch, and the matched CarSim ablation supports the benefit of richer dynamics. I also appreciate the qualification that improved clearance screening does not establish better collision-risk prediction.

Two questions remain:

  1. Controls are inferred, not measured. Different vehicle parameters, source-road friction, and controls may fit the recorded motion similarly but predict different responses after editing. CarSim validates forward dynamics, not the recovered inputs. Could you test whether gap changes and collision outcomes remain stable across plausible configurations with comparable replay errors?
  2. An edit that is adverse under frozen controls may be easily handled by a policy that brakes earlier. Do the added feedback policies use rendered images or simulator states? A small closed-loop test with an existing vision-based policy would show whether the edited scenes expose policy failures and whether rendering contributes to the evaluation. Otherwise, please clarify the concrete use of fixed-control analysis and limit the safety claims accordingly.
AuthorsOpenReview ↗

Part 1-2- Inferred Controls

We thank the reviewer for the constructive follow-up questions. We conducted additional experiments to address two concerns: the robustness of counterfactual outcomes to uncertainty in recovered controls and the contribution of rendered images to closed-loop driving evaluation.

1. Robustness to Inferred Controls

We appreciate the reviewer's concern regarding uncertainty in recovered controls. To examine whether different parameter–control configurations that reproduce the same recorded motion lead to consistent counterfactual predictions, we conducted an additional robustness evaluation.

We evaluated 40 Waymo scenes and 636 road edits, including 13 near-contact scenes and 27 randomly selected scenes. We sampled 16 configurations from a nine-dimensional parameter space covering vehicle mass, load distribution, CG height, tire characteristics, rolling radius, suspension, wheel inertia, and source-road friction (0.65–1.00).

For each configuration, we separately recovered the controls needed to reproduce the recorded trajectory, then applied identical road interventions to compare their counterfactual responses.

All configurations reproduced the recorded motion with the same reported median speed RMSE of 0.011 m/s. Despite the ambiguity in recovered parameters and controls, the counterfactual outcomes remained highly consistent:

MetricResult
Direction agreement for consequential minimum-gap changes99.6%
Nominally harmful edits remaining harmful across all configurations190/190
Rank correlation with nominal results (median / minimum)0.998 / 0.990
Range of gap changes across configurations (median / p90)0.04 / 0.17 m
Nominal collision-producing edits remaining collisions across all configurations12/13

As a control, applying the nominal controls directly to alternative vehicle configurations increased factual replay error to 0.34 m/s and produced collisions on the unedited road in 27% of cases. This confirms the importance of recovering controls separately for each configuration.

Sensitivity near friction limits. We further investigated edits exhibiting unusually large variations and corrected an evaluation-window truncation issue caused by brief wheel unloading. After recomputing the affected cases, 5 of 636 edits (0.8%) showed genuine configuration sensitivity. All involved braking on a low-friction sag (edited μ = 0.4), where some configurations produced lateral departures or substantially different stopping positions. One edit changed its collision classification, so our predefined collision-stability criterion was not fully satisfied.

We retain these cases in the reported statistics and will document the post-hoc evaluation-window correction and sensitivity near friction limits.

Overall, although the recovered vehicle parameters and controls are not unique, the direction and ranking of TerrainForge's counterfactual responses remain highly consistent across the sampled configurations. These results indicate that the main counterfactual conclusions do not depend strongly on a particular recovered control configuration, while identifying conditions where collision outcomes require greater caution.

AuthorsOpenReview ↗

Part 2/2 — Vision-Based Closed-Loop Evaluation

2. Vision-Based Closed-Loop Evaluation

We appreciate the reviewer's suggestion to further evaluate TerrainForge with a vision-based driving policy. Building on our previous experiments with state-based feedback policies, we extended the evaluation to vision-based closed-loop driving, where policies receive TerrainForge-rendered observations and adapt their controls accordingly. We conducted two additional experiments to examine how road edits affect driving decisions and whether the rendered scenes can expose policy failures.

(a) Does rendering affect an end-to-end vision-based driving policy?

We integrated the publicly released SimWAM-Waymo-IL policy without retraining. Every 0.25 s, the policy receives a newly rendered front-camera image, ego motion state, and route command, predicts a 5-second driving trajectory, and controls the vehicle through TerrainForge's 32-state dynamics model. The entire closed-loop evaluation uses updated rendered observations rather than prerecorded video.

We evaluated 13 Waymo scenes, of which 10 passed predefined zero-edit validation checks. We compared three conditions: (A) unedited road, (B) edited road with corresponding rendered images, and (C) edited road with unedited imagery as a visual ablation. Each condition was evaluated with three seeds.

In 26 of 30 edited-road conditions, the policy exhibited consistent driving responses exceeding seed-level variation, including 17 of 20 design-plausible edits. Under these plausible interventions, vehicles were a median 3.1 km/h slower at the curve apex and traveled 11.5 m less over 20 s.

Visual ablation further confirms the contribution of rendering. When edited-road physics was retained but edited imagery was replaced with unedited imagery, 22 of the 26 robust responses no longer exhibited a corresponding robust change.

These results demonstrate that edited visual observations contribute to the policy's decisions and resulting closed-loop vehicle motion, beyond the effects of road physics alone.

(b) Can edited scenes expose vision-related braking failures?

To further examine the role of visual–physical consistency, we evaluated a YOLO-based braking controller in a scenario involving a stationary pickup ahead. Unlike the end-to-end SimWAM experiment, this detector-based setup allows us to directly trace how changes in hazard visibility affect detection, braking, and collision outcomes.

Across 36 combinations of road geometry, friction, and initial speed, frozen controls resulted in collisions in 14/36 conditions, all of which were avoided by a state-feedback stopping controller.

The paired evaluation identified two visibility-sensitive braking cases under blind-summit, low-friction conditions. Under identical edited-road physics, the controller successfully stopped when supplied with unedited imagery, whereas the edited renderings delayed hazard detection and led to collisions.

At 50 km/h, the edited rendering reduced the hazard detection distance from 79 m to 33.5 m, resulting in a collision at 23 km/h. At 60 km/h, the hazard became visible only after the last feasible stopping point.

These results illustrate why visual and physical responses must be evaluated together. Under the same edited-road physics, using unedited imagery would miss the changes in visibility and braking outcomes introduced by the road intervention. The stress-test cases demonstrate how TerrainForge can expose vision-related driving failures that a physics-only evaluation would overlook.

Together, these experiments show that TerrainForge's rendered observations can influence closed-loop driving decisions and expose vision-related braking failures under challenging road conditions. They extend our earlier physical-response analysis to evaluating how road interventions affect visual perception and closed-loop driving behavior.

We appreciate the reviewer's suggestion, which helped strengthen the downstream validation of TerrainForge. We will incorporate these experiments into the revised manuscript to provide a more complete evaluation of its downstream applications.

Reviewer_5k1WOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

TerrainForge edits road elevation and friction in reconstructed driving scenes, then recomputes vehicle motion and camera views. It also trains a surrogate to screen road edits before running the full simulation.

Critical strengths

The setting is novel and under-explored: road geometry treated as a causal intervention. a substantial amount of engineering work to ensure coherent editing. A shared road representation drives vehicle dynamics, scene deformation, and camera motion, avoiding edits that change appearance without changing vehicle behavior. Critical Weaknesses:

Critical weaknesses

The main critical weakness is that it is evaluated on fixed control playback, Vehicles replay inferred controls; they do not brake, steer, or adapt to the changed road or neighboring vehicles. This supports controlled mechanical counterfactuals, but limits realistic traffic and safety conclusions.

Questions for authors

How do the multi-vehicle results change if all vehicles use a closed-loop (e.g., IDM) driver? What exactly does TerrainForge add over muSync-GS? Can you ablate the 32-state model against a point-mass grade model? Can you add a realism or artifact evaluation of the edited renders, especially near curbs and building contacts? Why train the surrogate on ego-only rollouts, and how well does it predict outcomes when all vehicles respond?

2 comments

We thank the reviewer for recognizing our road-intervention contribution and suggesting further evaluations.

Fixed-control playback and reactive driving. We appreciate the reviewer's question about how driver adaptation may affect multi-vehicle outcomes. Our primary goal is to isolate the physical effects of road geometry and friction changes. Replaying the same recovered controls on original and edited roads lets us compare vehicle responses under identical driving inputs, isolating road-induced effects from changes in driving decisions. While real drivers may adapt their behavior to modified conditions, closed-loop driving provides a complementary perspective rather than replacing this controlled comparison.

To examine adaptation, we evaluated four policies in 20 Waymo scenes (343 vehicles). Here DD is the median across vehicle pairs of the peak discrepancy in road-induced ego–lead gap change between ego-only and all-vehicle rollouts:

PolicyCrest D (m)Sag D (m)
Frozen1.521.41
Speed keeping0.240.24
IDM0.600.58
PDM-inspired0.891.07

Feedback reduces but does not eliminate this discrepancy (crest: 1.52 m frozen vs. 0.60 m IDM). Thus, fixed controls isolate road-induced effects, while adaptive policies show why multi-vehicle propagation still matters.

Distinction from muSync-GS. Both methods couple road edits with vehicle dynamics. TerrainForge adds (i) route-scale edits of grades, crests, and sags beyond local bumps/depressions; (ii) multi-vehicle propagation for relative-motion analysis; (iii) 32-state four-wheel dynamics capturing lateral/yaw/roll, independent suspension, and coupled tire/load effects; and (iv) consequential-edit screening using an intervention–response dataset and surrogate. This enables multi-vehicle road counterfactuals.

Dynamics fidelity ablation. We compared four models—kinematic, point-mass, half-car, and 32-state—on 34 CarSim cases (12 OOD holdouts), with matched controls, states, and edits:

Regime (n)Speed PM/32 (m/s)Travel PM/32 (m)Load PM/HC/32 (N)Pitch Kin/HC/32 (°)
Long curves (7).030/.007.37/.1120/58/14.024/.015/.014
Braking (11).010/.001.05/.01219/90/24.243/.108/.020
Hump/sag (4).107/.020.76/.141264/276/132.390/.084/.037
OOD bump (4).051/.010.40/.081000/171/112.803/.055/.051
Dry stop (3).015/.014.03/.01835/167/521.446/.212/.069
Low-μ ABS (5).281/.0921.41/.50692/199/521.222/.264/.089

PM: point-mass; HC: half-car; Kin: kinematic; 32: full model. Speed/load/pitch: RMSE; travel: distance error. The 32-state model has the lowest errors. While simpler models approximate smooth curves, the full model better captures tire-load and suspension coupling under braking, bumps, and low friction. Across 3,924 speed-bump edits, the point-mass model captured a median 5% (IQR: 2–14%) of the 32-state terminal-gap response magnitude. Across 1,572 candidate groups (788 scenes), top-ranked gap-increasing edits agreed in 41.9% of groups (95% CI: 38.6–45.4%; random: 10.8%); most disagreements involved different bump placements. Thus, fidelity also affects downstream edit selection.

Rendering realism and artifacts. We evaluated rendering artifacts near curbs and buildings. Across 30 Waymo scenes (17,838 training views), zero-edit rendering achieves 29.7 dB PSNR, 0.889 SSIM, and 0.147 LPIPS. Detection recall under crest/sag edits is 0.832/0.853 versus 0.845 on recorded frames. For moderate edits, the 3-pixel curb-alignment rate falls only 0.3–0.8 percentage points from zero-edit rendering; median error remains 1.73 px. At building contacts, transparent-gap ratio stays below 0.04% and 95th-percentile foundation discontinuity is 1.1 px. Moderate edits show limited degradation, with some local artifacts.

All-vehicle surrogate screening. The original ego-only surrogate served as a low-cost first-stage filter. For all-vehicle screening, we generated 28,736 all-vehicle labels across 1,922 scenes, with 21,364 edits (1,422 scenes) for training and 7,372 (500 scene-disjoint scenes) for testing. We added ten pre-intervention interaction features.

ModelRecall@20 ↑AP ↑
Ego-only.403.654
All-vehicle.503.813
Interaction-aware.517.860

All-vehicle labels drive the main gain; interaction context helps further. The benchmark-specific Recall@20 ceiling is .573; .517 reaches ~90% of this ceiling. For direction-aware harmful edits, the interaction-aware model reaches Recall@20=.775 and AP=.637, versus .563/.370 for the two-vehicle physics prior. This improves multi-vehicle consequential-edit screening.

Additional Reactive-Driving and Safety Evaluation

To further address the reviewer's concern about how fixed-control playback affects multi-vehicle interactions and safety-related outcomes, we conducted an additional reactive-driving evaluation building on our initial response. While our earlier IDM and PDM-inspired experiments examined changes in ego–lead distance, this expanded evaluation directly measures how reactive driving affects traffic-conflict outcomes across 335 Waymo scenes and 5,088 road edits.

We additionally incorporated Gipps' car-following model to provide longitudinal feedback, allowing vehicles to respond to road-induced changes and their leaders' altered motion. We adopted Gipps for this larger-scale evaluation because the residual IDM controller exhibited oscillations at short headways when coupled with tire-actuation dynamics. Gipps provided a more stable feedback baseline in this setting.

We compared traffic-conflict outcomes under fixed-control playback and reactive driving:

Safety-related outcome (5,088 edits)Fixed ControlReactive DrivingChange [95% CI]
Ego–lead conflict (new TTC < 2 s, increased TTC exposure, or collision)27369−204 [−274, −141]
New ego–lead collision5626−30 [−70, +4]
Adverse edit (new collision or ≥0.1 m reduction in ego clearance to any vehicle)633494−139 [−252, −29]
New overlap among other vehicles225238+13 [−57, +88]

95% confidence intervals are computed using 2,000 scene-level bootstrap resamples. The reported outcomes are simulation-based traffic-conflict indicators.

With Gipps feedback, ego–lead conflicts decrease by 75% (273 → 69), showing that driver adaptation can substantially reduce close-following conflicts. However, the number of adverse edits involving surrounding vehicles decreases more moderately (633 → 494, a 22% reduction), while changes in overlaps among other vehicles are not statistically conclusive. These results show that driver adaptation changes traffic-conflict outcomes, but road interventions continue to affect multi-vehicle interactions.

These results show that while driver adaptation changes traffic-conflict outcomes, road interventions continue to affect multi-vehicle interactions, highlighting the value of TerrainForge in modeling and evaluating road-induced physical responses under different driving policies.

Reviewer_wymNOpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

They proposed a framework for generating counterfactual driving scenarios by modifying road geometry and surface conditions in reconstructed real-world driving scenes. A single road field (elevation, slope, friction) both deforms the Gaussian reconstruction and drives a 32-state four-wheel vehicle model. Controls inferred from each recorded trajectory are frozen and replayed on the edited road, so any change in motion and camera pose is attributable to the a different scenario. The dynamics are validated against CarSim, the road profile against ZOD/LiDAR, and multi-vehicle propagation on Waymo. A boosted-tree surrogate trained on 15,758 edits predicts terminal-gap change with 22–40% lower MAE than predicting no change.

Critical strengths
  • Careful, component-wise validation. Each piece is checked against something it wasn’t fitted on: the vehicle model against CarSim on blind holdouts with pre-declared gates (the one failure is reported, not dropped), the recovered road profile against an independent LiDAR ground fit on ZOD, and the deformed scenes against the authored geometry with a flat edit as the measurement floor. This makes it possible to tell where an error comes from, which is rare in this area.

  • A useful problem formulation. Most controllable driving generators (MagicDrive, Panacea, DrivingDiffusion, and the HD-map/BEV-conditioned line of work) treat the road as a layout prior and let you move agents around on it; the road itself stays fixed background. Reconstruction-based editors like Real2Sim and CARLA-GS add physics but still mostly edit objects. Posing the road as the intervention and asking how a whole recorded traffic episode responds, with everything else held fixed, is a sensible next step that none of those systems can do.

  • Authors are honest about what doesn’t work: sustained grades barely show in the image, 2% edits do nothing laterally, the clearance change can’t be predicted, and the surrogate is only a first-pass filter.

Critical weaknesses
  • Image realism is never measured. Every image-side number is geometric (far-field optical flow, VGGT camera height, apex depth in the splats, displayed-vs-metric box offset). There is no PSNR/SSIM/LPIPS of the render against the recorded frames, not even for the zero-edit case where it should match almost exactly; no FID; no human study; no downstream check such as a detector behaving the same on render and recording. The display chain in Appendix A is a long series of heuristics, and the one number given means a car floats or sinks by nearly half a meter in one frame in twenty. Only one episode is rendered end-to-end with multi-vehicle response. For a paper whose output is video, this is better to be addressed.

  • Although multi-vehicle simulation is presented as a central contribution, the experiments in Section 4.3 keep the recorded plan-view paths fixed and independently replay the recovered longitudinal controls for each vehicle. Steering is zero in the road-aligned frame, and other vehicles do not react to the modified behavior of the ego vehicle or surrounding traffic. However, it might not be in the scope of this work.

Questions for authors

Please address the weaknesses and the questions below:

  • The ego stops within the horizon in only 8.5% of edited rollouts. For the other 92%, what does ∆G at an arbitrary horizon cutoff tell us about safety?

  • Since the multi-vehicle experiments replay fixed controls and recorded paths, how realistic are the resulting counterfactual interactions? How would the safety outcomes change if surrounding vehicles could react to the road modifications and to each other’s altered trajectories?

4 comments

We thank the reviewer for recognizing the value of our road-intervention formulation and validation. We conducted additional experiments to address the main concerns.

1. Rendering realism and artifacts. Across 30 Waymo scenes (17,838 training views), zero-edit rendering achieves 29.7 dB PSNR, 0.889 SSIM, and 0.147 LPIPS. FID is 27.2 for zero-edit and 40.0/39.9 for crest/sag edits. Vehicle-detection recall remains close to recordings (0.832/0.853 vs. 0.845).

MetricZero-editCrestSag
Curb alignment @ 3 px ↑0.6400.6320.637
Curb median error (px) ↓1.731.731.73
Building-contact transparency ↓0.009%0.017%0.031%
Building-contact flicker (/255) ↓3.413.523.43

Across 20 curb groups and 14 building-contact groups, alignment decreases by only 0.3–0.8 percentage points; building-contact transparency stays below 0.04%, and foundation discontinuity is 1.1 px (p95).

Multi-vehicle rendering and artifacts. We processed 34 additional candidate episodes and improved vehicle placement. The main artifact sources were distant-vehicle elevation scaling, depth ordering, and mismatch between the road-height field and visible road surface. After correction, 33/34 scenes have p95 display–metric deviations of 0.28–0.37 m (original-rule references: 0.47–0.93 m); all 34 have zero detected depth-order inversions.

Where paired measures are available, >10-cm wheel sinking falls from 21–50% to 0–9%, road-hidden wheels from 21–37% to 5–12%, and YOLO recall improves from 0.82–0.91 to 0.90–0.96. Scene 767010 remains outside this range (1.01→0.80 m), mainly due to degraded underlying reconstruction in adverse-weather imagery rather than road editing.

2. Interpretation of ΔG. We agree that terminal ΔG is not itself a safety metric. Since only 8.5% of edited ego rollouts stop, ΔG otherwise measures a finite-horizon road-induced longitudinal response, not completed stopping distance. Many targets are roadside objects, for which clearance is more relevant.

We added minimum TTC, time exposed to TTC < 2 s (TET), clearance, and simulated collisions. Among 97 edits identified as adverse by traffic-conflict proxies, 26 have nonnegative terminal ΔG, showing how a terminal metric can miss transient conflicts.

In a separate 335-scene, 5,088-edit in-path lead-vehicle evaluation, 92.5% of samples have both rollouts stopping, and terminal ΔG correlates with minimum-gap change (Spearman ρ = 0.976). Horizon sensitivity varies: ρ = 0.96 for sag (exit + 2 s vs. terminal) but ρ = 0.04 for hump (exit vs. terminal). We will distinguish motion response from safety-related traffic-conflict proxies.

3. Fixed-control playback and reactive driving. Our aim is to isolate road-induced physical effects. Replaying identical recovered controls on original and edited roads holds driving inputs fixed, so motion changes reflect geometry/friction interventions rather than altered decisions. This is a controlled counterfactual, not a claim that real drivers would behave identically.

To examine adaptation, we tested four longitudinal policies in 20 Waymo scenes (343 vehicles). DD is the median across vehicle pairs of the peak discrepancy in road-induced ego–lead gap change between ego-only and all-vehicle rollouts:

PolicyCrest D (m)Sag D (m)
Frozen1.521.41
Speed keeping0.240.24
IDM0.600.58
PDM-inspired0.891.07

Feedback reduces but does not eliminate multi-vehicle discrepancies (crest: 1.52 m frozen vs. 0.60 m IDM). It also changes simulated conflicts: two sag rear-end closures disappear under feedback, while one crest merging conflict emerges.

These results show that driver adaptation affects traffic interactions, while fixed-control replay remains useful for isolating road-induced physical effects.

**Additional Reactive-Driving and Safety Evaluation **

To better address the reviewer's concerns regarding the safety implications of terminal-gap changes and the realism of multi-vehicle interactions, we extended our evaluation to 335 Waymo scenes and 5,088 road edits. We further incorporated reactive driving and directly compared trajectory-level traffic-conflict outcomes under fixed-control and reactive-driving settings.

Building on our earlier IDM and PDM-inspired experiments, we further incorporated Gipps' car-following model and expanded the reactive-driving evaluation to 335 Waymo scenes and 5,088 road edits. In the larger-scale evaluation, the residual IDM controller exhibited oscillations at short headways when coupled with tire-actuation dynamics. We therefore adopted Gipps as a more stable longitudinal-feedback baseline, allowing vehicles to respond to road-induced changes and their leaders' altered motion.

Safety-related outcome (5,088 edits)Fixed ControlReactive DrivingChange [95% CI]
Ego–lead conflict (new TTC < 2 s, increased TTC exposure, or collision)27369−204 [−274, −141]
New ego–lead collision5626−30 [−70, +4]
Adverse edit (new collision or ≥0.1 m reduction in ego clearance to any vehicle)633494−139 [−252, −29]
New overlap among other vehicles225238+13 [−57, +88]

95% confidence intervals are computed using 2,000 scene-level bootstrap resamples.

Reactive driving substantially reduces ego–lead conflicts. The number of conflicting edits decreases by 75% (273 → 69), demonstrating that longitudinal feedback can effectively mitigate close-following conflicts.

However, multi-vehicle effects remain significant. Although adverse edits decrease from 633 to 494, the reactive-driving count remains 78% of the fixed-control count. Furthermore, new overlaps among other vehicles do not show a statistically established reduction, suggesting that longitudinal car-following alone cannot resolve all interactions involving surrounding traffic.

These findings complement our earlier results: driver adaptation changes the magnitude of counterfactual traffic outcomes but does not eliminate the importance of propagating road interventions across multiple vehicles.

Fixed-control replay provides a controlled basis for isolating road-induced physical effects, while reactive-driving evaluation further characterizes their consequences under behavioral feedback. We will incorporate these additional results and clarify the corresponding limitations in the revised manuscript.

Thank you for your response and for the revisions. I will take them into account in my final assessment.

Thank you for your follow-up and for taking the time to consider our responses and revisions. We appreciate your constructive feedback and careful evaluation of our work.

Comments on the paper

2 comments

Dear Reviewers,

Please take time to review the responses from the authors and engage with them for further questions and comments if any.

Time left is not much and please do shortly.

Thanks

AC

I. Acknowledgments

We sincerely thank the Area Chair and all three reviewers for their careful evaluations and constructive feedback. Their suggestions have strengthened TerrainForge's experimental validation and clarified the scope of our claims.

We especially appreciate Reviewer E38J's constructive engagement. Following our initial rebuttal, the reviewer acknowledged that "the all-vehicle screening evaluation addresses the original label mismatch" and "the matched CarSim ablation supports the benefit of richer dynamics." The reviewer subsequently raised two additional questions on inferred-control robustness and vision-based closed-loop evaluation, which we addressed with further experiments.

During the rebuttal, we responded to all three reviewers and conducted additional evaluations targeting their main concerns. We summarize the discussion below for the AC's convenience.

II. Key Strengths Recognized by Reviewers

  • Novel problem formulation. Reviewer 5k1W described the road-intervention setting as "novel and under-explored"; wymN recognized the value of editing the road itself rather than treating it as fixed background.
  • Unified physical and visual modeling. 5k1W and E38J highlighted the shared road representation linking scene deformation, vehicle dynamics, and camera motion.
  • Careful validation and transparency. wymN commended independent component-level validation and explicit reporting of failures; E38J recognized the substantial validation and transparent experimental design.

III. Key Concerns and Our Responses

Key concernReviewersAdditional evidence
Rendering realism and artifactswymN, 5k1WEvaluated 30 Waymo scenes: zero-edit PSNR 29.7 dB, SSIM 0.889, LPIPS 0.147; detection recall under edits remains close to recordings. Added curb/building artifact checks and improved multi-vehicle rendering across 34 scenes.
Fixed controls and driver adaptationAll threeTested four feedback policies, then expanded reactive evaluation to 335 scenes / 5,088 edits. Ego–lead conflicts fell 273 → 69 (−75%) and adverse edits 633 → 494 (−22%). Driver adaptation changes, but does not eliminate, multi-vehicle effects.
Meaning of terminal-gap changeswymN, E38JDistinguished longitudinal response from safety outcomes; added TTC, TTC exposure, clearance, and simulated collisions to characterize traffic conflicts directly.
Full multi-vehicle screening5k1W, E38JGenerated 28,736 all-vehicle labels and evaluated on 500 scene-disjoint test scenes. Consequential-edit Recall@20 improved 0.403 → 0.517, AP 0.654 → 0.860, and B@80 decreased 55.6% → 35.9%. Harmful-edit Recall@20 reached 0.775 vs. 0.563 for a physics prior.
Value of 32-state dynamics5k1W, E38JCompared four models across 34 CarSim cases (12 OOD holdouts). The 32-state model had the lowest reported errors; reduced models can materially alter road-edit responses and candidate rankings.
Inferred-control robustnessE38J (follow-up)Tested 16 parameter–control configurations, 40 scenes / 636 edits: 99.6% response-direction agreement, with 12/13 nominal collision edits remaining collisions across configurations. Reported low-friction sensitivity.
Vision-based closed-loop evaluationE38J (follow-up)Integrated SimWAM-Waymo-IL with updated TerrainForge renders. Across 10 validated scenes, 26/30 edited conditions produced consistent driving responses; 22/26 were not reproduced with unedited imagery. Additional braking tests exposed two visibility-sensitive collision cases.

IV. Commitment to Revision

We have substantially revised the manuscript to incorporate the new experiments and address the concerns raised by all three reviewers. The revisions strengthen rendering, dynamics, multi-vehicle, screening, robustness, and closed-loop evaluations.

We also clarified experimental settings, distinguished physical-response measures from safety-related outcomes, refined claims, and documented relevant limitations. All changes are highlighted in the revised manuscript for ease of reference.

We sincerely thank the Area Chair and reviewers for their valuable feedback, time, and consideration.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.