- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
TerrainForge studies road-geometry and friction counterfactuals in reconstructed driving episodes. A shared road field drives Gaussian-scene deformation and vehicle dynamics; inferred controls are held fixed across paired baseline and edited runs, and their motion difference updates the recorded vehicle poses and camera trajectory. The main multi-vehicle experiments replay plan-view paths while recomputing longitudinal and vertical response, rather than modeling reactive drivers. The evaluation combines CarSim comparisons, ZOD/LiDAR road-profile checks, geometry verification in Waymo reconstructions, and multi-vehicle gap analysis. A scene-disjoint bank of 15,758 ego-only road edits trains surrogates for terminal-gap change and consequential-edit detection. The reported surrogate results support prediction under this intermediate protocol; their transfer to the proposed full multi-vehicle screening pipeline remains to be established.
Critical strengths
-
A useful, coherent formulation of fixed-control road counterfactuals. The shared road representation connects geometric editing, vehicle response, and rendering instead of treating terrain as appearance alone. Under the primary edits, recomputing surrounding vehicles changes the estimated ego–lead gap response: the median peak discrepancy from ego-only propagation is 1.52 m for crests and 1.41 m for sags across 18 episodes (Section 4.3). These are peak gap discrepancies, not demonstrated safety improvements, but they substantiate the importance of propagating a shared road edit beyond the ego vehicle.
-
Substantial component validation and transparent reporting. The CarSim study distinguishes development from held-out cases, includes steering and vertical-alignment tests, and reports raw-time errors and numerical checks. Independent scene-level splits and bootstrap intervals strengthen the surrogate evaluation, while analytical and statistical baselines help identify what is learned. The paper also reports model-domain failures, weak minimum-clearance prediction, and differences among experimental protocols rather than presenting all results as a single end-to-end validation. This supports confidence in the demonstrated component-level findings.
Critical weaknesses
-
Screening effectiveness is not established for the final multi-vehicle outcomes. Section 4.5 and Appendix H train and evaluate the surrogate with recorded target motion, whereas Section 4.3 and Appendix E.2 show that allowing the lead to respond can substantially change, and in the illustrated episode reverse, the gap-change estimate. Recall@20% = 0.80 in Table 15 therefore measures retention of ego-only labels, not retention of consequential edits under full multi-vehicle simulation. Re-simulating only shortlisted candidates cannot recover important edits discarded by the screen. Moreover, the label is direction-agnostic, minimum-clearance prediction does not improve reliably over no change (Table 14), and only 7.0% of bank rows have both rollouts stop (Appendix H), so terminal-gap accuracy should not be equated with stopping-margin or collision-risk accuracy. A bounded end-to-end shortlist evaluation, including total computational cost, is needed to support the proposed two-stage use.
-
Accuracy of the coupled model is better supported than its incremental benefit for the reported counterfactual tasks. The CarSim holdouts validate the 32-state model, and Appendix G demonstrates lateral/load responses; my concern is not an absence of dynamics validation. However, the real-road test uses the inherited half-car, and kinematic road following has numerically lower test world-frame pitch RMSE than that half-car (0.236 versus 0.258 degrees; Table 8). As the authors note, this supports road-attitude recovery, not an advantage of richer dynamics. The main multi-vehicle bank fixes plan-view paths, and Appendix E.2 already explains much of the load response through a curvature-based formula. What remains insufficiently isolated is when richer coupling materially improves gap, wheel-load, or screening decisions relative to a matched reduced model. A small comparison on existing cases would clarify the added value and appropriate operating regime of this central component.
Questions for authors
-
On a representative held-out subset, how much of the full multi-vehicle consequential-edit set is retained by the surrogate shortlist? Please report recall at a fixed simulation budget against random selection and a simple physics/context baseline, distinguish harmful outcomes where sample support permits, and include feature-extraction and rollout costs. This targeted test would determine whether the current intermediate-label results support the proposed screening use; otherwise, would the authors restrict that claim to ego-response screening?
-
Beyond the existing CarSim accuracy tests and curvature-formula analysis, which outcomes benefit materially from the 32-state model over reduced longitudinal/half-car alternatives? If no direct comparison exists, could a bounded ablation on existing crest/sag and hump cases, with the same interventions, controls, and horizons, quantify changes in gap and wheel-load extrema relative to CarSim where available? This would clarify where the additional coupling is necessary without requiring a new benchmark.
4 comments
Thank you for the additional experiments. The all-vehicle screening evaluation addresses the original label mismatch, and the matched CarSim ablation supports the benefit of richer dynamics. I also appreciate the qualification that improved clearance screening does not establish better collision-risk prediction.
Two questions remain:
- Controls are inferred, not measured. Different vehicle parameters, source-road friction, and controls may fit the recorded motion similarly but predict different responses after editing. CarSim validates forward dynamics, not the recovered inputs. Could you test whether gap changes and collision outcomes remain stable across plausible configurations with comparable replay errors?
- An edit that is adverse under frozen controls may be easily handled by a policy that brakes earlier. Do the added feedback policies use rendered images or simulator states? A small closed-loop test with an existing vision-based policy would show whether the edited scenes expose policy failures and whether rendering contributes to the evaluation. Otherwise, please clarify the concrete use of fixed-control analysis and limit the safety claims accordingly.
Part 1-2- Inferred Controls
We thank the reviewer for the constructive follow-up questions. We conducted additional experiments to address two concerns: the robustness of counterfactual outcomes to uncertainty in recovered controls and the contribution of rendered images to closed-loop driving evaluation.
1. Robustness to Inferred Controls
We appreciate the reviewer's concern regarding uncertainty in recovered controls. To examine whether different parameter–control configurations that reproduce the same recorded motion lead to consistent counterfactual predictions, we conducted an additional robustness evaluation.
We evaluated 40 Waymo scenes and 636 road edits, including 13 near-contact scenes and 27 randomly selected scenes. We sampled 16 configurations from a nine-dimensional parameter space covering vehicle mass, load distribution, CG height, tire characteristics, rolling radius, suspension, wheel inertia, and source-road friction (0.65–1.00).
For each configuration, we separately recovered the controls needed to reproduce the recorded trajectory, then applied identical road interventions to compare their counterfactual responses.
All configurations reproduced the recorded motion with the same reported median speed RMSE of 0.011 m/s. Despite the ambiguity in recovered parameters and controls, the counterfactual outcomes remained highly consistent:
| Metric | Result |
|---|---|
| Direction agreement for consequential minimum-gap changes | 99.6% |
| Nominally harmful edits remaining harmful across all configurations | 190/190 |
| Rank correlation with nominal results (median / minimum) | 0.998 / 0.990 |
| Range of gap changes across configurations (median / p90) | 0.04 / 0.17 m |
| Nominal collision-producing edits remaining collisions across all configurations | 12/13 |
As a control, applying the nominal controls directly to alternative vehicle configurations increased factual replay error to 0.34 m/s and produced collisions on the unedited road in 27% of cases. This confirms the importance of recovering controls separately for each configuration.
Sensitivity near friction limits. We further investigated edits exhibiting unusually large variations and corrected an evaluation-window truncation issue caused by brief wheel unloading. After recomputing the affected cases, 5 of 636 edits (0.8%) showed genuine configuration sensitivity. All involved braking on a low-friction sag (edited μ = 0.4), where some configurations produced lateral departures or substantially different stopping positions. One edit changed its collision classification, so our predefined collision-stability criterion was not fully satisfied.
We retain these cases in the reported statistics and will document the post-hoc evaluation-window correction and sensitivity near friction limits.
Overall, although the recovered vehicle parameters and controls are not unique, the direction and ranking of TerrainForge's counterfactual responses remain highly consistent across the sampled configurations. These results indicate that the main counterfactual conclusions do not depend strongly on a particular recovered control configuration, while identifying conditions where collision outcomes require greater caution.
Part 2/2 — Vision-Based Closed-Loop Evaluation
2. Vision-Based Closed-Loop Evaluation
We appreciate the reviewer's suggestion to further evaluate TerrainForge with a vision-based driving policy. Building on our previous experiments with state-based feedback policies, we extended the evaluation to vision-based closed-loop driving, where policies receive TerrainForge-rendered observations and adapt their controls accordingly. We conducted two additional experiments to examine how road edits affect driving decisions and whether the rendered scenes can expose policy failures.
(a) Does rendering affect an end-to-end vision-based driving policy?
We integrated the publicly released SimWAM-Waymo-IL policy without retraining. Every 0.25 s, the policy receives a newly rendered front-camera image, ego motion state, and route command, predicts a 5-second driving trajectory, and controls the vehicle through TerrainForge's 32-state dynamics model. The entire closed-loop evaluation uses updated rendered observations rather than prerecorded video.
We evaluated 13 Waymo scenes, of which 10 passed predefined zero-edit validation checks. We compared three conditions: (A) unedited road, (B) edited road with corresponding rendered images, and (C) edited road with unedited imagery as a visual ablation. Each condition was evaluated with three seeds.
In 26 of 30 edited-road conditions, the policy exhibited consistent driving responses exceeding seed-level variation, including 17 of 20 design-plausible edits. Under these plausible interventions, vehicles were a median 3.1 km/h slower at the curve apex and traveled 11.5 m less over 20 s.
Visual ablation further confirms the contribution of rendering. When edited-road physics was retained but edited imagery was replaced with unedited imagery, 22 of the 26 robust responses no longer exhibited a corresponding robust change.
These results demonstrate that edited visual observations contribute to the policy's decisions and resulting closed-loop vehicle motion, beyond the effects of road physics alone.
(b) Can edited scenes expose vision-related braking failures?
To further examine the role of visual–physical consistency, we evaluated a YOLO-based braking controller in a scenario involving a stationary pickup ahead. Unlike the end-to-end SimWAM experiment, this detector-based setup allows us to directly trace how changes in hazard visibility affect detection, braking, and collision outcomes.
Across 36 combinations of road geometry, friction, and initial speed, frozen controls resulted in collisions in 14/36 conditions, all of which were avoided by a state-feedback stopping controller.
The paired evaluation identified two visibility-sensitive braking cases under blind-summit, low-friction conditions. Under identical edited-road physics, the controller successfully stopped when supplied with unedited imagery, whereas the edited renderings delayed hazard detection and led to collisions.
At 50 km/h, the edited rendering reduced the hazard detection distance from 79 m to 33.5 m, resulting in a collision at 23 km/h. At 60 km/h, the hazard became visible only after the last feasible stopping point.
These results illustrate why visual and physical responses must be evaluated together. Under the same edited-road physics, using unedited imagery would miss the changes in visibility and braking outcomes introduced by the road intervention. The stress-test cases demonstrate how TerrainForge can expose vision-related driving failures that a physics-only evaluation would overlook.
Together, these experiments show that TerrainForge's rendered observations can influence closed-loop driving decisions and expose vision-related braking failures under challenging road conditions. They extend our earlier physical-response analysis to evaluating how road interventions affect visual perception and closed-loop driving behavior.
We appreciate the reviewer's suggestion, which helped strengthen the downstream validation of TerrainForge. We will incorporate these experiments into the revised manuscript to provide a more complete evaluation of its downstream applications.
We thank the reviewer for recognizing our unified road representation and component validation. We address both concerns with new experiments.
1. Full multi-vehicle screening. We appreciate the distinction between ego-response and all-vehicle screening. We generated 28,736 all-vehicle labels across 1,922 scenes: 21,364 edits/1,422 scenes for development and 7,372 edits/500 scene-disjoint B2 scenes for evaluation. R@20 is recall in the top 20%; B@80 is the budget for 80% recall.
For consequential edits, 3c reduces B@80 from 55.6% (original) to 35.9% (random: 80.5%). With 34.9% positives, the R@20 ceiling is .573. With identical training scenes and inputs, changing only ego-only to all-vehicle labels improves R@20 .255→.482 and AP .435→.781. Adding the two-car prior raises R@20 to .503; pre-intervention interaction features then improve AP .813→.860 and B@80 43.8%→35.9%. Features use logs and edit-free replay only. Pre-registered paired 95% CIs: [+.031,+.064] for consequential AP (3c vs all-vehicle) and [+.138,+.291] for harmful R@20 (H4 vs two-car prior).
Harmful edits. We define harmful edits as a new footprint collision or a ≥0.1 m decrease in minimum signed ego-to-other-vehicle footprint clearance (edited minus baseline; common horizon; all movers re-simulated with frozen controls). Terminal gap is excluded. Among 719 positives, 196 are new collisions; the remaining 523 have a median clearance decrease of 0.45 m. Raising the threshold to 0.5 m preserves H4's advantage over the two-car prior (R@20 .845 vs .658; 95% CI of difference [+.107,+.279]). For new collisions or clearance deterioration ending within 1 m, the R@20 difference is not statistically significant (.692 vs .638; CI [-.020,+.153]), while the prior has higher AP (.413 vs .390). Thus, we claim improved clearance-deterioration screening, not superior near-collision or collision-risk prediction.
Computational cost. Costs were measured on 335 B-group scenes/5,088 edits (x86); B2 used different hardware. Screening costs 10.3 CPU-s/edit: 9.85 s for shared edit-free ego baseline replay, 0.45 s for interaction features, and ~0.02 s for other features/prior/inference. A full all-vehicle rollout costs 662 CPU-s, baseline included. We charge scene-fixed work once per selected scene without double-counting. At matched 80% recall, 3c uses 41% of exhaustive compute (2.4×) for consequential edits, versus 66% (1.5×) for the original model and 83% (1.2×) for random. H4 uses 29% (3.5×) for harmful edits, versus 69% (1.5×) for the two-car prior. These are modeled CPU-time speedups, not wall-clock measurements; claims concern lead-following scenes.
2. Incremental value of 32-state dynamics. We compared kinematic (Kin), point-mass (PM), half-car (HC), and full 32-state models with matched edits, controls, initial states, and horizons in 34 CarSim cases (12 OOD holdouts). Errors are against CarSim.
Across all reported regimes and metrics, the 32-state model achieves the lowest errors. Although reduced models already approximate smooth-curve longitudinal motion well, the benefits of 32-state dynamics are more pronounced under braking, bumps, and low friction, where suspension, tire forces, and load transfer are strongly coupled. Across 3,924 speed-bump edits, the median PM/32-state terminal-gap response ratio is 5% (IQR 2–14%). Top-ranked gap-increasing edits agree in 41.9% of 1,572 candidate groups (788 scenes) (95% CI 38.6–45.4%; random: 10.8%). Fidelity changes edit priorities, though disagreement alone does not prove superiority. As the reviewer notes, Table 8 evaluates real-road attitude recovery with the inherited half-car; the new matched CarSim comparison instead identifies where higher-order coupling provides a measurable benefit.