- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper presents SimTTA for test-time adaptation of end-to-end driving planners under sensor corruptions. SimTTA comprises three components. Geometric Trajectory Refinement (GTR) first generates candidate trajectories by scaling and rotating the predicted trajectory. Simulation-based Self-Verification (SSV) then scores these candidates using a scene reconstructed from the model’s perception outputs. Finally, Temporally-Verified Entropy (TVE) adapts the model’s batch normalization parameters by minimizing entropy on temporally consistent BEV predictions.
Critical strengths
-
The breadth and consistency of the empirical improvements constitute the strongest aspect of the paper. On DiffusionDrive, SimTTA improves mean PDMS from 78.9 to 85.4 on OpenScene and from 86.7 to 90.7 on nuScenes-C. The additional results on TransFuser and UniAD further strengthen the evidence that the approach generalizes beyond a single planner. Together, these experiments make a convincing case for its effectiveness under the corruptions tested.
-
The paper addresses a worthwhile problem. Model adaptation does not necessarily yield a better trajectory for the current input. Explicitly refining and evaluating that trajectory is therefore a sensible direction. Moreover, evaluating candidates using the model’s own perception outputs, rather than relying on ground-truth future agent trajectories, enhances the approach’s practical relevance.
Critical weaknesses
-
The methodological contribution appears incremental. Geometric perturbation, simulation-based candidate scoring, and temporally filtered entropy minimization form a reasonable combination, but I remain unconvinced that the paper establishes a sufficiently distinct contribution for ICLR. Section 4.5 provides a partial rationale for the geometric search, yet this analysis remains tied to DiffusionDrive and the PDMS setting. As shown in Table 4, GTR+SSV accounts for most of the improvement, raising mean PDMS from 78.9 to 84.1, whereas adding TVE further raises it to 85.4. Although TVE provides a measurable benefit, its planning gains are uneven across corruptions. Moreover, the experiments do not establish how much temporal verification adds beyond simpler adaptation objectives under the same candidate-selection pipeline. TVE also requires BEV class predictions and adaptable BN parameters. This does not invalidate the method, but it narrows the scope of the claim that the complete framework is model-agnostic.
-
The evaluation does not establish the claimed benefits in a full closed-loop setting. Section 4.1 refers to a “closed-loop protocol,” but the cited NAVSIM evaluation queries the planner at the initial observation and evaluates a fixed trajectory through non-reactive simulation. It does not test repeated planning from observations produced by the planner’s own actions. This concern is especially pertinent because SSV relies on potentially corrupted perception and constant-velocity predictions of other agents. The PDMS improvements and ranking agreement support the usefulness of this approximation, but they leave open whether the gains persist under execution-induced changes in observations, accumulated errors, and interactions. Since verification before execution is central to the paper’s motivation, the absence of such evidence materially weakens its safety claims.
Questions for authors
- Contribution of TVE: Holding GTR+SSV, BN statistics, and the adaptation budget fixed, how much does TVE improve upon standard entropy minimization or confidence-only filtering? Such an ablation would help determine whether temporal verification is essential to the reported gains.
- Evaluation protocol: Does the reported evaluation involve repeated replanning from sensor observations that change in response to the executed trajectory? If so, please clarify the setup and provide the corresponding results. If not, qualify the protocol description and the claims regarding safe execution accordingly.
- Architectural requirements: What outputs and model components does the complete framework require? For planners that lack dense BEV class predictions or adaptable BN layers, which components can be applied directly, and which would require additional modules or training?
8 comments
Response to Reviewer 48sm (Part 2)
W1-2: As shown in Table 4, GTR+SSV accounts for most of the improvement, raising mean PDMS from 78.9 to 84.1, whereas adding TVE further raises it to 85.4. Although TVE provides a measurable benefit, its planning gains are uneven across corruptions. Moreover, the experiments do not establish how much temporal verification adds beyond simpler adaptation objectives under the same candidate-selection pipeline.
Q1: Contribution of TVE: Holding GTR+SSV, BN statistics, and the adaptation budget fixed, how much does TVE improve upon standard entropy minimization or confidence-only filtering? Such an ablation would help determine whether temporal verification is essential to the reported gains.
(Answer W1-2 & Q1) We would like to respectfully emphasize that TVE provides both a standalone adaptation benefit and an additional improvement when combined with GTR+SSV. To show the effectiveness of the TVE loss itself, we gather the results of Table 2 and Table 4 of the main paper in Table R3.3.
Table R3.3
| Methods | Snow | Rain | Fog | FOV Lost | Gaussian Noise | Moving Object | Motion Blur | Temporal Misalignment | Mean |
|---|---|---|---|---|---|---|---|---|---|
| Source (No Adapt.) | 70.7 | 84.7 | 81.2 | 71.8 | 73.5 | 87.9 | 87.2 | 74.5 | 78.9 |
| BN Adapt. | 82.7 | 84.4 | 83.5 | 68.9 | 78.0 | 84.9 | 83.1 | 72.8 | 79.8 |
| SSA (ICLR'25) | 82.1 | 84.7 | 84.2 | 69.9 | 78.2 | 85.2 | 83.6 | 72.7 | 80.1 |
| TVE alone | 82.4 | 86.3 | 85.3 | 70.2 | 77.4 | 86.6 | 85.2 | 73.8 | 80.9 |
| GTR+SSV | 78.1 | 87.0 | 83.8 | 83.9 | 80.8 | 88.6 | 88.2 | 82.0 | 84.1 |
| GTR+SSV+TVE (SimTTA) | 85.1 | 88.2 | 87.4 | 82.7 | 81.3 | 88.4 | 87.8 | 82.1 | 85.4 |
As shown in Table R3.3, TVE alone improves mean PDMS from 78.9 (1st row) to 80.9 (4th row), outperforming BN Adapt. (2nd row) and SSA (3rd row), the strongest adaptation baseline in Table 2 of the main paper. Adding TVE to GTR+SSV further improves mean PDMS from 84.1 (5th row) to 85.4 (6th row). We acknowledge that this additional benefit varies across corruptions and does not improve every corruption type.
The larger improvement from GTR+SSV highlights the importance of evaluating candidate trajectories before execution. Adaptation alone cannot verify whether a trajectory is safe to execute. GTR+SSV addresses this through simulation-based candidate selection, while TVE complements it by adapting the representation used for planning and scene reconstruction.
To address the comparison with simpler adaptation objectives, we additionally conduct an experiment that holds GTR+SSV, BN statistics, and the adaptation budget fixed.
Table R3.4
| Methods | Snow | Rain | Fog | FOV Lost | Gaussian Noise | Moving Object | Motion Blur | Temporal Misalignment | Mean |
|---|---|---|---|---|---|---|---|---|---|
| Tent (Shannon Entropy) | 79.6 | 84.7 | 83.2 | 75.7 | 74.1 | 85.1 | 84.0 | 75.4 | 80.2 |
| Confidence-only Filtering | 79.3 | 84.7 | 83.2 | 72.8 | 73.3 | 84.9 | 84.1 | 75.4 | 79.7 |
| TVE w/o Temporal Verification | 81.7 | 86.0 | 84.8 | 77.8 | 76.5 | 86.6 | 85.3 | 77.5 | 82.0 |
| TVE (SimTTA) | 85.1 | 88.2 | 87.4 | 82.7 | 81.3 | 88.4 | 87.8 | 82.1 | 85.4 |
As shown in Table R3.4, TVE achieves 85.4 mean PDMS (4th row), outperforming standard entropy minimization at 80.2 (1st row) by 5.2 points and confidence-only filtering at 79.7 (2nd row) by 5.7 points. TVE outperforms all three alternatives across all eight corruption types. Furthermore, removing only temporal verification (3rd row) while retaining the other TVE components reduces mean PDMS from 85.4 to 82.0. This confirms that temporal verification itself contributes substantially to the performance gains.
A central difficulty in adaptation under corruption is that the predictions used to update the model can themselves be wrong. Standard entropy minimization can reinforce an incorrect prediction, while confidence-only filtering can retain it if its confidence is sufficiently high. TVE adds a temporal consistency condition. After ego-motion alignment, a BEV prediction must agree with the corresponding prediction from the previous observation, in addition to satisfying the confidence threshold. A prediction that is confident in the current frame but inconsistent across observations is therefore excluded from entropy minimization.
These results demonstrate that the choice of adaptation objective substantially affects planning performance even when candidate generation and verification are held fixed. They support the effectiveness of the complete TVE objective over simpler alternatives, while the additional ablation confirms the independent contribution of temporal verification.
We added this comparison in Table 20 of Appendix A.11 and updated Section 4.3 to refer to it.
Response to Reviewer 48sm (Part 3)
W1-3: TVE also requires BEV class predictions and adaptable BN parameters. This does not invalidate the method, but it narrows the scope of the claim that the complete framework is model-agnostic.
(Answer W1-3) We would like to respectfully clarify that the complete framework applies to planners satisfying specific architectural requirements. GTR requires trajectory outputs, SCR/SSV requires agent detections, an HD map, and ego poses, and TVE requires dense BEV class predictions and adaptable BN affine parameters. A planner without a dense BEV prediction head can still use GTR+SSV if the required scene information is available, while applying TVE would require a suitable BEV head. Similarly, applying TVE to a planner without BN layers would require modifying the adaptation mechanism.
We updated Section 4.1 and Appendix A.3 to make these conditions explicit and distinguish the requirements of each component.
W2: Absence of closed-loop evaluation
W2-1: The evaluation does not establish the claimed benefits in a full closed-loop setting. Section 4.1 refers to a "closed-loop protocol," but the cited NAVSIM evaluation queries the planner at the initial observation and evaluates a fixed trajectory through non-reactive simulation.
(Answer W2-1) We would like to respectfully clarify that our use of "closed-loop protocol" followed terminology used in prior NAVSIM-based work, including DiffusionDrive, which describes its evaluation using "closed-loop evaluations" and "closed-loop metrics." We recognize that NAVSIM evaluates trajectories through non-reactive simulation reconstructed from recorded driving logs, without repeated planning from observations generated by the executed actions. Our wording referred to this simulation-based evaluation and was not intended to imply a fully interactive closed-loop rollout.
We updated Section 4.1 to avoid ambiguity by replacing "closed-loop protocol" with "NAVSIM simulation protocol" to clarify the scope of the evaluation.
Response to Reviewer 48sm (Part 1)
W1 & Q1: Methodological contribution, effectiveness of TVE, and scope of model agnosticism
(Answer W1-1) We would like to respectfully emphasize that SimTTA advances TTA by addressing both the reliability of adaptation signals and the quality of trajectories selected for execution. Prior methods such as Tent, MOS, and SSA focus on improving adaptation, but under corrupted perception, model confidence or consistency can be misleading, and adaptation alone cannot verify whether a trajectory is suitable for execution. SimTTA addresses these issues through temporally verified adaptation and trajectory refinement and verification using a scene reconstructed from planner outputs.
TVE uses agreement across ego-aligned observations to exclude confident but temporally inconsistent predictions from adaptation. Meanwhile, GTR+SSV generates alternative trajectories and evaluates their outcomes in the reconstructed scene without ground-truth agent annotations. Thus, the two components address distinct failure sources during TTA, as further examined below.
Additionally, as per the reviewer's comment that the geometric analysis is limited to DiffusionDrive, we extend the analysis to three planners across OpenScene and nuScenes-C.
Table R3.1
As shown in Table R3.1, scaling and rotation jointly explain 96.5-96.6% of the coordinate variance of PDMS-improving perturbations across DiffusionDrive, TransFuser, and UniAD on OpenScene (1st-3rd rows), and 99.3-99.6% on nuScenes-C (4th-6th rows). The same dominant directions emerge across all six configurations, supporting the search design across models and datasets. These percentages describe coordinate variance rather than PDMS variance, and the analysis remains within the PDMS-based evaluation setting.
Table R3.2
Furthermore, the equal-budget comparison in Table R3.2 provides direct empirical support for the search design. Under the same 31-candidate budget, our GTR setting achieves 85.4 PDMS (5th row) compared with 83.6 when combining all perturbation types (3rd row). This supports concentrating a limited candidate budget on scaling and rotation, without claiming that other trajectory changes cannot improve PDMS. Together, these results support GTR as an empirically justified search strategy rather than an arbitrary heuristic, demonstrating consistent geometric patterns across planners and datasets and superior performance compared with alternative search strategies.
We added the PCA analysis across planners and datasets in Table 18 of Appendix A.9 and the equal-budget comparison in Table 7 of Section 4.5, and updated Section 4.5 to refer to them.