acceptodds
Withdrawn from ICLR 2027

CroTTA: Cross-Task and Cross-Frame Test-Time Adaptation for End-to-End Driving

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

End-to-end autonomous driving has achieved significant milestones, yet its reliance on the i.i.d. assumption renders it vulnerable to real-world distribution shifts. While Test-Time Adaptation (TTA) offers a promising paradigm for on-the-fly enhancement, its application to driving is hindered by two barriers: multi-task adaptation dependency, where adapting tasks in isolation propagates uncoordinated updates, triggering task-level interference; and (ii) continuous adaptation instability, where recursive online updates over long horizons cause compounding parameter drift and model collapse. To address these challenges, we propose CroTTA, a plug-and-play TTA framework for the robust online adaptation of end-to-end driving models. CroTTA introduces a joint adaptation objective that integrates perception multi-modal uncertainty with planning temporal consistency; by formulating cross-task constraints, it ensures that adaptation signals from the upstream perception remain consistent with downstream planning, thereby enhancing the adaptation performance under diverse distribution shifts. Furthermore, to ensure continuous adaptation stability, we develop a teacher-student architecture where the teacher model guides student updates across two time scales: it maintains a global anchor via EMA to suppress long-term cumulative drift, while acting as a short-term gatekeeper to discard transient, destabilizing updates through a per-step filter. Extensive experiments across multiple models demonstrate that, by updating only 0.15‰ of parameters with a 20ms latency, CroTTA achieves average recovery rates of 24.7%, 26.9%, and 11.8% for L2, collision rate, and PDMS, respectively, under severe distribution shifts. In comparison with existing TTA methods, CroTTA obtains an additional 20.5 and 18.2 percentage points of L2 and collision recovery. Our code is available at: https://anonymous.4open.science/r/CroTTA-88D0/.

Primary area:
applications to robotics, autonomy, planning
Keywords:
End-to-End Autonomous Driving, Test-Time Adaptation, Distribution Shifts, Cross-Task Coordination

Reviews

Among:
Shape:
1234this paper, Bottom 10%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.67 over 3 reviews, above 10% of the others.
Reviewer_rsoNOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper studies test-time adaptation of end-to-end autonomous driving models under distribution shifts, focusing on task dependencies between perception and planning and potential parameter drift during continual updates. The authors propose CroTTA, which combines binary entropy minimization of perception outputs with a delayed planning consistency loss based on observed ego-vehicle displacement. The adaptation process is constrained by an L∞L_\infty projection centered on exponential moving average (EMA) parameters and a stepwise update acceptance mechanism. Only a subset of LayerNorm affine parameters is updated. The experiments cover six models on nuScenes and five planners on NAVSIM, evaluated under various visual corruptions, time-varying perturbations, localization noise, cross-city transfer, and scenarios involving strong interactions. The results show improvements in planning error and collision-related metrics in most reported settings. However, the continual adaptation protocol needs clearer documentation, and the attribution of the gains and the verifiability of some reported recovery rates require further support.

Critical strengths

1.The method combines practically meaningful adaptation signals for end-to-end driving and achieves substantial improvements. Linking perception uncertainty to observed ego-vehicle motion is more task-specific than simply increasing prediction confidence. For example, under severe Gaussian noise, UniAD’s average L2 error decreases from 1.38 m to 1.25 m, and its average collision rate decreases from 0.95% to 0.72%, corresponding to relative reductions of approximately 9.4% and 24.2%, respectively. Table 5 also shows that combining the two losses yields better collision metrics than using either loss alone. These results directly support the method’s potential value. (Pages 7 and 9; Tables 2 and 5.)

2.The experiments provide broad coverage and useful evidence beyond successful cases. In addition to evaluating multiple models on nuScenes and NAVSIM, the appendices include experiments on clean data, fog, brightness, snow, time-varying contrast, joint perturbations involving GPS/yaw noise and motion blur, and geographic transfer from Boston to Singapore. The authors also present cases where adaptation increases error and acknowledge that severe localization errors and inappropriate learning rates can cause failures. These experiments help establish the method’s applicability and limitations, increasing the practical value of the findings. (Appendices A–C and G–H.)

Critical weaknesses

1.The execution protocol and stability evidence for the core online adaptation mechanism are insufficient, limiting assessment of the main contribution. Equations (8)–(13) rely on the discrepancy between the previous frame’s plan and the currently observed motion, but do not specify the exact sequence of differentiable recomputation, historical state handling, candidate update validation, and final scoring. This particularly affects understanding of how cross-frame adaptation is implemented in NAVSIM. Meanwhile, projection around a moving EMA center directly constrains only the parameters’ deviation from the current center; it does not, by itself, establish bounded long-term parameter drift or prevent model collapse. The existing ablations introduce EMA, projection, and update filtering jointly as P, while the explicitly described experiment with time-varying perturbations spans only 20 frames. Thus, the mechanisms claimed to support cross-task coordination and long-term stability have not been sufficiently isolated and validated. (Pages 5–7, Equations (8)–(13); Appendices A.5 and E.) 2.The comparisons underlying the key gains and the quantitative reporting contain substantive issues that hinder verification. Appendix F.1 explicitly restricts the main baselines to updating LayerNorm parameters in the perception classification branch, whereas the exact parameters updated by CroTTA for each architecture are not specified. Tent in Table 4 and the Lmm-only variant in Table 5 produce different results; their implementation differences need to be clarified to attribute the gains to the new mechanisms accurately. Furthermore, the VAD recovery rates in Table 2 are incompatible with the clean reference values in Table 7. Table 15 calculates recovery rates for the subset involving strong interactions using reference values from the full validation set, so these rates cannot be directly interpreted as recovery from corruption-induced degradation on that subset. The results for the full method on SparseDrive also differ between Tables 2 and 20. These issues reduce confidence in the magnitude of the contribution and its reproducibility. (Pages 7, 9, 15, 19, and 21–22.)

Questions for authors

Q1. Baseline comparisons and attribution of contributions — clarification and validation are needed. Please specify the parameter modules updated by each method, the number of update steps, and the optimization settings. Why do Tent in Table 4 (1.37/0.94) and the Lmm-only variant in Table 5 (1.33/0.92) produce different results? When the updated parameters and delayed motion feedback are matched, does the proposed stabilization mechanism still offer advantages over existing continual adaptation mechanisms? Q2. Long-term stability — targeted validation is needed. Could you separately ablate EMA, the L∞ projection, and the acceptance/rollback mechanism, and report performance, parameter drift, and update acceptance rates over longer continuous streams or repeated domain switches, explicitly without resetting parameters? Do these results support the claim of preventing model collapse, or would it be more appropriate to limit the conclusion to empirical improvements under the tested conditions? Q3. Recovery rates and consistency of results — corrections or explanations are needed. Please verify the VAD model versions and clean reference values in Tables 2 and 7, the results for the full method on SparseDrive in Tables 2 and 20, and the denominators used to calculate recovery rates in Table 15. Please also provide the clean reference scores for NAVSIM and uncertainty estimates for the task metrics, so that readers can assess the small absolute gains and verify the corresponding recovery rates. Q4. Efficiency and deployment claims — clarification is needed. What hardware and numerical precision were used for the timing measurements in Table 6? Do these measurements include recomputation of the previous frame, backpropagation, candidate update validation, and final prediction? What is the total end-to-end latency with adaptation enabled? In real closed-loop operation, where displacement feedback is generated by the policy’s own actions, how do you characterize the validity of Ltemp and its relationship to planning safety?

Reviewer_HYRsOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
3
Read review
Paper summary

CroTTA addresses online test-time adaptation of end-to-end driving models under corrupted observations. It updates LayerNorm affine parameters using a weighted sum of perception confidence entropy and a temporal planning loss that compares the previously predicted ego-position with subsequently observed ego-motion. To stabilize these online updates across frames, CroTTA also uses a teacher–student architecture that constrains parameter drift and rejects updates that worsen temporal planning consistency.

Critical strengths
  1. Useful adaptation signals for a consequential application. Combining perception entropy with subsequently observed ego-motion is a plausible way to constrain confidence-driven adaptation. On UniAD, the joint objective improves collision performance more than either loss alone, providing some empirical support for their complementary value.

  2. Reasonable breadth of empirical evaluation. Results across multiple architectures, two benchmarks, additional corruption types, and clean inputs make the evidence stronger than a single-model Gaussian-noise demonstration. The small trainable parameter fraction also makes the approach practically interesting if the full execution cost and causal implementation are confirmed.

Critical weaknesses
  • The plug-and-play claim needs qualification because CroTTA relies on updating LayerNorm affine parameters. The paper does not clearly specify which layers are updated in each architecture or whether their locations affect performance. Equation 2 also leaves the perception-to-planning interface unclear: the downstream planner still receives the original input, while the description emphasizes its dependence on intermediate perception features.

  • The reproducibility and practical value of the online adaptation remain unclear. The Gaussian corruption is described mainly by its severity level, with insufficient detail about how the noise is generated or whether the results are consistent across random seeds. Although fine-tuning during inference is a valid TTA approach, the reported learning-rate sensitivity raises questions about how reliably it can operate under changing deployment conditions without labeled data for tuning.

Questions for authors
  1. Which LayerNorm layers are updated in each architecture, and how much does their location affect performance? Does the method require particular LayerNorm placements to work?

  2. In Equation 2, what information does the planner receive through the original input in addition to the perception features? How does this correspond to the pipeline described in the text?

  3. How is the Gaussian noise generated and applied across cameras and consecutive frames? Are the reported improvements consistent across different random seeds?

  4. How are the adaptation hyperparameters selected before deployment without labeled target data? Given the learning-rate sensitivity, how reliably does the method adapt when the corruption type or severity changes?

Reviewer_5DD6OpenReview ↗
Rating
1
Confidence
2
Informativeness
2
Presentation
3
Soundness
1
Read review
Paper summary

This paper proposes CroTTA, a test-time adaptation method for end-to-end driving models that updates only LayerNorm parameters online. It combines perception-head entropy with a planning loss comparing the previous frame's planned waypoint to the observed ego position. An EMA teacher constrains each update, and updates are kept only if they lower the planning loss. Under synthetic Gaussian noise on nuScenes and NAVSIM, the authors report moderate recovery of the corruption-induced degradation across eleven planners, and better results than existing TTA methods on UniAD.

Critical strengths

The method is evaluated on eleven end-to-end planners of very different designs, with improvements in a consistent direction across all of them. This supports the plug-and-play claim and is the paper's main strength.

Critical weaknesses
  1. Main weakness: The evaluation does not distinguish adaptation from trajectory supervision. The experiments appear to use offline, non-reactive evaluation on nuScenes and NAVSIM, although the protocol is not explicitly described. In this setting, the temporal loss (Eq. 9) compares the previous planning prediction with the observed ego-state. If this state comes from the logged human trajectory, CroTTA effectively receives a one-step-delayed supervision signal derived from the same trajectory used for evaluation. The per-step filter (Eq. 13) also relies on this signal, while the TTA baselines do not use it. This also raises concerns about the evaluation metrics. The open-loop L2 metric measures agreement with the logged human trajectory, which may favor a method that adapts using previous positions from that same trajectory. The reported improvements may therefore partly reflect additional trajectory information rather than better adaptation to distribution shifts. Similarly, collision metrics evaluated against recorded traffic do not necessarily reflect safety in closed-loop driving, where other vehicles may react to the ego-vehicle's actions. The ablation results make this concern particularly relevant:
  • The largest additional L2 improvement comes from the teacher–student architecture. In the UniAD ablation (Table 5), combining perception entropy and temporal consistency achieves 13.6% L2 recovery, the same as temporal consistency alone. Adding the teacher–student architecture, which includes the per-step filter based on the temporal loss, increases recovery to 29.5%. However, the ablation does not isolate the contribution of the filter from the other stabilization components.
  • Without the temporal loss, L2 improvements are comparable to those of existing TTA methods. Using only perception entropy achieves 11.4% L2 recovery on UniAD (Table 5), compared with 9.1% for the strongest conventional TTA baseline on this metric (EATA–E, Table 4). The substantially larger improvement of the full method suggests that the additional temporal objective and stabilization mechanism, both of which rely on the observed ego-state, play an important role, although their individual contributions are not fully isolated. These results do not establish that the improvements come from delayed supervision, but the current evaluation cannot rule out this explanation. In closed-loop driving, observed ego-motion would reflect the vehicle's executed actions rather than the logged human trajectory, so the temporal loss may no longer provide the same information about planning correctness. A baseline using the same delayed ego-pose observations and a closed-loop evaluation are needed to determine whether the reported gains support the paper's claims about robust online adaptation.
  1. Minor weakness: Limited experimental validation. Comparisons with existing TTA methods are restricted to UniAD under a single corruption setting (Table 4), with baselines adapting only perception-related components while keeping the planner frozen. Several reported improvements are also relatively small (e.g., 0.03–0.04 m in L2 error for stronger planners and 0.3–0.8 points in NAVSIM PDMS), without reporting variability across runs or confidence intervals. Furthermore, the evaluation focuses primarily on synthetic distribution shifts, while the only natural cross-city shift (Boston to Singapore) is tested using a single model. These limitations make it difficult to assess the statistical reliability of the smaller improvements and the method's generalizability to more realistic driving conditions.
Questions for authors
  1. Temporal supervision and closed-loop evaluation. Is the observed ego-position in Eq. 9 obtained from the logged nuScenes/NAVSIM poses? If so, how can the reported improvements be distinguished from gains due to delayed trajectory supervision, for example through a baseline that fine-tunes online on the logged poses? How would this loss behave in closed-loop driving, where observed motion depends on the model’s executed actions?
  2. Cross-task interference. The paper identifies task-level interference as a central motivation, but combining the losses does not improve L2 error over the better individual loss on either UniAD (Table 5) or SparseDrive (Table 20). How does the joint objective specifically mitigate interference? An analysis of gradient alignment or perception performance before and after adaptation would help substantiate this claim.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.