CroTTA: Cross-Task and Cross-Frame Test-Time Adaptation for End-to-End Driving

End-to-end autonomous driving has achieved significant milestones, yet its reliance on the i.i.d. assumption renders it vulnerable to real-world distribution shifts. While Test-Time Adaptation (TTA) offers a promising paradigm for on-the-fly enhancement, its application to driving is hindered by two barriers: multi-task adaptation dependency, where adapting tasks in isolation propagates uncoordinated updates, triggering task-level interference; and (ii) continuous adaptation instability, where recursive online updates over long horizons cause compounding parameter drift and model collapse. To address these challenges, we propose CroTTA, a plug-and-play TTA framework for the robust online adaptation of end-to-end driving models. CroTTA introduces a joint adaptation objective that integrates perception multi-modal uncertainty with planning temporal consistency; by formulating cross-task constraints, it ensures that adaptation signals from the upstream perception remain consistent with downstream planning, thereby enhancing the adaptation performance under diverse distribution shifts. Furthermore, to ensure continuous adaptation stability, we develop a teacher-student architecture where the teacher model guides student updates across two time scales: it maintains a global anchor via EMA to suppress long-term cumulative drift, while acting as a short-term gatekeeper to discard transient, destabilizing updates through a per-step filter. Extensive experiments across multiple models demonstrate that, by updating only 0.15‰ of parameters with a 20ms latency, CroTTA achieves average recovery rates of 24.7%, 26.9%, and 11.8% for L2, collision rate, and PDMS, respectively, under severe distribution shifts. In comparison with existing TTA methods, CroTTA obtains an additional 20.5 and 18.2 percentage points of L2 and collision recovery. Our code is available at: https://anonymous.4open.science/r/CroTTA-88D0/.