The Confidence Illusion: Rethinking Entropy Minimization for Continual Test-Time Adaptation

Continual Test-Time Adaptation (CTTA) aims to adapt a source model to continuously changing target domains using unlabeled test streams. Existing CTTA methods largely rely on entropy minimization, assuming that confident predictions provide reliable adaptation signals. However, we reveal a hidden confidence illusion: confident predictions are not always trustworthy. Under distribution shifts, misclassified samples can also exhibit low entropy and be repeatedly optimized, causing error accumulation. Moreover, such misleading confidence can gradually bias prediction distributions toward a few dominant classes, leading to long-term adaptation instability. To address this issue, we rethink entropy minimization from the perspective of reliable confidence optimization and propose Reliability-Aware Balanced Entropy (RBE). RBE distinguishes trustworthy confidence from misleading confidence through two complementary designs. Specifically, Consistency-aware Entropy Exploration (CAE) estimates prediction reliability by combining weak-augmentation consistency and teacher-student agreement, and selectively emphasizes reliable samples during entropy minimization. Meanwhile, Balanced Prediction Regularization (BPR) softly penalizes persistent concentration in a temporally aggregated prediction distribution, thereby mitigating class-dominant collapse. Extensive experiments on multiple CTTA benchmarks demonstrate that RBE achieves competitive overall performance and stable adaptation under continuously evolving target domains. Our findings highlight the importance of moving beyond confidence maximization toward reliable confidence modeling for continual adaptation.