ProtoTTA: Explainable Test-Time Adaptation via Prototype Guidance

Deep networks that rely on prototypes—interpretable representations that can be related to the model input—have gained significant attention for balancing high accuracy with inherent interpretability, which makes them suitable for critical domains such as healthcare. However, their prototypes are limited by their reliance on training data, which hampers their robustness to distribution shifts. While test-time adaptation (TTA) improves the robustness of deep networks by updating parameters and statistics, the prototypes of interpretable models have not been explored for this purpose. We introduce ProtoTTA, a general framework that minimizes the binary entropy of prototype similarities to encourage confident, prototype-specific activations, while adaptively balancing this objective with output entropy based on a label-free estimate of prototype reliability. We stabilize these updates by geometric filtering and weighting by prototype importance and model confidence. Experiments across four prototypical backbones on four benchmarks in fine-grained vision, histopathology, and NLP demonstrate that ProtoTTA improves robustness over standard output entropy minimization while restoring the correct grounding of prototypes. We also introduce novel interpretability metrics and a vision-language model (VLM) evaluation framework to quantify and explain our test-time updates, finding positive associations between prototype alignment and VLM-rated reasoning quality. Code will be released upon publication.