Active Test-time Adaptation for Temporal Action Segmentation

Temporal action segmentation (TAS) models often suffer from performance degradation during deployment due to temporal distribution shifts and boundary ambiguity. Active test-time adaptation has shown promising results in image recognition by strategically querying sparse human feedback for adaptation. However, extending this paradigm to TAS is challenging due to long temporal span and action boundary ambiguity in procedural videos. To address these challenges, we present the first active test-time adaptation framework for TAS to strategically query sparse human annotations to ground the adaptation process. Our approach is based on a principled analysis of prediction dynamics, revealing that model hesitation under temporal smoothing constraints manifests as informative entropy peaks at action boundaries. We then propose directional boundary overwrite with agnostic and semantic variants that convert local feedback into adaptation labels, offering different trade-offs between annotation effort and supervision reliability. Our results demonstrate that annotating a few key temporal anchors significantly improves performance, establishing a new foundation for active adaptation of TAS models.