TT-ACS: Training-Free Task-Aware Adaptive Compression Scheduling for Vision Transformers

Token compression in vision transformers involves two distinct decisions: which tokens to compress and how much to compress. Existing methods primarily focus on how tokens are selected and compressed, while the compression budget is typically manually set and fixed across inputs. This input-agnostic design can under-compress redundant inputs while over-compressing information-rich ones. We introduce TT-ACS, a training-free framework for task-aware adaptive compression scheduling that determines the compression budget using task-relevant signals already computed by the model. Specifically, TT-ACS leverages attention interactions to estimate, for each input, the boundary between informative and redundant tokens, and maps this boundary to an adaptive compression budget. The resulting budget can be directly applied to existing token compression operators without modifying their internal mechanisms. On ImageNet-1K, TT-ACS consistently improves performance across all 13 operator–backbone configurations spanning four representative compression operators under matched compute budgets, with gains of up to 0.52 percentage points. Without task-specific retraining, TT-ACS also outperforms fixed-budget methods across multiple tasks, including object detection, video classification, and audio classification, with gains at all operating points on detection and video, and at light-to-moderate audio compression. These results demonstrate the value of compression scheduling as a distinct and complementary dimension of token compression, enabling compression strength to adapt to the task-relevant redundancy of each input. The code is available at https://anonymous.4open.science/r/TT-ACS/.