PACE: Preference-Aware Early Exiting for Concept Bottleneck Models

While concept bottleneck models (CBMs) offer concept-level explanations of model predictions, modern CBMs typically rely on computationally intensive vision-language backbones to construct and align their concepts. This makes efficient deployment more difficult, especially when inference must preserve informative concept-level evidence rather than produce only class labels. To address this problem, we present PACE, a preference-aware multi-exit framework that supports early exiting in CBMs. Specifically, PACE keeps the visual backbone fixed and trains additional concept exits at intermediate depths to reduce inference cost while preserving the intermediate concept bottlenecks. Building on our core training mechanism, Concept-Preference Guided Distillation (CPGD), PACE addresses the difficulty of learning shallow concepts by exploiting preferences regarding concept depth. Instead of forcing every shallow concept to imitate the ultimate bottleneck representation, CPGD constructs concept-specific teachers at multiple depths and uses fusion-confidence-weighted supervision to better train shallow exits. At deployment, PACE introduces Readiness-Calibrated MaxProb (RCMax), a lightweight decision rule that retains MaxProb as the primary early-exit criterion while calibrating exit decisions based on key-concept support extracted from the current bottleneck. Comprehensive experiments on well-known benchmarks show that PACE achieves a superior balance between accuracy and efficiency compared to state-of-the-art (SOTA) baselines, with speedups ranging from 1.73× to 2.48×.