acceptodds
Under review as a conference paper at ICLR 2027

PACE: Preference-Aware Early Exiting for Concept Bottleneck Models

Abstract

While concept bottleneck models (CBMs) offer concept-level explanations of model predictions, modern CBMs typically rely on computationally intensive vision-language backbones to construct and align their concepts. This makes efficient deployment more difficult, especially when inference must preserve informative concept-level evidence rather than produce only class labels. To address this problem, we present PACE, a preference-aware multi-exit framework that supports early exiting in CBMs. Specifically, PACE keeps the visual backbone fixed and trains additional concept exits at intermediate depths to reduce inference cost while preserving the intermediate concept bottlenecks. Building on our core training mechanism, Concept-Preference Guided Distillation (CPGD), PACE addresses the difficulty of learning shallow concepts by exploiting preferences regarding concept depth. Instead of forcing every shallow concept to imitate the ultimate bottleneck representation, CPGD constructs concept-specific teachers at multiple depths and uses fusion-confidence-weighted supervision to better train shallow exits. At deployment, PACE introduces Readiness-Calibrated MaxProb (RCMax), a lightweight decision rule that retains MaxProb as the primary early-exit criterion while calibrating exit decisions based on key-concept support extracted from the current bottleneck. Comprehensive experiments on well-known benchmarks show that PACE achieves a superior balance between accuracy and efficiency compared to state-of-the-art (SOTA) baselines, with speedups ranging from 1.73× to 2.48×.

Primary area:
interpretability and explainable AI
Keywords:
concept bottleneck models, early exiting, adaptive inference, explainable AI, efficient inference, vision-language models, representation learning

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_6fPLOpenReview ↗
Rating
3(+1)
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE which is an early-exit framework for concept bottleneck models. It freezes the visual backbone and shares a fixed concept-to-class classifier across intermediate exits. Its training component, Concept-Preference Guided Distillation (CPGD), constructs concept-specific teachers by combining outputs from deeper exits according to estimated depth preferences. Its inference component, Readiness-Calibrated MaxProb, augments classification confidence with a class-conditioned concept-readiness gap. Experiments on five recognition datasets show improved accuracy–computation trade-offs over adapted early-exit baselines, supplemented by component ablations, multi-seed evaluations, and concept-level experiments.

Critical strengths
  • The paper addresses a meaningful problem: accelerating CBMs while retaining intermediate concept representations, rather than producing only early class predictions.

  • The comparisons control several important factors, including the backbone, concept vocabulary, exit locations, exit-head capacity, and shared classifier. The forced-exit and matched-computation ablations help separate training-side and gating-side effects.

  • CPGD consistently improves shallow predictive accuracy over final-only distillation across the reported dataset–seed pairs. The transfer experiments also suggest that its training-side benefit is not restricted to the main CBM implementation.

Critical weaknesses
  • CPGD closely resembles the preference estimation and sparse multi-layer fusion mechanisms in MVP-CBM, which is cited by the paper. Group-level preferences, their combination with concept similarities, cross-layer weighting, and adaptive sparse thresholding already appear in that work.
  • The preference scores are derived from intermediate visual–text similarities. Higher similarity or a larger activation does not necessarily imply a more accurate concept prediction, particularly when intermediate features are mapped through the frozen final-layer projection.
  • The reported speedups are cumulative-MAC ratios obtained from offline exit decisions, rather than measured latency improvements. The practical deployment is a central motivation, such as dynamic batching, sample compaction, memory access, and the execution of preceding exit heads can affect actual speedups.
Questions for authors
  • How does CPGD compare with MVP-CBM-style teacher fusion under the same distillation protocol?

  • Do the preference scores and fusion-concentration weights correlate with actual teacher concept accuracy?

  • Does cumulative-MAC accounting include all preceding exit heads, and what latency improvement is achieved by online execution?

7 comments

We thank the reviewer for recognizing the importance of the research problem addressed in this paper, as well as our experimental efforts in controlled evaluation settings, improved shallow predictions, multi-seed validation, and transfer across CBM frameworks. To address your three technical concerns, we have conducted an additional comparison with MVP-CBM-style teacher fusion under the same distillation protocol, verified the relationship between preference scores and fusion concentration with teacher concept accuracy, and measured online inference latency on real GPU hardware.

AW1: Methodological Novelty and Substantive Differences from MVP-CBM

Our design for concept-depth preference estimation and sparse fusion was indeed inspired by MVP-CBM, and we have explicitly cited this work in Section 2 of the main text and Appendix B. It is worth noting that the two methods differ substantively in their core computational paradigms and optimization objectives: MVP-CBM adopts bottom-up multi-layer feature aggregation to serve a single final classifier, still requiring execution of the full visual backbone at inference; in contrast, PACE adopts top-down multi-exit knowledge transfer to build shallow concept bottlenecks capable of independent prediction and early exiting. Specifically, this is reflected in the following three aspects:

(1) From cross-layer aggregated prediction to heterogeneous, independent multi-exit concept bottlenecks. MVP-CBM aggregates concept activations from different visual layers into a single fused concept representation for final classification. In contrast, PACE constructs six independently predictive concept exits; tailored to the characteristics of visual representations at different depths, it employs Local Perception Heads (LPH) at shallow layers and Global Aggregation Heads (GAH) at deeper layers, while sharing a single frozen sparse linear classifier. This design enables intermediate layers to independently reconstruct interpretable concept bottlenecks compatible with the shared classifier without relying on deep feature fusion. Table 11 in the original paper further validates the critical role of heterogeneous exit heads in shallow concept recovery.

(2) From end-to-end activation fusion to decoupled knowledge transfer for shallow students. MVP-CBM adopts end-to-end joint optimization, directly learning a fused representation for classification via classification cross-entropy, concept cross-entropy, and sparsity constraints. In contrast, PACE adopts a decoupled training framework that freezes the visual backbone, final exit, and shared classifier, training shallow students via CLIP concept KL, final-layer class KD, and CPGD concept distillation. More importantly, CPGD utilizes deeper candidate teachers solely during the training phase to construct concept-specific distillation targets for each shallow exit, rather than directly using cross-layer fused representations for inference classification. Building upon the concept-preference idea, CPGD further introduces a candidate teacher count constraint, differentiable selection, normalized concept fusion, and supervision weighting based on fusion concentration, thereby achieving stable knowledge transfer from deep teachers to independent shallow bottlenecks.

(3) From fixed full-depth inference to concept-supported dynamic early exiting. MVP-CBM must compute all network layers to complete the final fused prediction; in contrast, PACE proposes the RCMax gate, which introduces a class-conditioned concept-support gap on top of MaxProb classification confidence to enable sample-wise dynamic exit decisions. Once the current shallow bottleneck satisfies the exit condition, neither subsequent visual layers nor deeper concept exits need to be executed, reducing inference computation overhead while preserving the concept bottleneck at the actual exit location for semantic evaluation and test-time intervention.

Controlled comparison with MVP-CBM-style teacher fusion under the same protocol. Following your suggestion, we implemented an MVP-CBM-style Teacher Fusion baseline under the exact same multi-exit distillation protocol. The experiment keeps the frozen visual backbone, six-exit architecture, heterogeneous LPH/GAH concept heads, shared classifier, and base distillation protocol identical, replacing only CPGD's teacher construction and corresponding fusion weights with MVP-CBM-style concept preference computation, threshold filtering, and exponential sparse weighting, thus eliminating interference from differences in exit-head capacity and base training settings.

MethodAvg. Early Acc. (%) ↑
MVP-CBM-style Teacher Fusion60.42
Full CPGD (Ours)61.68

On CIFAR-100, directly adapting MVP-CBM's exponential sparse fusion achieves an average shallow accuracy of 60.42%, whereas full CPGD reaches 61.68%, improving over MVP-CBM-style Teacher Fusion by 1.26 percentage points. Because both methods adopt the exact same student architecture and base distillation protocol, this controlled experiment demonstrates the effectiveness of CPGD in normalized teacher construction and concentration-weighted supervision.

AW2: Verification of the Relationship Between Preference-Guided Fusion, Concentration Weighting, and Teacher Concept Accuracy

Yes, our supplementary experiments provide direct evidence from two dimensions: concept-depth preferences guide the construction of fused teachers with higher ground-truth concept accuracy, while higher fusion concentration consistently corresponds to lower teacher concept prediction error rates. In PACE, preference scores are not used to directly single out an individual teacher, but rather to construct sparse multi-teacher fusion targets as auxiliary supervision signals for shallow concept learning, while fusion concentration is used to dynamically adjust the strength of this auxiliary supervision. To directly verify the relationship between these two mechanisms and ground-truth concept quality, we conducted two sets of supplementary analyses on the CUB dataset based on human-annotated attributes.

AW2(a): Concept Semantic Quality of Preference-Guided Fused Teachers

Using 107 deduplicated human-annotated concept targets from CUB, we constructed the corresponding deep fused teachers for the three shallow student exits L2, L4, and L6, and directly evaluated the concept balanced accuracy of these fused teachers against ground-truth annotations. The experiment used the pre-fixed seed 42, and simultaneously computed the expected performance of random single-teacher selection and random sparse fusion strategies.

Teacher ConstructionConcept BAcc. (%) ↑
Random Single Teacher65.34
Random Sparse Fusion65.54
Preference-guided Sparse Fusion66.16

The results show that preference-guided sparse fusion clearly outperforms both random single-teacher selection and random sparse fusion strategies in ground-truth concept balanced accuracy. This indicates that its advantage does not stem solely from multi-teacher information aggregation; concept-depth preferences provide useful teacher selection and fusion information, thereby constructing concept supervision targets with higher semantic quality.

AW2(b): Relationship Between Fusion Concentration and Concept Prediction Error Rate

We further analyzed the relationship between fusion concentration and concept prediction error rates by dividing sample–concept pairs into Low and High groups according to fusion concentration and measuring the prediction error rates of the corresponding fused teachers on ground-truth concepts.

Fusion ConcentrationL2 Error (%) ↓L4 Error (%) ↓L6 Error (%) ↓
Low34.5534.5435.17
High34.1434.0134.49
Error Reduction (pp)0.410.530.68

Across all three exits L2, L4, and L6, the high fusion concentration group exhibits lower ground-truth concept prediction error rates, confirming a consistent positive association between fusion-weight concentration and teacher concept quality, which directly supports CPGD's design of using fusion concentration to adjust distillation supervision weights. These two concept-level analyses also corroborate the final classification gains observed on CIFAR-100.

Reviewer_PXfROpenReview ↗
Rating
2
Confidence
2
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper proposes PACE, a preference-aware early-exiting framework for Concept Bottleneck Models (CBMs) that aims to improve inference efficiency while preserving meaningful concept representations. PACE introduces two components: (1) Concept-Preference Guided Distillation (CPGD), which uses concept-specific multi-depth teachers to improve shallow concept representations, and (2) Readiness-Calibrated MaxProb (RCMax), which incorporates concept-level evidence into early-exit decisions. Experiments on five classification benchmarks demonstrate improved accuracy–efficiency trade-offs over adapted early-exit baselines, achieving cumulative-MAC speedups of 1.73×–2.48×. Additional ablations and concept-level evaluations support the effectiveness of the proposed framework.

Critical strengths

1.Meaningful problem setting. The paper addresses an important limitation of conventional early-exit methods in CBMs: confident predictions do not necessarily imply reliable concept representations. The frozen-backbone design and shared classifier provide a controlled setting for studying this problem.

2.Solid empirical results. PACE achieves competitive accuracy–efficiency trade-offs across five datasets. The matched-speedup ablations and multi-seed experiments further support the effectiveness and consistency of the proposed approach.

Critical weaknesses
  1. Limited methodological novelty.The concept-depth preference mechanism is closely related to prior work such as MVP-CBM. Moreover, Table 2 shows that uniform multi-depth supervision already provides a 3.03-point improvement over final-only distillation, while full CPGD adds only another 1.12 points. The specific contribution of preference-aware selection therefore remains unclear.

  2. Insufficient evidence for concept preservation. Although preserving meaningful concept evidence is a central motivation, the reported semantic improvements are relatively small (e.g., +0.50 AUROC and +0.14 mAP on CUB). The simulated intervention results also show limited absolute success rates. More importantly, concept quality is not directly evaluated at the dynamically selected exit points, leaving the main interpretability claim only partially supported.

Questions for authors
  1. Contribution of preference-aware distillation. In Table 2, uniform multi-depth supervision already improves Avg. Early Acc. from 57.53% to 60.56%, while full CPGD further increases it to 61.68%. Could the authors compare uniform multi-depth supervision + RCMax against full PACE at matched speedups? This would help clarify the contribution of preference-aware teacher selection beyond using multiple teacher depths.

  2. Reliability of concept-aware exiting. In Table 4, CPGD improves classification accuracy substantially, but concept-level improvements are relatively modest (+0.50 AUROC and +0.14 mAP). Since these evaluations use forced exits rather than the RCMax-selected exits, could the authors evaluate concept fidelity and intervention effectiveness at the actual selected exits? This would provide more direct evidence for the central claim of concept-supported early exiting.

  3. Practical inference efficiency. Section 4.1 and Appendix C report speedups based on cumulative MACs (1.73×–2.48×). Could the authors clarify whether these estimates account for all intermediate exit heads and gating operations evaluated before termination? How do these theoretical speedups translate into actual inference latency, particularly at batch size one?

5 comments

We thank the reviewer for recognizing the importance of the research problem addressed in this paper, as well as our experimental results in terms of controlled evaluation settings, accuracy–efficiency trade-offs, multi-seed validation, and shallow concept learning. To address your concerns regarding the disentanglement of training-side contributions, concept quality at dynamic exit locations, and practical inference efficiency, we have conducted targeted supplementary experimental analyses.

AW1: Methodological Positioning and Effectiveness of Preference-Guided Distillation

First, we further clarify the overall methodological positioning of PACE. Although MVP-CBM provided important inspiration for concept-depth preference modeling, the two methods differ substantively in their computational paradigms and optimization objectives: MVP-CBM adopts bottom-up multi-layer feature aggregation to serve a single final classifier, still requiring execution of the full visual backbone at inference; in contrast, tailored to the dynamic inference requirements of CBMs, PACE constructs a complete framework comprising independent multi-exit concept bottlenecks, a top-down concept-level knowledge transfer mechanism, and a concept-supported dynamic exit strategy. Unlike directly using cross-layer fused representations for final prediction, PACE transfers deep concept knowledge into independently predictive shallow bottlenecks via CPGD, and combines this with RCMax to achieve adaptive early exiting without computing subsequent deeper exits.

To further verify the independent contribution of each component of CPGD within this framework, we conducted training-side supervision disentanglement ablations and, following your suggestion, added dynamic inference comparisons under matched computation budgets to progressively isolate the effects of additional concept distillation supervision, multi-depth teacher fusion, and the complete preference-guided mechanism.

(1) Disentangling Training-Side Supervision Contributions. The UD baseline in the original paper uses the CLIP concept KL loss and the final-exit class-level KD loss, whereas Uniform Multi-depth additionally introduces the MSE distillation loss on concept logits. Therefore, the improvement of Uniform Multi-depth over UD in Table 2 of the original paper encompasses the combined benefits of introducing concept MSE supervision and using multi-depth teacher fusion.

To accurately separate these two contributions, we added a Final-only + MSE baseline, which retains the exact same concept MSE distillation loss formulation but uses only the final exit as the sole concept teacher.

MethodAvg. Early Acc. (%) ↑
UD57.53
Final-only + MSE58.89
Uniform Multi-depth60.56
Full CPGD (Ours)61.68

On CIFAR-100, introducing final-layer concept MSE supervision alone improves average shallow accuracy by 1.36 percentage points; further extending to uniform multi-depth teachers brings a 1.67-percentage-point improvement; and incorporating full CPGD's preference-guided selection and concentration weighting yields an additional 1.12-percentage-point improvement on top of that. This stepwise comparison clearly isolates the contributions of different supervision signals, demonstrating that beyond additional concept supervision and multi-teacher fusion, CPGD's preference-guided teacher construction and weighted training design bring a clear, independent gain.

(2) Dynamic Inference Comparison under Matched Computation Budgets. Following your first question, we directly compared the dynamic inference performance of Uniform Multi-depth and full PACE under the same RCMax exit strategy and matched cumulative-MAC computation budgets.

MethodAccuracy (%) ↑MAC Speedup ↑
Uniform Multi-depth + RCMax74.191.772×
Full PACE (Ours)74.961.776×

At nearly identical computation speedups, full PACE still achieves a +0.77-percentage-point advantage in classification accuracy. Because both methods share the exact same multi-exit architecture, concept classifier, and RCMax exit rule, this result directly demonstrates that CPGD's training-side preference-guided gains effectively carry over to the complete dynamic inference pipeline.

AW2: Concept Fidelity and Concept Intervention Effectiveness at Dynamically Selected Exits

Following your second suggestion, to directly evaluate concept quality during actual dynamic inference, we conducted concept semantic alignment and concept intervention evaluations at the actual exit locations dynamically selected by the model for each sample, rather than testing only at fixed forced shallow exits.

Specifically, we evaluated across all 5,790 CUB test images using the 132 concepts mapped to human-annotated attributes from the original 370-dimensional concept vocabulary over 3 random seeds (1, 42, and 1024). For each test image, we directly extracted and evaluated the concept bottleneck representation at the layer where the gating policy actually triggered the exit.

(1) Concept Fidelity at Actual Dynamic Exit Locations.

MetricUD + MaxProbPACEImprovement
Classification Acc. (%) ↑55.4459.49+4.05 pp
Concept AUROC (%) ↑69.3469.75+0.41 pp
Weighted AUROC68.8669.24+0.38 pp
Concept mAP29.7129.88+0.16 pp
Concept BAcc64.0664.33+0.27 pp

The results show that under actual dynamic early exiting, PACE substantially improves classification accuracy by 4.05 percentage points while all four metrics of its actually output concept bottlenecks—AUROC, Weighted AUROC, mAP, and BAcc—remain stable and achieve consistent improvements across the board. Compared with the fixed-layer evaluations in the original paper, this three-seed experiment directly confirms that PACE effectively balances classification gains with concept semantic quality at the actual output layers during dynamic inference.

(2) Concept Intervention Effectiveness at Actual Dynamic Exit Locations. To further verify concept intervenability at dynamic exit locations, we followed the simulated human-attribute intervention protocol in Appendix D.4, applying the same 5-concept query budget at the actual exit dynamically selected for each image, and evaluated over three random seeds on CUB test samples initially misclassified by all compared methods.

MethodGuided Success@5 ↑Oracle Success@5 ↑
UD + MaxProb3.88 ± 0.149.28 ± 0.49
CPGD + MaxProb4.13 ± 0.439.82 ± 0.78
PACE (Ours)4.20 ± 0.429.88 ± 0.80

The results show that at actual dynamic exit locations, PACE outperforms UD + MaxProb in both Guided Success@5 and Oracle Success@5, while achieving comparable or higher intervention success rates than CPGD + MaxProb. This directly demonstrates that the concept bottlenecks selected by PACE preserve meaningful semantic alignment and remain amenable to test-time concept intervention, with higher Guided and Oracle Success@5 than UD + MaxProb.

AW3: Cumulative Computation Cost Accounting and Practical Hardware Online Inference Efficiency

First, we confirm that in our cumulative-MAC cost accounting, the computation cost for a sample exiting at a selected exit fully accounts for all preceding concept exit heads executed along the path prior to termination, as well as the computation overhead of the shared classifier.

To further verify how theoretical speedups translate onto real hardware, we conducted online early-exit measurements with batch size = 1 on an NVIDIA RTX 4090 GPU, using the same model configurations and validation-selected operating points as in Table 1 of the original paper, and fully including the concept head, shared classifier, and RCMax gating computation overhead at each visited exit during actual execution. The measured average latencies for the full-depth model and the dynamic PACE model are: CUB 1.931 ms / 1.166 ms; TinyImageNet 1.844 ms / 0.965 ms, respectively.

DatasetMAC Speedup ↑Latency Speedup ↑
CUB1.73×1.656×
TinyImageNet2.00×1.910×

Measurement protocol: 5 paired measurement runs were conducted using the same 1,024 test images; reported latencies are the means of the per-run average latencies, speedups are the means of the per-run ratios of unrounded latencies, and latencies are displayed to three decimal places.

The experimental results show that PACE achieves 1.656× and 1.910× online inference speedups on a real GPU, reducing actual latency by approximately 39.6% and 47.6% compared with full-depth inference, respectively. Meanwhile, the deviations between the measured hardware speedups and the theoretical cumulative-MAC speedups are only approximately 4.25% and 4.48%, indicating that the theoretical computation savings reported in the paper translate consistently into low-latency gains in practical deployment. Furthermore, the above results were obtained without low-level operator fusion or custom scheduling optimization for the dynamic exit paths, leaving room for further gains when combined with system-level optimizations in the future.

Reviewer_H81FOpenReview ↗
Rating
3
Confidence
4
Informativeness
2
Presentation
2
Soundness
3
Read review
Paper summary

The paper presents an end-to-end framework that supports early exiting in CBMs for improved inference efficiency. Specifically, the authors propose a novel learning strategy for training intermediate concept bottleneck layers with guidance from deeper layers, and an inference-time early-exit strategy based on concept activations rather than class probabilities alone for improved exit decisions.

Critical strengths
  1. The proposed framework for early exiting in CBM is both important and novel, and authors have shown the effectiveness of both proposed training and inference strategy on multiple benchmarks.
Critical weaknesses
  1. The work is missing important evaluations: i) The ablation study for the proposed CPGD approach does not compare training intermediate concept layers using only CLIP-derived concept predictions, and it unclear if CGPD can significantly improve intermediate concept learning over directly learning from CLIP-predicted concept activations. ii) The paper is missing results on the full ImageNet benchmark, following seminal methods in the field[1,2,4].

  2. Recent studies [3, 4] have shown that CBM training can leak information through the bottleneck layer, allowing models to achieve high accuracy without actually learning meaningful concepts. To address this issue, [4] introduced the A-NEC metric to enable fair comparisons between CBMs based on their concept learning ability and proposed method should be evaluated using this metric for a fair comparison. This is particularly important for the proposed method, as it learns concept bottlenecks at intermediate layers, and it remains unclear whether the learned concepts are faithful. Further, the manuscript is missing visualization of interpretable decisions, and it is unclear if the concept bottleneck layers are being faithfully learned with the proposed training approach..

[1] Yang, Yue, et al. "Language in a bottle: Language model guided concept bottlenecks for interpretable image classification." Proceedings of the IEEE/CVF conference on computer vision and pattern recognition. 2023.

[2] Oikarinen, Tuomas, et al. "Label-free concept bottleneck models." arXiv preprint arXiv:2304.06129 (2023).

[3] Yan, An, et al. "Learning concise and descriptive attributes for visual recognition." Proceedings of the IEEE/CVF International Conference on Computer Vision. 2023.

[4] Srivastava, Divyansh, Ge Yan, and Lily Weng. "Vlg-cbm: Training concept bottleneck models with vision-language guidance." Advances in Neural Information Processing Systems 37 (2024): 79057-79094.

Questions for authors

My primary concerns are the lack of ablation comparing proposed CPGD method to direct learning from CLIP activations, missing results on ImageNet benchmark, lack of visualization of learned concepts, and potential information leakage in learned bottleneck layers. Please see weaknesses for more details.

3 comments

We thank the reviewer for recognizing the significance of efficient inference for concept bottleneck models (CBMs), as well as the effectiveness of our training and inference strategies demonstrated across multiple datasets. To address your concerns regarding direct shallow concept supervision comparisons, concept faithfulness and potential information leakage, and qualitative visualizations and evaluation scope, we have conducted targeted comparative experiments and clarified the existing evidence.

AW1: Ablation Comparison Using Direct CLIP Concept Supervision Only

To examine whether direct CLIP-derived concept supervision alone is sufficient to train effective shallow concept bottlenecks, we introduced an additional CLIP-only baseline.

This experiment retains the exact same frozen visual backbone, concept vocabulary, heterogeneous concept exit heads, and frozen shared classifier as PACE; when training shallow exits, it uses only the CLIP-derived concept KL loss, removing the final-exit class-level knowledge distillation loss and the CPGD concept distillation loss.

MethodAvg. Early Acc. (%) ↑
CLIP-only28.10
UD57.53
Full CPGD (Ours)61.68

On CIFAR-100, CLIP-only achieves an average shallow-exit accuracy of only 28.10%, which is substantially lower than UD's 57.53% and full CPGD's 61.68%.

This result indicates that under the setting of independent multiple exits and a shared fixed classifier, directly fitting the CLIP concept distribution alone is insufficient to recover discriminative shallow concept bottlenecks. Combining class-level distillation with CPGD's cross-layer concept knowledge transfer provides crucial supervisory gains for recovering shallow concept bottlenecks.

AW2: Concept Bottleneck Faithfulness and ANEC Evaluation under Controlled Effective Concept Counts

To further examine whether potential information leakage exists in shallow concept bottlenecks, we followed your suggestion and adopted the Number of Effective Concepts metric introduced by VLG-CBM, comparing the performance of UD and CPGD while strictly controlling the number of effective concepts used by the classifier. Meanwhile, because PACE adopts a decoupled two-stage training scheme where the shared sparse classifier is pre-trained and kept frozen before training the shallow concept exits, this training mechanism also prevents shallow exits and the classifier from leaking non-concept information through joint co-adaptation.

We evaluated the ANEC-5 and ANEC-avg performance at shallow exits L2, L4, and L6 on the CUB dataset across three random seeds:

ExitANEC-5: UD / CPGD ↑ANEC-avg: UD / CPGD ↑
L217.21 / 18.8523.03 / 24.44
L426.55 / 29.7839.05 / 40.11
L634.64 / 37.4050.91 / 53.85

The results show that across all three shallow exits, CPGD consistently outperforms UD on both ANEC-5 and ANEC-avg metrics. Specifically, at L2, L4, and L6, ANEC-5 improves by 1.64, 3.23, and 2.76 percentage points, respectively, while ANEC-avg improves by 1.41, 1.06, and 2.94 percentage points, respectively. This demonstrates that when strictly limiting the number of effective concepts used by the classifier, CPGD still maintains a clear shallow prediction advantage, and its performance gains do not stem from utilizing more concept dimensions to smuggle soft information.

Furthermore, Section 4.4 and Appendix D.4 of the original paper conducted direct concept semantic evaluations based on human-annotated ground-truth attributes on CUB, covering 132 mapped concepts and 107 deduplicated semantic targets from the original 370-dimensional vocabulary. The results similarly show that while CPGD substantially improves shallow classification performance, concept AUROC, mAP, and balanced accuracy remain stable and improve slightly. The newly added controlled ANEC experiments and the human ground-truth attribute alignment evaluations complement each other, confirming the faithfulness of shallow concept bottlenecks from the perspectives of controlled concept sparsity utilization and alignment with human-defined attributes, respectively.

AW3: Qualitative Visualization of Interpretable Decisions and Dataset Evaluation Scope

Regarding the visualization of interpretable decisions, Figure 4 in Appendix E of the original paper presents qualitative cases on the CUB dataset, illustrating the predicted classes, the evolution of highly activated concepts across different depths from L2 to L12, and the actual exit layers selected by PACE for each sample. These cases visually demonstrate how concept evidence progressively evolves with depth from coarse-grained generic features to fine-grained class-supporting features, and how PACE exits early once an intermediate concept bottleneck already provides sufficient support. We will move this visualization analysis into the main text in the revised manuscript to more intuitively showcase interpretable dynamic exit decisions.

Regarding the dataset evaluation scope, this paper evaluates the accuracy–computation trade-offs on CUB, CIFAR-100, CIFAR-10, TinyImageNet, and the large-scale scene recognition dataset Places365 (comprising 365 classes and 2,544 concepts), and verifies the transferability of CPGD to recent CBM frameworks (DN-CBM and CFM) in Appendix D.5, covering visual recognition tasks across diverse scales and granularities. We agree that evaluating on the full ImageNet-1K benchmark would further test the generalization ability of our method under an ultra-large-scale class space; constrained by computational resources at the current stage of study, this work primarily focuses on validating across the five representative benchmarks and two recent CBM architectures mentioned above, and we will clarify this evaluation scope more explicitly in the revised main text while incorporating ImageNet-1K as an extension in future work.

Dear Reviewer H81F,

As the discussion period progresses, we wanted to briefly follow up and see if our supplementary experiments have addressed your questions. To briefly summarize our updates:

  1. Direct CLIP Supervision Ablation (AW1): Training shallow exits with CLIP-only supervision yields 28.10% average early accuracy on CIFAR-100, confirming that cross-depth distillation in CPGD (61.68%) is essential for recovering discriminative shallow bottlenecks.
  2. A-NEC Faithfulness Evaluation (AW2): Across 3 seeds on CUB, CPGD consistently improves over UD on both ANEC-5 (+1.64 to +3.23 percentage points) and ANEC-avg (+1.06 to +2.94 percentage points) across shallow exits L2,L4,L6L_2, L_4, L_6, confirming that our gains stem from faithful concept learning rather than bottleneck leakage.
  3. Visualizations & Scope (AW3): We will move the progressive concept visualization cases (Appendix E, Figure 4) into the main text and clarify our large-scale benchmark scope in the revision.

Please let us know if you have any remaining questions. Thank you again for your valuable feedback and support!

Best regards, Paper 21131 Authors

Comments on the paper

1 comment
AuthorsOpenReview ↗

General Response

We sincerely thank the Area Chair and all reviewers for your time, constructive feedback, and shared recognition of the importance of enabling efficient adaptive inference while preserving meaningful concept bottlenecks in CBMs. We are glad to have directly addressed the main weaknesses raised across the reviews through targeted experiments and clarifications, including Reviewer 6fPL's concerns as well as Reviewer PXfR's questions on CPGD ablations and real-hardware GPU latency. For Reviewer PXfR's single remaining follow-up regarding the standalone effect of RCMax under a fixed CPGD model, we have also directly answered it with a matched-budget controlled comparison.

As the discussion concludes, we briefly summarize our core contributions and the key experimental evidence added during the rebuttal:

  1. Methodological Distinction and CPGD Ablations (Reviewers 6fPL and PXfR): We clarified that unlike MVP-CBM, which aggregates features bottom-up for a single final classifier and requires full backbone execution, PACE uses decoupled top-down preference-guided distillation (CPGD) into independent exits for sample-wise early termination. Under the exact same multi-exit protocol on CIFAR-100, Full CPGD (61.68%) outperforms adapted MVP-CBM-style Teacher Fusion (60.42%, +1.26 pp), CLIP-only (28.10%), UD (57.53%), Final-only + MSE (58.89%), and Uniform Multi-depth (60.56%). Under dynamic inference at a matched ~1.77x speedup, Full PACE (74.96%) also outperforms Uniform Multi-depth + RCMax (74.19%, +0.77 pp).

  2. Controlled CPGD vs. RCMax Comparison and Concept Faithfulness (Reviewers PXfR, H81F, and 6fPL): Evaluating directly at the dynamically selected exits on CUB (3 seeds, matched ~1.73x MAC speedup) using the exact same CPGD models isolates the gains of our primary training method and complementary inference gate: Concept AUROC improves from 69.34% (UD + MaxProb) to 69.59% (CPGD + MaxProb, +0.25 pp from CPGD), and further to 69.75% (CPGD + RCMax, +0.16 pp from training-free RCMax calibration), alongside 59.49% dynamic accuracy (+4.05 pp over UD + MaxProb) and the highest Guided@5 (4.20%) and Oracle@5 (9.88%) intervention rates. Across CUB shallow exits L2, L4, and L6, CPGD also consistently beats UD under controlled concept sparsity (ANEC-5 +1.64 to +3.23 pp; ANEC-avg +1.06 to +2.94 pp) and achieves higher ground-truth teacher concept BAcc (66.16% vs. 65.34% and 65.54% random baselines), confirming faithful concept recovery without bottleneck leakage.

  3. Real-Hardware Online GPU Latency (Reviewers 6fPL and PXfR): Explicitly accounting for all preceding exit heads, shared classifier calls, and online RCMax gating on an NVIDIA RTX 4090 (batch size = 1, 5 paired runs of 1,024 images), PACE reduces average latency from 1.931 ms to 1.166 ms on CUB (1.656x speedup) and from 1.844 ms to 0.965 ms on TinyImageNet (1.910x speedup), within 4.5% of theoretical MAC speedups.

We will incorporate all of these new results, analyses, and clarifications into the final version of the manuscript. Thank you again for your time and dedicated service.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.