- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper proposes Concept Envelope, a method for learning low-dimensional representations that aim to preserve semantic information beyond what is required for prediction. The method jointly optimizes a task prediction loss and a covariance-separation penalty to learn a linear subspace, followed by a post-hoc concept grounding procedure to interpret the learned representation. The evaluations are conducted on seven visual benchmarks using predictive accuracy, concept activation vector retention, and edit fit, while further considering constructed day-of-week and month-of-year prediction tasks. While the results demonstrate improvements in retaining certain predefined semantic directions, the methodological novelty and the broader significance of the proposed approach remain insufficiently established.
Critical strengths
- The paper considers the distinction between predictive sufficiency and semantic preservation, which is a practical problem to investigate in representation learning.
- The proposed objective is relatively straightforward to implement and combines task supervision with a covariance-separation penalty, making the approach easy to understand and evaluate.
- The evaluation considers several aspects of representation quality beyond predictive accuracy, including concept direction retention and the transferability of semantic edits.
Critical weaknesses
- The proposed method primarily combines supervised subspace learning with a covariance-separation penalty motivated by existing envelope methods, followed by post-hoc concept grounding. The paper does not sufficiently establish what fundamental methodological insight or capability distinguishes this combination from existing approaches.
- The optimization objective explicitly penalizes cross-covariance between the retained subspace and its orthogonal complement, but the paper's central claim concerns semantic preservation. The connection between these two objectives remains insufficiently justified, as covariance separation alone does not guarantee that the retained directions correspond to meaningful or comprehensive semantic factors.
- The proposed semantic metrics primarily evaluate predefined concept directions and selected paired interventions, while the language experiments rely on constructed tasks. Although the results demonstrate improvements under these specific settings, they provide limited evidence that the method consistently preserves meaningful semantic structure or offers broader practical benefits.
Questions for authors
I expect the authors to address my concerns in the weaknesses section.
2 comments
Thanks for your response. However, after carefully considering the rebuttal, my main concerns regarding the novelty and significance of the work remain insufficiently addressed. Therefore, I will maintain my original score.
We thank the reviewer for the thoughtful assessment.
Q1: Is there an insight beyond combining known components?
A1: Yes. The insight is to read a pretrained representation through the envelope of its predictive subspace. Envelopes were introduced in statistics for efficient estimation in regression. We apply them to a different problem: making a compact part of a pretrained representation both predictive and readable as concepts, where a concept may span several directions. Existing approaches each give up one side of this goal. Concept bottlenecks keep interpretability by forcing prediction through scalar scores for predefined concepts, and dimension reduction keeps accuracy by retaining only what the label needs. Concept Envelope learns, from the task label alone, a subspace that is both predictive and readable, and names its directions afterwards. A2 explains why the envelope has this property. Making this work with pretrained representations also requires a new estimator, which learns the subspace jointly with a nonlinear predictor under covariance separation.
Q2: Why does covariance separation preserve semantics?
A2: The connection rests on the linear representation hypothesis, which we use as a working assumption. If semantic factors are approximately linear, the covariance of the representation is organized by concepts, and penalizing cross-covariance discourages the subspace from keeping one direction while dropping another correlated with it. We agree that covariance separation alone does not guarantee semantic recovery, as stated in Section 2. The envelope keeps the concepts used for prediction and those correlated with them, but not every semantic factor. Empirically, the directions it adds are concepts defined independently of the method, such as the Waterbirds background, which a prediction-only subspace barely retains.
Q3: Does the evidence reach beyond controlled settings, and what is the practical use?
A3: We evaluate semantic preservation on datasets whose concept factors are controlled by construction, with interventions whose true change is known, and on tasks with a known state geometry, because these are the settings where preservation can be verified. This is the type of evidence we provide, and it does not cover every task. The practical use is to see, from task labels alone, which concepts a predictive representation carries, including context entangled with the label that a prediction-only representation would drop. On the real-image benchmarks, post hoc grounding of the learned subspace gives such readings (Appendix Table 2). The CelebA blond-hair subspace contrasts light hair with dark hair and facial hair, a male-associated trait, and the CUB species subspace is organized first by habitat, perched woody vegetation versus aquatic or shoreline settings. They come without any concept annotation.
We hope these clarifications address the reviewer's concerns, and we would be grateful for a reconsideration of the assessment.