- Rating
- 1
- Confidence
- 4
- Informativeness
- 2
- Presentation
- 1
- Soundness
- 2
Read review
Paper summary
The paper proposes an approach for quantifying the loss of information of a concept, when another concept is removed from the representation.
To provide some background: In deep learning (DL) we frequently rely on representations from large deep neural networks (DNNs) for various tasks (e.g. classification). We might want to remove a concept from such a representation (e.g. removing gender from the representations of CVs generated by a text encoder).
This paper seeks to quantify when you remove gender from this representation by achieving linear guardedness (e.g. no linear model can do better than a constant predictor to predict gender), how much information do you lose on other concepts (such as experience etc.). It quantifies this information via the , and lower-bounds the decrease in for any projection that achieves linear guardedness (with an equality for the LEACE projection). The paper further argues that label correlation is insufficient to explain the loss in predictability of other concepts.
In the experimental section, it shows that this the terms used for the bound (the ) are more predictive of the ability to predict another concept (measured via AUC-ROC) than label correlation. They argue that this gives a computationally cheap way to screen beforehand which concepts to remove.
Critical strengths
The paper investigates a very important topic: namely, how much information is lost by the removal of a selected concept.
It has a solid approach to address this topic. Focusing on provides an analytical expression for quantifying information loss, which in turn reduces computation time. I also appreciated the investigation of LEACE, as it is a relevant method for this subfield.
The paper also provides an extensive empirical evaluation across a range of datasets and concepts, including numerical verification of the theoretical results. The experimental design is well motivated and appropriate for evaluating the proposed approach.
Critical weaknesses
I found this paper extremely hard to read and understand. Below, I will give some examples of this. My intention is not to be overly critical, but rather to offer constructive feedback that I hope will help the authors improve the presentation. The list below is not meant to be exhaustive but rather to illustrate a broader point.
- The abstract contains many terms that are technical or specific to this literature that I would not expect all interested readers to know, such as ‘target attribute’, ‘probe’ ‘post-erasure accessibility matrix’, (I would in general avoid mathematical notation and terms in the abstract and introduction).
- Similarly, for the introduction, there are many terms where I am still unsure what they mean. For example:
- ‘a linear readout’ – while used in the literature, I encourage you to be more concrete for unfamiliar readers.
- ‘second-order sense’ I do understand what you mean here but I do not think this is the accessible way of describing it.
- ‘Linearly accessible means recoverable from the representation’ what does recoverable mean here?
- ‘Some practical tasks’ What tasks?
- In the methodology, I find the notation hard to follow: the notation does not distinguish between scalars, vectors, and matrices through capitalization and boldface. I would encourage the authors to adopt the conventions suggested by ICLR.
- The paper frequently introduces theoretical results without explaining the concrete question they answer or intuition. For example, Theorem 1 could be introduced through this question: if we remove gender information from a representation, how much predictive power for occupation must necessarily be lost? This would help readers understand why the subsequent result matters. Similarly, I would encourage the authors to provide more intuition for Corollary 3 and Proposition 1, particularly their practical implications.
In terms of methodology, I believe the central weakness is that there is relatively little done to motivate that is a good metric for measuring information loss.It is not clear this is an good metric for (binary) classification tasks, where we typically rely on logistic regression. Indeed, Figure 1 suggests that while the decrease in is somewhat correlated with the decrease in AUROC, there is quite some dispersion, meaning that similar can correspond to different changes in the performance. I appreciate that the authors empirically investigate this limitation and acknowledge it. Nevertheless, I would encourage them to explain more clearly what their theoretical guarantees do and do not imply about downstream classification performance, and to better motivate why is the appropriate quantity to study for the intended applications.
Finally, I am not fully convinced of the usefulness of the proposed screening approach. The authors provide a runtime comparison in Section 5.3, showing that evaluating a candidate erasure takes 0.10 ms with their approach, compared to 3,339 seconds (less than an hour) for applying the erasure and evaluating downstream performance (for 12 concepts). This is a substantial speedup – but often we only care about a smaller number of concepts, and fitting a small number of linear probes is feasible. The advantage of fitting probes is that it provides a direct measurement of the loss, rather than a prediction based on . I would encourage the authors to provide a more convincing practical example where the computational savings justify this trade-off.
Questions for authors
Why do all the experiments focus on frozen representations? I am not necessarily opposed to it, but it would be interesting to see if the experimental results also translate to the setting where we have fine-tuned the model. An experiment on a single dataset would already be informative.
4 comments
I thank the authors for their effort and novel manuscript. Below, I briefly go over the original concerns I raised and the extent to which I believe they have been resolved.
- I still find that the writing in the manuscript is overly technical, and hard to understand. Again, not to nitpick, but all the examples of unclear writing mentioned in the original review remain in the current version.
- I agree that R^2 is a legitimate proxy for downstream performance of logistic regression, and the additional results confirm this. As long as the authors are crystal clear about this limitation, I believe this is fine.
- I am still puzzled when you would want to 'screen many candidate concept sets before selecting which interventions to evaluate' - in most cases we have a small set of concepts (e.g. gender, race) that we definitinely want to remove. Could the authors provide an actual, real-world example of where one wants to screen this many concepts, and where it is an option to not remove some?
- The non-inclusion of finetuned representations is fine as long as it is mentioned as a limitation.
Overall, my strongest objection to the paper (quality of writing) remains, hence I will keep my score.
Dear Reviewer mvD9,
Thank you for revisiting our submission. We appreciate that our clarifications regarding and frozen representations have addressed two of your original concerns.
On presentation. We understand that the manuscript remains difficult to follow and acknowledge that our revision has not sufficiently addressed this concern. We appreciate your specific feedback and will use it to improve the exposition.
On screening. We agree that screening is less relevant when a fixed set of protected attributes must be removed. Our intended use case is optional shortcut removal, for example in medical imaging. Multi-site imaging representations can encode nuisance attributes such as hospital identity, scanner manufacturer, and acquisition year, which may contribute to shortcut learning (Zech et al., 2018; Hill et al., 2024).
Unlike mandatory protected attributes, these nuisance concepts need not all be erased. An auditor may consider multiple candidate subsets and seek interventions that preserve clinically relevant information. Our method ranks candidates by predicted collateral cost, allowing expensive erasure and downstream evaluation to focus on a shortlist. It does not determine which shortcuts should be removed or replace final validation.
We will clarify this distinction and strengthen the motivation in a future revision.
Thank you again for your constructive feedback.
I appreciate the authors getting back quickly. On screening: I believe you provide a great example of where screening might be useful. Would be a great addition to put this in there, accompanied by an experiment/dataset with multiple shortcuts (see for instance [1]).
I also appreciate the authors acknowledging the feedback on the writing and wish them good luck with a future revision. I would encourage them to continue working on this topic, I believe the general approach (quantifying linear information loss for many concepts) is definitely interesting.
[1] Li et al. 2022, https://arxiv.org/abs/2212.04825
Dear Reviewer mvD9,
Thank you for the careful and detailed review. We especially appreciate your suggestions for improving the exposition and your questions about the connection between our theoretical metric and practical downstream performance. We agree that these distinctions should be communicated more clearly.
On presentation and intuition. We agree that the abstract and introduction introduce too much technical terminology before explaining the practical question. We will restructure the presentation around a practical example: if gender information is removed from a representation, how much linear predictive information about occupation must necessarily be lost? Theorem 1 answers this question exactly at the level of optimal linear least-squares accessibility.
Our goal is also to bring the main theoretical results on linear concept erasure together in one framework. This connects the geometry of the removed subspace, the maximum retained accessibility achievable by any guarding map, the conditions under which this optimum is attained, and the edit cost and footprint of LEACE. In particular, LEACE is shown to attain a universal optimum on retained accessibility, while the Schur term measures the collateral loss that no linear guarding map can avoid.
We will make this progression more intuitive, define concepts, probes, and linear accessibility earlier, clarify notation for scalars, vectors, and matrices, and introduce the theoretical results through the practical questions they answer.
On the choice of R2 and downstream relevance. We agree that linear R2 is not equivalent to AUROC or logistic-regression performance. It is nevertheless the natural theoretical quantity for our setting because linear guardedness itself is defined through the inability of an affine least-squares predictor to improve on a constant predictor. This makes accessibility loss exactly characterizable, whereas downstream classification degradation depends on the probe, regularisation, and data distribution.
Our empirical contribution tests whether this exact quantity is nevertheless informative about downstream performance. In the expanded single-concept evaluation, accessible coupling predicts held-out AUROC degradation better than squared label correlation (pooled Spearman 0.459 versus 0.212, two-way 95% CI for the difference [0.201,0.294]). We agree that this is a statistical association rather than a downstream performance guarantee, and will make that limitation more prominent.
On the practical value of screening. We agree that retraining a few probes may be entirely feasible when only a small number of candidate erasures are considered. Our intended application is different: screening many candidate concept sets before selecting which interventions to evaluate. The full Schur expression provides an exact pre-erasure score for linear accessibility loss, allowing candidate sets to be ranked without repeatedly applying erasers and retraining downstream probes.
To test whether screening works beyond the extreme candidates originally reported, we have now evaluated 30 additional pre-specified interior CelebA triples across ten representations, adding 300 actual erasures. On these non-extreme sets alone, the pre-erasure Schur score achieves Spearman ρ=0.912 (95% two-way CI [0.76,0.96]) against held-out mean AUROC degradation, and ρ=0.909 (CI [0.77,0.96]) against balanced-accuracy degradation. The measured AUROC drop increases monotonically across five predicted-score bins, from 0.044 to 0.108. This provides stronger evidence that the score is useful for candidate prioritization beyond the initially tested extremes. We do not claim calibrated downstream magnitudes or that screening replaces direct evaluation of selected interventions.
On frozen versus fine-tuned representations. Our focus on frozen representations is deliberate: the problem is to remove an attribute from an existing representation before training a downstream predictor. The theoretical results apply equally to a representation obtained from a fine-tuned encoder, provided that representation is subsequently held fixed. We agree that an explicit fine-tuned-encoder experiment would be informative, but it is not included in the present evaluation. We will clarify this scope and avoid implying that the reported empirical results establish transfer to that setting.
We appreciate your suggestions and will use them to make the paper's central message clearer: the framework characterizes the unavoidable collateral loss of linear guarding, identifies LEACE as attaining the optimal bound, and provides a practical way to screen candidate erasures before evaluating their downstream consequences. The accessibility guarantees are exact, while their relationship to downstream classification performance is empirical and necessarily more limited.