acceptodds
Under review as a conference paper at ICLR 2027

Interpretable Discriminative Text Representations via Agreement and Label Disentanglement

Abstract

Interpretable text representations should be both predictive and usable as named constructs that independent analysts can apply. Existing approaches typically satisfy only one of these goals: anonymous discriminative bases predict well, while topic models and concept-bottleneck pipelines attach names to features but often validate interpretability only post hoc. We propose LFD (LLM-assisted Feature Discovery), which operationalizes interpretability through two testable properties: conceptual clarity, requiring a feature definition to transfer across independent annotators with high Cohen's , and label disentanglement, requiring the feature's meaning to be distinct from the target label. LFD discovers candidate features from contrastive, outcome-opposed text pairs, screens them using cross-LLM agreement, and selects only those that improve held-out residual predictive accuracy. We show theoretically that a -based clarity screen bounds annotation-noise-induced generalization-gap inflation; at the substantial-agreement threshold , the inflation factor is at most . Across ten text-classification tasks, LFD yields compact, named feature bases that approach full-embedding performance on rule-articulable tasks while using one to two orders of magnitude fewer dimensions. Human audits confirm that LFD features are substantially clearer and more label-disentangled than those from a leading LLM concept-bottleneck baseline: on iSarcasm, human-human agreement is for LFD versus for TBM, and human raters score LFD features as more disentangled on all ten tasks. These results suggest that interpretability in text representations can be made falsifiable through cross-rater reproducibility and explicit separation from the prediction target.

Primary area:
interpretability and explainable AI
Keywords:
interpretability, LLM, text features

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_qTsrOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

This paper introduces operational interpretability for text representations, focusing on conceptual clarity and label disentanglement. The authors propose LFD, an LLM-assisted feature discovery framework that generates candidate features from contrasting text groups, evaluates their consistency through cross-LLM agreement, and selects features based on incremental predictive gains.

Critical strengths

S1. Whether named features are truly interpretable is an interesting research direction.

S2. The proposed method is straightforward and well motivated.

S3. The human evaluation is a useful addition, especially the comparison of annotation agreement between humans and the model.

Critical weaknesses

W1. 177 of the 202 features are lexical, with only 25 semantic features. Could the higher human agreement be mainly due to these simpler rule-based features rather than the cross-LLM examination?

W2. The paper emphasizes label disentanglement, but the method does not explicitly check whether a feature simply restates the target label. Such features could still pass the selection process.

W3. LFD uses LightGBM, while TBM uses logistic regression. A comparison using the same classifier would help clarify whether the performance differences come from the extracted features.

W4. The human evaluation covers only three datasets, with typically three annotators and around 25 texts per feature. The limited evaluation scale makes it difficult to draw strong conclusions about conceptual clarity.

W5. The paper tests different LLMs, but does not repeat feature discovery under the same settings. It is hard to tell whether different runs would produce similar features and results.

W6. The agreement threshold is set to 0.70, but the sensitivity analysis is limited to CFPB. It would be useful to test different thresholds on other datasets as well.

Questions for authors

Please refer to Weaknesses

Reviewer_pnMAOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
2
Soundness
1
Read review
Paper summary

The paper tackles an important issue in current interpretable models for text classification: the automatic extraction of interpretable features from text and whether they are sufficiently coherent among texts and aligned to human perception to actually represent an interpretable feature. Stemming from social science best practices, which require agreement among different human annotators, they propose LFD, a pipeline translating this approach to LLMs by employing three different LLMs with three different roles -- feature proposer, labeler and examiner -- and computing the agreement of the latter two for LLM-measured features. They also propose a double stage for the feature proposer (bootstrap and residual), which allows extracting coherent features that are still useful for predicting the final task, while encouraging diversity and label disentanglement. Experiments against a few baselines, and in particular against TBM, show that the proposed method achieves similar accuracy while extracting features more consistently aligned to human annotators' labels (on three datasets).

Critical strengths
  1. Novelty and Impact: The paper proposes an interesting approach to the discovery and evaluation of textual features. The proposed evaluation is relevant for a wide variety of approaches like mechanistic interpretability and label-free CBMs, which are currently massively using automatic annotation of concepts and neurons. Furthermore, these approaches often evaluate them through qualitative metrics or metrics based on a single LLM annotating the "meaning" of a neuron: the proposed approach instead checks whether the annotation is coherent among different annotators (LLMs and humans) and whether it matches the values actually used by the model (e.g., the neuron behaviour). The agreement metrics, although they are an extension to LLM annotations of well-established protocols in social sciences, are novel in this context to the best of my knowledge.
  2. Incremental concept representation, coherent and aligned to human annotations: the proposed model builds the concept representation incrementally, with a heuristic but quite effective strategy: first extracting features that are capable of discriminating the classes and later extending them with complementary cues to improve the current basis, while retaining only the features on which two independent LLMs agree. As a result, the extracted concepts are more coherent and more aligned with human annotations than those of a related approach (TBM).
Critical weaknesses

Experimental Evaluation

  • The scientific validity of the experimental evaluation is low: the paper reports results on a single seed and on 200 test texts per dataset. This is absolutely not in line with the best practice in machine learning, which requires always validating an approach on several repetitions of the same experiment and reporting the mean and standard deviation for each experiment.
  • In the related work the authors position their method w.r.t. several concept-based approaches (both standard and label-free), but the only concept-based baseline in the experiments is TBM, a method only available as an arXiv preprint. This is highly insufficient to attest the validity of the proposed method w.r.t. methods already existing in the literature. Moreover, the comparison against TBM is not fair either: LFD employs a much more complex task predictor (LightGBM vs. logistic regression), and it also employs a different LLM for feature proposal. All the components of the pipeline not related to the concept/feature representations should remain the same to ensure a fair comparison.
  • The pipeline phases are not validated and remain quite heuristic, as no ablation study on the different phases introduced is reported (bootstrap vs. residual phase, examiner screen, contrastive sampling). The only evidence for the residual phase is Fig. 1, which reports test (rather than validation) accuracy on a single dataset (CFPB) and a single run, with variations within the range of noise expected on 200 test texts (standard error around 0.035).
  • No intervention experiment is reported, although it is mentioned among the motivations in the introduction (l.41–43).

I would suggest the authors reduce the number of datasets and substantially improve the quality of the experiments on a few selected ones, with multiple repetitions, real comparisons against concept-based methods (e.g., FELIX, CB-LLM), and possibly on datasets with already available concept annotations (e.g., CEBaB).

Questions for authors
  1. I don't get why the second LLM is called "examiner". To the best of my understanding, it does not see the other LLM's labels, right? Also, have you checked whether the two LLMs make similar errors? You write explicitly that "different model providers do not imply statistically independent errors" (Sec. 2, l.133–134).
  2. I think that the choice of CFPB for Fig. 1 is far from ideal, as the model predictions seem not to improve much when adding further features in the residual phase. Why have you chosen this dataset? Is the result similar in other datasets as well, so that the first (bootstrap) features already account for most of the final accuracy?
  3. The results in Table 8, Appendix E, seem to indicate that the lexical features discovered by LFD are the ones that are most aligned among human annotators (with the higher κ). Do you think that this is due to the automatic parsing? Also, is the automatic parsing something that can scale in your opinion? Could you also report the HM–P κ separately for lexical and semantic features?
  4. Is the PCA reported as baseline a standard PCA or the outcome-weighted one listed in Table 1?
Reviewer_KKiBOpenReview ↗
Rating
1
Confidence
3
Informativeness
2
Presentation
1
Soundness
1
Read review
Paper summary

The paper introduces operational definitions for feature interpretability for text classification: conceptual clarity and label disentanglement. Furthermore, they introduce a novel algorithm for textual concepts discovery, LFD. They measure conceptual clarity, label disentanglement as well as predictive utility for the concepts discovered with their method, and they compare it to a method from literature TBM. They show that LFD seem to provide better features for interpretability.

Critical strengths
  • The paper aims to introduce operational definitions of interpretability. This is very important to be able to at least measure interpretability.
  • Their method aim to provide textual rules, or definitions to define the textual interpretable features, which I agree are far more interpretable than simple annotations, e.g. from bottleneck models.
  • Human studies, not only LLMs, are used for evaluations.
Critical weaknesses
  • Overall clarity: the paper started well. I found it well motivated and I appreciated the connection to social sciences. However, afterwards it got more difficult to understand. Major concerns are:

    • Overall phrasing and structure are difficult to follow. Many parts are difficult to parse at first or second read in my opinion.
    • Some concepts, definitions or acronyms are not well introduced or explained. E.g. TBM is mentioned very early, but its literature reference appeared far from it. Proposer, Labeler and Examiner were not clear to me what they were when they were first mentioned.
    • Some mathematical definitions seem perhaps superfluous, e.g. notation at L. 157, and some are undefined, e.g. in Assumption 1.
  • Perhaps not really a weakness, but it is not clear to me how LFD was designed to actual improve on the operational definitions of interpretability. As it stands, the paper seems to propose the definition, and then propose a method, that just happens to be better on the two interpretability dimensions defined by the authors. More questions below.

  • Experiments and evaluation seems limited, especially in terms of baselines and statistical power.

  • I find the definition of label disentanglement in Definition 1, still quite vague and not operational.

Questions for authors
  1. As I said, I appreciate the operational definitions of interpretabilty. However, I think they would need more validation. Why I understand and at least partially agree that inter-rater agreement could make sense as a proxy for concept clarity, I believe that more validation is needed. Could the authors comment on this?
  2. I am wondering if label disentanglement is always needed. E.g. if an AG news text is classified as Sci/Tech, I think it is fair to allow tech companies to be mentioned in the feature. I guess it would depend on how disentanglement is actually defined. Could the authors expand on this?
  3. Table 2: wouldn't it be more fair if TBM and LFD had the same "width". Also, oftentimes model interpretability (which I understand is not in scope of this paper as mentioned in L. 257) depends on the number of features used. Table 2, AG News Sport, using 33 features while TBM used 2 seems excessive and perhaps overwhelming from a user interpretability perspective.
  4. From the experiment, it is left unclear whether the perceived conceptual clarity (assuming that it is correctly assessed, cf. question 1) is a result of the overall LFD algorithm, or just thanks to the particular phrasing used by the LLM. If TBM was simply extended by providing textual rules rather than simple annotation, would that weaken the author's result?

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.