- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
This paper introduces operational interpretability for text representations, focusing on conceptual clarity and label disentanglement. The authors propose LFD, an LLM-assisted feature discovery framework that generates candidate features from contrasting text groups, evaluates their consistency through cross-LLM agreement, and selects features based on incremental predictive gains.
Critical strengths
S1. Whether named features are truly interpretable is an interesting research direction.
S2. The proposed method is straightforward and well motivated.
S3. The human evaluation is a useful addition, especially the comparison of annotation agreement between humans and the model.
Critical weaknesses
W1. 177 of the 202 features are lexical, with only 25 semantic features. Could the higher human agreement be mainly due to these simpler rule-based features rather than the cross-LLM examination?
W2. The paper emphasizes label disentanglement, but the method does not explicitly check whether a feature simply restates the target label. Such features could still pass the selection process.
W3. LFD uses LightGBM, while TBM uses logistic regression. A comparison using the same classifier would help clarify whether the performance differences come from the extracted features.
W4. The human evaluation covers only three datasets, with typically three annotators and around 25 texts per feature. The limited evaluation scale makes it difficult to draw strong conclusions about conceptual clarity.
W5. The paper tests different LLMs, but does not repeat feature discovery under the same settings. It is hard to tell whether different runs would produce similar features and results.
W6. The agreement threshold is set to 0.70, but the sensitivity analysis is limited to CFPB. It would be useful to test different thresholds on other datasets as well.
Questions for authors
Please refer to Weaknesses