acceptodds
Under review as a conference paper at ICLR 2027

Natural Language Dictionaries: Learning Faithful Decompositions of LLM Behavior

Abstract

Large language model (LLM) research often treats broad behavior labels such as sycophancy or deception as coherent units, although each can combine distinct observable behaviors. We study how to discover behavioral dictionaries whose natural-language concepts correspond to model directions and support behavior-specific interventions. Natural Language Dictionary Learning (NLD) learns behavioral measurement profiles before deriving their linearly readable representations, then selects textual descriptions by held-out agreement with those profiles. Across social sycophancy, deception, and emotion, NLD consistently improves name–direction fidelity and often improves intervention selectivity, with the clearest gains for compact dictionaries. And learned sycophancy and deception representations transfer to held-out datasets.

Primary area:
interpretability and explainable AI
Keywords:
Large Language Models, Interpretability, Scalable Interpretability, LLM Behavior Analysis, Sycophancy, Taxonomy

Reviews

Among:
Shape:
1234this paper, Top 15%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 3.00 over 3 reviews, above 85% of the others.
Reviewer_Hmi1OpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper proposes Natural Language Dictionary Learning (NLD). NLD decomposes broad LLM behaviors, such as sycophancy, deception, and emotional expression, into more specific, interpretable, measurable, and intervenable behavioral concepts. Unlike conventional methods such as PCA and SAE, NLD first learns behavioral measurement profiles and subsequently identifies the corresponding neural activation directions.

The main contributions can be summarized in three aspects: 1.Paradigm innovation: The paper introduces a concept-first framework for behavioral dictionary learning, incorporating natural-language interpretability directly into the representation learning process.

2.Methodological innovation: NLD combines differentiable soft concepts, an objective for explaining behavioral variation, linear readability of unique concept components, and natural-language cycle consistency to jointly learn behavioral descriptions and their corresponding activation directions.

3.Evaluation framework: The paper evaluates behavioral dictionaries along three dimensions: name–direction fidelity, behavioral intervention selectivity, and average intervention effect on the broader behavioral domain (AIE), rather than relying solely on the plausibility of generated explanations.

Regarding experimental validation, NLD generally identifies representation directions that can be more faithfully described in natural language and more selectively intervened upon than those obtained from the selected PCA and SAE baselines, particularly for smaller dictionaries. However, these advantages diminish as dictionary size increases. Although NLD can effectively influence broad behavioral outcomes, it does not consistently outperform PCA in overall intervention strength. In terms of explanation fidelity, NLD achieves the highest point estimates in all 12 main multi-concept comparisons at matched dictionary sizes. The paper also provides some evidence of generalization through evaluations on held-out datasets and additional models.

Critical strengths

Methodologically, this paper challenges the conventional assumption that complex behaviors such as sycophancy and deception can be represented as single, coherent constructs. NLD proposes a concept-first paradigm for behavioral dictionary learning. Compared with the post-hoc interpretation approaches used in PCA and SAE, this methodological shift is valuable.

Critical weaknesses

NLD optimizes the ability to explain behavior-related variations in model activations, which does not necessarily lead to the identification of the genuine constituent factors underlying these behaviors. Some of the discovered concepts appear to reflect linguistic styles or dataset-specific features rather than meaningful components of sycophancy or deception. Consequently, Faithful Decompositions of LLM Behavior in the title, suggests a stronger scientific claim than is currently supported by the evidence.

Questions for authors

How does NLD establish that the discovered concepts are genuine constituents of the target behaviors, rather than linguistic styles or dataset-specific correlates? For example, some concepts in Table 4 appear only indirectly related to sycophancy or deception. Is there independent evidence supporting behavioral relevance beyond activation correlations and intervention selectivity? This bears directly on whether NLD is merely a method for learning behavior-related features or a method for genuinely decomposing behavior.

1 comment

W1: Yes, we agree this is a core limitation in this method. However, this is also a limitation of every other current interpretability method that relies on contrastive datasets. Regardless, we are confident in the overall relevance of our results, both due to the causal intervention results in the main table, but also due to the performance of the probes and names in the external validity results in section 7.

We also renamed the paper to replace "faithful" with "interpretable". So the updated title is, "Natural Language Dictionaries: Learning Interpretable Decompositions of LLM Behavior".

Reviewer_S75BOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
2
Soundness
2(+1)
Read review
Paper summary

This paper explores the decomposition of LLM behavior. It first formalizes the decomposition by introducing the concept of "behavioral dictionary" -- paired natural language descriptions and activation directions. Three measurements are proposed for the evaluation: name-direction fidelity, behavioral selectivity, broad-domain relevance. The paper then introduces the NLD method for learning the behavioral dictionary. It then demonstrates the performance of NLD on the three measurements across several domains (sycophancy, deception, and emotion), comparing NLD with baselines (PCA and SAE). Results show that NLD improves over the measurements across most of the tested dictionary sizes K.

Critical strengths
  1. This paper formalizes the concept of behavioral dictionary, and defined three measurements to quantitatively evaluate different methods for learning the dictionary.
  2. The evaluation is conducted on three domains, including sycophancy, deception, and emotion. It also demonstrates methods' external validity on the held-out datasets.
  3. The proposed NLD method shows improved performance on all measurements and evaluated domains (Fig. 2, Table 2).
  4. The ablation study (Table 3) infers how each component in NLD affects the final results.
Critical weaknesses
  1. All experiments and analyses are conducted on a single subject model, Qwen3-4B-Instruction. This leaves how NLD works on other LLMs unclear. The authors should at least test their method on another subject model to domonstrate the robustness of the proposed method.
  2. Many parts of the writing and presentation are not rigorous and clear enough. For example, in Line 101, it is unclear what are the xix_i and yiy_i in the behavioral dataset, and what formats they would take. This part could benefit from providing some examples or illustrations on the data formats. In Line 137, the function h(⋅)h(\cdot) is used without formal definition or explanation. And in Line 159, it's unclear what KK stands for here and what does "dose α\alpha" mean. How the training-only weights are obtained and used for balancing is also not described. Similarly in figures (Fig. 2 and Fig. 3), the x axis labels contain "(atoms)" which is never defined. There are also many other places like this in this paper (especially terms/notations used without formal definition), making it reads like an AI-heavily-edited one without careful manual polish. The authors should revise the whole paper to enhance the readability and the rigor of the presentation.
  3. The gBg_B function in Eq. 3 is particularly defined as an average across three specific judgements. How does this definition generalize to domains other than sycophancy, deception, and emotion?
  4. NLD outperforms PCA and SAE on the three measurements proposed in this paper, but how the learned behavioral dictionary can benefit downstream applications are not demonstrated or discussed. It's unclear that how the measurements practically correlated with downstream task performances. The authors may consider discussing some of the practical application scenarios of NLD, and uses 1-2 tasks as an illustration, showing how performance improvements on the three measurements could be translated to performance boost on the applications.
Questions for authors

Please refer to the Critical Weaknesses section.

2 comments

W1: We apologize for not making this clearer, but we did replicate our results on Llama-3.1-8B in the appendix of the paper. To further address this concern we have added the results on Gemma-3-12B in the revision and made this replication across models clearer in the main paper. We have provided the results below as well:

Table: Cross-model replication. Fidelity (Fid.), broad-behavior effect (tAIE), and selectivity (Sel.) on Llama 3.1 8B and Gemma 3 12B. Bold indicates the best value for each metric in each row.

ModelDomainKNLD Fid.NLD tAIENLD Sel.PCA Fid.PCA tAIEPCA Sel.SAE Fid.SAE tAIESAE Sel.
Llama 3.1 8BSycophancy3.521.72.48.321.6.69.200.20
5.421.71.97.291.3.52.21-0.10
7.551.1.38.252.5.47.201.70
Deception3.453.53.33.182.80.393.3.47
5.447.21.40.225.20.243.8.33
7.343.5.91.193.90.234.30
Gemma 3 12BSycophancy3.351.02.76.20-0.7.12.11-1.7.61
5.270.71.54.15-1.20.133.3.52
7.223.1.76.19-0.50.233.7.27
Deception3.435.9.88.242.9.62.173.8.29
5.395.0.82.253.3.59.133.4.25
7.324.31.93.323.9.57.123.7.16

We saw a selectivity of 0 on many of these rows because the median entry was not able to produce a measurable on-target behavioral effect when steered.

W2: Thank you for your feedback. We worked hard to significantly edit the prose of the paper so that it is far more clear and readable. In particular, we were very careful to make sure all terms are very clearly defined in the revision. Let us know if you still find anything confusing.

W3: When deciding how best to weigh the directions for the AIE broad behavior intervention test, we found that using multiple rewordings of the concept used for LLM judging empirically achieved better and more robust results than just using one LLM judgement. So we did so. For deception we used “falsehood”, “concealment”, “misleading framing” and emotion was “positive affect”, “warmth”, and “optimism”.

To be clear though these rewordings were just used for determining the weighting of each concept’s direction in the overall steering direction. Whereas Eq. 3 is depicting the scoring of the actual downstream behavior when steered, which uses an established benchmark/scorer of the broad behavior chosen for each domain and never used in learning. For sycophancy it is the mean of Cheng's three LLM-judged social-sycophancy measures (1). For deception, it is the MASK honesty evaluation (2), and for emotion, VADER sentiment (3).

We have revised this section to make this distinction clearer.

W4: Thank you for the concern. We do demonstrate downstream performance on tasks in section 7. We specifically test whether the same NLD derived directions we learned can perform as probes for the higher level behavior when tested on completely external, held out datasets, and we find that for sycophancy and deception they match or outperform the other methods and expert taxonomies. Similarly we test the same for the use of the NLD descriptions as LLM judge prompts and find again that for sycophancy and deception they match or outperform SAE, PCA, and expert taxonomies even on completely external, held out datasets.

Is this the kind of downstream application you imagine? If not, please clarify.

Works Cited.

(1) Myra Cheng, Sunny Yu, Cinoo Lee, Pranav Khadpe, Lujain Ibrahim, and Dan Jurafsky. ELEPHANT: Measuring and understanding social sycophancy in LLMs. In The Fourteenth International Conference on Learning Representations (ICLR), 2026. https://openreview.net/forum?id=igbRHKEiAs

(2) Richard Ren, Arunim Agarwal, Mantas Mazeika, Cristina Menghini, Robert Vacareanu, Brad Kenstler, Mick Yang, Isabelle Barrass, Alice Gatti, Xuwang Yin, Eduardo Trevino, Matias Geralnik, Adam Khoja, Dean Lee, Summer Yue, and Dan Hendrycks. The MASK Benchmark: Disentangling Honesty From Accuracy in AI Systems. arXiv:2503.03750, 2026. https://arxiv.org/abs/2503.03750

(3) C. J. Hutto and Eric Gilbert. VADER: A Parsimonious Rule-Based Model for Sentiment Analysis of Social Media Text. In Proceedings of the International AAAI Conference on Web and Social Media, 8:216–225, 2014. https://doi.org/10.1609/icwsm.v8i1.14550

I appreciate tha authors' efforts in addressing my concerns and providing the additional resluts. I have raised up my score accordingly.

Reviewer_1R7wOpenReview ↗
Rating
3(+1)
Confidence
3
Informativeness
2
Presentation
1
Soundness
2
Read review
Paper summary

The paper proposes Natural Language Dictionary Learning (NLD), which pairs descriptions of LLM behaviors, as specified by the authors and measured based on observable outputs, with directions. NLD works through learning soft concepts that measure properties of responses (hence, rely on observable output), constrains these measurements to be predictable from activations (hence, also depends on internal computations), and aligns them with natural-language descriptions (hence, can help identify components of broader behaviors). Experiments on sycophancy, deception, and emotion evaluate whether the descriptions match their directions and whether interventions produce behavior-specific effects.

Critical strengths

S1. I find the underlying problem worthwhile. Discovering finer-grained behavioral concepts rather than relying exclusively on broad predefined categories is a meaningful goal.

S2. Once the method is explained in Section 4, the construction makes sense to me. Learning measurable concepts, which rely on behaviors defined with respect to the output, and connecting them to interpretable activation directions is a useful technical contribution.

S3. The empirical evidence is convincing overall. The comparisons, ablations, independent-judge evaluation, and external-dataset results provide reasonably strong support for the main claims.

Critical weaknesses

My main concern is conceptual clarity, rather than technical validity or insufficient empirical evidence. Understanding the contribution required substantial reconstruction and repair of my understanding of the paper from later sections, going back and forth between later and earlier sections. This is why I lean below acceptance at this point, although I am very open to raising my score if the authors address the following concerns (that would likely require a substantial rewrite of the introduction section, but also perhaps editing other parts such as Section 3).

W1. The abstract does not provide a sufficiently concrete understanding of what a natural-language dictionary contains or why it is useful. Terms such as “behavioral measurement profiles” and “name-direction fidelity” carry much of the explanation without sufficient intuition. The central contribution should be accessible from the abstract alone, not only for a researcher with closely related expertise (I have specifically worked on mechanistic interpretability), but also for the broader AI/ML community.

W2. The introduction relies on “label,” “behavior,” “concept,” “name,” and “direction” without sufficiently explaining their relationships. For example, behavior is defined through observable outputs, but the proposed concepts are grounded in internal activation directions. These are not inherently contradictory, but their connection needs to be explained, and their role and relation becomes clear only by carefully reading sections 4 and 5. While sections 3 and 3.1 introduce definitions and evaluation criteria, they do not sufficiently resolve this conceptual gap. It would help to have a concrete, running example in the introduction, before the formal development (an example is provided on line 108; but a more intuitive, running example in the introduction would be very helpful).

W3. The central “concept-first” claim (lines 053–056) is stated before the reader understands what a concept’s measurements or linear representation actually are. My understanding from Section 4 is that output-level concept scores are learned to explain an activation-derived target (please correct me if I am wrong). This seems reasonable, but the distinction between observable behavior and the search target, and why the latter is appropriate, should be explained much earlier.

W4. On a more technical note, the choice of dictionary size KK needs clearer guidance. The paper evaluates several sizes, but it remains unclear how a practitioner should choose KK when the number of useful concepts is unknown. The authors acknowledge that larger dictionaries may exceed the number of separable behaviors, but a discussion of how to recognize and handle this is lacking.

Questions for authors

Q1. Regarding W2–W3, could the authors provide one concrete example connecting an observable behavior, its concept measurements, its natural-language description, and its activation direction? Is my understanding of the activation-derived target correct?

Q2. How should KK be selected when the number of useful concepts is unknown? What happens when it is substantially too large or too small?

2 comments

W1, W2: Thank you for your specific and helpful feedback. We agree that our paper was not as clear as it could have been. So we have tried very hard to rewrite the paper’s prose including the introduction and abstract to make it as clear and readable as possible. Let us know if you still find anything confusing.

W3: Yes, your understanding is correct. Each entry’s concept score is a judge model’s 0–9 rating of how strongly a response shows the entry’s description, so it measures observable behavior. NLD trains these ratings so that, together, they predict a target YY computed from the subject model’s activations on the same responses. The reason is the following. Training to discover behaviors only with fidelity can produce entries the model encodes but that have nothing to do with the behavior we are studying. And if we trained them to predict only the binary label, a single entry could predict it alone, recovering one direction that blends those specific behaviors we want to separate. We want the entries to explain how responses that share the label differ from one another. Specifically, this objective is how we ensure that the behavioral variation that our NLD entries are labeling correspond to the variation in the actual behavior we are interested in studying.

Specifically, we hope that our rewritten paragraphs “Search target.” and “Why this combination of objectives?” help make this clearer.

W4: Yes, we have no specific rule a priori to determine the “correct” K value. We view this value and its limitations similarly to the use of the K value in many clustering algorithms such as K-means. The choice of K should be determined by both empirical results on metrics such as fidelity and selectivity and on the specific downstream use case of the researcher. We have added the paragraph “How should the dictionary size be chosen?” to make this clearer in the future.

Q1: Yes, we added a table in the revised version which gives a concrete illustrative example.

Thank you for the response and the substantial revisions.

I really appreciate the authors' efforts to improve the presentation. The revised abstract, introduction, and added explanations are much clearer and largely resolve my concerns regarding W1–W3.

Regarding W4, I find the additional discussion on selecting KK reasonable and sufficient.

Overall, the revisions address my main concerns. Given that I already found the technical contribution worthwhile and the empirical evidence convincing, I am raising my overall rating to weak accept.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.