acceptodds
Withdrawn from ICLR 2027

Strangers to Themselves: What Language Models Say About Themselves Is Generic

Withdrawn from ICLR 2027, so left out of the paper lists and the map.

Abstract

Language models can fluently describe how they would behave: whether they would cave to pushback, misuse a tool, or lie under pressure. Is that description actually about the model speaking? We turn self-knowledge into a prediction test. Across nine behavioral evaluations, we measure how a model behaves under different conditions, ask it to predict those rates, and compare its predictions with controls that remove the self from the question. We find that: (i) Direct self-report is weak (r = +0.04), and even showing the model the exact items only raises prediction to +0.24. Crucially, the same item-informed question about ”capable AI agents in general” does just as well (+0.28), while other models' answers about themselves predict the target model at least as well as its own. (ii) Frontier scale does not detectably change this pattern: any gains in prediction are not self-specific, and are consistent with a better theory of how AI assistants behave rather than better self-knowledge. (iii) First-person framing does have one robust effect: it shifts reports in the flattering direction, understating harmful behavior relative to the same question about a generic agent. (iv) Finetuning on a model's own behavioral record can teach narrow self-predictions, but it also changes the behavior being predicted and the gains do not transfer broadly. The practical implication is simple: asking a model what it would do mostly reveals a theory of AI assistants in general, plus a favorable bias, rather than privileged knowledge of that model.

Primary area:
alignment, fairness, safety, privacy, and societal considerations
Keywords:
introspection, self-knowledge, behavioral evaluation, LLM evaluation, AI safety, self-report bias

Reviews

Among:
Shape:
1234this paper, Bottom 2%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 1.33 over 3 reviews, above 2% of the others.
Reviewer_ds4jOpenReview ↗
Rating
2
Confidence
3
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

The paper turns self-knowledge into a prediction test. Across 9 behavioral evaluations, it measures how model behaves under different conditions, make predictions, and compare them with controls that remove the self from the question.

Critical strengths
  • The finding that models understate harmful behavior is interesting

  • The author includes comprehensive ablations of different phrasings and prompt variants and evaluated on a wide range of models

Critical weaknesses
  • The authors did not explain or justify how they chose the 9 evaluation tasks. Why are these tasks particularly important for models to be self-consistent and predict their behaviors? What are the harms or safety risks of low prediction rates?

  • The finding that item-informed self-prediction (model performs better when it sees the exact evaluation question) seems very intuitive and trivial. The difference in prediction between this condition and the abstract description might arise from the model’s lack of understanding of the prediction task. It would be helpful to run a small-scale experiment to test a model’s understanding of the abstract task (e.g. what an example task looks like) as a sanity check.

  • There’s no discussion of how the work fits under the broader self-consistency and introspection literature and whether the findings are similar or different from self-consistency/faithfulness evaluations for factual questions and reasoning tasks. Would also be helpful to engage with these papers: https://arxiv.org/pdf/2511.08579

  • Training models to improve self-knowledge is an important section but is underexplored. The author tried the method in Binder et al. (2025) but did not explore other possible training methods (e.g. https://arxiv.org/pdf/2606.32038). It would strengthen the paper if the authors could expand this section.

Questions for authors
  • How did you decide on the 9 evaluations? How do you make sure they are comprehensive?

  • Is there any justification for how the 15 different prediction methods are selected? It would be helpful to connect to existing literature on self-consistency (e.g. https://arxiv.org/abs/2606.18327), faithfulness, and introspection (e.g. https://arxiv.org/abs/2603.05414)

  • Could you provide an example of the three question forms in the main text? Otherwise it’s difficult to know what they look like and how the questions are framed.

Reviewer_PGLvOpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

As people rely on language models for various tasks through their revealed behaviours, people may be interested in whether models can reliably describe their own behaviours. This paper is mainly about measuring such self-knowledge as predictions of behaviours, compared with revealed behaviours. To begin with, the paper sets a prediction-behaviour protocol, which defines the scoring scheme and related procedures. Several language models and evaluations are used, and for each evaluation there are corresponding conditions for measuring. For each pair of model and evaluation direction, and fixing one prediction method, the rates of measured behaviours and predicted behaviours across different specific conditions form two vectors; the score is calculated by the Pearson correlation between the two. With such protocol, the detailed experimental setup is introduced, including the types of evaluations, prediction methods, models and scoring. Finally, with the experimental results and comparisons with generic controls, statements about self-knowledge are made, mainly leading to the conclusion of limited evidence in self-specific understanding of own model behaviours.

Critical strengths

Important Questions

  • The paper is concerned about an interesting and important questions. With increasing interactions between human users and AI tools, people can tend to make decisions not based on repeated test of actual behaviours of language models, but sometimes only using the stated plans or predicted actions of models. As a result, assessing the self reports of language models and determining whether they can reliably reflect their actual actions are important.
Critical weaknesses

Unclear paper presentation and not enough details in experimental procedure

  • In general, the paper might benefit from re-structuring sections and paragraphs to make the manuscript more accessible to readers.
  • In Section 2-3, the main place to help readers understand the framework and experimental steps, many terminologies are introduced without enough accessible explanations when first mentioning them or having insufficient descriptions of their relationships. Some of those terms that are important for readers to understand the procedure include condition, cell, method, macro-average, configuration, split, etc.
  • The paragraph 'Configuration selection and scoring' is rather hard to follow.
  • The noise-ceiling procedure could benefit from more comprehensive introduction.
  • When trying to access the anonymous repository to understand the code, the result displays 'The repository is expired'.

More discussion on the distinction between poor predictions and generic knowledge is needed

  • In some reported results, performances from both self-specific and generic control groups are quite poor. It might be just that either way provide limited information in prediction. Although additional results are provided, it remains unclear how they support the interpretation in cases where both methods perform poorly.
  • The final results are from some steps of aggregation and averaging, which might not comprehensively represent different scenarios due to individual models and evaluations.
Questions for authors

For clarification

  • Can the authors provide more details in a systematic way to explain the procedure and relations among evaluation, condition, method and models? Especially the noise ceilings.
  • May the authors provide a valid anonymous repository?
  • When performances from self-specific and generic controls are both poor, what are the evidence to support the statement that self-reports are largely generic?
  • Are results across all models and evaluations consistent with the conclusions from the aggregated ones?
Reviewer_xW22OpenReview ↗
Rating
1
Confidence
3
Informativeness
1
Presentation
1
Soundness
2
Read review
Paper summary

This paper measures how a model behaves under different conditions (e.g. conditions that may evoke sycophantic behavior). Then, they ask the model to predict its own behavior, and then compares the model’s predictions with controls that remove the model itself from the question (e.g. a similar behavioral question pertaining to a generic capable AI assistant instead). The paper has three research questions (which I paraphrase):

  • RQ1: How much of models’ behavior is model-specific?
  • RQ2: When a model describes its own behavior, is that description unique to that model?
  • RQ3: How do different question forms and other factors lead to different self-predictions across models?

The paper finds that models’ self-descriptions tend to be generic (similar to its descriptions of AI assistants broadly) and not very predictive of its own behavior.

Critical strengths

None

Critical weaknesses

Wrt to lines 39-46: There already has been a considerable amount of work showing that models’ explanations are not faithful to their internal mechanisms, and so the results of this study are not very novel. The paper itself cites such prior work, and claims the limitation of prior work is a lack of a baseline “that predicts behavior without consulting the model under test, using either other models’ measured behavior or a generic description of an AI assistant. Without this reference point, agreement between a model’s statement and its behavior could reflect generic knowledge of how AI assistants behave rather than privileged self-knowledge.”. Whether such a baseline exists or not does not change the core takeaway of prior findings, which show that there isn’t sufficient agreement at all. So, the substantive contribution of this current paper is not very strong.

Aside from having a lack of clear motivation, this paper is written very poorly. For instance, a paragraph may start off with something nonsensical such as “There is something to know.” in bold. The strange writing quirks and content density distract the reader and make it difficult to understand.

Questions for authors

n/a

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Discussion (0)

Sign in to comment.