- Rating
- 2
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 2
- Soundness
- 2
Read review
Paper summary
The paper turns self-knowledge into a prediction test. Across 9 behavioral evaluations, it measures how model behaves under different conditions, make predictions, and compare them with controls that remove the self from the question.
Critical strengths
-
The finding that models understate harmful behavior is interesting
-
The author includes comprehensive ablations of different phrasings and prompt variants and evaluated on a wide range of models
Critical weaknesses
-
The authors did not explain or justify how they chose the 9 evaluation tasks. Why are these tasks particularly important for models to be self-consistent and predict their behaviors? What are the harms or safety risks of low prediction rates?
-
The finding that item-informed self-prediction (model performs better when it sees the exact evaluation question) seems very intuitive and trivial. The difference in prediction between this condition and the abstract description might arise from the model’s lack of understanding of the prediction task. It would be helpful to run a small-scale experiment to test a model’s understanding of the abstract task (e.g. what an example task looks like) as a sanity check.
-
There’s no discussion of how the work fits under the broader self-consistency and introspection literature and whether the findings are similar or different from self-consistency/faithfulness evaluations for factual questions and reasoning tasks. Would also be helpful to engage with these papers: https://arxiv.org/pdf/2511.08579
-
Training models to improve self-knowledge is an important section but is underexplored. The author tried the method in Binder et al. (2025) but did not explore other possible training methods (e.g. https://arxiv.org/pdf/2606.32038). It would strengthen the paper if the authors could expand this section.
Questions for authors
-
How did you decide on the 9 evaluations? How do you make sure they are comprehensive?
-
Is there any justification for how the 15 different prediction methods are selected? It would be helpful to connect to existing literature on self-consistency (e.g. https://arxiv.org/abs/2606.18327), faithfulness, and introspection (e.g. https://arxiv.org/abs/2603.05414)
-
Could you provide an example of the three question forms in the main text? Otherwise it’s difficult to know what they look like and how the questions are framed.