acceptodds
Under review as a conference paper at ICLR 2027

Do LLM Self-Ratings Depend on the Internal Representations They Track?

Abstract

Self-rating scales designed for people are increasingly administered to large language models (LLMs) to assess their psychological states, yet whether such self-ratings measure anything inside the model has rarely been tested. Under the causal theory of measurement, a self-rating measures a state only if variation in the state produces variation in the self-rating; a correlation is not enough, because both may depend on a common cause. Because many concepts correspond to linear directions in LLM activations, this dependence can be tested by editing those directions. We study six emotions and six mental-health-related constructs in Llama-3.1-8B-Instruct and Olmo-3.1-32B-Instruct, using 24 scenario templates in which a single quantity, such as an exam pass rate, sets the intensity of the situation and the model rates its state on a 0–100 scale. Construct vectors extracted from independent stories pass the usual checks: their readouts at the final prompt token correlate closely with the self-ratings (mean within-template r = 0.899 and 0.791), and adding them shifts open-ended text toward the target state. Yet setting the construct readout to its value in the opposite-intensity scenario leaves the self-rating essentially unchanged: across 384 edits it never moves by more than 0.9 points, against natural score gaps of 30 to 41 points, whereas replacing the full hidden state at 75% depth transfers most of the gap (medians 91.9% and 99.7%). The self-ratings thus track these construct representations without depending on them, contrary to what measurement requires. Correlation and steering therefore do not establish that an LLM self-rating measures the state it names; until such dependence is shown, self-ratings are better treated as outputs of a rating task than as measurements of internal states.

Primary area:
interpretability and explainable AI
Keywords:
Human Simulation, Mechanistic Interpretability, Psychometrics, Large Language Models, Representation Probing, Measurements

Reviews

Among:
Shape:
1234this paper, Top 30%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.67 over 3 reviews, above 70% of the others.
Reviewer_mX7nOpenReview ↗
Rating
3
Confidence
3
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The paper asks whether numerical LLM self-ratings causally depend on the internal construct representations with which they correlate. It studies 12 emotional and mental-health-related constructs using 24 scenario templates in Llama-3.1-8B-Instruct and Olmo-3.1-32B-Instruct. Construct directions are extracted from independent stories and evaluated through readout–rating correlations and qualitative steering of open-ended responses. The central intervention holds the prompt fixed while matching a construct readout to its value under the opposite endpoint of the same scenario. Across 384 such edits, no self-rating changes by more than 0.9 points, despite average natural score gaps of approximately 30–41 points. In contrast, full-state replacement at 75% depth transfers median fractions of 91.9% and 99.7% of the natural gaps. Random-direction comparisons, dose diagnostics, and gradient analyses support the dissociation. The contribution is an empirical demonstration that correlation and steering do not establish natural-magnitude dependence on these candidate representations, motivating stronger causal validation before interpreting self-ratings as internal-state measurements.

Critical strengths
  1. The intervention controls make the near-zero effects interpretable. Across all 384 matching edits, score changes remain below 0.9 points, compared with natural gaps averaging approximately 30–41 points. At 75% depth, full-state replacement transfers median fractions of 91.9% and 99.7% of those gaps. Together with accurate readout matching, sham interventions, and checks on the full numerical answer distribution, this establishes a substantial dissociation between changing the tested construct readout and changing the information that determines the rating. This controlled comparison is the strongest basis for acceptance (Figure 4; Tables S5–S6).

  2. The paper turns a methodological concern into a concrete empirical test. It separates correlation across natural inputs, the ability of large vector additions to alter outputs, and dependence under interventions that match naturally occurring readout values. The candidate directions are extracted from independent stories, and the matching targets come from the opposite scenario endpoint rather than optimization to produce a desired rating change. This makes the test informative about whether these independently defined readouts support the proposed measurement interpretation. The high correlations obtained by random directions and the deterioration of numerical output format under some large edits give the distinction practical substance. Even if the candidates do not recover the representations actually used for rating, the results demonstrate why correlation and qualitative steering are insufficient validation.

Critical weaknesses
  1. The validity and specificity of the candidate construct directions remain uncertain. Apart from happiness, pairwise direction cosines at 75% depth are approximately 0.65–0.95, and another construct's direction often tracks the queried rating at least as well as its own. These findings are compatible with substantial shared affective or narrative content. The open-response steering validation is qualitative and uses larger, repeated interventions across generated positions, whereas the central rating intervention edits one position. Consequently, the validation does not establish that the edited direction is the construct representation used during natural rating computation. Full-state replacement transfers all information in the donor state, potentially including scenario and answer-related information; it cannot establish construct-specific mediation or show that a comparably small, low-dimensional edit can transfer the rating. The acknowledged absence of a single-direction positive control therefore matters. These limitations weaken the interpretation that the model bypasses a valid construct representation, while leaving the narrower critique of correlation and steering intact.

  2. The intervention sites and task design limit identification of the overall reporting mechanism. The main interventions edit only the final prompt token at 50%, 75%, and final depth. A construct could have influenced the rating earlier, with its information subsequently transformed or copied into other representations, making later edits to the tested readout ineffective. Thus, the results do not uniquely identify parallel report generation. In addition, the evidence covers two models and 24 templates that explicitly ask the model to imagine being an adult and vary one salient quantity, potentially favoring direct scenario-to-score computation. The operationalization also matters: matching a cosine readout need not match the donor's raw projection and generally changes the edited state's norm. These choices warrant a targeted robustness comparison and clear limits on generalization. The results should not be interpreted as showing that all LLM self-reports lack internal grounding, or as settling questions about introspection or subjective experience.

Questions for authors
  1. Clarification of causal scope. Is the intended conclusion restricted to insensitivity to natural-magnitude edits of the tested directions at the specified answer positions? Which existing result, if any, distinguishes parallel report generation from a mechanism in which the construct influenced computation earlier and its information was subsequently transformed or copied? This distinction materially affects my assessment of the mechanistic claims.

  2. Additional evidence relevant to the current assessment: a low-dimensional positive control. Could the authors provide a rating direction or low-dimensional subspace learned on independent data and evaluated on held-out templates, with interventions at the same position and comparable edit norms? A substantial transfer under this control would strengthen the interpretation that the small effects are specific to the construct directions, rather than a limitation of the intervention protocol's ability to capture low-dimensional causal dependence.

  3. Additional evidence relevant to the current assessment: projection matching. Equation (3) matches the target cosine but generally does not match the target projection onto the construct direction. Do the conclusions persist when directly replacing that projection, using h′=h+(v⊤htarget−v⊤h)vh'=h+(v^\top h_{\mathrm{target}}-v^\top h)v? Please also report perturbation magnitudes and changes in hidden-state norms for the two procedures. This comparison would clarify whether the near-zero effects depend on the particular cosine-based operationalization.

These questions concern uncertainties in the present evidence. Broader model coverage and evaluation with full psychological questionnaires would be useful future work, but are not prerequisites for my recommendation.

4 comments

Dear Reviewer,

Thank you for your careful assessment and constructive suggestions. Your comments have helped us clarify the causal scope of our findings and strengthen the experimental controls. We address your questions below.

Q1. Causal scope and earlier computation

Thank you for highlighting the importance of when construct information contributes to the rating. Our method tests whether changing an independently identified construct readout to its value in another natural scenario, while holding the prompt fixed, changes the self-rating. Several results bear on your earlier-computation explanation:

Full-state replacement at 75% depth transfers 91.9% and 99.7% of the natural score gap in Llama and Olmo (medians). The new single-direction control below also substantially changes ratings at this position. Matching at 50%, 75%, and final depth, separately and jointly, produces changes below 0.9 points across all 384 edits. At every answer-position layer, the final expected-score gradient is backpropagated through all subsequent layers. A local contribution along an earlier construct direction therefore remains included even if its information is later transformed or copied. The low construct–gradient alignment across layers complements the finite interventions.

Together, these results constrain earlier-computation explanations operating through the tested directions at the answer position. They support the conclusion that these readouts track ratings without substantially controlling them under natural-readout matching.

The current results do not fully rule out mechanisms involving untested directions, other token positions, or nonlinear/redundant representations. We have clarified this causal scope in the revised manuscript.

Q2. Single-direction positive control and matched edit norms

Thank you for suggesting this additional control. We have added a rating-relevant direction for each construct and model at 75% depth, learned from ten independent scenario templates using normalized high-versus-low rating quartile differences. We evaluated the same readout-matching procedure at the same answer position on two held-out templates.

The rating-relevant directions produced mean absolute rating changes of 28.28 points in Llama and 20.42 in Olmo, compared with 0.057 and 0.147 points, respectively, for the story-derived directions. After enlarging each story-direction edit to match the norm of the corresponding rating-direction edit, the mean absolute changes remained much smaller: 0.90 points in Llama and 1.35 in Olmo.

Rating-direction effects followed the natural rating difference in 48/48 conditions in Llama and 42/48 in Olmo. Thus, the protocol can detect substantial single-direction causal influence, and edit magnitude alone does not explain the contrast. This control provides some evidence for the directions' causal relevance to ratings, rather than establishing construct-specific mediation.

We have incorporated this control and the norm-matched comparisons into Section 4.4 and Appendix D.2 (Table S9).

Q3. Direct projection matching

Thank you for raising this concern about the matching procedure. Following your suggestion, we have repeated all 384 conditions using direct projection replacement. The results are as follows.

Mean absolute rating changes remained small: 0.059–0.196 points in Llama and 0.043–0.173 in Olmo.

At 75% depth, median relative edit norms were comparable between cosine and projection matching: 1.524% versus 1.555% in Llama, and 2.537% versus 2.542% in Olmo. The magnitudes of relative state-norm changes were also comparable: 0.0151% versus 0.0152% in Llama, and 0.1762% versus 0.1703% in Olmo, with cosine matching listed first throughout.

The weak rating effects therefore persist under raw-projection matching, with comparable perturbation magnitudes and state-norm changes in the reported comparison.

W1. Representation quality

We appreciate your careful comments on the validity and specificity of the construct directions. To address this concern, we have added held-out discrimination tests using 1,200 stories spanning six emotions and ten topics. At 75% depth, within-topic macro-AUROC was 0.776 in Llama and 0.809 in Olmo. Using the same English-derived directions on Chinese, German, French, and Spanish translations yielded AUROCs of 0.775–0.787 and 0.796–0.812, respectively. These tests provide direct evidence of emotion discrimination despite the shared components suggested by the high direction cosines.

We have included these discrimination and cross-language evaluations in Section 4.2 and Appendix B.2.

W2. Intervention sites and task scope

Thank you for emphasizing the importance of intervention sites and task design when interpreting the findings. The experiments test two models and 24 controlled templates that ask the model to imagine being an adult and vary one salient quantity. This design enables controlled causal comparisons but may favor direct scenario-to-score computation. We have explicitly limited our conclusions to this setting in the Discussion (Section 5), without generalizing to all self-reports or subjective experience.

Regarding intervention timing, our multi-depth interventions and all-layer gradient analysis constrain earlier-computation explanations through the tested answer-position directions, as discussed in Q1. The additional projection-matching results reported here in Q3 address the cosine-based operationalization. More complex scenarios, longer conversations, and broader model coverage remain directions for future work.

Thank you for the detailed response and the additional experiments. I appreciate the low-dimensional positive control, the direct projection-matching experiments, and the additional validation of the construct directions. These additions address several of my concerns and strengthen the evidence supporting the paper.

My main remaining concern is the construct validity of the extracted representations, which is also reflected in Reviewer 9deK’s comments. A direction can correlate with self-ratings and influence emotional expression while still being an incomplete or lossy representation of the construct. If the construct-related information used in rating computation was not captured by the extraction method, the small intervention effects could reflect this limitation rather than the model bypassing the construct representation. Therefore, the results do not fully establish that self-ratings are independent of the relevant emotional constructs, or that parallel report generation is necessarily the correct explanation.

At the same time, I recognize that this identification problem is difficult to resolve conclusively. In human psychology, anxiety can be studied through self-reports, behavior, and physiological responses, yet these measures do not always agree. A related difficulty arises when identifying constructs in AI systems. I therefore would not make definitive proof that an extracted direction is the “true” construct representation an unrealistic requirement for acceptance. I regard this as an important limitation on the interpretation of the findings.

Taking the additional evidence and this remaining limitation into account, I will maintain my original rating of 3 (Weak acceptance). Thank you again for the thoughtful response and the revisions.

Dear Reviewer,

Thank you very much for your thoughtful feedback and continued positive assessment of our paper. We sincerely appreciate your careful consideration of our additional experiments and your recognition of their contributions.

We agree that identifying the representations actually involved in rating computation remains an important open challenge. Your comments have helped us better clarify the scope and limitations of our findings, and we plan to make stronger construct validation a key direction of our future research.

Thank you again for your time, encouragement, and constructive suggestions, which have greatly helped us strengthen our work!

Reviewer_1WKqOpenReview ↗
Rating
3
Confidence
3
Informativeness
3
Presentation
3
Soundness
2
Read review
Paper summary

This paper investigates whether LLM self-ratings of emotions and mental-health-related constructs actually depend on the internal representations of these constructs. The authors study 12 constructs across two LLMs, using controlled scenarios in which a numerical variable changes the intensity of the situation. They first identify internal construct vectors and show that their readouts are strongly correlated with the models' self-ratings. They then introduce a readout-matching intervention, where they modify the hidden states to match the construct readout of another scenario while keeping the prompt unchanged. They find that these interventions have almost no effect on the self-ratings, while replacing the entire hidden state produces large changes. Additional gradient analyses support these findings. The authors argue that strong correlations between internal representations and self-ratings do not necessarily mean that the ratings causally depend on these representations.

Critical strengths
  • The paper addresses an interesting and important question about how LLMs generate self-ratings and whether these can be considered measurements of internal states. The paper is very clearly written, and I particularly liked how the authors introduce the two hypotheses and explain what their experiments can and cannot show. The results are presented carefully, without making overly strong claims on a topic where drawing conclusions is not straightforward.

  • The experiments are carefully designed, with several controls and additional analyses to support the findings. The combination of readout matching, full-state replacement, and gradient analysis gives evidence from different angles, each helping examine the main question from a different perspective. I also appreciated the discussion of limitations (in general, highlighted throughout the Discussion section, not only under the Limitations paragraph). Most of the concerns I had noted while reading the paper were eventually acknowledged by the authors themselves, which I think adds to the quality of the work.

Critical weaknesses
  • While the experiments are carefully designed, some aspects of the methodology could affect how we interpret the results. My main concern is whether the extracted construct vectors actually capture the internal representations that the models use when generating their self-ratings. If the relevant information is encoded elsewhere, or across several dimensions, the lack of an effect from the interventions would not necessarily mean that the ratings do not depend on internal representations of the constructs.

  • I also think that the negative results would be more convincing with a positive control showing that the proposed intervention can successfully identify a known causal relationship.

The authors already acknowledge some of these limitations, but I think they are important because they could affect the interpretation of the main findings. I have included more specific questions about these points below.

Questions for authors
  1. Is it possible that the self-ratings depend on other representations of the same constructs that your method does not identify? As you also mention, the high correlations between different construct vectors and the correlations with random directions raise some questions about what these vectors actually capture. Your full-state replacement experiments show that changing the hidden state at the selected position can strongly affect the self-rating. Could you try to identify a smaller subset of dimensions, or a direction/subspace, that actually influences the rating, and check whether it is related to the respective construct? I understand that the gradient analysis already looks at which directions affect the ratings, but have you checked whether these directions also capture the corresponding constructs? This could help clarify whether the model does not rely on its construct representations, or whether the extraction method simply misses the relevant ones.

  2. Have you tested your readout-matching method in a case where we already know that a particular internal representation affects the model's output? The full-state replacement experiment is a useful control, but it would also be interesting to see whether your method can identify a known causal relationship. Since the main results are negative, I think such a test would make them more convincing, showing that the method had the potential to "steer" the model's behavior, but in this particular case, it doesn't because the model doesn't rely on these representations. Without a positive example, there is the question whether the method cannot deliver what we expect in general.

  3. Could it be that both hypotheses are valid, depending on the situation? For example, in relatively simple scenarios like the ones used here, the model might calculate the rating directly from the prompt, while in more complex scenarios it might rely on its internal representations. Have you considered testing this with more complicated scenarios, or with longer conversations? I understand that the simple templates make the experiments easier to control, but I wonder whether the findings would hold in other settings.

  4. How did you select the ranges and step sizes for the variables in Table S2? Also, did you check whether changing X actually produces the expected change in the corresponding construct across the whole range? For example, having 95 or 100 weeks before a GRE exam might lead to essentially the same anxiety, since both are very far away. Similarly, a teacher might find a 100% exam pass rate suspicious, rather than becoming happier. So the relationship might not always be monotonic, especially for extreme values. Most of the scenarios seem reasonable, and I don't see this as a major issue, but I was wondering whether you verified these assumptions somehow, perhaps through human judgment or another independent evaluation.

Additional Comments:

  • A minor comment is that maybe the authors should consider adding a mention of emotions, internal/psychological states, or something related in the title, as the title is too general (about self-rating), while the paper is focused on a very specific topic. With such a title, I would expect to see validation on different setups beyond the ones presented here.

  • I would also suggest having a look at the paper "The Grounding Gap: How LLMs Anchor the Meaning of Abstract Concepts Differently from Humans" by Chlapanis et al. (NeurIPS 2026). The paper uses psycholinguistic experiments and mechanistic interpretability to study how LLMs represent and use information related to abstract concepts. In particular, it finds cases where models appear to have access to certain grounding information, but do not necessarily use it during generation. I think this is related to your distinction between having an internal representation of a construct and actually relying on it to generate an output, and could be worth discussing.

My overall impression of the paper is very positive; I just have certain important concerns that prevented me from directly giving it an "Accept" (4) score. However, I am very happy to discuss with the authors and, if these concerns are addressed, raise my score accordingly.

6 comments

Dear Reviewer,

Thank you for your positive assessment of our paper and your detailed methodological suggestions. Your questions have helped us further clarify the distinction between “directions associated with a construct” and “directions that can causally influence ratings.”

Q1. Identifying rating-relevant directions

Thank you for highlighting the need for a single-direction positive control. We agree that full-state replacement alone does not establish the effectiveness of readout matching along an individual direction.

We have therefore added a rating-relevant direction for each construct and model, extracted from ten new scenario templates, as a single-direction positive control. We focus here on the results at 75% depth. Within each new template, we computed the mean hidden-state difference between the highest and lowest self-rating quartiles, normalized these differences, and averaged them equally before final normalization. This captures state changes associated with higher ratings across scenarios.

Q2. Single-direction positive control

On the two held-out templates, we applied the same readout-matching procedure: holding the input fixed, we matched the direction’s cosine readout to the opposite natural endpoint. We also compared the new and story-derived directions on existing open-response tasks at α = ±0.5.

The new directions produced mean absolute rating changes of 28.28/20.42 points in Llama/Olmo, versus 0.057/0.147 for the story-derived directions. Changes followed the natural rating difference in 48/48 and 42/48 conditions.

To control for the larger edits required by the new directions, we matched the story-direction edit norm separately for each condition; rating changes remained much smaller (0.90/1.35 points). Thus, edit magnitude alone cannot explain the contrast.

Qualitative inspection of open responses did not reveal comparably clear changes in target-state expression.

These results provide a positive control for readout matching on held-out templates: the weak effects along the story-derived directions are not a general failure of the method.

We have added these positive-control results to Section 4.4 (Figure 4c), with the norm-matched comparisons reported in Appendix D.2 (Table S9).

Q3. More complex scenarios and longer conversations

Thank you for raising this possibility. This is an insightful suggestion. We acknowledge that the experiments in this paper use scenario templates designed to be as controlled and clear as possible. Conclusions drawn from this type of experiment, which draws on the approach of cognitive psychology, may face external-validity issues when generalized to real-world scenarios. We have narrowed the scope of the relevant conclusions to the tested models, controlled templates, and answer-position directions.

We have also identified more complex scenarios and longer conversations as directions for future work.

Q4. Variable ranges, step sizes, and non-monotonicity

The scenario variables and ranges were developed with input from psychology researchers and manually reviewed to ensure that changes in X could plausibly affect the target emotion. Ranges and step sizes were chosen pragmatically for stable comparisons.

We have inspected all 24 complete self-rating and activation-readout curves, which showed plateaus, abrupt changes, or fluctuations rather than uniform monotonicity. In the GRE anxiety template, both measures initially decreased rapidly as time until the exam increased, then flattened beyond approximately X = 50. We have also retained Olmo-3.1's anomalous response in the fear/dog-distance template. We have provided the exact ranges and step sizes in Table S2 and the complete curves for all 24 templates in Appendix C.2 (Figures S8–S9).

W1. Scope of the title

We thank the reviewer for this thoughtful suggestion. We agree that the previous title was broader than the scope supported by our paper. We have revised it to “Do LLMs’ Self-Ratings of Emotion and Mental Health Depend on the Internal Representations They Track?” to state the specific domain and causal question more precisely.

W2. The Grounding Gap and related work

Thank you for pointing us to this highly relevant work. We have read Chlapanis et al. (2026) and added a discussion of their findings to Section 5.

Their study highlights a distinction between the availability of grounding-related information and its actual use during generation. We observe a related dissociation: emotion-related construct vectors can steer open-ended responses, yet numerical self-ratings remain largely insensitive to changes along these same vectors within their natural activation ranges.

A possible explanation for this dissociation is that the presence of concept-related information in a model's hidden states does not guarantee that this information lies on the computational pathway used to produce a particular output. Different tasks may recruit different pathways, and a model may generate an appropriate response directly from contextual cues without relying on an intermediate representation of the corresponding construct. Furthermore, representations identified through correlation and steering may serve as imperfect proxies for the information used during natural computation. Our findings are consistent with these possibilities, although identifying the precise mechanisms behind this dissociation remains an important direction for future work.

Dear authors, thank you for taking the time to refine the paper based on my feedback. I hope you found it constructive. I will keep my positive evaluation of the paper, and I believe that this paper would be a good fit for ICLR 2027.

Reviewer_9deKOpenReview ↗
Rating
2
Confidence
4
Informativeness
2
Presentation
3
Soundness
2
Read review
Paper summary

The authors study the internal computational mechanism behind LLM self-ratings of its own emotion. They test two hypotheses: representation-mediated reporting and parallel report generation. The key difference is in the role of construct representation in LLM's self-report. Based on the mechanistic interpretability methods and controlled experiments, the authors support the hypothesis that the construct representations have little effect on the self-reports.

Critical strengths

The clearest strength is in novelty.

  • Conceptually, incorporating causal theory of measurement, mechanistic interpretability, and LLM psychometrics to study the internal mechanisms behind how LLMs rate their own emotions is novel
  • The proposed method, readout matching, is also somewhat novel, although its empirical advantages are claimed not demonstrated.
  • The problem being studied is also interesting. There have been reports about discrepancies in LLM behavior vs self-reports. This study adds a new layer to the landscape.
Critical weaknesses

My most significant concern is on the soundness of the experimental setup used to draw the authors' conclusion.

  • The soundness of the authors' conclusion (in support of the "parallel report generation hypothesis") depends heavily on the quality of the extracted feature vectors, but the empirical or theoretical test of the quality is limited.
  • Since the vectors were extracted via a simple and quite uncontrolled method, I think it is plausible that the extracted vectors do not capture the constructs well enough within LLMs. This doesn't mean that the vectors do not represent the constructs. What I mean is that they may not be 'sufficiently' good representations. While the authors showed that the extracted vectors are correlated with self-ratings about the constructs and that they can causally steer LLM behavior, poor and lossy representations can still attain the same empirical qualities that the authors demonstrated (particularly without strong baselines).
  • What this means is a risk in result interpretation. If (1) the entire hidden-state vector incorporated much stronger information about the construct than the extracted feature and if (2) that difference in information quality drove the experimental outcome difference, the results are reduced to the simple issue of feature extraction method's performance and not the test of the two competing hypotheses.
  • Figure S1 is in indirect support of my concern. The extracted features are 'dull', with high inter-feature similarities. Lower similarities between positive-negative emotions (happy vs others), compared to the ones in negative-negative emotions, show that the extracted features make sense, but this is insufficient evidence for the quality of the features.
  • (this is more minor and theoretical) it is worth thinking about whether the 'construct representation' can be reduced to a single vector in LLM activation space. I understand that the field norm is to naively treat the extracted feature vectors as the right construct. However, I believe the translation from technical implementation to the broader hypothesis requires more care and meticulous language.

The first concern, by itself, encourages me to rate the paper as weak accept. Understandably, a paper cannot perfectly rule out an alternative hypothesis, and I agree that this paper still makes a convincing and interesting case. However, I lean toward weak reject due to the second concern. Even assuming that the authors' conclusions are correct, I think this paper has limited impact.

  • What practical insights can one draw from the conclusions of this study?
  • The authors state, 'LLM self-ratings are better treated as outputs of a rating task than as measurements' to critique Wekenborg et al (2026). But this implication is a straw man. Wekenborg et al (2026)'s central claim was 'the promising tools for modelling affective processes relevant to human psychopathology', which irrelevant to whether the outputs are of a rating task or internal measurement.
Questions for authors

None

6 comments

Dear Reviewer,

Thank you for carefully reading our paper and providing constructive feedback. We understand that you have two main concerns:

(1) the extracted directions have not been sufficiently validated

(2) the practical significance of this work remains unclear.

We address these concerns below and agree that our conclusions need to be defined more precisely.

Q1. Representation quality and causal interpretation

Q1.1. Representation-quality assessment

We agree that the previous manuscript provided limited evidence regarding representation quality. The evaluation of representation quality has long been lacking in mechanistic interpretability. We therefore adapt the neuroscience framework of representation proposed by Pohl et al. (2026) to linear representations in LLMs to assess their quality.

Sensitivity: Whether representation strength changes with stimulus intensity. At 75% model depth, our analysis of all 24 templates covering twelve constructs yielded mean absolute correlations between scenario intensity X and the construct readout of 0.801 and 0.747 in Llama and Olmo, respectively.

Specificity: Whether a representation is selectively activated. For the six emotions, we generated a total of 1,200 additional stories across ten held-out topics. Within each topic, we used each emotion direction to distinguish stories about the target emotion from stories about the other five emotions. At 75% depth, the macro-averaged AUROC was 0.776 in Llama and 0.809 in Olmo.

Invariance: Whether a representation is activated by different forms of stimuli. We translated the above stories into Chinese, German, French, and Spanish, and directly tested specificity using the same directions extracted from English data. The macro-averaged AUROC ranged from 0.775 to 0.787 in Llama and from 0.796 to 0.812 in Olmo.

Functionality: Whether a representation can influence behavior. In the original open-ended generation experiment, we intervened at α = −0.5, 0, and +0.5. Qualitative inspection showed changes in emotion-related content and response style.

Together, these results support the view that the extracted directions respond to changes in the scenario, distinguish different emotions after controlling for topic, generalize across languages, and influence open-ended responses.

We have added the topic-controlled discrimination and cross-language evaluations to Section 4.2 and Appendix B.2 (Figure S2 and Table S5).

Q1.2. Positive control using rating-relevant directions

We also acknowledge that full hidden-state replacement and single-direction interventions transfer different amounts of information. We have added a positive control using rating-relevant directions. For each construct, we used ten new templates to extract a direction from the high- and low-rating quartile groups and performed the same natural-endpoint readout matching on two held-out templates. At 75% depth, the new directions produced mean absolute rating changes of 28.28/20.42 points in Llama/Olmo, whereas the story-derived directions produced changes of only 0.057/0.147 points. Therefore, the difference cannot be explained solely by the ineffectiveness of single-dimensional editing.

We have incorporated this single-direction positive control into Section 4.4 and Appendix D.2.

Q1.3. Similarity between construct directions

Regarding Figure S1, our “construct stories minus neutral stories” method retains components shared across emotions and does not explicitly subtract the mean of the other emotions. Thus, the high similarity between different directions may reflect shared affective or narrative information rather than poor extraction quality.

Q1.4. Causal scope

Finally, we acknowledge that the representations extracted using the current method cannot fully capture the constructs. Whether a single linear vector is sufficient to fully represent a psychological construct has recently been a topic of active discussion in the field. We have clarified that the extracted directions are candidate, possibly partial, representations of the constructs, and have restricted our causal conclusions to the tested directions, positions, and intervention procedures, while retaining the possibility of distributed or nonlinear encoding.

Q2. Practical significance and prior research

Q2.1. Behavioral simulation and internal measurement

We thank the reviewer for this clarification. We acknowledge that our earlier wording did not sufficiently distinguish behavioral simulation from internal-state measurement. LLMs can be useful behavioral simulators without sharing human psychological mechanisms, and we do not dispute Wekenborg et al.’s (2026) contribution in this respect.

Our question is narrower: do numerical self-ratings causally depend on the candidate internal states they are interpreted as measuring? This distinction matters when researchers move from reproducing human-like responses to using those responses to validate state induction, explain behavior, or select interventions.

For example, a prompt that increases both self-rated anxiety and biased responding establishes effects on two outputs; it does not establish that an internal anxiety state mediates the behavioral change. The prompt may influence the two outputs through different pathways.

Our experiments show that even strong readout–rating correlations and successful open-response steering can coexist with very weak rating sensitivity to the tested construct readouts. This also suggests a possible explanation for self-report–behavior dissociations documented in recent studies (Han et al., 2026; Kocielnik et al., 2026): self-reports may respond to contextual cues without faithfully tracking the internal processes that guide behavior.

Q2.2. Practical implications

Our contribution is to use the causal theory of measurement to distinguish behavioral simulation from internal-state measurement and specify the additional evidence needed to extend self-rating results to claims about mechanisms or interventions.

These findings offer a methodological caution for research on LLM self-reports, with particularly important implications for AI safety. A reassuring self-report is not, by itself, evidence of a safer model. Interpreting reports of confidence or risk awareness as measurements of internal states requires evidence of causal dependence, consistent with the grounding criterion articulated by Lindsey (2025). Claims of improved safety additionally require behavioral validation: in a different setting, Nguyen et al. (2026) found that targeted finetuning could improve an intention-reporting metric while increasing adversarial-prefill attack success on most tested models.

We have clarified this distinction and the scope of our conclusions in the Introduction and Discussion (Sections 1 and 5). Applying our causal tests to safety-related self-reports remains future work.

References

Wekenborg, M. K., et al. (2026). Large language models as experimental systems in human psychopathology: a modelling study. The Lancet Digital Health. doi:10.1016/j.landig.2026.101014.

Han, P., et al. (2026). The Personality Illusion: Revealing Dissociation Between Self-Reports & Behavior in LLMs. Proceedings of the 43rd International Conference on Machine Learning, PMLR 306, 39867–39903.

Kocielnik, R., et al. (2026). Rethinking Psychometric Evaluation of LLMs: When and Why Self-Reports Predict Behavior. arXiv:2606.12730.

Lindsey, J. (2025). Emergent Introspective Awareness in Large Language Models. Transformer Circuits Thread.

Nguyen, Q. M., Ahmed, U., & Kim, T. (2026). Can LLMs Reliably Self-Report Adversarial Prefills, and How? arXiv:2606.23671.

AuthorsOpenReview ↗

Construct Validity Concerns in LLM Self-Reports

Applying human-developed self-report scales to large language models remains a topic of considerable debate.

On the one hand, LLMs learn linguistic patterns associated with human personality, emotions, and attitudes from large-scale text corpora, enabling them to exhibit human-like patterns in their questionnaire responses when completing psychological questionnaires (Pellert et al., 2024; Serapio-García et al., 2025).

On the other hand, concerns remain about the construct validity of this approach. For example, depression scales assume that respondents have experienced emotional and psychological symptoms over a certain period. However, producing human-like responses does not necessarily imply that LLMs possess the corresponding psychological processes (Lin, 2025).

Discrepancies between self-reports and actual behavior (Han et al., 2026), as well as sensitivity to prompt variations (Gupta et al., 2024), suggest that current evidence is insufficient to directly interpret LLM self-ratings as valid measurements of their internal psychological states.

This issue has practical implications. If researchers interpret changes in self-ratings as evidence of successful internal-state induction and subsequently use them to explain behavioral changes, they may mistakenly attribute the common effects of a prompt on ratings and behavior to the causal influence of an internal psychological state (Borsboom et al., 2004; Lin, 2025). Similarly, in AI safety research, improvements in self-reporting metrics do not necessarily indicate improvements in actual model safety (Nguyen et al., 2026).

We believe our study provides a distinct causal perspective that can help psychological researchers understand how LLMs generate self-report responses and determine what psychological interpretations these responses can support, while also helping clarify the appropriate boundaries of their use in human-participant simulation and AI safety research.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.