- Rating
- 3
- Confidence
- 3
- Informativeness
- 2
- Presentation
- 3
- Soundness
- 2
Read review
Paper summary
The paper asks whether numerical LLM self-ratings causally depend on the internal construct representations with which they correlate. It studies 12 emotional and mental-health-related constructs using 24 scenario templates in Llama-3.1-8B-Instruct and Olmo-3.1-32B-Instruct. Construct directions are extracted from independent stories and evaluated through readout–rating correlations and qualitative steering of open-ended responses. The central intervention holds the prompt fixed while matching a construct readout to its value under the opposite endpoint of the same scenario. Across 384 such edits, no self-rating changes by more than 0.9 points, despite average natural score gaps of approximately 30–41 points. In contrast, full-state replacement at 75% depth transfers median fractions of 91.9% and 99.7% of the natural gaps. Random-direction comparisons, dose diagnostics, and gradient analyses support the dissociation. The contribution is an empirical demonstration that correlation and steering do not establish natural-magnitude dependence on these candidate representations, motivating stronger causal validation before interpreting self-ratings as internal-state measurements.
Critical strengths
-
The intervention controls make the near-zero effects interpretable. Across all 384 matching edits, score changes remain below 0.9 points, compared with natural gaps averaging approximately 30–41 points. At 75% depth, full-state replacement transfers median fractions of 91.9% and 99.7% of those gaps. Together with accurate readout matching, sham interventions, and checks on the full numerical answer distribution, this establishes a substantial dissociation between changing the tested construct readout and changing the information that determines the rating. This controlled comparison is the strongest basis for acceptance (Figure 4; Tables S5–S6).
-
The paper turns a methodological concern into a concrete empirical test. It separates correlation across natural inputs, the ability of large vector additions to alter outputs, and dependence under interventions that match naturally occurring readout values. The candidate directions are extracted from independent stories, and the matching targets come from the opposite scenario endpoint rather than optimization to produce a desired rating change. This makes the test informative about whether these independently defined readouts support the proposed measurement interpretation. The high correlations obtained by random directions and the deterioration of numerical output format under some large edits give the distinction practical substance. Even if the candidates do not recover the representations actually used for rating, the results demonstrate why correlation and qualitative steering are insufficient validation.
Critical weaknesses
-
The validity and specificity of the candidate construct directions remain uncertain. Apart from happiness, pairwise direction cosines at 75% depth are approximately 0.65–0.95, and another construct's direction often tracks the queried rating at least as well as its own. These findings are compatible with substantial shared affective or narrative content. The open-response steering validation is qualitative and uses larger, repeated interventions across generated positions, whereas the central rating intervention edits one position. Consequently, the validation does not establish that the edited direction is the construct representation used during natural rating computation. Full-state replacement transfers all information in the donor state, potentially including scenario and answer-related information; it cannot establish construct-specific mediation or show that a comparably small, low-dimensional edit can transfer the rating. The acknowledged absence of a single-direction positive control therefore matters. These limitations weaken the interpretation that the model bypasses a valid construct representation, while leaving the narrower critique of correlation and steering intact.
-
The intervention sites and task design limit identification of the overall reporting mechanism. The main interventions edit only the final prompt token at 50%, 75%, and final depth. A construct could have influenced the rating earlier, with its information subsequently transformed or copied into other representations, making later edits to the tested readout ineffective. Thus, the results do not uniquely identify parallel report generation. In addition, the evidence covers two models and 24 templates that explicitly ask the model to imagine being an adult and vary one salient quantity, potentially favoring direct scenario-to-score computation. The operationalization also matters: matching a cosine readout need not match the donor's raw projection and generally changes the edited state's norm. These choices warrant a targeted robustness comparison and clear limits on generalization. The results should not be interpreted as showing that all LLM self-reports lack internal grounding, or as settling questions about introspection or subjective experience.
Questions for authors
-
Clarification of causal scope. Is the intended conclusion restricted to insensitivity to natural-magnitude edits of the tested directions at the specified answer positions? Which existing result, if any, distinguishes parallel report generation from a mechanism in which the construct influenced computation earlier and its information was subsequently transformed or copied? This distinction materially affects my assessment of the mechanistic claims.
-
Additional evidence relevant to the current assessment: a low-dimensional positive control. Could the authors provide a rating direction or low-dimensional subspace learned on independent data and evaluated on held-out templates, with interventions at the same position and comparable edit norms? A substantial transfer under this control would strengthen the interpretation that the small effects are specific to the construct directions, rather than a limitation of the intervention protocol's ability to capture low-dimensional causal dependence.
-
Additional evidence relevant to the current assessment: projection matching. Equation (3) matches the target cosine but generally does not match the target projection onto the construct direction. Do the conclusions persist when directly replacing that projection, using ? Please also report perturbation magnitudes and changes in hidden-state norms for the two procedures. This comparison would clarify whether the near-zero effects depend on the particular cosine-based operationalization.
These questions concern uncertainties in the present evidence. Broader model coverage and evaluation with full psychological questionnaires would be useful future work, but are not prerequisites for my recommendation.
4 comments
W1. Representation quality
We appreciate your careful comments on the validity and specificity of the construct directions. To address this concern, we have added held-out discrimination tests using 1,200 stories spanning six emotions and ten topics. At 75% depth, within-topic macro-AUROC was 0.776 in Llama and 0.809 in Olmo. Using the same English-derived directions on Chinese, German, French, and Spanish translations yielded AUROCs of 0.775–0.787 and 0.796–0.812, respectively. These tests provide direct evidence of emotion discrimination despite the shared components suggested by the high direction cosines.
We have included these discrimination and cross-language evaluations in Section 4.2 and Appendix B.2.
W2. Intervention sites and task scope
Thank you for emphasizing the importance of intervention sites and task design when interpreting the findings. The experiments test two models and 24 controlled templates that ask the model to imagine being an adult and vary one salient quantity. This design enables controlled causal comparisons but may favor direct scenario-to-score computation. We have explicitly limited our conclusions to this setting in the Discussion (Section 5), without generalizing to all self-reports or subjective experience.
Regarding intervention timing, our multi-depth interventions and all-layer gradient analysis constrain earlier-computation explanations through the tested answer-position directions, as discussed in Q1. The additional projection-matching results reported here in Q3 address the cosine-based operationalization. More complex scenarios, longer conversations, and broader model coverage remain directions for future work.
Thank you for the detailed response and the additional experiments. I appreciate the low-dimensional positive control, the direct projection-matching experiments, and the additional validation of the construct directions. These additions address several of my concerns and strengthen the evidence supporting the paper.
My main remaining concern is the construct validity of the extracted representations, which is also reflected in Reviewer 9deK’s comments. A direction can correlate with self-ratings and influence emotional expression while still being an incomplete or lossy representation of the construct. If the construct-related information used in rating computation was not captured by the extraction method, the small intervention effects could reflect this limitation rather than the model bypassing the construct representation. Therefore, the results do not fully establish that self-ratings are independent of the relevant emotional constructs, or that parallel report generation is necessarily the correct explanation.
At the same time, I recognize that this identification problem is difficult to resolve conclusively. In human psychology, anxiety can be studied through self-reports, behavior, and physiological responses, yet these measures do not always agree. A related difficulty arises when identifying constructs in AI systems. I therefore would not make definitive proof that an extracted direction is the “true” construct representation an unrealistic requirement for acceptance. I regard this as an important limitation on the interpretation of the findings.
Taking the additional evidence and this remaining limitation into account, I will maintain my original rating of 3 (Weak acceptance). Thank you again for the thoughtful response and the revisions.
Dear Reviewer,
Thank you very much for your thoughtful feedback and continued positive assessment of our paper. We sincerely appreciate your careful consideration of our additional experiments and your recognition of their contributions.
We agree that identifying the representations actually involved in rating computation remains an important open challenge. Your comments have helped us better clarify the scope and limitations of our findings, and we plan to make stronger construct validation a key direction of our future research.
Thank you again for your time, encouragement, and constructive suggestions, which have greatly helped us strengthen our work!
Dear Reviewer,
Thank you for your careful assessment and constructive suggestions. Your comments have helped us clarify the causal scope of our findings and strengthen the experimental controls. We address your questions below.
Q1. Causal scope and earlier computation
Thank you for highlighting the importance of when construct information contributes to the rating. Our method tests whether changing an independently identified construct readout to its value in another natural scenario, while holding the prompt fixed, changes the self-rating. Several results bear on your earlier-computation explanation:
Full-state replacement at 75% depth transfers 91.9% and 99.7% of the natural score gap in Llama and Olmo (medians). The new single-direction control below also substantially changes ratings at this position. Matching at 50%, 75%, and final depth, separately and jointly, produces changes below 0.9 points across all 384 edits. At every answer-position layer, the final expected-score gradient is backpropagated through all subsequent layers. A local contribution along an earlier construct direction therefore remains included even if its information is later transformed or copied. The low construct–gradient alignment across layers complements the finite interventions.
Together, these results constrain earlier-computation explanations operating through the tested directions at the answer position. They support the conclusion that these readouts track ratings without substantially controlling them under natural-readout matching.
The current results do not fully rule out mechanisms involving untested directions, other token positions, or nonlinear/redundant representations. We have clarified this causal scope in the revised manuscript.
Q2. Single-direction positive control and matched edit norms
Thank you for suggesting this additional control. We have added a rating-relevant direction for each construct and model at 75% depth, learned from ten independent scenario templates using normalized high-versus-low rating quartile differences. We evaluated the same readout-matching procedure at the same answer position on two held-out templates.
The rating-relevant directions produced mean absolute rating changes of 28.28 points in Llama and 20.42 in Olmo, compared with 0.057 and 0.147 points, respectively, for the story-derived directions. After enlarging each story-direction edit to match the norm of the corresponding rating-direction edit, the mean absolute changes remained much smaller: 0.90 points in Llama and 1.35 in Olmo.
Rating-direction effects followed the natural rating difference in 48/48 conditions in Llama and 42/48 in Olmo. Thus, the protocol can detect substantial single-direction causal influence, and edit magnitude alone does not explain the contrast. This control provides some evidence for the directions' causal relevance to ratings, rather than establishing construct-specific mediation.
We have incorporated this control and the norm-matched comparisons into Section 4.4 and Appendix D.2 (Table S9).
Q3. Direct projection matching
Thank you for raising this concern about the matching procedure. Following your suggestion, we have repeated all 384 conditions using direct projection replacement. The results are as follows.
Mean absolute rating changes remained small: 0.059–0.196 points in Llama and 0.043–0.173 in Olmo.
At 75% depth, median relative edit norms were comparable between cosine and projection matching: 1.524% versus 1.555% in Llama, and 2.537% versus 2.542% in Olmo. The magnitudes of relative state-norm changes were also comparable: 0.0151% versus 0.0152% in Llama, and 0.1762% versus 0.1703% in Olmo, with cosine matching listed first throughout.
The weak rating effects therefore persist under raw-projection matching, with comparable perturbation magnitudes and state-norm changes in the reported comparison.