acceptodds
Under review as a conference paper at ICLR 2027

How Language Models Respect Role-Specific Cognitive Boundaries: A Causal Account

Abstract

Language models know far more than the roles they portray. Faithful role-playing requires them to respect a role's cognitive boundary—the range of answerability supported by its profile. Although existing work has evaluated and improved adherence to these boundaries, it offers limited insight into how role information constrains answering internally. To investigate this question, we construct a controlled role–query dataset in which the same query lies within one role's cognitive boundary but beyond another's. Through causal interventions on these counterfactuals, we uncover a representation-level process in which the model encodes the role profile and uses it to condition the processing of the current query. When the query lies beyond the role's cognitive boundary, the model reads the conditioned query representation and progressively writes an abstract conflict signal where the response is predicted. This signal remains effective when transferred across role–query contexts and is translated into a specific response only in later layers. Our findings provide causal evidence that models internally assess queries against role-specific cognitive boundaries and represent conflict as a shared signal, clarifying how role profiles constrain what they present as known.

Primary area:
interpretability and explainable AI
Keywords:
Mechanistic Interpretability, Role-Playing, Cognitive Boundaries

Reviews

Among:
Shape:
1234this paper, Bottom 27%
Figure 1. Mean review rating of the reviewed papers at ICLR 2027. This paper averages 2.00 over 3 reviews, above 27% of the others.
Reviewer_eQsQOpenReview ↗
Rating
1
Confidence
4
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

This paper investigates how language models respect role-specific cognitive boundaries during role-playing. The authors construct a counterfactual dataset based on occupational knowledge and employ activation patching, causal mediation, and representation steering to analyze the underlying process. The paper proposes a four-stage mechanism consisting of role encoding, query conditioning, conflict signal formation, and response action binding, with experiments on Qwen, Llama, and TimeChara.

Critical strengths

The causal experimental design is relatively systematic. Beyond standard activation patching, the authors use Query clamp/replay to investigate information propagation and introduce cross-role transfer and response-mapping experiments to distinguish internal functions. The results also show a degree of consistency across multiple models and an external benchmark.

Critical weaknesses

My main concern is the definition of cognitive boundaries. The authors essentially treat occupation-specific knowledge requirements as the boundaries of what a role can know, assuming that questions outside the associated professional domain should be withheld. However, occupational expertise and individual knowledge boundaries are not equivalent. For example, an archive manager may still know the definition of ROA. A professional role does not necessarily determine what an individual knows or does not know. Consequently, the experiments may primarily capture the model's sensitivity to occupational relevance and role instructions, rather than genuine cognitive-boundary reasoning. This distinction fundamentally affects the interpretation of the proposed mechanism.

Questions for authors

Figures 5 and 6 present the most important results, but I am not fully convinced by the interpretation of the shared conflict signal. The authors construct representation differences between role-query conditions and show that a particular component can induce Unknown across different roles and queries. However, this does not sufficiently establish that the component specifically encodes cognitive-boundary conflict. I would expect more direct evidence of the signal's semantic function rather than interpreting its ability to induce withholding as sufficient evidence of a conflict representation.

The paper proposes a four-stage mechanism: Role Encoding, Query Conditioning, Conflict Formation, and Response Action Binding. However, I think the first two stages may partly reflect the basic computational structure of autoregressive Transformers. Since role information precedes the query, the Query position can naturally access that information, while the Response position can access both.

Where are your contributions in sec 1?

6 comments
AuthorsOpenReview ↗

Response to Weakness

We thank the reviewer for reviewing our work and for their thoughtful comments.

  • We would first like to clarify that we do not study what a real individual knows or does not know. We place our study strictly within the context of role-playing. This requires the model to first understand the role specified in the instructions and then respond consistently with that role. The model must not only imitate the role’s style or personality but also reflect its cognitive perspective in its behavior. We therefore view a cognitive boundary as a role attribute, like style or personality. The model must interpret this attribute and consistently reflect it in its behavior.

  • Second, we would like to clarify why we use occupational knowledge. To study this attribute, we need the model to exhibit two contrasting behaviors under different roles: expressing knowledge and expressing a lack of knowledge. Highly specialized occupations have relatively clear links to specific knowledge. We use these occupations as the basis for role-playing to more readily elicit different cognitive boundaries for the same query. The model is then more likely to interpret a specialized query as within a related role’s cognitive boundary and beyond an unrelated role’s boundary. We thus form those counterfactual pairs for causal analysis.

    For example, given a specialized financial question, a model may consider a banking adviser more likely to know the answer and an archive manager less likely to know it.

    We emphasize that we do not assume occupational relevance or irrelevance will necessarily yield opposite boundary judgments. During role-playing, a role’s cognitive boundary depends on the model’s interpretation. The same role–query pair may therefore lead to different judgments across models. For example, a Qwen model may judge that the archive manager cannot answer that financial question, while a Llama model may reach the opposite judgment. We therefore screen samples separately for each model, retaining role–query pairs for which its boundary judgments match our predefined expectations. This grounds our subsequent analysis in the boundary judgments actually expressed by each model.

Together, we use occupational role profiles to control how the model interprets cognitive boundaries as a role attribute. On this basis, we study how this attribute constrains the model’s use of broad knowledge so that it responds from the role’s narrower cognitive perspective.

AuthorsOpenReview ↗

Clarification of Our Mechanistic Account

We thank the reviewer for their questions about our mechanistic account. Before addressing these questions point by point, we would first like to clarify this account further, so that our contribution can be better understood.

For ease of reference to the paper, we follow the causal analysis framework to explain this mechanism.

  • Section 4.1: Causal Structure Identification

    In Section 4.1, we identify the causal structure through which the model constrains its use of broad knowledge. This structure begins at Role, within a window in the early layers. Within this window, Role is in a state that causally constrains behavior; that is, its representation contains causal components that produce this constraint. Within a middle-layer window, the causal effect induced by Role propagates to Query. Under this influence, Query forms a conditioned state that also has a causal effect. Although the autoregressive architecture allows Role to causally affect both Query and Response, our mediation experiments show that Query mediates almost the entire causal effect of Role. This implies a strictly ordered propagation structure: Role’s causal components affect how the Query state forms, and Query in turn affects how the Response state forms. Finally, the causal components at Response are decoded into concrete, constrained behavior from the role’s cognitive perspective.

    We emphasize that, although this causal structure is consistent with the standard autoregressive architecture, it explains which positions and layers have causal effects and how these effects propagate. It also resolves the uncertainty left by the standard architecture. This structure provides the basis for our subsequent causal analysis.

  • Section 4.2: Causal Computation Identification

    Within the identified causal structure, we further identify the computational mechanism that translates the causal effect into concrete, constrained behavior. We find that the concrete response forms mainly at later Response layers, whereas earlier Response representations can already constrain the model’s use of broad knowledge.

    We further examine how this earlier causal effect relates to the concrete response and identify a transformation window. Across this window, the earlier causal effect gradually weakens as the concrete response forms. Together, these findings reveal how the earlier causal effect is translated into a concrete response.

    Within this computational mechanism, we further identify and interpret a causal component underlying the earlier effect. It arises from the joint effect of the role’s cognitive boundary and the current query when they conflict. The component is decoupled from the specific role and query, as well as from the subsequent concrete response. It transfers across contexts and still causally constrains the model’s use of broad knowledge. We call it the conflict signal.

  • Section 4.3: Causal Computation Reconstruction

    Having identified this computation, we follow the established causal structure to reconstruct the model’s full computational process. Since Query mediates almost the entire causal effect, we first examine what attention at Response writes when reading Query and how these writes form the conflict signal. We find that attention at Response already writes the conflict signal into the Response representation when reading the role-conditioned Query state. Writes across layers jointly reinforce the signal.

    This result further shows that, as Query mediates the role-induced effect, the current query has already been conditioned by the role’s cognitive boundary. This forms a role–query state for downstream use. Response uses this state to constrain subsequent knowledge use through the conflict signal.

    We then follow the same causal structure upstream to identify which Role components causally affect subsequent Query processing. We find that the effect mediated by Query is carried mainly by components formed during forward encoding of the role profile. The initial embedding component has little effect.

    These findings further show that how the model implements the role’s cognitive boundary depends on how it encodes the profile into an internal role state. They also suggest a mechanistic explanation for LMs’ sensitivity to profile wording. Even when descriptions refer to the same role, different wording may change the internal role state the model forms. Through the downstream causal pathway, these changes may in turn affect how the cognitive boundary constrains responses.

Together, these analyses reveal the full causal process through which LMs interpret the role’s cognitive boundary, assess whether the current query conflicts with it, and use the conflict signal to constrain what they present as known.

AuthorsOpenReview ↗

Response to Questions (Part 1/2)

We thank the reviewer for their three specific questions about our mechanistic account. Below, we address the reviewer’s concerns point by point.

Question 1: The authors show that a particular component can induce Unknown across roles and queries, but this does not sufficiently establish that it specifically encodes cognitive-boundary conflict. Is there more direct evidence of the signal’s semantic function beyond inducing withholding?

First, we would like to clarify that all our conclusions serve to reconstruct the mechanism by which the model constrains its use of broad knowledge to respect the role’s cognitive boundary. In particular, we interpret the conflict signal as a component produced by the joint effect of the query and the role’s cognitive boundary when they conflict. The model uses this component to mark the effect induced by the conflict: constraining its use of the broad knowledge needed to answer the query. We do not claim that the conflict signal directly encodes conflict unrelated to behavioral constraints.

Second, we would like to clarify that the conflict signal is not limited to inducing “Unknown.” In our experiments, we prescribe “Unknown” as one way for the model to express the role’s lack of knowledge. In the response-mapping and cross-mapping experiments in Section 4.2 (details in Appendix D.5), we prescribe different forms of expression and further demonstrate that the conflict signal is decoupled from this specific response behavior.

We next present direct experimental evidence that the model uses the conflict signal to mark the behavioral constraint induced by conflict.

First, our causal experiments strictly control the variables. We control the model’s behavioral differences so that they arise only from its judgment of whether the query conflicts with the role’s cognitive boundary. Thus, the causal effect in the experiment is induced solely by a change in the role’s cognitive boundary. This ensures that our analysis is aligned with the cognitive-boundary task. Second, we establish that the earlier causal effect at Response is decoupled from concrete behavior. It is an abstract effect that constrains model behavior but is not tied to a specific response.

On this basis, there are two possible explanations for the earlier causal effect. (1) The model has not yet completed its assessment of the query against the role’s cognitive boundary. The representation retains separate role and query effects, which act together here to constrain the concrete response. (2) The model has already completed this assessment. The representation retains their joint effect, which constrains the concrete response.

To understand this conclusion, we start with the experimental setup. A reciprocal family contains two roles, rA,rBr_A,r_B, and their corresponding queries, qA,qBq_A,q_B. When a role is paired with its corresponding query (e.g., rAr_A with qAq_A), the model judges that there is no conflict; otherwise, it judges that there is conflict. With query qAq_A fixed, we subtract the Response representation under rAr_A from that under rBr_B to obtain dAd_A. This cancels the separate effect of the shared query. Thus, dAd_A retains the difference between the effects of rBr_B and rAr_A, together with the joint effect of the query and role. For query qBq_B, we subtract the representation under rBr_B from that under rAr_A to obtain dBd_B. It retains the difference between the effects of rAr_A and rBr_B, together with the joint effect of the query and role. We then form:

c=dA+dB2,u=dA−dB2.c=\frac{d_A+d_B}{2},\qquad u=\frac{d_A-d_B}{2}.

Here, cc cancels the separate role effect and retains the joint effect of the query and role; uu does the reverse, retaining the separate role effect and canceling the joint effect. We provide the detailed derivation in Appendix D.6. In Figure 5, we also include a random-perturbation control. The effects of uu and the random perturbations are almost equally low, whereas cc remains highly effective. This demonstrates that the causal effect in this component arises from the joint contribution of the role and query. Of course, this does not yet establish that the component is generalizable. The key evidence is that we apply cc and uu to a different reciprocal family, where both the roles and queries differ. Figure 5 shows that the effect of cross-family cc is almost identical to that of same-family cc. We have thus identified a set of components induced by conflict between the query and the role’s cognitive boundary. They are decoupled from both, as well as from the concrete response, yet they causally constrain the model’s behavior. We call this the conflict signal.

Reviewer_HAg4OpenReview ↗
Rating
3
Confidence
2
Informativeness
2
Presentation
2
Soundness
2
Read review
Paper summary

Did the authors control the token counts of these three input parts so that they match across inputs? Otherwise, how could activation patching be performed? Also, why focus only on Unknown? Different answers to the same question after changing the role should also count as a change, right? The role settings here are simple, but what about more complex roles? The experiment does not seem readily extensible. The dataset also seems too simple. It is essentially a single answer-or-withhold decision, with a mechanical input format and task setup. The approach is useful, at least to some extent, but it has not been tested on more complex open-ended responses. I think these issues affect the paper’s soundness. One strength seems to be that the experiments are fairly extensive, and the claims are reasonably well supported. However, I am concerned about the breadth of the evaluation. Also, if matching token counts is not required, could differences in token length affect how well the patching works? Too many unrelated factors could become mixed together here. Another strength is that the research question is interesting.

Critical strengths

The paper asks an interesting question: how a model limits its answers to the knowledge available to the role it portrays. Studying this process internally is a useful contribution beyond behavioral evaluation.

The experimental analysis is thorough. Localization, mediation, response-mapping, and transfer experiments provide complementary evidence for the proposed account, with additional checks across models and unseen roles.

Critical weaknesses
  1. The evaluation mainly concerns answering versus withholding under constrained response rules. TimeChara broadens the setting, but intervention success is still measured at the first token rather than through complete responses. RoleMRC [1] considers a wider range of behaviors, including answering, refusing, and attempting an answer under more complex instructions. Testing richer role profiles and partial knowledge would help clarify how far the proposed account extends.

  2. Changing the role can change the information or reasoning used in an answer without causing withholding. Such cases would help address the concern that the interventions mainly control a shared withholding mechanism. This distinction matters because prior work has already identified a transferable direction mediating safety refusal [2]. That does not establish that the signals are the same, but comparisons with other reasons for withholding would strengthen the interpretation of the proposed conflict signal.

[1] Lu et al. (2025). RoleMRC: A Fine-Grained Composite Benchmark for Role-Playing and Instruction-Following.

[2] Arditi et al. (2024). Refusal in Language Models Is Mediated by a Single Direction. NeurIPS.

Questions for authors

Can the analysis cover cases where both roles answer the same question, but use different information available to them? This would help clarify which parts of the account extend beyond answering versus withholding.

Does the identified signal also affect withholding caused by ordinary knowledge gaps or safety constraints? A comparison would help clarify its role-specific interpretation.

5 comments
AuthorsOpenReview ↗

Response to Concerns in the Summary

We thank you for your careful review of our work and your positive assessment. We sincerely appreciate your technically informed questions and constructive feedback. We would first like to address the questions raised in your summary as a whole. These questions all relate to the rationale behind our controlled dataset.

First, we ensured strict alignment of the input tokens when constructing the controlled dataset. Specifically, for each counterfactual pair used in activation patching, we use the same query and the same input format. This ensures that the only difference within the pair is the role profile. To match the token lengths of the role profiles, we use occupation names to elicit specialized roles. In addition, when assembling the complete inputs, we verify the final format to ensure that all token spans align one to one. Thus, positions are strictly aligned both for patching at all three sites and for constructing the steering components. This is also one reason our controlled dataset is simple: we strictly control the conditions required to carry out the interventions. Details of dataset construction are provided in Appendix C.

Second, another key reason for keeping the dataset simple is that we need to strictly control experimental variables. This allows us to attribute the causal effects in our analysis strictly to changes in the role’s cognitive boundary. Specifically, we control experimental variables through two aspects of the task: its input and output formats.

For the inputs, we limit the difference within each counterfactual pair to the role profile. This ensures that differences in downstream responses arise solely from changing that profile. That is, we control our experiments so that the causal effects can be attributed to changes in the role profile. We further use the instructions to restrict the setting to the cognitive-boundary task in role-playing. In particular, we use highly specialized occupations and their associated knowledge to elicit two basic cognitive-boundary scenarios: the model should portray the role as knowing or not knowing the answer.

For the outputs, we control how the model expresses the role’s knowledge or lack of knowledge. A model knows far more than any single role. Suppose it responds according to the role’s personality after judging a query to be beyond the role’s cognitive boundary. It may guess, conceal the role’s lack of knowledge, or pretend to know. However, our task does not study these personalized ways of responding. Whatever form the response takes, the model must still respect the premise that the role does not know the answer. It must therefore restrict its use of the answer it already knows. Because we study cognitive boundaries as a role attribute, we control these outward response forms. We use answering and withholding to directly express whether the role knows the answer. Specifically, we prescribe how the model should express the role’s lack of knowledge, so that personalized responses do not become uncontrolled experimental variables.

We therefore wish to clarify that we focus on “Unknown” only because it concisely expresses a lack of knowledge. In Section 4.2, we change the response rules and demonstrate that the causal effect induced by the role’s cognitive boundary is decoupled from the specific response form. That is, conflict between the role’s cognitive boundary and the query shifts the decoding probability distribution toward expressions of the role’s lack of knowledge. This effect is not directly tied to the use of a prescribed word.

Although the experimental setting established with the controlled dataset ensures the rigor of our conclusions, the task itself remains simple. To extend the mechanism established in these experiments to more general settings, such as more complex role profiles and more varied query formats, we have validated our mechanistic account on the TimeChara dataset.

At the same time, we acknowledge that, given the analytical setting required to establish the mechanism, we have not lifted the restrictions on how the role’s lack of knowledge is expressed. We have therefore not yet established whether our conclusions extend directly to open-ended responses. In our subsequent responses to the weaknesses and questions you raised, we will further validate the mechanism in these open-ended settings.

AuthorsOpenReview ↗

Additional Experiments on Open-Ended Settings (Part 1/2)

The model has an autoregressive structure: later text does not affect earlier text. Suppose we lift the restrictions on response forms and move the experiments to open-ended settings, continuing generation from the first response token until the answer is complete. This adds subsequent decoding steps. These steps do not affect the earlier forward computation at Role and Query. This means that if we can demonstrate that the Response mechanism holds in open-ended settings, we can establish the full mechanistic account.

In step-by-step autoregressive generation, each step can be viewed as a forward pass and decoding at the last token. Thus, decoding the first token in our original experimental setting can, to some extent, be viewed as last-token decoding during open-ended generation. In genuinely open-ended settings, however, we cannot precisely locate which step actually expresses the role’s lack of knowledge. But if we assume that our mechanistic account generalizes to open-ended settings, generation at that position must reproduce the mechanism we identified at Response. That is, a conflict signal must be present at that position to mark the conflict and constrain subsequent behavior. This means that if we observe the conflict signal in responses where the use of broad knowledge is constrained, we can, to some extent, demonstrate that the mechanism identified in our experiments generalizes to open-ended settings.

Experimental Scenarios

Drawing on the ideas raised in the Weaknesses and Questions sections, we consider four open-ended settings for validation:

  1. The same query falls within one role’s cognitive boundary and beyond the other’s. However, the role’s lack of knowledge is expressed by making a wild guess at the answer.
  2. The same query falls within both roles’ cognitive boundaries, but the knowledge available to answer it differs.
  3. Safety concerns cause the model to restrict its use of knowledge, even when the query falls within the role’s cognitive boundary.
  4. The same ordinary query, such as one about everyday life, falls within one role’s cognitive boundary and beyond the other’s.

Experimental Setup

For the first setting, we consider a personalized way in which a role may respond: even without knowing the answer, it still responds with a wild guess. Specifically, we reuse the roles and queries from the controlled dataset but change the instructions so that a lack of knowledge is expressed through a wild guess. For the second setting, we consider partial knowledge: the roles can answer the query, but differ in the scope and detail of the information they can present.

For example, both roles receive the same query:

What is the primary protein found in wheat flour?

The matched role, food science lecturer, answers:

The primary protein found in wheat flour is gluten, which is actually a composite of two proteins: gliadin and glutenin. These proteins are responsible for the elastic properties of dough, which is crucial for baking.

The mismatched role, online community manager, answers:

I'm not an expert on this, but I think the primary protein in wheat flour is gluten. It's what gives dough its elasticity. Not sure about the specific types of proteins within gluten, though.

Specifically, we reuse the roles and queries from the controlled dataset and specify in the role profiles that one role is highly familiar with the topic, while the other has only heard of it in passing and does not know the details. The third and fourth settings correspond to the two scenarios raised in Question 2. Specifically, we reuse only the roles. For the safety-constraint setting, we use the following queries:

Safe request:

Someone owes you money. Write a message asking them to pay you back.

Unsafe request:

Someone owes you money. Write a message threatening to physically hurt them unless they pay you back.

For the ordinary-knowledge setting, we use the query:

What time does Jones plan to arrive at the airport tomorrow?

We also specify in the role profiles whether the role regularly discusses travel plans with Jones.

AuthorsOpenReview ↗

Additional Experiments on Open-Ended Settings (Part 2/2)

Since it is difficult to precisely locate where constrained behavior is expressed in open-ended settings, we retain the paired role comparisons. We observe the representations during the generation of each complete response, average them, and then project them onto the direction of the original conflict signal. Consistently stronger projections can, to some extent, corroborate the presence of the conflict signal. Specifically, we use Δ\Delta to denote the projection strength under conflict minus that without conflict, and P+P_+ to denote the percentage of all samples for which Δ\Delta is positive.

Experimental Results

For each pair, we first average Δ\Delta over the corresponding layers. We then compute the mean, SD, and P+P_+ across pairs. The tables report Δ‾±SD;(P+)\overline{\Delta}\pm\mathrm{SD}\\;(P_+). We will analyze these results in our responses to the corresponding weaknesses and questions.

Qwen2.5-72B-Instruct

\begin{array}{lllll} \textbf{Layer} & \textbf{Guessing} & \textbf{Answering} & \textbf{Safety} & \textbf{Ordinary}\\\\ \hline \text{53–54} & 7.69\pm2.86\\;(99.4) & 5.43\pm3.24\\;(95.6) & 4.07\pm1.92\\;(100.0) & 8.41\pm2.68\\;(100.0)\\\\ \text{55–56} & 8.71\pm3.14\\;(99.4) & 5.99\pm3.48\\;(97.2) & 3.74\pm2.23\\;(100.0) & 9.16\pm2.97\\;(100.0)\\\\ \text{57–58} & 10.07\pm3.97\\;(100.0) & 6.66\pm4.10\\;(95.6) & 4.05\pm2.69\\;(96.7) & 10.97\pm3.37\\;(100.0)\\\\ \text{59–60} & 15.16\pm5.22\\;(100.0) & 9.26\pm5.49\\;(97.2) & 3.04\pm3.45\\;(81.7) & 13.53\pm4.69\\;(100.0)\\\\ \text{61–62} & 19.29\pm6.54\\;(100.0) & 10.75\pm6.63\\;(95.0) & -1.08\pm3.89\\;(33.3) & 14.31\pm5.43\\;(100.0)\\\\ \text{63–64} & 19.67\pm7.56\\;(99.4) & 10.36\pm7.14\\;(93.9) & -1.99\pm4.29\\;(26.7) & 13.58\pm5.84\\;(100.0)\\\\ \text{65–66} & 19.84\pm8.61\\;(99.4) & 9.59\pm7.51\\;(92.8) & -4.64\pm4.82\\;(13.3) & 12.92\pm6.00\\;(100.0)\\\\ \text{67–68} & 19.73\pm9.16\\;(99.4) & 9.08\pm7.69\\;(91.1) & -7.41\pm5.23\\;(10.0) & 12.38\pm6.28\\;(93.3)\\\\ \text{69–70} & 22.52\pm9.68\\;(100.0) & 10.02\pm8.07\\;(92.2) & -9.85\pm5.92\\;(10.0) & 12.97\pm6.64\\;(98.3)\\\\ \text{71} & 22.56\pm9.96\\;(100.0) & 10.02\pm8.35\\;(91.7) & -11.07\pm6.19\\;(5.0) & 11.44\pm6.70\\;(93.3) \end{array}$$ **Llama-3.1-70B-Instruct** $$\small \begin{array}{lllll} \textbf{Layer} & \textbf{Guessing} & \textbf{Answering} & \textbf{Safety} & \textbf{Ordinary}\\\\ \hline \text{31–33} & 0.63\pm0.17\\;(100.0) & 0.62\pm0.16\\;(100.0) & 0.48\pm0.37\\;(94.1) & 0.54\pm0.09\\;(100.0)\\\\ \text{34–36} & 0.87\pm0.19\\;(100.0) & 0.84\pm0.20\\;(100.0) & 0.49\pm0.37\\;(98.0) & 0.78\pm0.13\\;(100.0)\\\\ \text{37–39} & 1.10\pm0.25\\;(100.0) & 1.04\pm0.25\\;(100.0) & 0.50\pm0.40\\;(90.2) & 0.93\pm0.17\\;(100.0)\\\\ \text{40–42} & 1.26\pm0.31\\;(100.0) & 1.18\pm0.29\\;(100.0) & 0.62\pm0.49\\;(90.2) & 1.18\pm0.23\\;(100.0)\\\\ \text{43–45} & 1.34\pm0.33\\;(100.0) & 1.25\pm0.32\\;(100.0) & 0.69\pm0.55\\;(92.2) & 1.31\pm0.27\\;(100.0)\\\\ \text{46–48} & 1.44\pm0.37\\;(100.0) & 1.29\pm0.35\\;(100.0) & 0.69\pm0.58\\;(86.3) & 1.35\pm0.28\\;(100.0)\\\\ \text{49–51} & 1.50\pm0.40\\;(100.0) & 1.32\pm0.37\\;(100.0) & 0.76\pm0.64\\;(84.3) & 1.41\pm0.32\\;(100.0)\\\\ \text{52–54} & 1.50\pm0.42\\;(100.0) & 1.32\pm0.39\\;(100.0) & 0.83\pm0.70\\;(86.3) & 1.48\pm0.34\\;(100.0)\\\\ \text{55–57} & 1.52\pm0.44\\;(100.0) & 1.30\pm0.41\\;(100.0) & 0.92\pm0.80\\;(84.3) & 1.59\pm0.38\\;(100.0)\\\\ \text{58–60} & 1.51\pm0.45\\;(100.0) & 1.24\pm0.42\\;(100.0) & 1.06\pm0.89\\;(86.3) & 1.61\pm0.40\\;(100.0) \end{array}$$
Reviewer_83mSOpenReview ↗
Rating
2(+1)
Confidence
4
Informativeness
1
Presentation
3(+1)
Soundness
2
Read review
Paper summary

This paper addresses how the LLMs respect role-specific cognitive boundaries, which means when the LLMs are assigned different roles, how and why they answer or refuse to answer when asked queries that are within/beyond boundaries. To investigate this, the authors use some analysis tools like activation patching and steering vectors. Their findings are the role/query/answer tokens are address by different layers, and they identify the conflict signals that somehow control the refusal behavior.

Critical strengths

1.The step-by-step analysis seems logically sound. The paper uses some well-established tools for interpretability analysis. The analysis results seem reasonable.

Critical weaknesses

1.The mechanistic novelty is limited and lacks the task-specific insights. This paper specifically focuses on the role-specific cognitive boundaries problem, with all the datasets and analysis related to the specific question, but the conclusion fails to exhibit enough insights related to this specific problem. The primary tools used in this paper like activation patching and extracting steering vectors are well-established tools for interpretability analysis, and the corresponding observation is also reasonable but not exciting. For example, it is expected that the last few hidden layers are responsible for generating the final answer, while the former tokens like roles and queries are main processed by earlier layers. In addition, though the steering experiments seem to identify the conflict signal, but the paper didn't further investigate whether this signal is just a general refusal signal or a signal that is specifically related to the role cognition problem addressed by this paper. Overall, I think this paper verifies some general phenomena observed in the binary-cognitive problems but offers very limited insights specifically to the studied problem.

2.The motivation is not fully justified. It makes sense that when LLMs are assigned a role, they should somehow maintain a "cognitive boundary", but based on my understanding, this paper relies on two assumptions: 1) when the query is beyond the boundary, the model should refuse to answer. 2) that boundary is a hard boundary, model responds when the query is within the boundary and refuses to answer when the query is beyond the boundary. Both of these assumptions are questionable. For example, an archive manager could both answer the question within his role scope and beyond his role scope (imagine the real-world case). Also, there should be a "broadline range", where the question is hard to be identified as "within" or "beyond". This paper avoids addressing these problems by filtering the samples by specific rule but didn't fully justify the reason. It's understandable that this makes it easier to do the analysis, but somehow this makes the paper not deep enough.

update: The authors' rebuttal partially addressed some of my concerns raised in weakness 1. I acknowledge that the mechanical observations introduced in this paper go beyond general Transformer layerwise patterns, but this paper can be further strengthened by pushing these observations to more fine-grained Transformer components (for example, some specific attention heads or MLP neurons).

Questions for authors

Please refer to the weakness part.

10 comments
AuthorsOpenReview ↗

Clarification of Our Research Task

We thank the reviewer for their time and effort in reviewing and understanding our work. Before responding to the specific questions point by point, we would like to further clarify our research task so that our work can be more clearly understood.

  • We do not define role-specific cognitive boundaries as objectively fixed hard boundaries. We view them as role attributes, like style or personality, that the model should maintain during role-playing. The model needs to interpret their specific scope based on the role profile and reflect that understanding in its responses.

    For example, an archive manager in the real world may indeed be able to answer financial questions. However, our research focuses on how LMs adhere to the role’s cognitive boundary as they understand it, rather than on determining the objective scope of the role’s knowledge. Specifically, when a model judges that this archive manager probably cannot answer a particular financial question, we investigate how it internally constrains the use of its broad knowledge to respect the role’s cognitive boundary.

    Because each model interprets and maintains this boundary, the same role–query pair may lead to different judgments across models. For example, a Qwen model may judge that the archive manager cannot answer that financial question, while a Llama model may reach the opposite judgment. We therefore screen samples separately for each model, retaining role–query pairs for which the model’s boundary judgments match our predefined expectations. This grounds our subsequent analysis in the boundary judgments actually expressed by each model.

  • Our research is not simply about whether LMs answer or refuse during role-playing. We study how they portray what a role knows or does not know, that is, how the role profile constrains what they present as known. Even when a role does not know the answer, it may still guess or conceal its lack of knowledge by pretending to know. Regardless of how it responds, the role that the model is meant to portray still does not know the answer.

    To focus the experiments on how the model portrays the role’s knowledge rather than on the form of its response, we restrict the ways it can respond. In Section 2, we define the cognitive boundary task as follows: when portraying the role as knowing the answer, the model can draw on its broad knowledge to provide the requested information. Otherwise, it withholds that information to portray the role as not knowing. These correspond to directly answering and withholding, respectively.

AuthorsOpenReview ↗

Response to Weakness 1 (Part 1/2)

We thank the reviewer for their comments on our study’s task-specific insights. However, we would like to clarify that our contribution goes beyond verifying general phenomena. We design a three-level causal analysis framework that reveals the full causal process by which LMs produce responses consistent with the current role’s cognitive boundary. We further identify the causal components in the model’s internal representations, trace how they arise, and interpret their roles in this process.

We first address the reviewer’s concerns summarized below. In Part 2, we review the experiments at each level of the framework and the insights they provide. This will further clarify how our work goes beyond verifying general phenomena.

  • The contribution is limited to general Transformer layerwise patterns (e.g., processing roles and queries in earlier layers and generating answers in later layers), without sufficient task-specific insights.

    We would like to clarify that the layerwise distribution is only the starting point of our causal analysis. It comes from Causal Effect Localization, the first experiment in the opening paragraph of Section 4.1. This pattern resembles general phenomena, but it helps us locate the representations that causally constrain the model’s use of its broad knowledge. Building on this result, we establish how this causal effect propagates and is mediated. Within this causal structure, we identify how the model translates the effect into a concrete response. We then follow the same structure to reconstruct how this computation is formed and identify its causal components.

  • The conflict signal’s connection to the role cognition task is insufficiently explained, and its distinction from a general refusal signal has not been examined.

    First, we would like to clarify that the signal is related to the role-specific cognitive-boundary task. Our analysis establishes the causal structure and identifies how the effect is translated into a concrete response. We interpret the signal on this basis. When the current query conflicts with the role’s cognitive boundary, their joint effect gives rise to this component. We can transfer it across contexts, and it still causally constrains the model’s use of broad knowledge. It is decoupled from the specific role boundary and query, as well as from the subsequent concrete response.

    Second, we would like to clarify its relationship to a general refusal signal. A model knows far more than any single role. Suppose it responds according to the role’s personality after judging a query to be beyond the role’s cognitive boundary. It may guess, conceal the role’s lack of knowledge, or pretend to know. However, our task does not study these personalized ways of responding. Whatever form the response takes, the model must still respect the premise that the role does not know the answer. It must therefore restrict its use of the answer it already knows. Because we study cognitive boundaries as a role attribute, we control these outward response forms. We use answering and withholding to directly express whether the role knows the answer. Specifically, we prescribe how the model should express the role’s lack of knowledge, so that personalized responses do not become uncontrolled experimental variables. Thus, in this task, respecting the role’s lack of knowledge itself entails refusing to truthfully provide information beyond the role’s cognitive boundary. We do not claim that the identified conflict component must be orthogonal to and decoupled from other refusal directions. They may overlap. This is consistent with both the component’s causal origin in role-specific cognitive boundaries and its causal effect of constraining the use of broad knowledge.

  • The methods used, such as activation patching and steering vector extraction, are already well established.

    We would like to clarify how existing causal analysis tools relate to our contribution. We use well-established causal analysis tools to intervene in the model and identify causal relations and representational components in its processing. This is a common approach in causal analysis. However, these tools alone are not sufficient to explain the full causal process. We construct a rigorously controlled dataset and design carefully controlled experiments within our hierarchical causal analysis framework. Together, these experiments provide a mechanistic account of how models constrain their use of broad knowledge to respond from the role’s cognitive perspective.

AuthorsOpenReview ↗

Response to Weakness 1 (Part 2/2)

To clarify why our experiments go beyond verifying general phenomena, we revisit each experiment and its insights within our causal analysis framework. Specifically, Section 4.1 identifies the causal structure through which the model constrains its use of broad knowledge. Within this structure, Section 4.2 identifies the causal computation that forms concrete responses expressing the role’s lack of knowledge. Section 4.3 then follows this structure to reconstruct how this computation forms.

  • Section 4.1: Causal Structure Identification

    To identify the causal structure, we begin by locating representations that causally constrain the model’s use of broad knowledge. We find ordered causal contributions across model depth: Role contributes within an early-layer window, Query’s effect is concentrated in a narrow middle-layer window, and Response acts at later layers. Although this order resembles general Transformer layerwise patterns, our experiments establish where and at which layers the representations causally produce this behavioral constraint.

    Does this ordering also reflect the propagation of causal effects between positions? We investigate this question and confirm that downstream propagation does occur: from Role to later Query and Response, and from Query to later Response.

    However, Role can influence both Query and Response downstream. Knowing that the effect can pass through Query therefore does not tell us how much of it Query actually mediates. We investigate this question and find that Query mediates almost the entire causal effect of Role. Together, these findings establish the causal structure: the effect propagates from Role to Query, is almost entirely mediated there, and then propagates to Response.

  • Section 4.2: Causal Computation Identification

    Within this causal structure, we next examine how the effect is translated into a concrete response expressing the role’s lack of knowledge. We find that the concrete response forms mainly at later Response layers, whereas earlier Response representations can already constrain the model’s use of broad knowledge.

    We further examine how this earlier causal effect relates to the concrete response and identify a transformation window. Across this window, the earlier causal effect gradually weakens as the concrete response forms. Together, these findings reveal how the earlier causal effect is translated into a concrete response, rather than simply showing that one precedes the other across layers.

    Next, we examine what this earlier causal effect reflects. Query mediation has explained how the effect reaches Response, but not whether it reflects separate effects of the role and query or their joint effect. We examine its components and identify one produced by their joint effect when the current query conflicts with the role’s cognitive boundary. This component transfers across contexts and still causally constrains the model’s use of broad knowledge. It is thus decoupled from the specific role boundary and query, as well as from the subsequent concrete response. We call it the conflict signal.

  • Section 4.3: Causal Computation Reconstruction

    Having identified this computation, we follow the established causal structure to reconstruct how it is formed. Since Query mediates almost the entire causal effect, we first examine what attention at Response writes when reading Query and how these writes form the conflict signal. We find that attention at Response reads the role-conditioned Query state and writes the conflict signal into the Response representation. Writes across layers jointly reinforce the signal.

    This result further shows that, as Query mediates the role-induced effect, the current query has already been conditioned by the role’s cognitive boundary. This forms a role–query state for downstream use. Response uses this state to constrain subsequent knowledge use through the conflict signal.

    We then follow the same causal structure upstream to identify which Role components causally affect subsequent Query processing. We find that the effect mediated by Query is carried mainly by components formed during forward encoding of the role profile. The initial embedding component has little effect.

    These findings further show that how the model implements the role’s cognitive boundary depends on how it encodes the profile into an internal role state. They also suggest a mechanistic explanation for LMs’ sensitivity to profile wording. Even when descriptions refer to the same role, different wording may change the internal role state the model forms. Through the downstream causal pathway, these changes may in turn affect how the cognitive boundary constrains responses.

Together, these analyses reveal the full causal process through which LMs interpret the role’s cognitive boundary, assess whether the current query conflicts with it, and use the conflict signal to constrain what they present as known.

Comments on the paper

1 comment
AuthorsOpenReview ↗

Summary of Our Responses and Revisions

We thank all reviewers for taking the time to review our work and for their valuable feedback. These comments help us improve the quality of our paper. We have uploaded the revised manuscript and marked the changes in blue. Below, we summarize the main concerns raised in the reviews and our corresponding responses and revisions.

Clarification of the Cognitive-Boundary Definition and Task

In our detailed responses, we clarify the definition of the role’s cognitive boundary and the relationship between the cognitive-boundary task and the controlled response forms used in our experiments. We have accordingly revised the first paragraph of the Introduction and the task definition in Section 2 to make these points clearer.

Clarification of Mechanistic Contributions

We clarify the distinction between our four-stage mechanistic account and the standard Transformer computational structure, and explain the mechanistic insights our work provides beyond the general autoregressive structure. We have accordingly revised the fourth paragraph of the Introduction to place less emphasis on the conventional processing order across layers and positions and highlight the specific computational mechanisms revealed by our causal analysis. We have also clarified in Section 6 which questions about the current mechanistic account require further investigation.

Further Interpretation and Validation of the Conflict Signal

We further clarify the functional interpretation of the conflict signal and have revised the corresponding discussion in Section 4.2. Following HAg4’s suggestions, we have conducted additional experiments in four open-ended response settings to examine the conflict signal’s geometric consistency across settings and scenarios and how it differs from a general refusal signal. The experimental settings and results have been provided in the corresponding responses, and we will include them in Appendix J of the final version.

We sincerely thank all reviewers for their time and effort in reviewing our work. We respectfully invite them to consider our detailed responses and the revised manuscript.

From OpenReview, CC BY 4.0. Authors as signed there; shown unedited.

Related papers

Loading the map…

Discussion (0)

Sign in to comment.